Stream Interface for Creating Torrents

[martindale]

Ideal interface, very similar to the README example:

var fs = require('fs');
var nt = require('nt');

var file = fs.readFile('foo.txt');
var torrent = nt.stream();

file.pipe( torrent );
torrent.pipe( fs.createWriteStream('mytorrent.torrent') );

[fent]

I thought about doing this a long time ago, which led me to make this: https://github.com/fent/node-kat

but I never integrated it into this module.

[martindale]

Awesome. I'll take a look at integrating it. Thanks!

I'm seeing this snippet in the readme:

var rs = nt.make('http://myannounce.net/url', __dirname + '/files');
rs.pipe(fs.createWriteStream('mytorrent.torrent'));

// callback style
nt.makeWrite('outputfile', 'http://announce.me', __dirname + '/files',
  ['somefile.ext', 'another.one', 'inside/afolder.mkv', 'afolder'],
  function(err, torrent) {
    if (err) throw err;
    console.log('Finished writing torrent!');
  });

Since the first 2 lines look like a streaming interface, is this issue "closed"?

[fent]

When creating a torrent, it returns a readable stream, that can be piped. But, a stream cannot be piped to it.

So it's like halfway closed.

Something like this might do the trick (maybe?)

var stream = require('stream');

var Q = require('q');  // any promises library should work.

/**
 * A writable stream for a file.
 */
function File(filename) {
  stream.Writable.call(this);

  this.firstChunk = true;

  // Metadata.
  this.filename = filename;

  // We'll start processing the file when this is resolved.
  this.deferred = Q.defer();

  // . . .
}

File.prototype._write = function(chunk, encoding, callback) {
  var _this = this;

  // Don't start reading until the promise is resolved.
  if (this.firstChunk) {
    return this.deferred.promise.then(function() {
      // Set the flag to false, so we can ignore this "if".
      _this.firstChunk = false;

      processChunk();
    }, callback);
  }

  processChunk();

  function processChunk() {
    // Do whatever you need to do with the chunk here.
  }
}

/**
 * The hasher at `hasher.js`.
 */
function Hasher() {
  // . . .

  // Array of `File` instances.
  this.files = [];

  // . . .
}

/**
 * Add a file to the hasher.
 */
Hasher.prototype.add = function(filename) {
  var f = new File(filename);

  this.files.push(f);

  return f;
};

/**
 * Start processing the files.
 */
Hasher.prototype.start = function() {
  // You start processing the files here.

  // Example: to start processing the first file:
  this.files[0].deferred.resolve();

  // If you want to process the file with some function defined
  // in this scope, you may:
  //
  // - Modify the `File` object to receive an arbitrary function, or
  // - Make it emit events
  //
  // or whatever tweaks that make the trick.
};

Trying to do all the file processing on their own objects (asynchronously), and use the object returned by make (currently the hasher) only to join the results and produce the final output.

The usage should be like this:

var fs = require('fs');
var nt = require('nt');

var t = nt.make('http://myannounce.net/url');

fs.createReadStream('file1').pipe(t.add('file1'));
fs.createReadStream('file2').pipe(t.add('file2'));

t.pipe('output.torrent');
t.start();

// or maybe:
// t.start().pipe('output.torrent') 

Not sure if this is possible, though. I don't fully understand how torrent files are built.

[fent]

I like the API.

A queue system like that already exist in kat, which I was planning to use for this.

But now I'm thinking it's not the fastest solution because sometimes a stream might stall for a while, during which a later stream could be read and its chunks hashed, but only if we know the size of the earlier stream.

So it's more complicated :/

[martindale]

I looked at implementing this on my own, but dealing with the existing codebase presented too much friction for the pace I want to move. @unusualbob wrote an implementation, so I'm using that version for now as I only need to create torrents.

[fent]

Is this still wanted? are there better existing implementations out there? I'd rather redirect people to a perfectly working solutions, specially since i don't personally use this anymore

[martindale]

I'd love to see this implemented cleanly, especially with support for updating torrents (one of BEPs implements this, can't recall specifically).