drive.download() does not work as expected

[gmaclennan]

Calling await drive.download() does not download the expected data if you do not wait for the drive to update first. Currently drive.update({ wait: true }) does not work as expected either. This is possibly due to hyperbee read streams not being updated to account for the change in default behaviour of hypercore changing to updates being local only by default.

This PR adds a failing test, but I'm not sure if the fix belongs here or in hyperbee.

[LuKks]

I think it's expected behaviour because you didn't receive the first version/length update yet

If I understand correctly, you can "update" only once for waiting to "bootstrap" into the network, so you receive the latest length from peers, but then subsequent updates won't work so you should not rely on force updating to receive a new length

I might be wrong, I was also confused with this for a long time haha

In other news, we're working to improve this so replicating becomes way easier!

[LuKks]

Funny enough, check this really old version of drives:
https://github.com/holepunchto/drives/blob/43a2f23fc39a0cce713b4d7a62779490d1ccc11f/download.js#L40
I experienced something similar

[LuKks]

Top summary: I would say it's ok to wait for the first update

We just need to fix Hyperbee so you can just do drive.update({ wait: true }) instead of accessing the db core, etc but we have a PR for that already

[gmaclennan]

Thanks @LuKks - the behaviour seemed to change with Hypercore 10.6.0 which made updates local only - at least I had a test that did something similar to this (without calling update({ wait: true }) first) that worked with prior to 10.6.0, but maybe I was using it wrong!

I do find the update() stuff to be confusing - I'm never sure when I need to call it to make sure things work as expected, and since we don't exclusively use hyperswarm for discovery (we also use mdns) I'm never quite sure how the findingPeers stuff is supposed to work.

[gmaclennan]

FYI I have been using this for implementing a "live" download functionality: https://github.com/digidem/mapeo-core-next/blob/main/lib/blob-store/live-download.js#L87

Is this something you would consider adding to hyperdrive?

[LuKks]

Once Hyperbee update method is fixed then just do like so:

const swarm = new Hyperswarm()
const done = drive.findingPeers()
swarm.on('connection', (socket) => drive.replicate(socket))
swarm.join(drive.discoveryKey)
swarm.flush().then(done, done)
await drive.update({ wait: true })

[LuKks]

Keep in mind something like this:

Let's say you already have the code above working and everything is ok, let's say you later re-execute again for a second time this:
await drive.update({ wait: true })

You expect the drive to PULL from other peers their latest length, right? That's the main issue, peers must PUSH to you new data in this case the latest version/length

So you only get background updates for new versions! I think you can always use the Hypercore 'append' event to know when you receive an update

[LuKks]

Ah, sorry, I didn't answer your doubt about how findingPeers works

First check this: https://github.com/holepunchto/hyperdrive#const-done--drivefindingpeers
Very important is that you don't depend on Hyperswarm

When you do this:
const done = core.findingPeers()
That does literally almost nothing, it's just an internal counter that gets increased

So all requests made after that are put on hold, until you call done(), so call it when your mdns thing has connected to all peers that it found, that's it

[gmaclennan]

Ah right, thanks for that @LuKks, helpful. I think in our case the main challenge is deciding when "done" happens - new devices can appear at any time, so it's never actually "done".

[LuKks]

When you start the mdns search, if there is a peer available, it's normally extremely fast to appear, right?

So you could just give 3-5 seconds of wait until a peer appears, otherwise continue without

When you enter a swarm topic i.e. drive discovery key and there is no peers, the update takes a few seconds as well, you should try it out