Note on improving download speed

[sambacha]

IPFS uses QUIC transfers. These connections can be limited by the size of the UDP receive buffer. This buffer holds packets that have been received by the kernel, but not yet read by the application (quic-go is what IPFS uses in this case). Once this buffer fills up, the kernel will drop any new incoming packet.

ensure you have configured

sysctl -w net.core.rmem_max=2500000

default is 208 kiB
you want: 2048 kiB

see https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size

[wmitsuda]

Did you notice significant improvement on time spent to download the sourcify repo? Just curious.

I'm on mac, it seems the default in my machine is 8MB:

$ sysctl kern.ipc.maxsockbuf
kern.ipc.maxsockbuf: 8388608

[sambacha]

Will have to benchmark to get specifics, OSX configuration is recommended to have about 30-35% more than what is needed per the documentation provided above

Benchmarking would be interesting, maybe will create a dummy repo with a small data set that way its reproducible, wdyt?

[wmitsuda]

Would be good, I don't have a linux machine so I can't reproduce, but if this is confirmed to improve download speed we can add this recommendation on build instructions, the raw sourcify repo is quite big right now (~11GB) and increasing.

[sambacha]

Would be good, I don't have a linux machine so I can't reproduce, but if this is confirmed to improve download speed we can add this recommendation on build instructions, the raw sourcify repo is quite big right now (~11GB) and increasing.

I found this very relevant repo: https://github.com/ConsenSys/ipfs-lookup-measurement
with latency for content retrieval graphed here

Also just to clarify, not using the flat store (using badger) and pub sub, any other important flags?

Cheers

[sambacha]

Taken from: https://medium.com/@CameronSparr/increase-os-udp-buffers-to-improve-performance-51d167bb1360

On BSD/Darwin systems you need to add about a 15% padding to the kernel limit socket buffer. Meaning if you want a 25MB buffer (8388608 bytes) you need to set the kernel limit to 26214400*1.15 = 30146560. This is not documented anywhere but happens in the kernel here.

To update the value immediately to 7.5 MB, type the following commands as root:

sysctl -w kern.ipc.maxsockbuf=8441037