Consider removing readable-stream dependency

[Fuzzyma]

Looking at your CI runs, its seems like the lowest Node Version that is supported is Node 18.
Afaik, Node 18 has a native ReadableStream implementation so I was wondering while readable-stream is still used.

bl is downloaded about 150M times a month and the traffic created by the readable-stream dependency alone is over 8TB. Considering that and 2 less deps to care about, I think it would be a win to get rid of readable-stream

I saw that you bumped the version of readable-stream in your latest major so if there was already a discussion about it, please point me to it :).

I would be happy to create a PR to do the necessary changes!

[rvagg]

Node has always had readable-stream in it, it started life as an extract of the streams library and now it's the source of truth (or last I checked). The reason for it's inclusion here, originally at least, was to have a stable streams library to work against instead of relying on the variability of whatever is bundled in Node, and to work without any bundling issues within that various browser bundling formats. If you have information that changes any of those dynamics then I'd be happy to hear them (although with limited mental bandwidth to absorb them because the whole browser bundling ecosystem still makes me grumpy).

[Fuzzyma]

Oh I wasn't aware that this package is used in the browser as well. However, as far as I know node 16 offers the WebStream api and since it's a browser standard it should be fairly stable and "just work"

I will have to check the minimum supported browsers tho

[mcollina]

Here is the TL;DR from the person shipping readable-stream:

1. the source of truth is node:stream
2. the source code of readable-stream is a copy of what it is shipped as v18, hopefully we'll update to v24 later this year once v18 phase out of LTS. This is not a problem for this module, as it currently supports Node v18+.
3. readable-stream is guaranteed to work in browsers, at least with some bundlers. node:stream is not polyfilled by bundlers anymore, therefore by dropping readable-stream browser support would break.

My recommendation is:

• use readable-stream for long-term node compatibility (way past EOL, we still ship fixes for 1.x, 2.x, 3.x lines)
• use readable-stream for browser compatibility

Consider what @Fuzzyma is asking, the solution here is relatively simple: split https://github.com/rvagg/bl/blob/master/bl.js (the browser thing) into its own module for browser usage, of which bl will depend upon. Then drop browser support for bl proper by switching to node:stream.

[v1rtl]

my 2c

1. for browser compat, Web Streams could be used, or for both node and web since Node.js supports web streams as well. (And the lowest supported version for bl would be v22)

2. it's not a package's job to polyfill node built-ins, that's a bundling tool's job (usually it's just a setting or two to enable polyfilling)

[dynst]

https://bundlephobia.com/package/bl@6.1.0

This dependency is currently 91.1% of the bundle size. Shedding that would be an enormous win.

[MichaelDeBoey]

@rvagg Any news on how you want to move forward for removing readable-stream from the dependencies?

Added bonus of removing readable-stream is that you can remove @types/readable-stream, which would resolve #145 as well

[rvagg]

No? Refer to what Matteo said above.

Look, how about we just have a separate export for browser, or bundlephobia-gaming use if people still really care about bundle size (sorry, but I'm surprised we're still caring about bundle size in the age of smart tree-shaking, ESM, etc. etc.). You could import bl from 'bl/ws' and just go all in on webstreams. Would that work?

[v1rtl]

Matteo's comment is explaining the reasons why readable-stream would be included in general, not for this specific module (if I understood correctly). bl.js already targets Node 18 which has the same stream implementation as native streams.

The question is, why does readable-stream have to stay as a dependency for the Node version, if the stream implementation in the supported range for BL is the same one as in readable-stream? It would make sense if bl targeted versions below 18. Is readable-stream used only to maintain browser support? Is BL.js even intended to work in browsers?

I don't think this discussion is anyhow relevant to bundle size. Using a polyfill like that certainly impacts bundle size, but that's not the only issue. It breaks type compatibility as mentioned in @MichaelDeBoey comment. That happens pretty frequently with libraries that depend on polyfills whose types differ from the ones provided in the global space/as native module.

Speaking of bundlers, they already polyfill readable-stream for the most part, either by default (like Vite) or wirh a plugin, see node-stdlib-browser. I haven't seen bundlers dropping support specifically for Node streams. Webpack 5 dropped all polyfills in general, not just streams, if that is what the statement was referring to.

That is not a call to action or a demand. I would like to clarify on whether browser support is actually intended. In case it isn't, then there is no practical benefit in depending on readable-stream (IMO). If the reason to not remove it is to not introduce additional maintenance burden, that's understandable

[rvagg]

OK, I feel like I'm missing the main point, or points here. Likely so because I keep on having to context switch into here and my attention span isn't long enough to absorb all of this (also, "works for me", so I'm having problems figuring out what I should be concerned about). So help me out.

What is the main complaint, or are we dealing with lots of complaints from different perspectives?

1. "I don't like my node_modules being so large and removing readable-stream from bl will make my life better"
   a. "150M x readable-stream just for bl = 8TB which is sad for .. the world, environment, networks?"
2. "I want to use bl in a browser bundle, and bundle size is too big
3. "readable-stream is currently too old in bl, so we have type and other incompatibilities"
4. "Having readable-stream as a dependency means we can't use native Node.js core types in here and that causes problems for consumers in some way"
5. Something else?

I'm more than happy to update readable-stream to a newer version, and if we still have Node.js v18 in CI it's because someone (me I suppose) has neglected to ditch old versions, I have no intention of the latest version of bl to support unsupported versions of Node.js.

But here's my position, YMMV, this is just me, so understand where I'm standing so we can have this discussion:

1. readable-stream is a stable target, Node.js versions are not. I can pin to a particular readable-stream version and get a stable streaming implementation. This may not be as acute a problem today as it used to be, but I quite like having a stable and predictable set of dependencies, not having to depend on a changable standard library if I don't have to (and in this particular case, I don't have to because of readable-stream).
2. I don't really care about the size of node_modules—I care much more about careful selection of dependencies than the size of them.
3. I don't really care about browser bundles for this project. Maybe I will one day, I probably don't do enough browser dev to have run into the kinds of problems that I originally wrote this library to solve. But then, I'd probably not be using 'buffer' to solve that problem in the browser, there's probably some variant of DataView I could use for faux-joined ArrayBuffers that would do what I need, or I'd just write that (and maybe park it in here!).

[MichaelDeBoey]

readable-stream is a stable target, Node.js versions are not

@rvagg As @mcollina (the person shipping readable-stream) mentioned, that's just not true:

1. the source of truth is node:stream
2. the source code of readable-stream is a copy of what it is shipped as v18

So using node:stream directly is actually the exact same thing as using readable-stream for this lib.

The added benefit of a smaller bundle + typing fixes makes this an absolute no-brainer to remove the readable-stream dependency imo.

[benmccann]

@MichaelDeBoey I'm not sure I understood the above the same way you did. I think it was being suggested that the readable-stream implementation rarely changes where as if you use the version from Node.js it might differ according to which version of Node you use.

5. Something else?

My main concern would be that readable-stream plus its dependencies is ten packages which can be published by a dozen different people. Given how often we see accounts compromised to publish malicious npm packages this introduces a lot of supply chain risk. The disk space is free in my personal opinion (though I know others will disagree), but the security risk isn't.

[MichaelDeBoey]

It might differ according to which version of Node you use

@benmccann If we were talking about libraries, you're absolutely correct that implementation could change drastically in future major updates

Given the history of Node.js, I can't think of them ever breaking a stable API in future versions though