| 18:17 | <MylesBorins> | how do people feel about something like bufferlist for Uint8Arrays? |
| 18:17 | <MylesBorins> | would this make sense for TC39? maytbe a whatwg thing? |
| 18:17 | <MylesBorins> | https://www.npmjs.com/package/bl |
| 18:19 | <Bakkot> | MylesBorins I can't entirely tell what that does? |
| 18:19 | <MylesBorins> | this was something that was created to work with Node.js buffers |
| 18:20 | <MylesBorins> | if you are receiving a stream of buffers you can use the bl interface to compile them into a list of buffers and operate on them |
| 18:20 | <MylesBorins> | Deno is starting to implement their own approach to buffers / solving groups of buffers |
| 18:20 | <MylesBorins> | I'm on the fence if this should be a language feature (and as such WHATWG might be more appropriate) |
| 18:21 | <devsnek> | so like |
| 18:21 | <devsnek> | optimized TypedArray concat? |
| 18:22 | <MylesBorins> | I've had a need for this when automating and concatenating output from child processes |
| 18:22 | <MylesBorins> | if you simply turn the buffer to string you can end up converting a buffer that has partial data |
| 18:22 | <MylesBorins> | so the buffer list allows you to collect all the buffers that make up the message before converting to string |
| 18:31 | <ljharb> | i've had use cases of being able to combine Maps/Sets without iterating over all of them, or WeakMaps/WeakSets, which can't be iterated over; which seems like the same kind of "lazy combination wrapper" as bufferlist? |
| 18:31 | <shu> | do you want a StringBuffer, but for Buffers? like a BufferBuffer? |
| 18:33 | <shu> | also unclear on why this needs to be a language feature |
| 18:33 | <devsnek> | i'm assuming performance |
| 18:33 | <devsnek> | typedarray.p.set is not ideal |
| 18:34 | <ljharb> | it's really annoying to implement a WeakSet that also wraps existing WeakSets |
| 18:34 | <ljharb> | totally doable, but not trivial |
| 18:34 | <shu> | are we talking about weaksets or buffers |
| 18:34 | <devsnek> | i'm talking about buffers |
| 18:34 | <shu> | also unclear you get performance benefits with a generic lazy combination buffer |
| 18:34 | <ljharb> | lol sorry, i guess i bogarted the topic a bit, let's go back to buffers |
| 18:34 | <devsnek> | ljharb: isn't there already a set union proposal |
| 18:35 | <ljharb> | devsnek: set, not weakset |
| 18:35 | <devsnek> | open an issue i gus |
| 18:35 | <devsnek> | guess |
| 18:35 | <devsnek> | shu: we have Buffer.concat in node for a reason :P |
| 18:35 | <Bakkot> | MylesBorins there might be something here but I'm not entirely sure what problems you have that this solves, which makes it hard to tell |
| 18:36 | <shu> | devsnek: and that reason is? |
| 18:36 | <Bakkot> | if the motivation is _just_ converting to strings, it sounds like a thing for TextEncoder (it could take a list of buffers instead of just a buffer), which is (somewhat unfortunately) a WHATWG thing |
| 18:36 | <devsnek> | its faster to make c++ do it |
| 18:37 | <devsnek> | actually i take this all back Buffer.concat was ported to js |
| 18:38 | <Bakkot> | MylesBorins I am definitely on board with adding more stuff for working with buffers in general though |
| 18:39 | <MylesBorins> | so at a very very high level thing here |
| 18:39 | <MylesBorins> | node has Buffer.concat |
| 18:39 | <MylesBorins> | the ecosystem has stuff like bl |
| 18:39 | <MylesBorins> | deno is starting to solve this problem too |
| 18:39 | <MylesBorins> | (I'm not as familliar with front end problems that might need this type of solution) |
| 18:40 | <MylesBorins> | but avoiding environments from continuing to diverge is a huge goal to me |
| 18:40 | <Bakkot> | if the problem is quickly making a new buffer from some existing buffers, sounds like it would be a good proposal, yeah |
| 18:40 | <shu> | Buffer itself has already diverged |
| 18:40 | <MylesBorins> | I think "is this a language feature" is a super fair question |
| 18:40 | <MylesBorins> | and why I'm not 100% convinced 262 is the right venue |
| 18:40 | <shu> | would the ability to concat ArrayBuffers ease transition from node buffers to ArrayBuffers? (which are pretty different ergonomically) |
| 18:41 | <MylesBorins> | shu TBH I'm not as deep into this part of the code base, so solid maybe |
| 18:41 | <shu> | so i see two divergence problems: deno and node, and server and front-end |
| 18:41 | <shu> | i *think* my feeling about the Buffer polyfill is that it's a scourage? |
| 18:41 | <shu> | scourge |
| 18:42 | <MylesBorins> | I got pinged by Mikeal Rogers about this, specifically because he is seeing all the work we are having to do in node to tease us off buffer |
| 18:42 | <MylesBorins> | and seeing deno head in the same direction |
| 18:42 | <shu> | "tease us off buffer" -> where does node wanna go? |
| 18:42 | <devsnek> | in terms of buffer we'd rather just use normal Uint8Arrays |
| 18:44 | <MylesBorins> | yeah, I defer to the devsnek here ::P |
| 18:44 | <devsnek> | the thing is |
| 18:44 | <devsnek> | when we hit some sort of usability problem along that pathway |
| 18:44 | <devsnek> | the question is whether its a problem with uint8array or a problem with node |
| 18:44 | <devsnek> | i think that's sort of what myles meant above? |
| 18:46 | <shu> | in the specific concat and BL cases, it does smell like a pre-generalized solution that mainly cares about strings. is that accurate? |
| 18:46 | <MylesBorins> | shu not neccessarily |
| 18:46 | <MylesBorins> | strings were just the example I have personal experience with |
| 18:46 | <MylesBorins> | my understanding is that folks who work with streaming interfaces use bl extensively as an intermediary format for the incoming data |
| 18:47 | <shu> | that gets into a separate domain of streams |
| 18:48 | <MylesBorins> | streams utilize this, even emitters utilize this as well |
| 18:48 | <MylesBorins> | if you have an emitter or a async iterable that is returning buffers you may want to deal with them as an opitmized list |
| 18:48 | <MylesBorins> | as opposed to shove them all in an array and concat later |
| 18:49 | <shu> | i guess i'm also missing what is the optimization opportunity |
| 18:49 | <shu> | the individual buffers are still all there and observable? |
| 18:50 | <MylesBorins> | I've asked mikeal to give me more examples |
| 18:50 | <MylesBorins> | tbh I'm serving much more as a middle person here |
| 18:53 | <devsnek> | the individual buffers are still there |
| 18:53 | <devsnek> | ideally this would be more like preallocating a buffer and passing it to the api |
| 18:53 | <devsnek> | but js doesn't have resizable buffer types yet |