| 00:15 | <Domenic> | ok, back, although i should probably head out for the day |
| 00:16 | <terinjokes> | Domenic: you should go home |
| 00:16 | <Domenic> | this Promise.resolve analogy soothes me greatly |
| 00:16 | <Domenic> | (in general any time I can analogy to promises, I feel more confident) |
| 00:17 | <Domenic> | we can let pipeTo only accept true WritableStreams, and if in the future that becomes limiting we can add WritableStream.cast or similar as a mechanism for coping |
| 00:24 | <wanderview> | Domenic: https://github.com/whatwg/streams/issues/321 |
| 00:24 | <wanderview> | have a good night! |
| 06:37 | <JakeA> | annevk: https://github.com/slightlyoff/ServiceWorker/issues/607#issuecomment-90819078 - I don't understand "wait with overrideRequestURL" |
| 06:53 | <annevk> | JakeA: just not implement it |
| 06:54 | <annevk> | JakeA: what you argue for would be a new special case that did not exist before; the case the implementers brought up was not considered when we figured it as a shorthand for short-circuiting redirects |
| 06:58 | <JakeA> | annevk: we've got some internal customers for this, I'll find out if the API works for them. But yeah, in that case it isn't exactly like a redirect. |
| 06:58 | <annevk> | JakeA: it would be interesting to hear why they can't use Response.redirect() |
| 07:05 | <JakeA> | annevk: yeah, I'm digging into that and building them an example |
| 08:00 | <annevk> | Is anyone using https://atom.io/? |
| 08:41 | <philipj> | annevk: I think the general situation is that Content-Type is ignored for both media elements and <track> |
| 08:42 | <philipj> | there's a bit of a history, but since Microsoft caved and started ignoring it in IE I think there's no turning back (which I'm happy about) |
| 08:42 | <annevk> | philipj: so for X-Content-Type-Options: nosniff we want to give server administrators the feature of enforcing Content-Type for resources so they can't be used in an unexpected context |
| 08:43 | <annevk> | philipj: this requires a whitelist of MIME types for audio/video and a separate set for track |
| 08:43 | <philipj> | uh, ok, I couldn't say how that currently works |
| 08:44 | <annevk> | philipj: it doesn't for those contexts |
| 08:44 | <annevk> | philipj: well, it might in IE |
| 08:44 | <annevk> | I haven't tested IE yet |
| 08:44 | <philipj> | for <track> there's just no code that looks at Content-Type, at least not in Blink, if it's not WebVTT then it won't work |
| 09:33 | <annevk> | philipj: okay, so we could just make the whitelist text/webvtt |
| 09:33 | <annevk> | philipj: for CSS it's text/css at the moment |
| 09:33 | <annevk> | philipj: only unclear thing then is the MIME types for audio/video which is somewhat of a trainwreck |
| 09:45 | <annevk> | text/vtt* |
| 09:57 | <annevk> | http://w3cdreams.tumblr.com/ |
| 10:01 | <jgraham> | Wow, well I was enjoying that and then the Service Worker thing guenuninely made me feel ill |
| 10:10 | <annevk> | jgraham: hypno cat? |
| 10:12 | <jgraham> | Possibly |
| 10:13 | <jgraham> | Also made me unable to spell genuinely |
| 10:17 | <zcorpan> | just need to let the eyes follow the circles a few laps and then you'll feel better again |
| 14:52 | <wanderview> | Domenic: you said there were user space Promise libs that were faster than browser implementations... can you point me at one you recommend for perf? |
| 14:52 | <bradleymeck> | wanderview: bluebird |
| 14:52 | <Domenic> | yep |
| 14:56 | <wanderview> | Domenic: should I just be able to drop this in with the stream reference impl? |
| 14:57 | <Domenic> | wanderview: ugh, the reference impl is so un-optimized... |
| 14:57 | <Domenic> | wanderview: but, yeah, if you do global.Promise = require('bluebird') it should be drop-in-able |
| 14:58 | <Domenic> | But I mean, all the asserts (in loops too!) and try/catches and the hilarious queue-with-sizes implementation will dwarf the performance change I think |
| 15:01 | <wanderview> | Domenic: meeting now... but I'll try to describe the case I'm concerned with... I can write something not use ref impl, but just something similar to tease out the Promise impact |
| 15:03 | <Domenic> | wanderview: OK cool. Be sure it does actual I/O too :) |
| 15:04 | <wanderview> | Domenic: javascript consumers may not do actual I/O |
| 15:05 | <Domenic> | Ultimately the stream should be grounded in some I/O, most likely |
| 15:06 | <Domenic> | E.g. maybe your stream vends JS objects but it's derived from some stream that read from a file/network |
| 15:36 | <wanderview> | Domenic: is it not reasonable for js to do in-memory operations like your stream demo where its searching for a particular value? that is not I/O related |
| 15:36 | <Domenic> | wanderview: how is that no I/O related? |
| 15:37 | <Domenic> | wanderview: the stream is a fetch stream |
| 15:37 | <wanderview> | Domenic: I thought you meant on the consumption side you wanted it to be I/O |
| 15:37 | <Domenic> | wanderview: oh, no, i just meant that each promise produced from the reader should correspond to doing some I/O |
| 15:38 | <Domenic> | i.e., don't test for (let i = 0; i < 1000; ++i) { Promise.resolve(5).then(foo); }; test for (let i = 0; i < 1000; ++i) { fs.read(fd, ...).then(foo); }) |
| 15:39 | <wanderview> | Domenic: the main case I can think of that concerns me is where you write data to a pipe buffer... and then want to read it starting at some later time... in that case the pipe .read() is not related to I/O until the buffer is drained... and draining the buffer will be much slower than with sync .read() |
| 15:39 | <Domenic> | "much" slower citation needed... especially since sync read() would need to use async .ready or similar for backpressure signals |
| 15:39 | <wanderview> | Domenic: thats why I want to test with real Promises :-) |
| 15:40 | <Domenic> | sounds good :) |
| 15:40 | <wanderview> | Domenic: I agree at least one async operation is needed... but doing it on every buffered chunk seems potentially not good |
| 15:41 | <wanderview> | Domenic: I'm going to put this in the batch read issue... I think some kind of .readAllAvailable() would solve this... give me an array of all available chunks... so you get an array of length 1 or greater |
| 15:41 | <wanderview> | for cases where you just want to read as fast as possible |
| 15:41 | <Domenic> | wanderview: sounds good, yeah. Or 0 chunks if end of stream. |
| 15:52 | <wanderview> | Domenic: I guess my concern here started by looking at the current rs.pipeTo() implementation... since it waits for each .read() to resolve before calling .read() again... you can't get any pipelining going that way... and then I started to wonder if it should do two .read() calls... so while its writing the first its already started the async process to get |
| 15:52 | <wanderview> | the next, etc |
| 15:53 | <Domenic> | wanderview: right, but that should only delay at most a microtask, and then it will resolve and immediately call read() again, and the stack only has to unwind one frame since we're already in the microtask loop... it's one extra frame per loop iteration basically. |
| 16:02 | <JakeA> | wanderview: I've been thinking about cache.add[All]. I've been seeing a lot of examples check the status code of responses before putting them into the cache. Maybe we should revisit the idea of checking response.ok before caching (as an option). This would of course fail on all opaque responses. Do you think it's still useful? |
| 16:05 | <wanderview> | JakeA: in a meeting |
| 16:05 | <JakeA> | no rush |
| 17:06 | <annevk> | JakeA: should it not be added to fetch() then? |
| 17:15 | <wanderview> | JakeA: that sounds fine to me... I don't really have an opinion... whatever developers want/expect |
| 17:15 | <wanderview> | not sure I understand the opaque response thing, though |
| 17:15 | <annevk> | Domenic: the post-ES6 specification plan sounds rather lovely |
| 17:15 | <Domenic> | :) |
| 17:15 | <Domenic> | we'll see if we can pull it off... |
| 17:16 | <annevk> | Domenic: I wonder if WHATWG can gradually convert to that as well |
| 17:16 | <wanderview> | what plan? |
| 17:16 | <annevk> | wanderview: https://esdiscuss.org/topic/the-great-tooling-revolution |
| 17:18 | <Domenic> | annevk: you mean the two-impls thing? |
| 17:18 | <annevk> | Domenic: the tooling and writing style |
| 17:18 | <Domenic> | annevk: ah interesting. |
| 17:19 | <annevk> | having said that, ES6 is rather hard to digest, but some of the algorithm shorthands you're introducing seem nice |
| 17:19 | <Domenic> | Yeah, I was going to say, "Ecmaspeak"'s evolution into something more user-friendly is still in progress (cf. https://streams.spec.whatwg.org/#conventions) |
| 17:19 | <Domenic> | And it's unclear the value of Ecmaspeak vs. some kind of "spec ES" |
| 17:20 | <wanderview> | Domenic: what does chrome's shipping Response body stream do for a Response returned from cache.match()? does it stream from disk? |
| 17:20 | <Domenic> | wanderview: good... question...... tyoshino are you awake perchance? |
| 17:20 | <wanderview> | I mean... is that implemented in this first version |
| 17:20 | <Domenic> | oh hi yhirano_ is in here too |
| 17:21 | <Domenic> | Maybe the thing to do is ask on the blink-dev Intent to Ship thread? |
| 17:27 | <wanderview> | Domenic: ok, will do... mainly just curious |
| 17:28 | <wanderview> | Domenic: what is WritableStream.getWriter() you to mention in that issue update? I don;t see that in the spec |
| 17:59 | <Domenic> | wanderview: https://github.com/whatwg/streams/issues/319 :-S |
| 18:00 | <wanderview> | ok... so not spec'd yet |
| 18:00 | <wanderview> | wasn't sure if it was coming or previous-and-gone |
| 18:01 | <Domenic> | heh... yeah |
| 18:01 | <Domenic> | wanderview: related https://github.com/whatwg/streams/commit/d757e05cf27f73488d13825e51ce5706a759ab27 |
| 18:04 | <wanderview> | Domenic: I didn't want to bikeshed more in the "getReader is named poorly" issue... but take() is also widely used for locking concepts :-) |
| 18:09 | <Domenic> | meeeeeeeeeeehhhhhh |
| 18:12 | <wanderview> | Domenic: getReader() does not unlock if the reader gets GC'd right? or rather, the whole stream has to be GC'd in addition to the reader? |
| 18:12 | <Domenic> | wanderview: right. (and in the latter case i'm not sure there's a difference.) |
| 18:12 | <Domenic> | wanderview: webkit has a test showing this |
| 18:13 | <Domenic> | unsure how/whether to put it in the general suite... |
| 18:13 | <wanderview> | Domenic: I really dislike stuff with explicit release like revokeObjectUrl()... so many leaks in code that use them... the error paths are atrocious |
| 18:14 | <wanderview> | from my experience with code in fxos trying to use that kind of API |
| 18:14 | <Domenic> | wanderview: well this case is pretty different, since it's not like you could successfully get another reader by waiting for GC of the first one, since that's unreliable |
| 18:15 | <Domenic> | that's a fair argument for the lifecycle management thread though |
| 18:15 | <wanderview> | yea |
| 18:15 | <wanderview> | I think I mentioned it there |
| 18:16 | <wanderview> | Domenic: is there a finally thing on promises for people to always return the reader? |
| 18:16 | <wanderview> | maybe that makes it less of a problem |
| 18:17 | <Domenic> | wanderview: we've been wanting to add finally to promises for a long time, and this cancelable promise stuff might push it over the line. |
| 18:19 | <wanderview> | Domenic: I guess its not a huge problem here... since lock is released automatically on close |
| 18:19 | <wanderview> | its optimized for single reader case... which is what we want |
| 18:24 | <Domenic> | wanderview: yeah, that's kind of the idea. if anyone's doing something tricky like parsing a header portion then passing the partially-read stream on to someone else, they can probably remember to release the lock. Although I agree finally would be nice. |
| 18:25 | <wanderview> | Domenic: that seems like something easier down with pipeThrough()... although it would be nice to remove the header transform piece after its done |
| 18:25 | <Domenic> | hmm yeah that's true |
| 18:26 | <wanderview> | ^down^done |
| 18:26 | <Domenic> | i guess most uses of readers are hidden behind the higher-level methods like pipe*() and tee() |
| 18:42 | <calvaris> | sorry, I just fell and I don't know if you read what I said but |
| 18:42 | <calvaris> | wanderview: I have a pull request with that test |
| 18:42 | <calvaris> | all custom tests that we had for WK are now there and waiting for Domenic to merge them |
| 18:43 | <wanderview> | calvaris: which test? |
| 18:44 | <calvaris> | the one for the garbage collection regarding the reader |
| 18:44 | <wanderview> | Domenic: btw, we apparently do implement microtask queue in gecko now |
| 18:44 | <wanderview> | calvaris: ah, cool |
| 18:45 | <Domenic> | wanderview: ah that's excellent |
| 18:45 | <wanderview> | Domenic: I still don't think our c++ can optimize around them, though... since js can call into c++ with js still on the stack |
| 19:59 | <trevnorris> | Domenic: "Cheaper than I/O" doesn't say a lot and Promise instantiation speed isn't the only thing to consider. It's the complete time from .push() to .read(). |
| 21:23 | <wanderview> | Domenic: what does it mean for a ReadableStream to .pipeTo() a WritableByteStream... what if the RS produces non-byte things? |
| 21:23 | <Domenic> | wanderview: calling .write() (or the equivalent internal operation if we move to that model) will error, causing the pipe to error. |
| 21:24 | <wanderview> | Domenic: but if the ReadableStream passes Uint8Array chunks, then its ok? |
| 21:24 | <Domenic> | wanderview: yes, I would really like that to work |
| 21:25 | <Domenic> | wanderview: maybe allow any of the "BufferView" types |
| 21:25 | <wanderview> | or just ArrayBuffer, I presume? |
| 21:28 | <Domenic> | wanderview: right, I forgot what the type was called that is a union of all typed arrays + ArrayBuffer + DataView |
| 21:28 | <Domenic> | "BufferSource" apparently |
| 21:30 | <wanderview> | Domenic: where is the latest ReadableByteStream proposal? |
| 21:30 | <Domenic> | wanderview: https://github.com/whatwg/streams/blob/asyncbytestream/BinaryExtension.md |
| 21:32 | <wanderview> | Domenic: so... its starting to feel to me like we should not have a ReadableByteStream at all... if the underlying source of the stream operates on bytes, then the specialization has already occurred... getReader() can just return a ByobReader |
| 21:32 | <wanderview> | or am I missing something? |
| 21:32 | <TabAtkins> | bring-your-own-beerReader |
| 21:33 | wanderview | didn't name it. :-) |
| 21:33 | <wanderview> | ByobByteStreamReader |
| 21:34 | <Domenic> | wanderview: hmm, but adding the BYOB layer changes things quite a lot, adding complexity specifically for the byte case. You would advocate building that complexity into the readable stream itself? |
| 21:35 | <wanderview> | Domenic: if a view is not passed to .read()... can it not just do a "get me the next chunk" semantics pretty easily? |
| 21:35 | <trevnorris> | Domenic: spec probably can't define any implementation details, can it? |
| 21:35 | <Domenic> | wanderview: for example the underlying source needs some kind of hook read(view) -> promise that fills view, and inside the stream/reader mechanisms you need something such that byobReader.read(view) transfers view to view2 then passes view2 to the read hook on the source |
| 21:35 | <Domenic> | trevnorris: it can define anything observable |
| 21:36 | <Domenic> | trevnorris: including creation APIs |
| 21:36 | <Domenic> | wanderview: the idea is you should opt in to one mode or the other (BYOB or auto-flowing) |
| 21:37 | <wanderview> | Domenic: then keep it getByobReader(), but have it reject on streams that don't have the right underlying source |
| 21:37 | <Domenic> | for example BYOB doesn't have a high water mark or queue in general (although it does probably keep one around since kernel buffers are finite... unsure if that's observable to spec or can be part of the underlying source details) |
| 21:37 | <Domenic> | wanderview: ick, that's a very bad API. It would be like Node having a tagName that throws an exception when you try to use it on Comment nodes |
| 21:37 | <trevnorris> | Domenic: my implementations tie a C++ class to every JS stream, on which a class method can be defined to receive incoming data. .pipe() can detect that and short circuit the JS call by passing from the internal C++ to the implementation defined method. |
| 21:38 | <trevnorris> | unfortunately it depends on certain V8-isms that i'm not sure are currently possible in other engines. |
| 21:38 | <Domenic> | trevnorris: right, in that case the spec's job is to make your short circuit unobservable |
| 21:38 | <Domenic> | trevnorris: and that means completely unobservable, even if someone overwrites dest.write to log when called or similar ;) |
| 21:38 | <wanderview> | Domenic: what does ReadableByteStream get you beyond the Byob capacility? It doesn't seem like anything... so maybe it should be a ByobReadableStream |
| 21:39 | <Domenic> | wanderview: yes, that is indeed the real delta. We figured it was a more user-friendly name? But no real opposition to renaming it |
| 21:39 | <wanderview> | Domenic: I guess the problem with that is the producer of the stream does not know if the user wants Byob semantics |
| 21:39 | <Domenic> | Actually we are kind of hoping not to use the term "BYOB" in public API. The fact that's in tyoshino's doc is largely just as a placeholder :) |
| 21:39 | <wanderview> | ^user^consumer |
| 21:40 | <wanderview> | I know |
| 21:40 | <wanderview> | Domenic: it just feels the "type" of the stream is determined by the underlying source... and having to have a special shell around particular underlying sources is clumsy... ReadableStream should just enable the extra features if its constructed with a byte oriented underlying source |
| 21:40 | <Domenic> | wanderview: the hope is that it is efficient for streams like fetch/fs/etc. that are bytes-backed to implement ReadableByteStream, and then consumers who want to opt in to BYOB behavior can use .getByobReader(), and normal people (including those agnostic to the chunk type) can just use .getReader() |
| 21:41 | <Domenic> | wanderview: I don't agree with that. The logical conclusion of that argument is that streams should be able to bake me a cake if I give them the right underlying source. |
| 21:42 | <Domenic> | Adding fundamentally new capabilities, instead of just different behavior, should require a new type. |
| 21:42 | wanderview | likes cake. |
| 21:42 | <wanderview> | Domenic: what do you anticipate being different about a WritableByteStream? |
| 21:43 | <Domenic> | wanderview: hazy right now, but one big thing is it will need to detach any passed-in buffers since they might be sent off thread |
| 21:44 | <wanderview> | Domenic: that... seems hard without including it in the WritableStream contract |
| 21:44 | <Domenic> | hmm how so? |
| 21:45 | <wanderview> | Domenic: isn't that an optimizaiton that should be negotiated by native underlying source to native underlying sink? |
| 21:45 | <Domenic> | wanderview: I am talking about var a = new Uint8Array([5, 10, 15, 20]); fileStream.write(a) |
| 21:46 | <Domenic> | I want to send the backing memory of a off into the thread that does file I/O |
| 21:46 | <Domenic> | and I am pretty sure we don't want to let people do a[0] = 7 and have that mutate the memory, maybe before the write, maybe after the write |
| 21:46 | <wanderview> | Domenic: you're talking about effectively making the chunk disappear from content... if under normal circumstances I can do ws.write(chunk); muchAroundWithChunk(chunk); ... how will code based just on WritableStream know that .write() suddently doesn't work that way because it has a WBS? |
| 21:47 | <wanderview> | Domenic: I guess I'm saying WritableStream.write() contract should say "don't expect the chunk to exist in its current form after this call" regardless of WritableStream vs WritableByteStream |
| 21:47 | <Domenic> | wanderview: that's fair, although I'm not sure in practice how worried we should be. But one easy fix is to add .getTransferringWriter() or similar, whereas .getWriter() does copies?? I dunno. |
| 21:48 | <Domenic> | wanderview: I just have no idea how to enforce that contract |
| 21:48 | <Domenic> | if it's not true for most writable streams, then people might assume it, no matter if the spec has some kind of "don't assume this please" note. |
| 21:48 | <wanderview> | Domenic: maybe its not enforceable... but we can at least say "told you so" when people' stuff breaks :-) |
| 21:48 | <Domenic> | heh, ok |
| 21:49 | <wanderview> | Domenic: it just seems stuff like ByobReader and maybe this TransferringWriter are opportunistic things... optimize in this fashion if its available... otherwise use the lesser stuff |
| 21:49 | <wanderview> | Domenic: and I guess in your mind the right way to "check if its available" is to do an instanceof on the prototype? |
| 21:50 | <Domenic> | wanderview: I don't see how to make ByobReader opportunistic. E.g. no way to do https://gist.github.com/domenic/65921459ef7a31ec2839#reading-a-file-chunkwise (old API I think) without explicitly doing things with JS |
| 21:50 | <Domenic> | wanderview: check if what is available? |
| 21:50 | <wanderview> | Domenic: check if ByobReader is available on a RS, for example |
| 21:50 | <Domenic> | wanderview: if (rs.getByobReader) { ... } |
| 21:52 | <wanderview> | Domenic: can't see much difference between that and getByobReader() returning null if its not supported |
| 21:53 | <Domenic> | wanderview: it's just basic API design... we don't add everything onto a single object. New classes of objects get ... new classes. |
| 21:53 | <Domenic> | Node vs. Element, etc. |
| 21:53 | <wanderview> | Domenic: I think I dislike mixing that with the revealing constructor pattern... the constructor is not revealing all the behavior |
| 21:53 | <wanderview> | Domenic: if we had a revealing factory method that could construct the right type... it might seem a bit better to me |
| 21:54 | <Domenic> | wanderview: I don't see how they're related. Revealing constructor pattern lets you customize a type's behavior. You can have different revealing constructors for different types. We don't use the same type for Promise and ReadableStream, even though they both have customizable behavior via the revealing constructor pattern. |
| 21:54 | <Domenic> | why would you say the constructor is not revealing all the behavior? |
| 21:55 | <wanderview> | Domenic: why would you ever do new ReadableStream(myByteSource)? shouldn't you *always* do new ReadableByteStream(myByteSource)? |
| 21:55 | <Domenic> | wanderview: yes? |
| 21:56 | <Domenic> | did i say otherwise? |
| 21:56 | <wanderview> | Domenic: no... but it feels weird to leave that as a footgun for people |
| 21:56 | <Domenic> | it's the same footgun as new Promise(myByteSource) ... not really worried. |
| 21:56 | <wanderview> | the type is really associated with the source, not the wrapper object |
| 21:56 | <Domenic> | or new WritableStream(myByteSource) |
| 21:57 | <wanderview> | Domenic: except those will fail... ReadableStream(myByteSource) will work, but just prevent consumers from optimizing |
| 21:57 | <Domenic> | The source is an adapter between the conceptual source and the concrete readable stream type |
| 21:57 | <Domenic> | wanderview: why would it work? |
| 21:58 | <wanderview> | Domenic: err... from what I can tell new ReadableStream() and new ReadableByteStream() expect the same properties on the passed duck typed source? |
| 21:58 | <wanderview> | is that not true? |
| 21:58 | <Domenic> | wanderview: the argument for ReadableByteStream() is not specced at all yet :) |
| 21:59 | <Domenic> | wanderview: it will probably be something like { start() { }, read(view), cancel(reason) { } } |
| 22:00 | <wanderview> | ok |
| 22:01 | <wanderview> | Domenic: going back to WritableByteStream... one optimization we have in our native streams is the writing side can pass its destination back into the person doing the write... so in theory this could be passed back to read(view) or something to all a ReadableByteStream to write directly into a WritableByteStream |
| 22:02 | <Domenic> | wanderview: good point, i forgot that was also something we definitely want out of ReadableByteStream + WritableByteStream pipes |
| 22:03 | <Domenic> | they can set up a small buffer pool and reuse buffers to limit total consumption and GC churn ... it will work beautifully ... /me waves his hands |
| 22:15 | <wanderview> | Domenic: sorry... I'm used to the DOM stuff which just enforces types and then says "using X's internal thing, do stuff" |
| 22:17 | <Domenic> | wanderview: np, just concerned about the layering in the design. Ideally *ByteStream should be additive and opt-in, both for consumers and from an architectural level. |
| 22:20 | <wanderview> | Domenic: to be honest, things like off-main-thread piping are mostly interesting to me for *ByteStream... for streams with potentially arbitrary js objects for chunks... not sure I can safely move those around off-thread |
| 22:20 | <wanderview> | not sure if that changes anything |
| 22:20 | <Domenic> | wanderview: oh, no, I was never really planning on them being off thread... but i want the model to not change drastically when you move from non-byte to byte streams |
| 22:21 | <Domenic> | wanderview: although I guess it could be pretty useful in some cases e.g. if a UA provided stream wants to pass metadata with each chunk like { remotePort, remoteAddress, data } or something instead of just data |
| 22:22 | <wanderview> | Domenic: I built a bunch of node stream libs to do that kind of thing before... not sure anyone really liked it much |
| 22:22 | <wanderview> | https://blog.wanderview.com/blog/2013/03/01/composable-object-streams/ |
| 22:23 | <Domenic> | wanderview: do you have a summary of where we are on https://github.com/yutakahirano/fetch-with-streams/issues/30 ? when i went to bed last night i think we were convering on new Request({ body: readableStream }) + fetch(request, wsRevealer) + cache.add(request, wsRevealer) or similar. But then it changed overnight and now I am confused. |
| 22:24 | <wanderview> | Domenic: I think DOM APIs would prefer to return a structured webidl object with a stream property (like Response) |
| 22:24 | <Domenic> | ? |
| 22:24 | <wanderview> | Domenic: I objected to putting wsRevealer on the consumer... because Request is no longer representative of the network request |
| 22:24 | <Domenic> | (I like the UDP object streams BTW!) |
| 22:25 | <Domenic> | Hmm was it ever? |
| 22:25 | <Domenic> | Isn't Request more like RequestMetadata? |
| 22:25 | <wanderview> | Domenic: no... before streams came into it, it contained all info to perform a network request |
| 22:25 | <wanderview> | including the body |
| 22:26 | <Domenic> | right, info to perform a network request, but not a network request itself.... |
| 22:26 | <Domenic> | which is why you can store it in a cache; you can't store a network request in a cache... |
| 22:27 | <wanderview> | Domenic: correct... it is a representation of a possible network request... it is not an actual in progress network request... but if you move the body ws-revealer to fetch() then Request no longer fully describes the possible network request |
| 22:27 | <Domenic> | hmmm |
| 22:27 | <Domenic> | i guess that's true |
| 22:27 | <Domenic> | and ws-revealer is really "a sequence of instructions for how to create a body" so it still goes in the category of "representation of a possible network request" |
| 22:27 | <wanderview> | Domenic: look at my last proposal... it combines your WritableStream wrapper without the hard coded type switch |
| 22:28 | <Domenic> | Although I doubt you'll store the ws-revealer in the cache... |
| 22:28 | <Domenic> | Yeah, I liked that, although unsure how it fits with the rest of the discussion up until that point |
| 22:28 | <wanderview> | Domenic: well, right now Cache only supports GET... so can't put a body in... but if you could, Cache would trigger ws-revealer to get the body data |
| 22:29 | <wanderview> | I still have all the code in gecko to store Request bodies |
| 22:29 | <Domenic> | hmm i see |
| 22:29 | <Domenic> | yeah i guess that makes sense |
| 22:29 | <Domenic> | you reify the body whenever the request gets "committed" somewhere |
| 22:30 | <wanderview> | those are fancy words, but I will nod my head |
| 22:30 | <Domenic> | :P |
| 22:30 | <wanderview> | does reify just mean normalize? |
| 22:30 | <Domenic> | nah ... i'm thinking of it as, make a real set of bytes out of a function that represents a way to get bytes |
| 22:31 | <wanderview> | ah, ok... serialize then |
| 22:31 | <Domenic> | ws-revealer is a "potential body" that gets reified into a real body when you fetch/cache-add |
| 22:31 | <Domenic> | yeah that i guess |
| 22:31 | <wanderview> | I should look up that term I guess |
| 22:31 | <wanderview> | I have more of an EE background so I tend to get lost in the CS theory world |
| 22:32 | <Domenic> | might be a mathematician thing, I dunno. Or just a pretentious thing :P |
| 22:32 | <Domenic> | so the current proposal is req.{setWriter/pipeTo} plus ... body(ws) { ... } which gets triggered after setWriter/pipeTo plus ... do we allow body: readableStream? |
| 22:33 | <wanderview> | Domenic: I still want body: readableStream, yes |
| 22:33 | <Domenic> | poor overloaded body: option |
| 22:34 | <wanderview> | Domenic: didn't you hear? fetch is mostly about sugar :-) |
| 22:35 | <wanderview> | Domenic: and to be clear... I only think we need this ws-revealer thing because of the desire to have progress notification that a pipe would obfuscate |
| 22:37 | <wanderview> | Domenic: btw... resolving the .write() promises when written to the kernel is going to be somewhat challenging... in gecko we get notification from the network code about progress in a different path from where we write... so we have to match that progress back up to the promises to resolve, etc |
| 22:37 | <wanderview> | maybe not challenging... but annoying |
| 22:39 | <Domenic> | :-S |
| 22:39 | <wanderview> | much easier to integrate that with a progress event (which is what it was designed for, of course) |
| 22:39 | <Domenic> | Well, there's https://github.com/whatwg/streams/issues/316 ... |
| 22:39 | <trevnorris> | wanderview: doubt this'll have any affect on the spec here, but in previous implementations I've had calls like .write() return a request object so you can trace the status at any point in the future. |
| 22:40 | <trevnorris> | e.g. .progress() to see how much has been written. |
| 22:40 | <trevnorris> | I use that in conjunction with timeouts to cancel writes that are taking too long. |
| 22:40 | <wanderview> | Domenic: for example, I think gecko network code throttles progress notifications to once every 50ms or something... so you will see batches of .write() promises resolves at the same time, etc. |
| 22:41 | <wanderview> | trevnorris: sounds like you want cancellable promises :-) |
| 22:41 | wanderview | trolls |
| 22:41 | <Domenic> | wanderview: lol, because the spec says 50 ms, good times :P |
| 22:41 | <trevnorris> | hehe. ;) |
| 22:45 | <trevnorris> | wanderview: is it possible to do something like: var req = ws.write(data); setTimeout(function(req) { if (req.status() != 'complete') req.abort(); }, 1000); ? |
| 22:47 | <wanderview> | trevnorris: I think you would have to call ws.abort() instead of req.abort() |
| 22:47 | <trevnorris> | wanderview: what if you did ws.write(data1) ws.write(data2) and I only wanted to abort writing data1? |
| 22:47 | <Domenic> | with cancelable promises you could do `var p = ws.write(data); setTimeout(() => p.cancel(), 1000). (Assuming calling p.cancel() does nothing on an already-settled promise) |
| 22:48 | <trevnorris> | already-settled promise? |
| 22:48 | <wanderview> | trevnorris: that seems racy to me... you may end up with data1 and data2 or just data2... also, not all write operations are abortable once they start, etc |
| 22:48 | <Domenic> | already fulfilled or rejected |
| 22:49 | <Domenic> | yeah, how does that work in POSIX? |
| 22:49 | <trevnorris> | wanderview: if you've queued up several chunks of data to be written, and only the first has actually been sent to the kernel it should be possible to remove any specific write req from the queue. |
| 22:50 | <wanderview> | trevnorris: in a multi-threaded environment... it may have been sent to the kernel and you just haven't been notified of it yet |
| 22:50 | <boogyman> | can you elaborate on a scenario where a Promise could be resolved while a "child" Promise has yet to resolve? |
| 22:50 | <bradleymeck> | its important to remember cancellation does not mean abrupt termination |
| 22:50 | <bradleymeck> | just that it should cancel at the next point |
| 22:50 | <Domenic> | boogyman: what is a child promise |
| 22:50 | <trevnorris> | wanderview: as soon as the data is handed off to something else I'll consider it unreachable. in the case of Node I know because we make the call to uv_write() directly. |
| 22:50 | <bradleymeck> | if there is not a clearly defines point of cancellation, something should not be cancellable |
| 22:51 | <bradleymeck> | defined* |
| 22:51 | <boogyman> | Domenic: however you define "p.cancel()" on an already resolved p |
| 22:51 | <Domenic> | i don't understand |
| 22:51 | <Domenic> | please phrase your question using code? |
| 22:51 | <wanderview> | trevnorris: "something else" is another thread? or the kernel? because in multi-process browsers it goes js->c++->IPC->c++->kernel with thread and process switches in there |
| 22:51 | <bradleymeck> | if cancellation is inteded to be similar to abort()/halting a thread it should be rethought |
| 22:52 | bradleymeck | can't type today |
| 22:52 | <wanderview> | right now ws.write() does not return a cancellable promise... |
| 22:52 | <wanderview> | this is all hand wving |
| 22:52 | <wanderview> | waving |
| 22:52 | <trevnorris> | wanderview: anything that takes control away from us over the lifetime of the data. but if I have an array of data chunks and only the first has been sent to uv_write() then the others should be able to be removed from their position in the queue. |
| 22:52 | <boogyman> | `var p = ws.write(data); setTimeout(() => p.cancel(), 1000). (Assuming calling p.cancel() does nothing on an already-settled promise) <-- Under what circumstance would be have already been resolved if it is dependent upon ws.write(data) |
| 22:53 | <boogyman> | would p.cancel()* |
| 22:53 | <wanderview> | trevnorris: does uv_write() do file writing on a separate IO thread or the main thread? |
| 22:53 | <Domenic> | boogyman: the adjective "resolved" does not apply to the function call p.cancel(), nor to its return value (which is undefined) |
| 22:53 | <trevnorris> | wanderview: main thread. |
| 22:53 | <bradleymeck> | wanderview: but I think at that point the cancellation point would be if the kernel gets it still, once it gets to the kernel it cannot be stopped, so attempts at cancellation would need to propagate to the C++ that flushes to the kernel, and if it has started flushing it is in an uncancellable state |
| 22:53 | <Domenic> | trevnorris: false? |
| 22:54 | <trevnorris> | wanderview: though I agree that if the data chunks were immediately sent to another thread to be written then we would have "lost control". thus cancel would only be a notification that we no longer need to be notified of its completion. |
| 22:54 | <trevnorris> | Domenic: eh? |
| 22:54 | <Domenic> | trevnorris: fs writes in io are done in a threadpool? |
| 22:54 | <wanderview> | trevnorris: ok, then you don't have to deal with the races I do... trying to pick out a single buffer to cancel is going to be hit or miss in browsers or other multi-threaded environments |
| 22:54 | <bradleymeck> | they are queued on the main thread though |
| 22:54 | <trevnorris> | Domenic: they're a special case. uv_wirte() and uv_try_write() is always done on the main thread. |
| 22:55 | <trevnorris> | *uv_write() |
| 22:55 | <bradleymeck> | ? |
| 22:55 | <Domenic> | trevnorris: ah, I thought we were talking about file I/O since wanderview asked "does uv_write() do file writing". I guess uv_write is for sockets? |
| 22:55 | <trevnorris> | yes. sorry I missed that. uv_write() is only for sockets. |
| 22:56 | <trevnorris> | filesystem I/O is a pain thanks to kernel incompatibilities. |
| 22:56 | <Domenic> | gotcha |
| 22:56 | <wanderview> | Domenic: what I am getting is they treat the "write is complete" state when it leaves main thread... which might not quite be to kernel |
| 22:56 | <Domenic> | yeah |
| 22:56 | <wanderview> | Domenic: which is different from your goal of "bytes written to kernel" |
| 22:56 | <wanderview> | I'm not sure to-kernel is all that much better than an app internal checkpoint |
| 22:56 | <Domenic> | if you are sure that writes respect backpressure, maybe "accepted and queued" is a good enough proxy for upload progress... |
| 22:56 | <wanderview> | but willing to try to support it |
| 22:57 | <trevnorris> | it's possible to share memory on the req across threads so read-only fields can be used to check its status. |
| 22:57 | <wanderview> | trevnorris: you have an atomic check-and-set for cancellation across threads? |
| 22:57 | <bradleymeck> | wanderview: I think at either point cancellation is a suggestion to w/e you handed it off to |
| 22:58 | <bradleymeck> | it needs to be able to continue the work if it has started side effects already |
| 22:58 | <bradleymeck> | no real need for locking to my knowledge |
| 22:58 | <wanderview> | bradleymeck: exactly... I guess I was just getting at I find it hard to reason about the need to cancel buffer1 and let buffer2 through... when you cannot know if you got to buffer1 in time |
| 22:58 | <trevnorris> | wanderview: right now only have it so if I've queued up many small buffers to be written I can check how many of them have actually been sent to the kernel. |
| 22:58 | <bradleymeck> | i don't think you should be allowed to know at the time of cancellation |
| 22:59 | <trevnorris> | wanderview: streaming video for example. say you're buffering data to be written. it's better that frames are lost and the data stays current then making sure all the data goes through. |
| 23:00 | <bradleymeck> | wanderview: to rephrase, I cannot think of a good reason you should be allowed to know that buffer1 was cancelled before it calls .finally |
| 23:00 | <wanderview> | Domenic: I think we are still not on the same page here: https://github.com/yutakahirano/fetch-with-streams/issues/30#issuecomment-91057900 |
| 23:01 | <bradleymeck> | which to me resolves the problem of reasoning |
| 23:01 | <wanderview> | I guess I'm happy we don't yet allow canceling individual .write() calls... and if we did, I think it would very much have to be a "best effort" |
| 23:02 | <wanderview> | bradleymeck: sure... I think I understand better that its a best effort cancel, for example the video streaming case mentioned |
| 23:02 | <bradleymeck> | which is good, I want "best effort" and not guaranteed |
| 23:03 | <bradleymeck> | cause once side effects start / external systems are involved you need to let them resolve back to valid states (say if you have not finished writing) |
| 23:03 | <bradleymeck> | guaranteed cancel would require external systems to stay in valid and buffered states which is a no-no |
| 23:03 | <wanderview> | Domenic: I'll respond in bug... but I think very much "setWriter()" should function like a pipeTo()... we could call it drainToWriter() or something if you want |
| 23:06 | <trevnorris> | wanderview: doing an atomic field set so the writing thread can check if the queue from another thread is still needed is very much possible. |