00:36
<sirisian>

Here's an example:

// Each item has a few pipelined actions that take time indicated by the number. So the first item does 3 operations each taking 100ms
async function* gen() {
  yield* [[100, 100, 100, 'a'], [100, 100, 100, 'b'], [10, 10, 10, 'c'], [10, 10, 10, 'd'], [10, 10, 10, 'e'], [10, 10, 10, 'f']];
}
for await (const i of gen.map(async item => {
	await new Promise(resolve => setTimeout(resolve, item.shift()));
  return item;
}).map(async item => {
	await new Promise(resolve => setTimeout(resolve, item.shift()));
  return item;
}).map(async item => {
	await new Promise(resolve => setTimeout(resolve, item.shift()));
  return item;
})) {
  // This will after summation of all the times in each array.
  console.log(item.shift());
}
00:51
<sirisian>
Handled in-order one must wait 300ms to see the first result. All the operations are handled sequentially taking a total 300 * 2 + 30 * 4 = 720ms. An out of order first come with a task limit of 5 for each step (assume these are network or single worker thread delays) would take 300ms. Even a task limit of 3 would take 300ms. in that example.
02:14
<bakkot>
sirisian: yeah it looks like you're trying to do things with sync iterables of promises, which isn't a concept the language ever really works with, right now. except in like Promise.all I guess.
02:45
<sirisian>
bakkot: https://jsfiddle.net/9xsyev61/ This isn't cursed at all.
02:48
<bakkot>
sirisian: lol
02:48
<bakkot>
looks like it dropped some though? I only see a single 1 in the console
02:48
<bakkot>
oh, wait, that's just the wrapping, sigh
02:49
<bakkot>
that still only works with finite iterators though
02:49
<bakkot>
finite underlying iterators, that is
02:49
<bakkot>
well... maybe it could be made to work with infinite ones, hmm
02:50
<sirisian>
yes it should be able to. Just pass in the iterator.
02:50
<sirisian>
take(5) kind of thing then next to get the next work.
02:54
<bakkot>
oh, but it doesn't compose, because it produces an async iterator rather than a sync iterator of promises
02:54
<bakkot>
so you can't do it twice with different mapping functions
03:26
<sirisian>
bakkot: https://jsfiddle.net/xs1brkL8/ Wait, I don't use these much. Can you detect the break in code like that to print "map done"?
03:27
<bakkot>
sirisian: i don't understand the question
03:32
<bakkot>
oh, do you mean, the break in the bottom loop? yes: it will cause the yield to be a return, and you can use a try/finally to do cleanup
03:32
<sirisian>
oh it throws. I see.
03:32
<bakkot>
it doesn't throw
03:32
<bakkot>
it puts a return
03:32
<bakkot>
a finally will trigger, but not a catch
03:33
<bakkot>
async function* map(tasks, n) {
  try {
    // Start n work
    const parallel = new Set();
    for (let i = 0; !tasks.done && i < n; ++i) {
      parallel.add(makeWork((await tasks.next()).value));
    }
    // Process the work yielding the first one done in the set of n work
    while (parallel.length != 0 && !tasks.done) {
      const task = await Promise.race([...parallel].map(t => t.promise));
      parallel.delete(task);
      yield task;
      // Add 1 more work
      parallel.add(makeWork((await tasks.next()).value));
    }
  } finally {
    console.log('map done');
    tasks?.return();
  }
}
03:34
<sirisian>
https://jsfiddle.net/xs1brkL8/1/ neat, is this composable?
03:35
<sirisian>
In the actual helper setup it would need to be on the iterator, but that should be fine. I could rewrite it do that I think.
03:37
<sirisian>
Oh wait, can you modify all iterators to add functions to them. (I know this would be bad).
03:38
<bakkot>
that's what the iterator helpers proposal is
03:38
<bakkot>
i.e. adding functions to all iterators (all iterators which inherit from iterator.prototype, anyway)
03:39
<bakkot>
I can't quite tell if your thing actually composes properly just from staring at it
03:40
<bakkot>
oh, !tasks.done isn't a thing, though
03:41
<sirisian>
whoops forgot that one. ~~https://jsfiddle.net/xs1brkL8/2/~~ https://jsfiddle.net/xs1brkL8/3/
03:41
<sirisian>
I mean can you edit the iterator prototype right now for testing polyfills?
03:41
<bakkot>
yes
03:42
<bakkot>
re^, you have to check the done property of the result every time you call .next()
03:42
<bakkot>
you have to check done before reading value
03:43
<bakkot>
(to obey the iterator contract, anyway; you don't have to but it's going to behave weirdly if you don't)
03:56
<sirisian>
Wait, how do you edit the iterator prototype? What is it called?
03:57
<bakkot>
it's called [].values().__proto__.__proto__
03:57
<sirisian>
That's how I know this is a good idea.
04:06
<sirisian>
Is the async iterator a different one then?
04:07
<bakkot>
yes
04:11
<bakkot>
(it's (async function*(){})().__proto__.__proto__.__proto__)
04:11
<sirisian>
ah, I was off one proto
04:24
<bakkot>
sirisian: ok I looked again and I am pretty sure this implementation will never start the third phase until at least 5 items from the second phase have finished, which I think is not what you intend
04:25
<bakkot>
where you have await tasks.next() you want to instead have a race with that and the existing work queue items, and if one from the work queue finishes you yield it right away
04:26
<sirisian>
https://jsfiddle.net/xs1brkL8/4/ My current one for reference. Thinking more. Updated: https://jsfiddle.net/xs1brkL8/10/
04:32
<sirisian>
ah, I think I see what you mean since the iterator can take a while to yield. I'll put a sleep in my makeALotOfWork just to make it more clear also and handle it.
05:29
<sirisian>
There's no way to get the Promise from Promise.race along with the value without creating another promise right? Like it returns the value, but if you have say 10 promises you don't know what promise that value corresponds to.
12:19
<Justin Ridgewell>
I made a few concurrent async iterable helpers at https://github.com/jridgewell/minx/blob/main/src/async-iterable-concurrent.mjs
12:40
<annevk>
Why does tc39.es not host ECMA-404?
12:47
<yulia>
That is probably just an oversight..
13:45
<annevk>
yulia: ta, filed https://github.com/tc39/tc39.github.io/issues/283 to track
13:46
<yulia>
yulia: ta, filed https://github.com/tc39/tc39.github.io/issues/283 to track
thanks, we are discussing it in #tc39-website:matrix.org
13:46
<yulia>
we don't seem to have an html source anywhere -- i think it predates that
13:46
<yulia>
so we likely need to write that up
13:46
<yulia>
We will get in touch with Chip
13:47
<yulia>
for now we might just host the pdf under the usual url
13:47
<annevk>
I figured that might be part of the problem, but PDF could be linked until hosting is sorted in the future
13:47
<yulia>
yeah
15:05
<bakkot>
sirisian: you can get substantially simpler, I think: https://gist.github.com/bakkot/6bee327466c06887d96a65f88f4cf728
15:14
<bakkot>
though I notice there's another way this could be configurable, which is, there is both "number of tasks running in the mapping function" and "number of unsettled promises read from the underlying iterator", and those don't actually need to be the same number
15:15
<bakkot>
in my gist I have it so that the sum of those two numbers is n, but they could reasonably be independently configured
15:23
<bakkot>
Justin Ridgewell: hmm, I wonder if we should make iterator helpers work like your mapper...
15:23
<bakkot>
probably not I guess? it would maybe be surprising.
15:25
<bakkot>
though I guess you're only exposed to the difference if you're manually pumping the iterator rather than just using for await
15:33
<Justin Ridgewell>
It’s really difficult to reason about my way.
15:34
<Justin Ridgewell>
You can get a done message before all the previous messages have completed
15:34
<Justin Ridgewell>
I had to write the cap iterator to help me reason about the end states when using the map and interleave
15:35
<Justin Ridgewell>
Even map, you have to be aware of reentrancy during any async portions, which was difficult
15:37
<Justin Ridgewell>
I do not recommend we make standard iterators that behave this way
15:37
<Justin Ridgewell>
Backpressure may be slow but it has a very predictable ordering
15:38
<bakkot>
Yeah, sounds good.
15:39
<bakkot>
Staring at this async code above has been making my head hurt also. Which says there's maybe a place for the language to help out, but as a power tool, not one of the simple helpers.
17:14
<sirisian>
bakkot: Very nice and compact. Also your idea about separating n above is excellent. These problems are definitely fun little puzzles to think about. Your use of first is nice. I asked a question yesterday, but deleted it, about getting the promise instance from a Promise.race when was thinking about things. ( https://jsfiddle.net/yt5n4gLd/ Pasted your solution into a polyfill kind of implementation)