01:10
<Justin Ridgewell>
the primary reason for doing so, aside from the self-contained nature of the resulting promise, is that it will also capture synchronously thrown errors in the conversion code and propagate them as promise rejections
We already lost that battle when we added Promise.resolve.
01:11
<Justin Ridgewell>
I’ve hit the queueing usecase enough times that it seems like a deferred struct is a good addition.
01:16
<bakkot>
ljharb: I lack experience here; what does misuse look like?
03:00
<ljharb>
I’ve seen tons of people use the pattern when it’s not needed; the number of times those legit use cases come up are very rare ime
03:15
<littledan>
I’ve seen this pattern often be needed when adapting a callback-based API to promises
03:15
<littledan>
Otoh I think people overused it back when promises were just coming out, async await was not yet standard, and people didn’t really understand .then
03:16
<littledan>
I think now would probably be a good time to add Promise.defer given the improved environment and continued need
03:17
<littledan>
I don’t think the legit use case is rare. It is just that, before async await, everyone was very confused all the time
03:21
<Kris Kowal>
There is some popular prior art: jQuery's `$.deferred()` exposes the reject/resolve methods, and a `.promise()` method to get the internal promise.
The chain of prior art continues. jQuery’s deferred() comes from the bad chain, because it conflated promise and resolver, allowing people to optionally separate them. jQuery took its cues from Python Twisted Deferred, which in turn took some of its cues from E promises. Whereas, Promise.defer() (from an early draft of the Promises proposal) takes its cues from Q.defer() is more like what @bakkot proposes, which in turn came from MarkM’s proposal for promises back in 2010, then from Tyler Close’s Waterken, which in turn took its cues from E.
03:23
<Kris Kowal>
That is to say, Promise.defer() => {promise, resolve, reject} is sound design (since by default promise and resolve should be held by different parties for POLA purposes), and while I named it “defer” originally as a nod to “Deferreds” in Python’s Twisted, it doesn’t suffer the same design error.
03:24
<littledan>
Yeah it is important that the callbacks returned close over the capability related to the individual promise (this was an issue in V8’s weird defer)
03:25
<littledan>
I was skeptical of defer at some point due to making all the callbacks but that was too much of a microoptimization I think
03:28
<Kris Kowal>
I find myself using Promise.defer() for async queues too, but it’s stuffed in a library and only gets used that way once. But I also get a lot of use of Promise.defer() for 1. broadcasting a drain event 2. broadcasting a fast moving state change to a slow consumer (replace the deferred when the consumer observes the current state) 3. chaining mutual exclusion for stateful protocols or “baton passing”
03:29
<Kris Kowal>
Async queue https://github.com/endojs/endo/blob/master/packages/stream/index.js#L31-L50
03:30
<Kris Kowal>
(At Agoric, we’re calling Promise.defer() makePromiseKit() but Promise.defer() is definitely the right name.)
03:34
<Kris Kowal>
I think we ended up where we were because the Promise constructor needed to have some behavior and Promise.defer() was duplicative, so it could be punted indefinitely. It’s trivial to make a defer() from Promise(), so it was a disappointing concession but easy to recover from.
03:36
<bakkot>
yeah, it makes sense how we ended up here. it's not that it's hard to do the thing given the Promise constructor, just annoying.
03:40
<Kris Kowal>
I for one thing it would be worth the cost to add Promise.defer() => {promise, resolve, reject} to the language.
03:41
<Kris Kowal>
Also noting that I’d also enjoy Promise.defer() => {promise, resolve} since resolve(Promise.reject(error)) recovers the absent reject.
06:24
<Ashley Claymore>
Also noting that I’d also enjoy Promise.defer() => {promise, resolve} since resolve(Promise.reject(error)) recovers the absent reject.
seems like a safe optimisation, Whenever I’ve used this pattern I’ve only ever needed to expose resolve
06:48
<bakkot>
i have definitely needed reject, and it seems odd to make you write resolve(Promise.reject(error)) in that case
09:26
<joepie91 🏳️‍🌈>
We already lost that battle when we added Promise.resolve.
huh? Promise.resolve does not do the same kind of thing - it produces an immediately-settled Promise. the problem with deferred Promises is that they produce pending Promises and expect some outside call to settle it
09:27
<joepie91 🏳️‍🌈>
and keep in mind that adding something like defer to the language spec doesn't just "make it available", it also expresses a strong endorsement of using it, and over the years I've repeatedly found "it's defined by the language" to be a stronger-weighing argument for people than "is this tool appropriate for the usecase".
15:35
<Justin Ridgewell>
How do you get the value you pass to Promise.resolve?
15:35
<Justin Ridgewell>
Likely it’s a function call.
15:36
<littledan>
you await it?
15:36
<Justin Ridgewell>
const value = fn();
Promise.resolve(value);
15:36
<Justin Ridgewell>
That fn() isn’t being caught on anything
15:37
<Justin Ridgewell>
So we’ve arrived at the same problem of not properly catching the rejection in your promise
15:37
<littledan>
well, this is something that the Promise constructor does
15:37
<littledan>
(or maybe I misunderstand what you are asking)
15:38
<Justin Ridgewell>
Exposing a deferred just makes the queue case easier, and it’s impossible to express in any other way.
15:38
<littledan>
(it's clearly already possible)
15:39
<Justin Ridgewell>
People don’t use the constructor, they use Promise.resolve when the value is simple. We already opened that door for them.
15:39
<Justin Ridgewell>
(it's clearly already possible)
How? How else besides storing a deferred and resolving it later?
15:40
<littledan>
right, I mean, it's possible with the Promise constructor. We're not talking about adding a new capability
15:40
<Justin Ridgewell>
Dan, we’re talking about two separate things.
15:40
<littledan>
sorry for my confusion
15:41
<Justin Ridgewell>
The promise constructor vs Promise.resolve. Then whether deferred allow you to not capture a thrown error.
15:42
<Justin Ridgewell>
The first is already there, and people prefer Promise.resolve.
15:42
<littledan>
well, deferred is more powerful than Promise.resolve, because... it's not resolved immediately. I think that's the difference.
15:42
<Justin Ridgewell>
Because of that, adding deferred isn’t opening up anything g that we haven’t already opened.
15:43
<Justin Ridgewell>
https://matrix.to/#/!wbACpffbfxANskIFZq%3Amatrix.org/%24ucmxi_iERR-4n8X_rOaL_yKvlbNpxiaWawsRESJDwcc
15:45
<joepie91 🏳️‍🌈>
I find "we already have a misuse-prone thing, so we might as well add another misuse-prone thing" to not be a terribly convincing argument, personally
15:46
<littledan>
(I'm not free for a call right now if this is what you're suggesting, but I'm happy to drop this subject until we can do so)
15:47
<littledan>
if you're arguing, adding Promise.defer would not be a huge change and would be aligned with what we already have, I agree
15:47
<Justin Ridgewell>
Yes.
15:48
<Justin Ridgewell>
I also don’t think many would reach for it, except those that explicitly need the queueing case.
15:48
<littledan>
you don't think adapting callback-based APIs is also a big use case?
15:48
<Justin Ridgewell>
Can you explain?
15:50
<James M Snell>
Just as a datapoint, I think we'd end up using Promise.defer() fairly extensively on the Node.js internals. We have about 40 or so cases where we currently use that pattern internally, and we have a createDeferredPromise() utility to accomplish this
15:50
<littledan>
like, if you want to make a Promise-based setTimeout, you can do something like let {promise, resolve} = Promise.defer(); setTimeout(resolve, 1000). IMO this is cleaner than using the Promise constructor
15:50
<littledan>
(this is where I've wanted it in the past)
15:51
<joepie91 🏳️‍🌈>
aside from "cleaner" being an ambiguous term, I disagree that it is more readable; you lose the visual grouping indicating that the setTimeout is internal logic for the Promise.
15:52
<joepie91 🏳️‍🌈>
more crucially, it is already possible to write precisely such a createDeferredPromise utility function today, trivially so, and there are several implementations of it in library form. this is not something that needs to be part of the language spec from a functionality perspective
15:53
<joepie91 🏳️‍🌈>
adding something to the language has serious implications and outsized ecosystem costs compared to shipping something in library form, and to be honest it's concerning to me how easily people gloss over the "encourages misuse" problem for something that's ultimately of extremely questionable value to have in the language to begin with
15:54
<James M Snell>
I would argue that the prevalence of utility functions to accomplish it are an argument in favor of adding the mechanism to the language. It would be nice to eliminate a dependency
15:54
<joepie91 🏳️‍🌈>
"a lot of people do this" is, in and of itself, not a good argument to add something to the language
15:54
<joepie91 🏳️‍🌈>
that's how you end up with, say, Python's standard library
15:55
<James M Snell>
well, just offering my opinion on it. I think it would be a useful addition to the language. Not really wanting to argue it beyond that.
15:56
<joepie91 🏳️‍🌈>
lots of things are 'useful additions to the language', though; but usefulness is just one of many factors that should be considered, considering the high costs of language/core additions
16:23
<bakkot>
joepie91 🏳️‍🌈: if you have thoughts on what misuse looks like here, I'm interested to hear
16:24
<bakkot>
the only example so far was just using it when not needed, which, while unfortunate, isn't the sort of misuse I'd be most concerned about, assuming it's similarly performant to the promise constructor
16:27
<bakkot>
"a lot of people do this" is, in and of itself, not a good argument to add something to the language
I definitely agree with this, but it's a necessary-but-not-sufficient sort of thing. if it's something a lot of people do and it's a reasonable thing to do and it's awkward to do with the current mechanisms and we don't think it's going to lead to a lot of confusion or bad patterns, that is a good reason to add it. here I think that might be the case, though with low confidence at the moment.
16:33
<joepie91 🏳️‍🌈>

bakkot: the problem I'm concerned about is one of maintainability rather than performance. for the "making something non-promisey work with promises" usecase, the optimal implementation from a maintainability perspective is one that is a) self-contained and b) minimal; that is, you'd be using new Promise, and its body would contain only the minimal wiring necessary to convert some other non-standard async API into a Promise. that way, the conversion itself can be entirely agnostic to the business logic, and all the actual business logic would exist in a 'safe' Promise-y context (ie. benefiting from the Promise error handling behaviour such as automatic propagation)

this already gets done wrong a lot due to lacking documentation on working with Promises, in that a lot of people tend to put additional business logic within the new Promise, sometimes nesting several levels of (unsafe) async callbacks before eventually resolving/rejecting something, or adding needless complexity by eg. having promise chains nested within a new Promise. but in this case, at least the design of the API encourages keeping resolve/reject logic within that callback - so the problem is usually contained to just the contents of the new Promise callback, which makes it easier to fix.

the problem with a defer API, then, is that the pattern it encourages is "pass your promise to one place, and your resolve/reject functions to another", which I've already seen lead to people passing resolve/reject functions through sometimes multiple layers of indirection (all of them using 'unsafe' async patterns), ending up with a huge pile of spaghetti async code that's nearly impossible to reason about or untangle even when you want to fix it. unlike with the new Promise API, there's nothing that 'nudges' people towards keeping it contained within a single place.

(note: I'm currently sick and someone down the street seems to have just decided to start hammering something again, so I hope that all makes sense, and I don't know how long I can keep being engaged in this discussion for)

17:01
<bakkot>

for the "making something non-promisey work with promises" usecase, the optimal implementation from a maintainability perspective is one that is a) self-contained and b) minimal; that is, you'd be using new Promise, and its body would contain only the minimal wiring necessary to convert some other non-standard async API into a Promise

hm, I think I do not agree with this, or rather I agree in theory if the wiring happens to be cleanly expressed that way (say your sync API is in the ((val, err) => body callback style) but that in many cases the wiring does happen to work out cleanly that way (say an event emitter style, where only certain events or events under particular conditions should be the trigger for the promise settling, and the other cases you're doing something else). this is kind of what your next paragraph gets at:

this already gets done wrong a lot due to lacking documentation on working with Promises, in that a lot of people tend to put additional business logic within the new Promise

I think this is actually encouraged by the design of the Promise constructor, and part of the point of this proposal would be to make it easier to not do that. in my experience the business logic is often pretty entangled with the wiring, and can't be cleanly separated - if you only want to settle after a certain number of events, say, then the current design means that the "business" logic which counts the events has to go inside of the callback to the Promise constructor (or you have to a Promise.defer-style helper), so that it has access to resolve.

which I've already seen lead to people passing resolve/reject functions through sometimes multiple layers of indirection (all of them using 'unsafe' async patterns), ending up with a huge pile of spaghetti async code that's nearly impossible to reason about or untangle even when you want to fix it

that does seem like a bad outcome, thanks for the example.

note: I'm currently sick and someone down the street seems to have just decided to start hammering something again, so I hope that all makes sense, and I don't know how long I can keep being engaged in this discussion for

yeah, no worries, if this does become a proposal there will be plenty of time to comment at that point

17:08
<joepie91 🏳️‍🌈>

I think this is actually encouraged by the design of the Promise constructor, and part of the point of this proposal would be to make it easier to not do that. in my experience the business logic is often pretty entangled with the wiring, and can't be cleanly separated - if you only want to settle after a certain number of events, say, then the current design means that the "business" logic which counts the events has to go inside of the callback to the Promise constructor (or you have to a Promise.defer-style helper), so that it has access to resolve.

to clarify, I would consider "counting the events" in that example to be part of the wiring logic, not the business logic, as it is necessary for the wiring the function - and therefore it belongs within the new Promise constructor. with "business logic" I am really referring to things that have nothing to do with the wiring, eg. subsequent transformations of the resolved value, multiple underlying asynchronous operations, that sort of thing.

this is not the greatest example, but it happens to be from yesterday so I still have it in my history: https://bpa.st/RZVQ#1L41 -- neither the Promise.all nor the JSON.parse/IrcBookState.FromJson logic really belongs inside of that new Promise, as both are business logic unrelated to wiring up non-Promise async APIs into a Promise. now in this specific case, there is no useful wiring within the new Promise callback and that shouldn't have been used at all, hence why it's not a great example :)

17:10
<joepie91 🏳️‍🌈>
but I regularly see basically the same error being made within code that does use new Promise for a legitimate reason, but just puts entirely too much stuff within it that doesn't need to be there
17:23
<bakkot>

joepie91 🏳️‍🌈: yeah, so, I feel like again the "there is too much stuff inside of new Promise" thing is a consequence of the design of the Promise constructor, and would be improved by Promise.defer in many cases.

like, to talk a little more concretely, consider the case of an event emitter which emits a series of chunks and then a done event. you could promise-ify it in a few different ways:

let emitter = makeChunkEmitter();
let chunks = [];
emitter.on('chunk', chunk => chunks.push(chunk));
return new Promise(resolve => emitter.on('done', () => resolve(chunks)));

which has the problem that the two on calls are weirdly separated, which makes it hard to follow, or

let emitter = makeChunkEmitter();
return new Promise(resolve => {
  let chunks = [];
  emitter.on('chunk', chunk => chunks.push(chunk));
  emitter.on('done', () => resolve(chunks))
});

which has the problem that there is logic unrelated to promise resolution inside of the call to the Promise constructor, and that if emitter is null it will incorrectly put the error inside of the Promise even though the problem wasn't inside of the emitter itself at all, or

let emitter = makeChunkEmitter();
let { resolve, promise } = Promise.defer();
let chunks = [];
emitter.on('chunk', chunk => chunks.push(chunk));
emitter.on('done', () => resolve(chunks));
return promise;

which I think is the cleanest, but which has the problem that Promise.defer does not exist.

19:12
<ljharb>
Justin Ridgewell: Promise.try handles that resolve case :-) https://github.com/tc39/proposal-promise-try