It's not just the timing of enabling PromiseHook. It is that the portion of an async function before the first await behaves differently from the rest of the function because the context capture will have already happened around the outer await before passing over the inner awaits. It's confusing to users that the context is not flowing out.

The proposal as it current stands is maintaining the lexical consistent value inside a async function body across await for AsyncContext.Variable, which is part of the approach aiming to address the dynamic scope concerns from the previous meeting's feedback.

Yes. That is actually what users keep telling me they are expecting.

(See the warnings about rust tracing::Span in async code https://docs.rs/tracing/latest/tracing/span/struct.Span.html#in-asynchronous-code for some consequences of this sort of design in the wild -- but they're better off because Futures are expressive enough to write Future::instrument without needing built-in help)

sure, people rewriting code from callbacks to promises might prefer resolution time, but what about people writing async/await code from scratch? Do we know what they need? Would we be breaking them?

Flowing around is essentially just local variable scope which, to me, doesn't really seem to have much value. I've asked a few times for an explanation or some use case why anyone would want that flow but have not yet got any answer on that. 🤔

even with .set(), I don't think this would be dynamic scoping, at least not in the sense that https://github.com/tc39/proposal-async-context/blob/master/SCOPING.md discusses

One concern I have with the mutable variable idea is how one is supposed to bound the mutation - I think it's common to make a change with a lifetime, which is why we've gravitated toward the run() encapsulation. As you've pointed out, with the flow-through semantics, run is very clearly the wrong name, and set is much more appropriate. But your example also hints at some mechanism whereby changes to the variable would still somehow go out of scope? I think that's important, but it seems like it would lead to lots of confusion. If some other scope is entered internally, would that inner scope accidentally prune off a branch of var mutations? That also doesn't seem right.

There's not really any writing on it presently, but basically because we don't flow into promise continuations we need to keep an object at the top-level of the request which we push spans into and basically guess what the parent is because, unlike with callbacks, the parent does not flow into the continuation. This means we have to look at what has been pushed into that list in the past and try to guess what the parent would be. Sometimes it's simply the last span in the list, but not always, so we need to do some additional analysis to figure out what was the last span which could be a realistic parent, which requires we also hold a bunch of the span metadata. Holding all this metadata is expensive given that we could have hundreds of thousands of concurrent spans. What we want is for the correct context to just be able to flow through the resolve path so we only need to store the IDs and can entirely eliminate that big bag of data which lives for the whole life of the request.

Unfortunately it is a form of dynamic scoping, but from callee to caller. This was another one of the points I had to discuss with the SES folks to get them on board with the proposal. The way they reasoned about it is complicated, but essentially boils down to an implicit param to every function that is the async context mapping.

Honestly I have to agree with others who pointed out that some in TC39 will see this version to be "not well behaved"

I just have to admit the use cases are valid and there are many places where We Need Both .... makes things way more complicated tho

my experience with mutable dynamically bound variables in Factor was negative. It can be surprising how much or little the write to the variable is propagated up/backwards and then read by someone else. .run has a simple answer to this question: it isn't propagated up/backwards.

so we ended up rediscovering that setting a dynamically scoped variable was an anti-pattern, and we moved towards doing .run instead

Yes, basically every other language flows this way with OTel, but other languages also provide things like bytecode manipulation so it's able to do this externally. With Node.js there is no solution.

I don’t understand this. What are the inner and outer awaits? In my mind, the async context is bound when we enter the function, it doesn’t matter where the awaits are.

Are there links to issue reports for this?

It’s a local variable scope that propagates to child calls automatically. I explicitly want this becasue it’s understandable, and easy to debug.

I think this is what I’m missing. I don’t understand why could would need to guess, I’m naively assuming its always follows-from the last child and child-of the current parent. What’s a case where this isn’t correct? How does the flows-through solve holding holding the hundreds of thousands of concurrent spans?

I'm not actually sure I understand this. As with the current proposal, you would still need to have access to an object in order to set or get, so isn't it exactly parallel to setting a variable defined in an outer scope?

What do you mean by breaking out of awaits?

As far as I'm aware, JavaScript is the only language that thinks (for some reason) that this needs to be handed over to the user rather than just letting them modify the current scope.

.NET also has AsyncLocal, which is basically exactly what was originally proposed in Node.js before AsyncLocalStorage was chosen instead, because people didn't trust the Node.js runtime to manage scopes correctly. https://learn.microsoft.com/en-us/dotnet/api/system.threading.asynclocal-1?view=net-8.0

I don't necessarily disagree, but set() establishes a communications channel that's not limited to one microtask and its outgoing scoped data flow

But this is also why I prefer instance-scoped bind as the default--you're not messing with a global graph and therefore influencing unrelated things.

See:

And yes, global wrap would still be available but, as I've said, I would strongly advise against its use.

but resolve and then would use global wrap, and it seems like they would still allow this channel

is there any documentation anywhere about what SES needs from new proposals?

lol no it’s more of a vibe. We have asked them for these docs several times over the years and their response is “yes that is a great idea, someone should definitely write that”

is there any documentation anywhere about what SES needs from new proposals?

and yet in https://github.com/endojs/endo/pull/1424 some of them were saying that it should be up to proposal authors to make sure their proposals work with SES??

I'm not actually sure I understand this. As with the current proposal, you would still need to have access to an object in order to set or get, so isn't it exactly parallel to setting a variable defined in an outer scope?

The spec, as it is presently, calls for capturing the context at the point where an await happens, not where the promise being awaited resolves. The context flows into that promise, and it would flow all the way to the point of resolving, but then we orphan that branch and return the context back to the captured value of that outer await.

In this particular example though, it's using store.enterWith(...) which effectively sets the value in the current scope without creating a new scope first. Because the initial segment of an async function executes synchronously, if that occurs before the first await of that internal function then the point at which the outer function captures its await will have already change its context and so will adopt that inner context. Whereas if that inner function modifies the current context after the first await within that inner function it will have already passed the point where the outer function captured the await context and so it would not use that value. The fact this sometimes changes the outer function context and sometimes doesn't is extremely confusing. Users are expecting it to always change the outer context, as indicated in the issue, because it is logically a continuation and should therefore behave the same as it would were it written with callbacks instead.

The issue I just posted yesterday was exactly this, and we get numerous reports expecting this behaviour in Datadog support escalations. I've seen it come up many times before in many places too--Slack conversations, conversations at collab summits...I've never seen anyone that expected the behaviour as it is presently. It's possible that no one talks about it because people don't complain about what they think is already working, but given that the vast majority of context management usage is APMs and all the APM devs I've talked to have complained about these context flows as problematic I feel like there's a pretty good chance we're getting it wrong.

Consider this code:

async function doSomething() {
  await doAThing() // first span
  await doAnotherThing() // no span
  await doYetAnotherThing() // second span should be follows-from first
}

If we have call doSomething() twice in a Promise.all(...), which we do not instrument or have any particular awareness of--maybe it's user code--then we'll get two spans being created for the first awaits of each, then when the second spans come in they will no have a way to differentiate because the doSomething() itself would have the same context when it runs both times and would be restoring that over the awaits, blowing away the context we had from the inner function calls which we otherwise could associate with if we had flow-through semantics.