Thanks, that helps. With this, I traced it down to https://github.com/jsdom/whatwg-url/blob/a4cb13309246ca9ecf03404fdbf0d23ecaf114dd/lib/urlencoded.js#L10-L33 and https://github.com/jsdom/whatwg-url/blob/a4cb13309246ca9ecf03404fdbf0d23ecaf114dd/lib/percent-encoding.js#L20 which indeed is doing the critically important thing of ignoring things outside the known valid range and leaving them in-tact, whereas EcmaScript's decodeURIComponent throws an error for anything unexpected after a % sign.

I did actually (briefly) consider using jsdom originaly, but found that it (understandably) kind of makes maximum use of other internals. It seems it goes down to the level of code points and then brings all of TextEncoder into it all as well, and UintArray, etc. I struggled to find a minimal way to port that over.

I managed to make currently-failing test cases pass by replacing the blanket decodeURIComponent call with a loop over the string and then if the two chars after % are in-range, call decodeURIComponent on just But then found that this doesn't work for multi-byte characters, e.g. decoding %7F%C3%BF is only accepted when done at once, not when done chunk by chunk. The URL spec doesn't say that, but then again, the URL spec doesnt say to use decodeURI of course anyway, it says to work on bytes and code points and not return to UTF-8 strings until all the way done, which I can't do within ES5 I think, or at least not without a ton of other emulation code.

I did try forcing it by using String.fromCharCode(parseInt(chunk.slice(1), 16));`, but that just produced garbage.

I hate myself for this, but I managed to get all tests passing both old and new cases, by using a regex. That seemed simpler than trying to eagerly buffer up multiple %- chunks or re-implemenitng lower level code point mapping for multi-bytes.

The regex looks only for valid in-range percent encoded chunks and eagerly repeats this so that we always pass decodeURI as many at once as possible.

	function percent_decode(bytes) {
			return bytes.replace(/((%[0-9A-Fa-f]{2})*)/g, function(m, p1) {
				return decodeURIComponent(p1);
			});
	
	}

I feel dirty now, but it seems to hold up in my testing so far. Curious if anyone here knows of a better minimal way to get the multi-byte code points to work in ES5 without "a lot of code" and/or can poke holes in the above, preferably holes that didn't already exist in the previous version of the FT polyfill (which simply calls decodeURIComponent on the whole thing).

Thanks, that gives me a bit more confidence. The use case that led me to discover this bug in the FT polyfill is actually very much related to bytes that are not UTF-8 bytes.

Wikipedia uses UTF-8 for canonical URLs everywhere now and has for almost two decades now. But some of the earliest language editions (Specifically en.wikipedia, pl.wikipedia, and zh.wikipedia) used to use a different encoding, which our backend web server (MediaWiki PHP) continues to support to ensure URLs don't break. If the decoded string isn't valid UTF-8, it will on those wikis fallback to a legacy encoding. For en.wikipedia that is windows-1252, for pl.wikipedia iso-8859-2 and for zh.wikipeida/zh-hans windows-936 and zh/zh-hant windows-950. As per https://codesearch.wmcloud.org/core/?q=fallback8bitEncoding%20%3D

For example, this URL is still supported: https://en.wikipedia.org/w/index.php?title=Apollo%96Soyuz_Test_Project

While we don't have any need to read the "correct" value of the title parameter in JavaScript, we do have many other query params we sometimes read client-side, and so having new mw.Uri (our in-house library) or new URL (polyfiled) throw unconditionally on those pages is a problem. It's fine if we can't read searchParams.get('title') correctly, so long as we can still do other things with it.

Our downstream task (which was meant to be closed by using the URL API) - https://phabricator.wikimedia.org/T106244

My PR (with docs now): https://github.com/Financial-Times/polyfill-library/pull/1173.

I'm still looking into improving the handling of incomplete multi-byte. We are unlikely to encounter those for any "valid" URLs, not even legacy ones, but it still seems better not to crash on those but do a replacement indeed like the standard does.