Support `Quotable<OsStr>` on Windows

[milesj]

@allenap

I see that these conversions are only available on Unix via #[cfg(unix)]: https://github.com/allenap/shell-quote/blob/master/src/lib.rs#L179 And I'm assuming that this is because it uses OsStrExt, but in practice, this makes interoperable code very awkward to work with.

On Unix I can use things "as-is" but on Windows, I always have to convert everything to a String (or equivalent), which is a lot of overhead.

Is there a reason that as_encoded_bytes() isn't used here? https://doc.rust-lang.org/std/ffi/struct.OsString.html#method.as_encoded_bytes Is it because of the note/caveats?

[allenap]

I didn't and still don't have Windows to hand to experiment so I didn't get around to implementing From for Quotable for this conversion, plus (this is assuming I wasn't aware of as_encoded_bytes at the time) it would have been a conversion that required an allocation – like you say, convert to a String first. But as_encoded_bytes looks like a good fit that could let this work on Windows too.

[allenap]

I've played with as_encoded_bytes() a bit and I'm getting a strong feeling (not yet a conviction) that it's not the right approach. I don't know what would be the right approach on Windows.

How does Windows actually invoke bash for example? Does Bash have to understand UTF-16LE strings? Claude suggests that:

1. Windows Process Creation (UTF-16)
   • Parent process calls Windows API with UTF-16 strings
   • Windows stores the command line as a single UTF-16 string
2. MSYS2/Cygwin Runtime Layer (UTF-16 → UTF-8)
   • Git for Windows uses MSYS2 runtime (a fork of Cygwin)
   • The runtime has a special startup code that runs before main()
   • This startup code:
     • Calls GetCommandLineW() to get the UTF-16 command line
     • Parses it into individual arguments
     • Converts UTF-16 to UTF-8 bytes
     • Constructs traditional Unix-style char** argv array
3. Bash Receives UTF-8
   • Bash's main(int argc, char** argv) receives UTF-8 byte strings
   • Bash treats them as byte sequences (Unix-style)
   • No knowledge of the original UTF-16 encoding

However, when Bash is writing to a pipe the data comes out as binary. MSYS2/Cygwin apparently forces the locale to a UTF-8 encoding, meaning the output can be treated as UTF-8. When writing to a console, MSYS2/Cygwin will convert from UTF-8 to UTF-16LE.

What does this mean for generating escaped strings? This crate is byte-oriented (so far; PowerShell support may change that, in part), and assumes UTF-8 (or at least ASCII without being too opinionated about non-ASCII bytes). That's not going to work for Windows native strings, which are 2-bytes per character; as_encoded_bytes() isn't going to work at all, i.e. I think a conversion from OsStr/OsString is necessary. Perhaps that can be made more ergonomic.

What does it mean that this crate outputs to Vec<u8> and bstr::BString on Windows? Those will contain UTF-8 (or ASCII plus other bytes). They too will require conversion back to Windows native strings for use in constructing command arguments, for example, but perhaps they're being used to write a script that will later be executed on a Unix machine. The user must be aware of that; perhaps the docs could be clearer about what is written to byte-oriented containers.

[milesj]

Thanks for the deep dive, very informative.

I suppose I can look into making everything String on my end to avoid the interoperable issue.

[allenap]

I've just re-read the std::ffi docs, specifically about conversions, and found this (emphasis is mine):

On all platforms, OsStr consists of a sequence of bytes that is encoded as a superset of UTF-8; …

I had not properly taken this onboard, and I was assuming that OsStr was a properly native string.

I'm circling back to your initial suggestion to use as_encoded_bytes().