Remote command on Windows gets stuck, doesn't display output

[Ambient-Impact]

Opening this in continuation of drush-ops/drush#4199.

Describe the bug or behavior
On Windows, attempting a drush @alias status (or any command really; @alias is a remote site alias) results in the process running indefinitely with no visible output most of the time. Occasionally, a single line of output appears but the process still doesn't exit.

To Reproduce
drush @alias status

Expected behavior
Process should display output and exit.

Actual behavior
Process doesn't exit, usually no output.

Workaround
Running with -vvv and copying/pasting the ssh command that's generated.

System Configuration

Q	A
Drush version?	10.2.0
Drupal version?	8.8.1
PHP version	7.2.13
OS?	Windows

Additional information
After a lot more digging in addition to the linked Drush issue, the exact location that execution is getting stuck is in Symfony\Component\Process\Pipes\AbstractPipes::write() on line 132:

$written = fwrite($stdin, $this->inputBuffer);

On Windows, Symfony\Component\Process\Pipes\WindowsPipes is used, which extends AbstractPipes and uses temporary files as a workaround for various Windows PHP issues. If you open the temporary files that Symfony generates, the remote site output is all there.

With that in mind, it would seem that this would be a Symfony issue, but then if you bypass Drush, creating and running a PHP file (either via drush php:script or directly) like so:

<?php

require '../vendor/autoload.php';

use Symfony\Component\Process\Process;

$process = new Process(<ssh command>);

$process->run();

print $process->getOutput();

the ssh command correctly executes and exits. Replace <ssh command> with the SSH command generated by Drush. This seems to indicate that there's something in Drush or site-process that's doing something a bit differently, possibly in RealtimeOutputHandler?

File system and stream stuff are outside of my area of expertise, so that's as far as I've gotten. I'm not really sure where to go from here, so I'll leave it here for now in case someone else wants to continue looking into this.

[greg-1-anderson]

What ssh command are you running in your test? Does the command use stdin in any way? I presume not, since your test script does not handle stdin, and site process is hanging trying to provide stdin.

It seems that the fix here is to recognize that $this->inputBuffer was not provided by the client, so we should skip trying to write it to the process object.

[Ambient-Impact]

I've tried both drush @remote-alias status and drush @remote-alias cr, where @remote-alias is, well, my remote site alias. 😛

[Ambient-Impact]

Oh wait, you're probably referring to the SSH command generated by Drush. Here's the full output, with some stuff redacted in uppercase:

λ drush @staging-remote status -vvv
 [preflight] Config paths: E:/Web/Ambient.Impact/Web/vendor/drush/drush/drush.yml
 [preflight] Alias paths: E:/Web/Ambient.Impact/Web/drupal/drush/sites,E:/Web/Ambient.Impact/Web/drush/sites,E:/Web/Ambient.Impact/Web/drush/sites
 [preflight] Commandfile search paths: E:\Web\Ambient.Impact\Web\vendor\drush\drush\src,E:/Web/Ambient.Impact/Web/drush
 [debug] Redispatch hook status [0.51 sec, 8.61 MB]
 [info] Executing: ssh -o PasswordAuthentication=no -i ~/.ssh/ambient.impact.drush ambientimpact@NOPE 'DRUSH_PATH_OMG status -vvv --uri=URI_LOL --root=REMOVED' [0.53 sec, 9.06 MB]

Copying and running the command after Executing: works correctly.

[greg-1-anderson]

So the interesting question is, what is the state of $this->inputBuffer when the fwrite call hangs on Windows? The difference between the working case (your short script above) and the non-working case (called through from Drush) is that Drush is trying to always provide stdin, just in case it is available / needed.

One thing to try: does it work better on Windows if you pass the -n (non-interactive) flag to Drush? Does it work better if you echo '' | drush ...?

Those things will affect how Drush is interacting with Symfony process. It is likely that in the non-working case on Windows, Drush it providing the tty from the terminal, and Symfony process is waiting for a control-D from the keyboard to indicate that stdin is finished. Try non-interactive, and try piped input, and try typing control-D when it's frozen, and see if any of those things affect the behavior.

The solution here is probably going to involve testing in advance (if possible) that there is no stdin available, and change how Drush calls Site Process when there is no stdin.

If the -n flag works, that might just have to be the workaround on Windows if we do not have any way (e.g. similar to the posix commands on Linux / Mac) to determine if there is any stdin available.

[Ambient-Impact]

Doing a crude print $this->inputBuffer at that point, it seems to be an empty string.

-n does not work. However, both echo '' | drush ... and control-D do! Hope that helps.

[greg-1-anderson]

Casual inspection of the code appears to indicate that Site Process handles $input in the same way as the working sample shown in the OP. More investigation into the root cause is needed.

[Ambient-Impact]

(nods) Anything else you'd like me to try?

[greg-1-anderson]

Change your working example above to use code that more closely matches the actual code in Site Process, and see if you can make it break.

Change the code in Site Process to be more like the code in your working example (removing features, just to test) and see if you can make it start working.

Trace through the code and see if you can find something that is setting a mode or changing the state of the Process object in any way that is causing it to break.

[Ambient-Impact]

I'll give those a try one of these days when I get a chance.

[Ambient-Impact]

After some trial and error, I've got this example working in all three cases, printing out the status output:

use Consolidation\SiteProcess\Util\RealtimeOutputHandler;
use Consolidation\SiteProcess\ProcessBase;
use Drush\Drush;
use Drush\Style\DrushStyle;
use Symfony\Component\Console\Input\ArrayInput;
use Symfony\Component\Console\Output\BufferedOutput;
use Symfony\Component\Process\Process;

$process = new Process("ssh -o PasswordAuthentication=no -i ~/.ssh/ambient.impact.drush ambientimpact@NOPE 'DRUSH_PATH_OMG status -vvv --uri=URI_LOL --root=REMOVED''");

$process->run();

print $process->getOutput();


print "\n------------------\n";

$siteAlias = Drush::aliasManager()->get('@staging-remote');

$drushProcess = Drush::drush($siteAlias, 'status');

$drushProcess->start();

$drushProcess->wait(function($type, $buffer) {
  print $buffer;
});


print "\n------------------\n";

$stdin = new ArrayInput([]);
$stdout = new BufferedOutput();
$stderr = new BufferedOutput();
$drushStyle = new DrushStyle($stdin, $stdout);

$process = new ProcessBase("ssh -o PasswordAuthentication=no -i ~/.ssh/ambient.impact.drush ambientimpact@NOPE 'DRUSH_PATH_OMG status -vvv --uri=URI_LOL --root=REMOVED'");

$process->setRealtimeOutput($drushStyle, $stderr);
$process->run($process->showRealtime());

print $stdout->fetch();

The last one is copied and modified from RealtimeOutputHandlerTest. Is there some way to get Symfony/Site Process/Drush to output directly to the console output or does that not really matter in this test script?

[greg-1-anderson]

Just pass STDOUT into setRealtimeOutput (just like the test, but omit making the buffered output objects) if you want output to go right to the console.

However, outputting to the console is not really testable, so I would way that it's not important to do it in this test script. It would be good to try it out once ad-hoc, just to confirm that it works. After that, you can take it on faith that STDOUT will work the same way as the buffered output stream used in the test.

[Ambient-Impact]

Sorry for disappearing. Been really busy but will get back to this when I have some more time.

[greg-1-anderson]

After that, you can take it on faith that STDOUT will work the same way as the buffered output stream used in the test.

Just to be clear, I mean for this particular test. In general, the behavior of exec / ssh, with tty vs without tty and etc. makes it difficult to predict how capturing vs. non-capturing modes work. This is why confirming that things work right with ad-hoc testing first is a very important step, to ensure that the tests -- which should guard against change in behavior -- are in fact testing against a known good / working state.