Wait-RSJob and Receive-RSJob pipeline streaming

[Ashnal]

Is there currently a way to output Jobs as they complete from the pipeline?

Currently, if you run code like this:
1..100 | Start-RSJob -Throttle 25 {Start-sleep -seconds 2; $_} | Wait-RSJob -ShowProgress | Receive-RSJob

You will see the progress track, but no output is produced until all jobs complete. Would it be possible for Wait-RSJob to output jobs as they complete, rather than all at once?

Receive-RSJob has this same problem. It collects all pipeline input during process and waits to spit it all out in the end block. I'd much rather see it output Jobs during the process block.

This is incredibly important when there is a chance of a job hanging. Some WMI calls can hang in a pretty bad way, and allowing streaming output means that any jobs that finished (usually all but one) can have their output written via Export-csv or similar, even if one hangs.

Is there a way to submit a fix for this? I ended up writing a working fix for both Wait-RSJob and Receive-RSJob before I pressed the submit button.

[proxb]

The Wait-Job was by design with everything being outputted at the end to mirror the PSJobs infrastructure with Wait-Job. That being said, if you want, you can fork my repo and add the changes and then submit a pull request to me so I can review the code.

I see your point about being able to stream the jobs as they complete onto the next cmdlet/function in line and will need to give this some thought. I think an extra parameter to support this might be a good idea to allow the jobs to stream through Wait-RSJob. With Receive-RSJob, there definitely is no reason why that shouldn't be streaming the output vs. display in the End block.

[martin9700]

I was going to suggest a switch to control the behavior. -Passthru or something. Not what Passthru traditionally does, but in this case you're no longer outputting the job object but the job output so I think it could work.

[Ashnal]

Is there a good reason for Wait-RSJob to behave that way beyond it mirroring Wait-Job?
I can't think of any case where streaming the output would be detrimental (this applies to Wait-Job as well). Whenever you are piping the output of Wait-RSJob to another pipeline enabled cmdlet/scriptblock there will always be a performance benefit and no downside. Anyone that wants all the jobs to be processed at once can simply assign the output of the Wait-RSJob pipeline to a variable, then process that variable after, no real need for a switch due to how the pipeline functions.

I'll take a look at the forking and pull request. I've not used Github at all beyond downloading. I have an edit that should keep all original functionality intact while enabling streaming output.

[proxb]

Well, to me the idea of a Wait type of command is that you are waiting for everything to complete before moving on. Allowing items to be passed thru the pipeline while it is supposed to be waiting means that it really isn't "waiting" anymore. This is based on the verb guidance here: https://technet.microsoft.com/en-us/library/system.management.automation.verbslifecycle.wait(v=vs.85).aspx

[Ashnal]

That's actually a good point. I admit I have never really used jobs since I read from many sources that they didn't perform well with the overhead for a large number of jobs. I jumped onto the runspace concept pretty much as I was learning powershell after seeing it in practice and reading about it on a number of articles (including your own), eventually settling on Invoke-Parallel for my needs.

When I look at Wait-RSJob I see it as a method to wait for a job to finish before returning. I also know that I would always want to see those jobs either in my console as they process (if I'm not piping it anywhere or capturing the output), or have the next command in line to immediately start processing on that object for performance.

To be precise, the description of Wait you linked specifies pausing an operation before continuing. That is a vague enough definition that it could also be applied to just the one object or the whole collection. Even if it is streaming output, it's still waiting for the job to complete before sending it off, and the command still won't return completely until all jobs are processed. It just helps performance within a pipeline without affecting its non-pipeline behavior. It's still waiting, just on a per object context, instead of a whole collection context. The usage and end result is still the same, but it provides a bit better pipeline performance and a built in countermeasure to hung runspaces.

Also, there isn't any functionality currently present in the module that allows for jobs to .be started and returned as soon as possible down a pipeline without writing your own custom version of a wait function.

[martin9700]

Another option would be a switch in Receive-Job: -Wait, or -Streaming, something along those lines that would change its behavior.

[mg07647]

One workaround is to do something similiar to the following after starting the jobs:

while (Get-RSJob)
{
Start-Sleep -Milliseconds 200
$done = Get-RSJob |
Where-Object State -in 'Completed','Failed','Stopped','Suspended','Disconnected'
$done | Receive-RSJob
$done | Remove-RSJob
}

[Ashnal]

True, you can do that. At that point you're literally just re-writing Wait-RSJob though. Wait-RSJob already does a loop like that, except it waits for all jobs to be finished and returns the jobs, rather than feeding them to Recieve-RSJob. It's expected that you do this after.

I'd much rather it either be implemented by default via the pull request I made, or made into a switch on Wait-RSJob as someone else suggested. My preferred workaround was to just alter the behavior of Wait-RSJob myself.

[proxb]

Have you been able to re-submit a PR against my latest updates? I have more people hit me up in favor of streaming for Wait-RSJob and would like to integrate that as well as RecieveRSJob into the module.

[Ashnal]

Sorry, I only just got back from a small 4 day vacation, and quite sick at that too. I'll see if I can make a new fork today and resubmit

[proxb]

Cool! I've merged the PR for Receive-RSJob. Thanks for that! If you want, go ahead and do another for Wait-RSJob.

[proxb]

Fixed with latest PR.