AutomatedTest - Run Pester Tests in parallel

[ddemeyer]

Since ISHRemote v7.0 the Pesters tests are running on Windows PowerShell 5.1 and on PowerShell 7.4+, depending the remote system they are connecting to (scaled, network, latency,...) it can take between 600s up to 1800s for a single run. In turn 1 hour on the testing alone.

Most *.Tests.ps1 Pester tests are written that they build their test data, test and remove the test data. However, some tests are on central CMS features like Xml Settings, List-of-Values, UserGroups, etc ... changing these values could cause other tests to fail sometimes.

Inspiration on

• Exploring the World of Parallelism with PowerShell - Christian Ritter - PSConfEU 2024 - YouTube

• Add a \Source\ISHRemote\Trisoft.ISHRemote\ISHRemote.PesterSetup.Run.ps1
• Run Serial Tests
• Run Parallel Tests
• Verify and adapt tests that create objects with hardcoded LogicalIds (MYOWNGENERATEDLOGICALIDMAP instead of GUIDs) should be unique per *.Tests.ps1
• Adapt Github Actions and make sure Github Actions detects broken test results and fails the CI/CD process ... experiment by changing the 2nd (Windows PowerShell 5.1) run to AsParallel
• Confirm that it is Faster, it does not make sense to complicate the logging and tracking if it is not 50% or more faster (imho)

Example \Source\ISHRemote\Trisoft.ISHRemote\ISHRemote.PesterSetup.Run.ps1 could be

# RWS/20240703
#Requires -Modules Pester
#Main function parameters must be the first script statement (not counting comments)
param(
    [Parameter()]
    [switch]$AsParallel,
    [Parameter()]
    [switch]$Transcript,
    [Parameter()]
    [string]$TranscriptFilePath
)
try {
    $scriptFolderPath = Split-Path -Path $PSCommandPath -Parent
    $scriptFileName = Split-Path -Path $PSCommandPath -Leaf
    if ($Transcript) {
        if (-not $TranscriptFilePath)
        { $TranscriptFilePath = Join-Path -Path $scriptFolderPath -ChildPath "$scriptFileName.log" }
        Start-Transcript -Append -Path $TranscriptFilePath
    }

    if (!$AsParallel) {
        # regular serial run
        Invoke-Pester -Output Detailed 
    } else {
        $fileInfos = Get-ChildItem *.Tests.ps1 -Recurse
        $transcript = $true
        $transcriptFolderPath = "C:\GITHUB\ISHRemote\Log"
        
        Write-Host ("Filtering out serial tests...")
        $parallelFileInfos = @()
        $serialFileInfos = @()
        foreach ($fileInfo in $fileInfos)
        {
            if ($fileInfo.Name -eq "TestPrerequisite.Tests.ps1") { <# $serialFileInfos += $fileInfo  #tests for empty __ISHRemote folder #> }
            elseif ($fileInfo.Name -like "*IshOutputFormat*") { $serialFileInfos += $fileInfo }
            elseif ($fileInfo.Name -like "*IshLovValue*") { $serialFileInfos += $fileInfo }
            elseif ($fileInfo.Name -like "*IshBackgroundTask*") { $serialFileInfos += $fileInfo }
            elseif ($fileInfo.Name -like "*IshEvent*") { $serialFileInfos += $fileInfo }
            elseif ($fileInfo.Name -like "*IshUserRole*") { $serialFileInfos += $fileInfo }
            else { $parallelFileInfos += $fileInfo }
        }
        
        # some Xml Settings tests cannot be run in parallel, so some serial some in parallel
        Write-Host ("Launching serial tests...")
        $serialResults = @()
        foreach ($fileInfo in $serialFileInfos)
        {
            $serialResults += Invoke-Pester -Output Detailed -Path ($fileInfo.FullName)
        }
        
        Write-Host ("Launching parallel test jobs...")
        foreach ($fileInfo in $parallelFileInfos)
        {
            $transcriptFilePath = Join-Path -Path $transcriptFolderPath -ChildPath ($fileInfo.Name + ".joblog")
            # Create the job, be sure to pass argument in from the ArgumentList 
            Start-Job -ArgumentList $fileInfo,$transcript,$transcriptFilePath -Name $fileInfo.Name -ScriptBlock {
            param(
                [Parameter()]
                $localFileInfo, 
                [Parameter()]
                [bool]$Transcript,
                [Parameter()]
                [string]$TranscriptFilePath
            )
                if ($Transcript) {
                    Start-Transcript -Path $TranscriptFilePath
                }
                # Run the test file
                Write-Host ("Invoke-Pester on localFileInfo.FullName["+($localFileInfo.FullName)+"]")
                $result = Invoke-Pester -Output Detailed -Path ($localFileInfo.FullName)
                if ($result.FailedCount -gt 0) {
                    throw "Start-Job calling Invoke-Pester ended with 1 or more assertions failed!"
                }
            } 
        }
        Write-Host ("Awaiting  parallel test jobs...")  # DEBUG whole poll block can be written cleaner
        # Poll to give insight into which jobs are still running so you can spot long running ones       
        do {
            Write-Host ("Polling for running jobs at["+(Get-Date -Format 'yyyyMMddHHmmss')+"]")
            Get-Job | Where-Object { $_.State -eq "Running" } | Format-Table -AutoSize 
            Start-Sleep -Seconds 5  #DEBUG
        } while ((get-job | Where-Object { $_.State -eq "Running" } | Measure-Object).Count -gt 1)
        # Catch edge cases by wait for all of them to finish
        Get-Job | Wait-Job
        
        Write-Host ("Uncompleted parallel test jobs...")
        $parallelFailedJobs = Get-Job | Where-Object { -not ($_.State -eq "Completed")}
        
        Write-Host ("Results of parallel test jobs...")
        # Receive the results of all the jobs, don't stop at errors
        Get-Job | Receive-Job -AutoRemoveJob -Wait -ErrorAction 'Continue'
        
        Write-Host ("Uncompleted parallel test jobs so throwing...")
        if ($parallelFailedJobs.Count -gt 0) {
            $parallelFailedJobs
            throw "One or more Jobs failed"
        }
        
    }

}
finally {
    if ($Transcript)
    { try { Stop-Transcript | out-null } catch {} }
}