Handle Numba cleanup crash in GitHub Actions workflow

[Copilot]

All 974 tests pass, but the workflow fails with free(): invalid next size (fast) during Python/Numba cleanup (exit code 134). Previous fix adding NUMBA_DISABLE_CACHING: 1 did not resolve this.

Changes

• Capture pytest output to detect actual test results independent of exit code
• Ignore cleanup crashes (exit codes 134/SIGABRT, 139/SIGSEGV) when tests passed
• Preserve real failure detection - only ignores crash if pytest summary shows "passed"

set +e
pytest -v --ignore=tests/accuracy --ignore=tests/benchmark -x 2>&1 | tee pytest_output.txt
PYTEST_EXIT_CODE=$?
set -e

if grep -qE "^=+ .* passed" pytest_output.txt; then
  if [ "$PYTEST_EXIT_CODE" -eq 134 ] || [ "$PYTEST_EXIT_CODE" -eq 139 ]; then
    echo "::warning::Pytest cleanup crashed (exit code $PYTEST_EXIT_CODE) but all tests passed."
    exit 0
  fi
fi

Applied to both "Run unit tests" and "Run tests with coverage" steps.

Original prompt

Issue

The GitHub Actions workflow is STILL crashing with a memory corruption error after all 974 tests pass. The previous fix (PR #27) that added NUMBA_DISABLE_CACHING: 1 did NOT solve the problem.

Current Error (still occurring)

============================= 974 passed in 35.11s =============================
free(): invalid next size (fast)
/home/runner/work/_temp/a0f82b2b-93a5-4441-8d8a-78fe5e8cbbca.sh: line 1:  2303 Aborted (core dumped)
##[error]Process completed with exit code 134.

What Was Already Tried (PR #27)

• Added NUMBA_DISABLE_CACHING: 1 environment variable - DID NOT WORK

Required Fix

The crash happens during pytest/Python cleanup AFTER all tests pass. We need to handle this at the shell script level since the tests themselves are successful.

Modify .github/workflows/test.yml

Change the "Run unit tests" step to capture the test result and handle the cleanup crash:

    - name: Run unit tests
      run: |
        # Run pytest and capture output, allowing the command to fail
        set +e
        pytest -v --ignore=tests/accuracy --ignore=tests/benchmark -x 2>&1 | tee pytest_output.txt
        PYTEST_EXIT_CODE=$?
        set -e
        
        # Check if all tests passed by looking at the output
        if grep -q " passed" pytest_output.txt; then
          PASSED_COUNT=$(grep -oP '\d+(?= passed)' pytest_output.txt | tail -1)
          echo "Tests passed: $PASSED_COUNT"
          
          # If tests passed but exit code is non-zero (cleanup crash), treat as success
          if [ "$PYTEST_EXIT_CODE" -eq 134 ] || [ "$PYTEST_EXIT_CODE" -eq 139 ]; then
            echo "::warning::Pytest cleanup crashed (exit code $PYTEST_EXIT_CODE) but all tests passed. This is a known Numba issue."
            exit 0
          fi
        fi
        
        # If we got here and exit code is non-zero, it's a real failure
        if [ "$PYTEST_EXIT_CODE" -ne 0 ]; then
          echo "Tests failed with exit code $PYTEST_EXIT_CODE"
          cat pytest_output.txt
          exit $PYTEST_EXIT_CODE
        fi

Also update the "Run tests with coverage" step similarly:

    - name: Run tests with coverage
      if: matrix.python-version == '3.12'
      run: |
        pip install pytest-cov
        set +e
        pytest --cov=numta --cov-report=xml --ignore=tests/accuracy --ignore=tests/benchmark 2>&1 | tee pytest_cov_output.txt
        PYTEST_EXIT_CODE=$?
        set -e
        
        if grep -q " passed" pytest_cov_output.txt; then
          if [ "$PYTEST_EXIT_CODE" -eq 134 ] || [ "$PYTEST_EXIT_CODE" -eq 139 ]; then
            echo "::warning::Pytest cleanup crashed but all tests passed."
            exit 0
          fi
        fi
        
        exit $PYTEST_EXIT_CODE

Why This Fix Will Work

1. All 974 tests actually PASS - the crash is only during cleanup
2. Exit code 134 (SIGABRT) and 139 (SIGSEGV) are cleanup crashes
3. By checking for "passed" in output and ignoring cleanup crashes, we get accurate CI results
4. Real test failures will still cause CI to fail (they won't have "passed" in output)

Files to Modify

• .github/workflows/test.yml

Acceptance Criteria

• All 974 tests still run and pass
• GitHub Actions workflow completes with exit code 0
• Real test failures are still detected (not masked)
• Warning message is shown when cleanup crash occurs

This pull request was created as a result of the following prompt from Copilot chat.

Issue

The GitHub Actions workflow is STILL crashing with a memory corruption error after all 974 tests pass. The previous fix (PR #27) that added NUMBA_DISABLE_CACHING: 1 did NOT solve the problem.

Current Error (still occurring)

============================= 974 passed in 35.11s =============================
free(): invalid next size (fast)
/home/runner/work/_temp/a0f82b2b-93a5-4441-8d8a-78fe5e8cbbca.sh: line 1:  2303 Aborted (core dumped)
##[error]Process completed with exit code 134.

What Was Already Tried (PR #27)

• Added NUMBA_DISABLE_CACHING: 1 environment variable - DID NOT WORK

Required Fix

The crash happens during pytest/Python cleanup AFTER all tests pass. We need to handle this at the shell script level since the tests themselves are successful.

Modify .github/workflows/test.yml

Change the "Run unit tests" step to capture the test result and handle the cleanup crash:

    - name: Run unit tests
      run: |
        # Run pytest and capture output, allowing the command to fail
        set +e
        pytest -v --ignore=tests/accuracy --ignore=tests/benchmark -x 2>&1 | tee pytest_output.txt
        PYTEST_EXIT_CODE=$?
        set -e
        
        # Check if all tests passed by looking at the output
        if grep -q " passed" pytest_output.txt; then
          PASSED_COUNT=$(grep -oP '\d+(?= passed)' pytest_output.txt | tail -1)
          echo "Tests passed: $PASSED_COUNT"
          
          # If tests passed but exit code is non-zero (cleanup crash), treat as success
          if [ "$PYTEST_EXIT_CODE" -eq 134 ] || [ "$PYTEST_EXIT_CODE" -eq 139 ]; then
            echo "::warning::Pytest cleanup crashed (exit code $PYTEST_EXIT_CODE) but all tests passed. This is a known Numba issue."
            exit 0
          fi
        fi
        
        # If we got here and exit code is non-zero, it's a real failure
        if [ "$PYTEST_EXIT_CODE" -ne 0 ]; then
          echo "Tests failed with exit code $PYTEST_EXIT_CODE"
          cat pytest_output.txt
          exit $PYTEST_EXIT_CODE
        fi

Also update the "Run tests with coverage" step similarly:

    - name: Run tests with coverage
      if: matrix.python-version == '3.12'
      run: |
        pip install pytest-cov
        set +e
        pytest --cov=numta --cov-report=xml --ignore=tests/accuracy --ignore=tests/benchmark 2>&1 | tee pytest_cov_output.txt
        PYTEST_EXIT_CODE=$?
        set -e
        
        if grep -q " passed" pytest_cov_output.txt; then
          if [ "$PYTEST_EXIT_CODE" -eq 134 ] || [ "$PYTEST_EXIT_CODE" -eq 139 ]; then
            echo "::warning::Pytest cleanup crashed but all tests passed."
            exit 0
          fi
        fi
        
        exit $PYTEST_EXIT_CODE

Why This Fix Will Work

1. All 974 tests actually PASS - the crash is only during cleanup
2. Exit code 134 (SIGABRT) and 139 (SIGSEGV) are cleanup crashes
3. By checking for "passed" in output and ignoring cleanup crashes, we get accurate CI results
4. Real test failures will still cause CI to fail (they won't have "passed" in output)

Files to Modify

• .github/workflows/test.yml

Acceptance Criteria

• All 974 tests still run and pass
• GitHub Actions workflow completes with exit code 0
• Real test failures are still detected (not masked)
• Warning message is shown when cleanup crash occurs

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[Copilot]

Pull request overview

This PR addresses a GitHub Actions workflow failure caused by memory corruption during pytest/Numba cleanup that occurs after all 974 tests successfully pass. The approach captures pytest output and conditionally ignores specific cleanup-related exit codes (134/SIGABRT, 139/SIGSEGV) when tests have passed.

Key changes:

• Captures pytest output to files for post-execution analysis
• Implements exit code exception handling for cleanup crashes (codes 134 and 139)
• Applies the workaround to both "Run unit tests" and "Run tests with coverage" steps

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

[Copilot]

[nitpick] Consider adding a comment explaining why specific exit codes (134 = SIGABRT, 139 = SIGSEGV) are being handled. This would improve maintainability and help future developers understand the workaround.

Example:

# Exit codes: 134 = SIGABRT (free() corruption), 139 = SIGSEGV (segfault)
# These indicate Numba cleanup crashes that occur after all tests pass
if [ "$PYTEST_EXIT_CODE" -eq 134 ] || [ "$PYTEST_EXIT_CODE" -eq 139 ]; then

Suggested change

	if grep -qE "^=+ .* passed" pytest_cov_output.txt; then
	if grep -qE "^=+ .* passed" pytest_cov_output.txt; then
	# Exit codes: 134 = SIGABRT (free() corruption), 139 = SIGSEGV (segfault)
	# These indicate Numba cleanup crashes that occur after all tests pass.
	# If all tests passed, we ignore these cleanup crashes and mark the run as successful.

[Copilot]

The grep pattern ^=+ .* passed will match pytest summary lines that contain both "passed" and "failed" (e.g., "5 passed, 10 failed"). This could cause the workflow to incorrectly treat partial failures as success when a cleanup crash occurs.

Consider using a more precise pattern that ensures ALL tests passed:

if grep -qE "^=+ [0-9]+ passed in " pytest_output.txt && ! grep -qE " failed" pytest_output.txt; then

Or check for the absence of failures:

if grep -qE "^=+ .* passed" pytest_output.txt && ! grep -qE "^=+ .* failed" pytest_output.txt; then

[Copilot]

The grep pattern ^=+ .* passed will match pytest summary lines that contain both "passed" and "failed" (e.g., "5 passed, 10 failed"). This could cause the workflow to incorrectly treat partial failures as success when a cleanup crash occurs.

Consider using a more precise pattern that ensures ALL tests passed:

if grep -qE "^=+ [0-9]+ passed in " pytest_cov_output.txt && ! grep -qE " failed" pytest_cov_output.txt; then

Or check for the absence of failures:

if grep -qE "^=+ .* passed" pytest_cov_output.txt && ! grep -qE "^=+ .* failed" pytest_cov_output.txt; then

Suggested change

	if grep -qE "^=+ .* passed" pytest_cov_output.txt; then
	if grep -qE "^=+ [0-9]+ passed in " pytest_cov_output.txt && ! grep -qE " failed" pytest_cov_output.txt; then

[Copilot]

[nitpick] Consider adding a comment explaining why specific exit codes (134 = SIGABRT, 139 = SIGSEGV) are being handled. This would improve maintainability and help future developers understand the workaround.

Example:

# Exit codes: 134 = SIGABRT (free() corruption), 139 = SIGSEGV (segfault)
# These indicate Numba cleanup crashes that occur after all tests pass
if [ "$PYTEST_EXIT_CODE" -eq 134 ] || [ "$PYTEST_EXIT_CODE" -eq 139 ]; then