Skip to content

fix(trainer): return TRAINJOB_COMPLETE when all steps are done#340

Open
priyank766 wants to merge 1 commit intokubeflow:mainfrom
priyank766:fix/local-job-status-338
Open

fix(trainer): return TRAINJOB_COMPLETE when all steps are done#340
priyank766 wants to merge 1 commit intokubeflow:mainfrom
priyank766:fix/local-job-status-338

Conversation

@priyank766
Copy link

What this PR does / why we need it:
LocalProcessBackend.__get_job_status() returns TRAINJOB_CREATED when all steps have finished, instead of TRAINJOB_COMPLETE. This causes wait_for_job_status() to always timeout (600s) on the local backend even when jobs complete successfully. This is a one-line fix in the else branch to return the correct status.

Which issue(s) this PR fixes:

Fixes #338, #315

Checklist:

  • Docs included if any changes are user facing

Copilot AI review requested due to automatic review settings February 28, 2026 06:00
@google-oss-prow
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andreyvelich for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions
Copy link
Contributor

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

  • If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
  • Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

@priyank766 priyank766 changed the title fix(local): return TRAINJOB_COMPLETE when all steps are done (#338) fix(local): return TRAINJOB_COMPLETE when all steps are done Feb 28, 2026
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a one-line bug in LocalProcessBackend.__get_job_status() where the else branch (reached when all steps are in TRAINJOB_COMPLETE state) incorrectly returned TRAINJOB_CREATED instead of TRAINJOB_COMPLETE. This caused wait_for_job_status() to always time out (after 600 seconds) on the local backend, even for successfully completed jobs.

Changes:

  • Fix the else branch of __get_job_status to return TRAINJOB_COMPLETE instead of TRAINJOB_CREATED when all steps have finished successfully.

status = constants.TRAINJOB_CREATED
else:
status = constants.TRAINJOB_CREATED
status = constants.TRAINJOB_COMPLETE
Copy link

Copilot AI Feb 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix correctly addresses the bug, but no test case covers the scenario where all steps reach TRAINJOB_COMPLETE status — which is exactly the path being fixed. The existing test_wait_for_job_status only tests the non-existent job error case. A test that mocks all step statuses as TRAINJOB_COMPLETE and asserts that __get_job_status (or indirectly get_job) returns TRAINJOB_COMPLETE would prevent this regression from recurring.

Copilot uses AI. Check for mistakes.
@priyank766 priyank766 changed the title fix(local): return TRAINJOB_COMPLETE when all steps are done fix(local): return TRAINJOB_COMPLETE when all steps are done Feb 28, 2026
@priyank766 priyank766 changed the title fix(local): return TRAINJOB_COMPLETE when all steps are done fix(trainer): return TRAINJOB_COMPLETE when all steps are done Feb 28, 2026
@priyank766
Copy link
Author

priyank766 commented Feb 28, 2026

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LocalProcessBackend.__get_job_status never returns TRAINJOB_COMPLETE

2 participants