Process waiting jobs #59
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit refactors the background queue processing to improve reliability and efficiency. It introduces a dedicated `processJob` method and incorporates broker publishing within the main processing loop. It also adds a mechanism to handle waiting jobs and ensures proper prioritization. Additionally, the changes include database connection checks, schema updates, and the removal of redundant code.
Updates the PHP version in the Dockerfile to 8.2 for improved performance and security. Refactors parameter handling within the BackgroundJob entity for better type safety and efficiency. Uses null-safe operator to simplify conditional formatting of datetime properties.
Adds `Exception` to the `@throws` annotation of the `process` method. This ensures that the method's potential to throw a generic `Exception` is properly documented, improving code clarity and maintainability.
Ensures callback jobs are correctly routed to their specified queues by appending the callback queue name to the base queue name. This prevents jobs from being placed in the default queue when a callback-specific queue is defined.
Ensures jobs are marked as ready before being published to the broker. This prevents potential race conditions where a job might be processed before its state is properly initialized.
Implements recurring background jobs by introducing an `isRecurring` flag to the `BackgroundJob` entity. Adds logic to re-publish the job after completion if the flag is set. This allows jobs to be automatically rescheduled, enabling periodic tasks. Also, instead of postponing waiting jobs, it clones and republishes them.
Adds a `@throws \Doctrine\DBAL\Exception` annotation to the `cloneAndPublish` method. This clarifies the method's potential to throw a Doctrine DBAL exception, which improves code maintainability and helps prevent unexpected runtime errors.
Replaces the boolean flags `isUnique` and `isRecurring` with a `ModeEnum` to represent the job's execution mode. This provides a more structured and extensible way to manage job behavior, allowing for easier addition of new modes in the future. It also removes the need for separate boolean checks, simplifying the code and improving readability.
Removes the `is_unique` and `is_recurring` columns from the background queue table. Replaces the `is_recurring` column with a more generic `mode` column, allowing for different execution modes beyond simple recurrence. This simplifies the schema and provides a more flexible way to manage different types of background jobs.
Ensures database compatibility by setting an explicit length for the mode column, preventing potential errors due to exceeded maximum length during data storage.
Removes the parameters format configuration and always stores parameters as JSON, falling back to serialization if JSON encoding fails. This change simplifies the logic and removes the need for format checking and conversion.
Ensures background job parameters can be serialized even when they contain objects that aren't directly JSON-encodable. It achieves this by first checking if the parameters are "JSONable". If not, it serializes the parameters using PHP's serialize function, providing a fallback mechanism for complex data structures.
Renames the aliased JsonException to its non-aliased version for better clarity and consistency. It also updates the PHPDoc for setParameters method to use the non-aliased exception.
Makes the `getUnfinishedJobIdentifiers` method public. This method retrieves identifiers of background jobs that are not in a finished state, allowing external processes to check for pending or failed jobs. This functionality is required to enable the processing of jobs stuck in a waiting state.
Simplifies queue naming logic by introducing a dedicated `getQueue` method, ensuring consistent queue name generation across the application. This change also allows consumers to process jobs from the default queue without specifying a queue name.
Moves queue name generation with priority to the Manager class for better consistency. The Manager class now provides `getQueueWithPriority` method which is used in both Producer and Consumer. This ensures that queue names are generated consistently across the application. Also, the Producer and Consumer classes are now readonly and dependencies are injected via constructor promotion.
Ensures that the queue parameter is always required when consuming messages. This prevents potential errors and makes the code more robust by explicitly enforcing the presence of a queue name.
Prevents errors when a background job is created without initial parameters. It skips the JSON encoding process if the parameters array is null or empty.
The check and trigger for processing waiting jobs was removed because it's now handled elsewhere, preventing potential duplicate triggers.
Ensures recurring jobs are rescheduled only after successful completion and only if a job with the same identifier doesn't already exist. This prevents duplicate job creation and ensures that recurring tasks are reliably executed.
Corrects the condition for re-publishing waiting jobs. It now correctly publishes a job only if it is not already unfinished, preventing potential duplicate processing.
Simplifies the process of fetching and transitioning waiting background jobs to ready state, improving efficiency. The changes include: - Modifies logic to find oldest unfinished job IDs by group. - Streamlines queue name generation.
Adds an `updated_at` field to the BackgroundJob entity and database table. This ensures that the last modification time of a background job is tracked, allowing for more efficient processing and potentially enabling features based on job update frequency. The update time is automatically updated whenever the entity is persisted.
Ensures that the 'updated_at' column in the database table defaults to the current timestamp. This change addresses a potential issue where the 'updated_at' value might not be initialized properly when a new record is created.
Ensures that the `updatedAt` field is properly initialized when a new background job is created. This allows to correctly track when a job was last modified for the process waiting jobs functionality.
Updates the entity's update timestamp only when the entity is in the 'ready' state. This prevents unnecessary updates to the database record when the entity is not yet ready for processing.
Refactors the job fetching logic to use a single method for retrieving jobs. This change improves code maintainability and reduces redundancy by consolidating the fetching logic into a single, reusable method. It also addresses potential inconsistencies and improves the overall reliability of job retrieval.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.