Skip to content

Comments

99: Basic functioning of multiple workflow types per worker at once.#100

Merged
MarkTNO merged 4 commits intomainfrom
99-allow-a-worker-internal-to-connect-to-multiple-workflow-types
Jun 25, 2025
Merged

99: Basic functioning of multiple workflow types per worker at once.#100
MarkTNO merged 4 commits intomainfrom
99-allow-a-worker-internal-to-connect-to-multiple-workflow-types

Conversation

@lfse-slafleur
Copy link
Member

No description provided.

@lfse-slafleur lfse-slafleur linked an issue Mar 27, 2025 that may be closed by this pull request
@lfse-slafleur lfse-slafleur self-assigned this Apr 8, 2025
@lfse-slafleur lfse-slafleur marked this pull request as ready for review April 8, 2025 08:32
@MarkTNO
Copy link
Contributor

MarkTNO commented Jun 14, 2025

This approach is ok for now, also because this functionality is needed asap.

It does have a draw back since celery tasks do not interleave between queues. The queue which is added first will have the highest priority. Jobs from a specific queue will be executed only when all other higher priority queues (to which the task is connected) are empty. This is not desirable.
This could be solved by having a single queue per worker type (optimizer and simulator queue). This will mean a worker will be connected to a single queue. The added benefit of this is that workers are more flexible and job prioritization will work better.
This is quite some work since the queue names are now equal to the workflow type name.

Any other suggestions are welcome!

@MarkTNO MarkTNO merged commit c2c894b into main Jun 25, 2025
32 checks passed
@MarkTNO MarkTNO deleted the 99-allow-a-worker-internal-to-connect-to-multiple-workflow-types branch June 25, 2025 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow a worker (internal) to connect to multiple workflow types

4 participants