Add possibility to create additional worker for own transports

[Wysselbie]

Improvement

Additional array in existing/new property in the Store spec, that launches an additional worker for specified transport: bin/console messenger:consume order_export mails.

Reason

In some projects transports are added to change priorities of messages or scale respective workers up for those specific messages.
Some have Symfony configs like this:

framework:
    messenger:
        transports:
            async: '%env(MESSENGER_TRANSPORT_DSN)%'
            mails: '%env(MESSENGER_TRANSPORT_MAILS_DSN)%'
            order_export: '%env(MESSENGER_TRANSPORT_ORDER_DATA_DSN)%'

        routing:
            'Symfony\Component\Mailer\Messenger\SendEmailMessage': mails
            'AcmeIntegrations\OrderExport\Messenger\BulkOrderExportMessage': order_export

Information

It would be enough for an MVP to set replicas to 1.

Definition of Done

StoreSpec is extended by a new array or object field, that lets the operator create one Deployment/ReplicaSet/StoreExec for every configuration in that new array/object. This new resource executes a bin/console messenger:consume <TRANSPORTS> with respectively given transports.

[TrayserCassa]

Hey thank you again for writing an issue @Wysselbie.

I think this problem can be solved by an issue which is currently work in progress.
Currently we are planning to use the prometheus metrics from shopware which is currently in development/discussion phase.

With this prometheus endpoint, the operator can ask how many messages are available for which queue and can scale new worker based on the message size and type of queue which would be the kubernetes/operator way of doing it. How does this sound for you?

Currently we are not planning to expose the worker command overwrite, since the better way would be to let the operator handle the scaling and then we also need to support it or break it. What do you think?

[Wysselbie]

The idea of letting the operator decide by the information from a prometheus endpoint sounds good in general.

I think here is a misunderstanding or I got something wrong. You say the operator should handle "[scaling] new worker based on the message size and type of queue". How does the operator know which type of transports exist? Does it read from Symfony parameters or do I need to provide this information to the operator by adding information to the Store CR?
No matter how, would this feature be optional? I mean the existence and usage of the prometheus metrics from shopware to do so. Since it is not mandatory for a shop to work, I would not see this as required. So the operator should be able to run different workers anyway.
And in this point I think you got me wrong. I do not want to expose an overwrite of something but just having a possibility to run other workers than the default one. Additional workers that handle specific other queues.
Or - another idea - to redistribute the set of queues. E.g. you have six queues and worker A handles the first three and worker B handles the second three. This would in fact have to overwrite the worker command. 😄

What do you say? 🙂

[TrayserCassa]

So currently they are implementing Prometheus metrics into shopware and this would be a requirement for the operator. I don't think there would be a good reason to not require it? 🤔 Because the implementation should be straight forward. You can already check the POC:
https://github.com/shopwareLabs/PrometheusExporterBundle

The plan is, that the metrics report back how many messages are in a specific queue and the operator will scale new workers if messages from a specific queue hit a threshold. At least this would be my preferred way of doing it because you don't need to care about the worker scaling then. And for a low performance shop it would be fine to run one worker with all queues I guess.

The threshold would be defined by the store resource, we will provide a default but you can specify this for each queue. This is also a POC for us so if this feature is not working out we can always provide a way of overwriting the worker command 😄

[Wysselbie]

Sorry, for leaving you hanging. ^^

Okay, now that sounds like a plan to me. And I like how you stand for that decision.
Looking forward to that. 🙂
No matter how, we will get it. 😄