Clustered Mode

[pote]

Having a conversation with @frodsan it came up that he'd like to have a clustered mode similar to Puma's so that he can benefit from running multiple processes easily and take full advantage of his Heroku Dynos.

This doesn't mean we'd get rid of Disc threaded mode - although there are plans to ditch Celluloid, but that's an entirely different discussion.

Initially, this seems like a good idea to me, as I've been thinking of implementing a forking model inspired by Havanna.

As always I'd like to have a space where I can hear opinions for or against this change, anybody have any thoughts on the matter? 😄

[felixbuenemann]

I think it's worth having for cases where you can't or don't want to spawn multiple processes yourself.
With Sidekiq I tend to run each worker process as a standalone upstart service, which allows for more fine grained monitoring and supervision. It also allows to do rolling restarts without andy software support. It also much more complex to manage than a single master process that takes care of his children ;-)

One interesting thing that a controlling master process enables is monitoing the resource usage and doing graceful recycling of workers if they exceed certain memory or cpu limits. This is much harder todo with third party tools, which have less insight into what's going on.

For example in Sidekiq I run my workers in upstart and monitor them with Inspeqtor. If I detect that memory usage is too high (let's say 500 MB), I first tell the worker not to accept new jobs and terminate it to be restated 2 minutes later. This is fine, because if I run 4 workers having one inactive worker is not so bad. But let's say, I have a some bad jobs, that exhausts too much resources on all available workers in a short time frame. Then I'm left with no workers to accept work until at east one has restarted. I's hard too avoid those edge cases in generic monitoring tools, but it's relatively trivial for a master process too decide when it's a smart or poor choice to recycle a worker.

[pote]

Yeap, I've been thinking about this for a while.

As I want to keep Disc as simple as possible, as a starting point I was thinking of including things like upstart/systemd/etc scripts in an examples directory like Sidekiq does so people can set up independent Disc worker processes easily.

However, it might be good to create a separate, optional way to run Disc workers with a master process that can leverage all the benefits you've described. I don't really know what the exact scope of this component would be just yet, but it's great to have a conversation about it here. :)

[felixbuenemann]

As long as you don't have any advanced daemon features that require a master process, sticking with upstart/systemd is fine and much better than reinventing the wheel and writing you own daemon code.

You get service supervision, service dependency management and logging/logrotation for free and there's no need to mess with possibly stale pid files.

Even on old sysv style /etc/init.d systems daemonizing can be handled by start-stop-daemon, but you need to add something like monit for supervision.

One drawback to those tools is that they usually require adding sudo rules to allow deploy users to control the services.