Excon::Error::NotFound - Heroku Delayed Jobs

[patricklindsay]

Since upgrading to 2.2.0 I've been receiving the following error when a job is created in Heroku.

Excon::Error::NotFound: Expected([200, 201, 202, 204, 206, 304]) <=> Actual(404 Not Found)

  File "/app/vendor/bundle/ruby/2.4.0/gems/excon-0.59.0/lib/excon/middlewares/expects.rb", line 7, in response_call
  File "/app/vendor/bundle/ruby/2.4.0/gems/excon-0.59.0/lib/excon/middlewares/response_parser.rb", line 9, in response_call
  File "/app/vendor/bundle/ruby/2.4.0/gems/excon-0.59.0/lib/excon/connection.rb", line 389, in response
  File "/app/vendor/bundle/ruby/2.4.0/gems/excon-0.59.0/lib/excon/connection.rb", line 253, in request
  File "/app/vendor/bundle/ruby/2.4.0/gems/heroics-0.0.24/lib/heroics/link.rb", line 111, in request_with_cache
  File "/app/vendor/bundle/ruby/2.4.0/gems/heroics-0.0.24/lib/heroics/link.rb", line 66, in run
  File "/app/vendor/bundle/ruby/2.4.0/gems/heroics-0.0.24/lib/heroics/resource.rb", line 28, in method_missing
  File "/app/vendor/bundle/ruby/2.4.0/gems/platform-api-2.1.0/lib/platform-api/client.rb", line 1431, in info
  File "/app/vendor/bundle/ruby/2.4.0/gems/workless-2.2.0/lib/workless/scalers/heroku.rb", line 24, in workers
  File "/app/vendor/bundle/ruby/2.4.0/gems/workless-2.2.0/lib/workless/scalers/heroku.rb", line 12, in up
  File "/app/vendor/bundle/ruby/2.4.0/gems/workless-2.2.0/lib/workless/scaler.rb", line 21, in block (2 levels) in included
  File "/app/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb", line 446, in instance_exec
  File "/app/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb", line 446, in block in make_lambda

On the brightside updating did fix DJ logging in Heroku. (#89) However will have to downgrade until can figure out what is causing this.

[paulcc]

We've had similar problems, as have the people reporting issues #104 and #105, and it basically is caused by https://github.com/lostboy/workless/blob/master/lib/workless/middleware/workless_checker.rb

There's a few related problems which (if you're unlucky) combine to cause havoc with apps, as outlined below.

I'm more than happy to help fix some of these! Despite issues like this, Workless is very useful for our apps and we'd like to see it improve. We've got partial fixes for most of them, and can help migrate those into the gem if anyone else is interested.

• It is attempting an API call on each request handled by the web dynos, so for an app with multiple dynos and a lot of traffic, this is painful. At the very least, it should be caching the results across the whole app and only retrying the API call every few minutes - very few apps need the background workers to be checked for every single request.

• The call to the API is completely unprotected, so when it fails due to API unvailability (including it being disabled and when you've exceeded the request rate), it throws an exception - and throws it at the rack level, which makes it a bit trickier to deal with from what I've seen.

• With no caching, this could end up throwing exceptions at a significant rate! which has a knock-on effect for whatever you are using to manage & monitor exceptions.

• Plus, it's probably not the end of the world if the scaling call does fail - perhaps the exception from the API level should be caught, logged, but not rethrown. In our apps, we have other mechanisms to track a backlog of jobs and this seems to work well.

• Is it appropriate to do the scaling check at the Rack level? Given the various things that can go wrong, wouldn't it be better as a before-filter in selected controllers? Plus, in settings like Heroku, such as something running every 10 minutes under the scheduler, so it might be better to have a range of options for triggering the scaling functionality and allow it to be configured according to the usage patterns of the app in question.

• We tried removing the checker from the app's middleware list, eg as in http://guides.rubyonrails.org/rails_on_rack.html#deleting-a-middleware, but it seems to have no effect. It looks like the Railtie def is adding the middleware later on - are you sure this is the right way to add in the middleware?

Another trick we use with workless is to only check for jobs that need running up to a few minutes ahead, so we never keep the dynos alive if no jobs need to be run in the next few minutes. It's an easy thing to add.

Hope this is useful!

[paulcc]

Also seems relevant to a few other issues, like #56 #87 #74 #72 #85

Worth addressing #81 as well.

[patricklindsay]

Interesting! @paulcc You have examples of the fixes? I'd be interested in forking the gem and fixing this. Currently we've stopped using the gem. Instead we're using Heroku Scheduler as a work around but the minimum check time is 10 minutes.

Workless is a great concept but pretty much unusable with that middleware checking on every single request. I agree moving it out into possibly a before_action and checking at a configurable interval sounds good.

@davidakachaos, @lostboy Anyone working on this/still maintaining?

[drsharp]

Hear hear! I'd love to have this working. I inherited an old Rails 3 app that used workless and I'm getting this exact same error (after upgrading to Ruby 2.3 and Rails 5). In my case, there is only one background job which handles the importing of data via CSV. And it's something that is actually done maybe once or twice a year, so workless was a good fit. I don't know of any other way to run a long-running process on Heroku but not keep the worker dyno up for all that time that nothing is happening.

I'd be happy to beta-test any fixes or a fork if need be. @paulcc feel free to let me know.

[patricklindsay]

@drsharp You could use Heroku Scheduler for that by the sounds of it.

Just set it up to run rake jobs:workoff every day or so.

[drsharp]

Thanks for that suggestion. That may work. The real problem, in my case (and for which workless was the perfect solution), is that this is a once-in-a-blue-moon action taken by an admin to upload a datafile and have it process in the background. If it was processed via scheduler, then they'd have to know that uploading the datafile just means that within the next 24 hours or so it'll get processed. Also, since this is a once or twice a year activity, then 363-364 days, the scheduler task would run and do nothing. Not really a big deal, but wasted cycles and some wasted cost (minimal I know).

My case is perfect for: very rarely, have a controller action spawn a background job to do some data processing, then otherwise be done with it. Again, at least as I understand it, this is exactly the use case that workless does well.

Anyway, for me, it's back to the drawing board or maybe use the scheduler as you suggest and just change the client's expectation.

[unmultimedio]

In case it works for some of you: https://stackoverflow.com/q/49022441/1755300 (same case, Excon issue when deploying in Heroku)

[patricklindsay]

I've created a fork which;

• Removes the Middleware, replacing it with a controller mixin which can be easily consumed and overridden if required. To prevent request rate issues the check to see if any workers are required is staggered by 1 minute. This is configurable via Workless.work_off_timeout
• The check now filters by run_at time as well as failed jobs. This prevents workers from being sat idle whilst waiting for a scheduled job.
• Updated docs

I haven't yet tested it in production and they're a couple doc issues which I need to update but I'm happy to create a PR if someone is willing to merge it. Otherwise I'll create a tag on the fork within a couple of days and people can use that if they wish.