Workflows stuck in "pending" in Vercel World

[zzvxdoteth]

Hello all —

I'm in a turborepo / Next.js setup and have a simple agent workflow working perfectly fine locally with workflows initiating and completing as expected. Upon deployment to Vercel, I am able to initiate workflows but they are perpetually stuck in "pending" state, with no logs emitted indicating a fault of sorts.

I'm not sure what other details to provide as it's a fairly vanilla setup to my understanding and there are no error logs indicating a directional issue — happy to provide clarification as needed

Thanks!

[zzvxdoteth]

For whatever it's worth, I'm on a Hobby pricing tier at the moment; I haven't seen anything in the documentation indicating this would be problematic but please correct me if I've missed something.

[zzvxdoteth]

Example run ID if helpful: 01KB4SD8XXFAW8A3JP9YQN7ZDF

[zzvxdoteth]

Looks like Pro tier is required (https://vercel.com/docs/workflow#observability) -- I'd recommend making that info visible on the useworkflow.dev site + within the observability portal as well (or at least not rendering the observability portal for Workflows in non-Pro tiers) as that was non-obvious!

More importantly though, I've upgraded this account to Pro and redeployed the project and the same issue still persists.

[MihaiSandor]

I’m seeing the same issue on Vercel. I also have a large number of jobs stuck in the queue:

wrun_01KB5HYM812EMJXF70AZTCQN34
wrun_01KB5HYD22YT23YB4QKX13NA9Y
wrun_01KB5HY2WN0AP37AC4E5AFMGTM

[hainsdominic]

Yes, no workflows can run, even with pro. Seems like its completely broken in production for now.

[hainsdominic]

@TooTallNate Can you check? Seems like a major incident.

[MihaiSandor]

cc: @pranaygp, @adriandlam, @rauchg

[pranaygp]

thanks for the ping here - we're looking into it right now

[pranaygp]

pro tier should not be required for workflow btw - so that's not the issue

[pranaygp]

@MihaiSandor @hainsdominic @zzvxdoteth what versions of workflow are you on?

[hainsdominic]

@MihaiSandor @hainsdominic @zzvxdoteth what versions of workflow are you on?

Beta 22, I upgraded from 17 to see if it would fix anything & it didn't.

[TooTallNate]

This does not appear to be a widespread outage, so there is something specific going on with your configuration. A few questions:

• Is this happening for all your workflow runs or only some?
• Do you see anything interesting in your deployment logs? I'm specifically curious in the route that calls the start() function.
• Could you share source code that can reproduce the issue?

[zzvxdoteth]

• Workflow version is beta 22
• No logs of note on startup nor on start() (successfully get back a Run object with a runId)
• AFK for a few hours so will share a reproducible snippet later when I can!

[zzvxdoteth]

• Happening consistently across workflows, but I only have one workflow in usage so that's not saying much

[zzvxdoteth]

Perhaps @MihaiSandor @hainsdominic have more useful threads to share!