Scheduling Workflows and activities with Postgres Fails sporadically

[radoslav-tomov]

I switched from SQLite to Postgres, and I've noticed that some workflow/activity instances fail to start sporadically. I looked deeper, and it seems that the underlying error is:

time=2026-01-05T07:55:18.986Z level=ERROR msg=mesg err="workflow ProvisionInstanceWorkflow not found"

The workflows and activities are being registered. The same code works for SQLite. I'm gonna try to debug it further, and provide more info, but at this point I lean towards the PostgreSQL integration.

[vr009]

Hi! This error shouldn't be related to the backend implementation - it occurs when the workflow function isn't found in the registry, which is essentially a plain Go map of registered workflows, so please verify there is no conditional logic that might skip registration.

[rajasekarv]

@vr009 I can reproduce the same with high number of parallel tasks per workflow. Not consistently though. If you run it enough times, the issue will surface one or two times. When I was porting this to Rust for a pet project, I found out a race condition in executor.go in workflow. It was sometime back. Let me see if I can find it again.

[radoslav-tomov]

Hi! This error shouldn't be related to the backend implementation. It occurs when the workflow function isn't found in the registry, which is essentially a plain Go map of registered workflows, so please verify there is no conditional logic that might skip registration.

This is what I thought as well, but I have added logging and debugged it. All is registered correctly. In runtime, one call is capable of finding the activity, and the second is not. Very weird. If I switch to SQLite, it works as expected, so I think it is most likely an issue with the Postgres integration. Maybe I'm doing something wrong, but I cannot debug through the entire stack. I suspect that there is a panic that happens somewhere, and due to the recovery, the debugger gets messed up.

[vr009]

@radoslav-tomov Could you please share your worker code snippets, or explain your use case in detail. I can't reproduce the issue so far even under pretty considerable load.