Test multiple PRs in one run of CI

[jwoudenberg]

Thank you for this great project. I've installed in on a repo and it's running beautifully!

One concern I have with using this for larger projects is that a queue like this is only able to merge as many PRs a day as it can run CI in a day. Ideally CI is quick, but when this isn't possible and the amount of PRs merged a day becomes large one might have to wait a long time for a PR to get merged.

I'd love Kodiak to take a page out of Zuul's book and be able to make us of parallel CI capacity where it is available. That might look something like this:

• Lets say we start out with 5 PRs ready to be merged.
• Kodiak pushes a new branch in which all 5 PRs have been merged together, and creates a PR for it against master (or whichever base branch is used). If CI passes for this combined branch it can be merged, and just like that 5 PRs are checked in a single CI run.
• Of course CI of the combined branch might fail. To hedge against this Kodiak doesn't just push a branch containing those 5 PRs, but also branches containing subsets of those 5 PRs. There's different strategies one might choose here, Zuul picks one that might be interesting to look at. CI will run for these 'fallback' options in parallel. If CI fails for branch containing all 5 PRs it tries to merge one of the other branches.

The benefit of such an approach would be that, given parallel CI capacity, many more PRs can be merged safely within the same period of time.

[chdsbd]

This sounds like a great idea and I see it being a requirement for high traffic projects.

I looked through some prior art and it seems like this feature is called "rollup" or "batch" in other systems. Bors seems to have this as well as derivatives like homu and bors-ng. So you may be able to use one of those solutions to get this feature if you need it right now.

For Kodiak, I think this feature goes beyond the scope of the project as I see it. For me Kodiak is a tool to automate the "merge" button and other things around GitHub PR workflows. I think it would definitely be possible to add this feature to Kodiak, but it would require a significant amount of time and it may be easier to start from scratch.

That said, I would welcome this change in Kodiak, but I don't see myself working on this unless it becomes a problem for me when using Kodiak at work.

[jwoudenberg]

That makes perfect sense. Thank you for your quick response, and the links!