Ask HN: How to scale our ci/CD process?

2 points by shlosky 4 years ago | 1 comment

We have a suite of integration tests that take about 2 hours to complete.

Currently our flow is that each feature branch runs the integration tests, and only if the branch is updated from master and the tests completed, it can be merged to master. So naturally, it happens quite a low that you start the tests when your branch is updated, but by the time it ends, there are new changes to master, and you must update your branch and re-run the tests...

We are obviously doing something wrong. But we are not sure what.

Should we allow to merge the feature branch even without updating and testing? it feels risky...

Should we allow to merge, but have another run on master before deployment to production?

detaro 4 years ago |

Few points, how viable they are really depends on what you're doing and how you are doing it.

Can you throw more hardware at it? Parallelize the tests as much as possible, 2 hours is annoyingly long either way.

> Should we allow to merge the feature branch even without updating and testing? it feels risky...

> Should we allow to merge, but have another run on master before deployment to production?

There's a lot of variations on those. Some CI systems can test stacked changes - i.e. they'll merge multiple changesets and test the result, if the test succeeds all included changes are applied. Merging into a staging branch on which the tests are run and only promoting to (deployable) master once tests succeed achieves pretty much the same thing. Some companies use feature-flags as much as possible, so that changes can be merged as early as possible but are only activated in production once they've been tested.

A highly interesting field is also systems to only run the tests that touch code impacted by a change, but that's difficult, esp. for integration tests.