7-Point CI Checklist
Is setting up and maintaining CI really worth it for your project? Not
unless you’re doing it right. Follow the CI checklist to see how yours
stacks up.
With these rules you’ll be well on your way to CI bliss:
-
Fixing it must be a priority
If you let your CI build languish while failing, it will quickly
become like that rotting month-old piece of meat in the back of the
fridge. No one wants it, but no one wants to touch it to throw it out.
-
Runs before and after every merge
A Continuous Integration suite isn’t much good if it isn’t
continuous, is it! You shouldn’t be merging failing code without a
good reason. But, just because two pull-requests passed independently,
doesn’t mean they will both pass together, so you should run the CI
after merge as well.
-
Lint
Often forgotten, linting is an important step in any CI pipeline. If
you want to enforce code styles (or check for common code bugs), CI is
the perfect place to do it.
-
Tests are reliable
False-fails, or flakey tests will make people ignore your CI
altogether. When under pressure developers will just hit rebuild to
“re-roll the dice”. No trust == no value.
-
Builds a reproducible release bundle
One CI run, with some given input should == one output bundle. The
same one, every time. That means no using timestamps in your release
bundle. No using randomness when building your release bundle.
Everything needs to be deterministic. The big advantage here is that
you gain a high degree of certainty that the code you are running is
the code you wrote.
-
Release bundle can be linked back to originating code and build
If your CI system does deployment, you should be able to tie back the
code which is running (or was running at any given time), to the CI
build which produced it. If you can’t, you really shouldn’t have any
confidence that what you’re running in production is actually what you
think.
-
Notifies people when it breaks
Last but not least, your CI build must notify someone when it breaks.
It shouldn’t spam everyone incessantly, especially if you still have
flakey, unreliable tests. That will cause people to tune it out and
ignore failures. Getting the right notification level is a balance.
Ideally it should only notify the person who last pushed (odds are
they broke the build). But the status should be visible passively
before doing a deployment.
Is setting up and maintaining CI really worth it for your project? Not unless you’re doing it right. Follow the CI checklist to see how yours stacks up.
With these rules you’ll be well on your way to CI bliss:
-
Fixing it must be a priority
If you let your CI build languish while failing, it will quickly become like that rotting month-old piece of meat in the back of the fridge. No one wants it, but no one wants to touch it to throw it out.
-
Runs before and after every merge
A Continuous Integration suite isn’t much good if it isn’t continuous, is it! You shouldn’t be merging failing code without a good reason. But, just because two pull-requests passed independently, doesn’t mean they will both pass together, so you should run the CI after merge as well.
-
Lint
Often forgotten, linting is an important step in any CI pipeline. If you want to enforce code styles (or check for common code bugs), CI is the perfect place to do it.
-
Tests are reliable
False-fails, or flakey tests will make people ignore your CI altogether. When under pressure developers will just hit rebuild to “re-roll the dice”. No trust == no value.
-
Builds a reproducible release bundle
One CI run, with some given input should == one output bundle. The same one, every time. That means no using timestamps in your release bundle. No using randomness when building your release bundle. Everything needs to be deterministic. The big advantage here is that you gain a high degree of certainty that the code you are running is the code you wrote.
-
Release bundle can be linked back to originating code and build
If your CI system does deployment, you should be able to tie back the code which is running (or was running at any given time), to the CI build which produced it. If you can’t, you really shouldn’t have any confidence that what you’re running in production is actually what you think.
-
Notifies people when it breaks
Last but not least, your CI build must notify someone when it breaks. It shouldn’t spam everyone incessantly, especially if you still have flakey, unreliable tests. That will cause people to tune it out and ignore failures. Getting the right notification level is a balance. Ideally it should only notify the person who last pushed (odds are they broke the build). But the status should be visible passively before doing a deployment.