Sugioarto

2015-02-19

Git Bisect Fails

git bisect is a nice Git tool to narrow down a bug to a single commit in logarithmic number of steps. After giving this tool an upper bound (for first known broken commit) and the lower bound (for last known working commit) you can move quickly and divide the commit set in two halfs. The next step will decide if the commit that breaks a feature is in the upper half or the lower one.

Determinism

git bisect works well, when you can reproduce the failure deterministically. It means that you need to know instantly if something works or not, for example by have a fixed procedure to make a test, even when you have to do it manually.

Non-determinism and probability

The tool is not going to work well when there is a chance that your test lies about if something works or not. Once you end up in the wrong half, you will not find the commit, but a totally irrelevant commit.

This is a totally annoying situation and you will find failures for same commits that you considered safe to use in earlier runs. But you will notice it far later when you get the idea to check commits twice, because it is usually hard to produce an overlapping bisect run in this situation. It can drive you crazy when you find out that in the end also the latest commit is sometimes without the bug that you are looking for.

Stateful failures

One of the worst kinds of failures are those which save a state somewhere and you don't notice it. Beginning with this commit, you often get bad commits and at the end of the search, you notice that you have not found the bug. Then you try to approach from other range of bisected commits and only find bad commits. When you notice that you cannot find any safe commit anymore you will perhaps find the state that causes this or... you if you have bad luck you will hit a commit that resets the state. And in this case, again, you get commits that are good that have been considered bad before, mostly after one full run that turned up nothing valueable.

Conclusion

Analyze the character of the bug. This is not always easy and sometimes not possible, but think about it if the bug can be non-deterministic or stateful.

When you notice something strange about a bisect run, try to look at what you search for and compare it to the log entries. Also go directly for the extremum and check from time to time if you would end up at the top or bottom commit. It saves time in these weird cases.

And finally, stop looking for such kind of bugs with git bisect as soon as possible, because it is a huge waste of time. Look for other ways how to reproduce them more reliably.