Git Bisect Fails
git bisect is a nice Git tool to narrow down a bug to
a single commit in logarithmic number of steps. After giving this tool an upper
bound (for first known broken commit) and the lower bound (for last known
working commit) you can move quickly and divide the commit set in two halfs.
The next step will decide if the commit that breaks a feature is in the upper
half or the lower one.
git bisect works well, when you can reproduce the failure
deterministically. It means that you need to know instantly
if something works or not, for example by have a fixed
procedure to make a test, even when you have to do it manually.
Non-determinism and probability
The tool is not going to work well when there is a chance that your test lies about if something works or not. Once you end up in the wrong half, you will not find the commit, but a totally irrelevant commit.
This is a totally annoying situation and you will find
failures for same commits that you considered safe to use
in earlier runs. But you will notice it far later when you
get the idea to check commits twice, because it is
usually hard to produce an overlapping
bisect run in
this situation. It can drive you crazy when you find out
that in the end also the latest commit is sometimes without
the bug that you are looking for.
One of the worst kinds of failures are those which save
a state somewhere and you don't notice it. Beginning with
this commit, you often get bad commits and at the end of
the search, you notice that you have not found the bug.
Then you try to approach from other range of
commits and only find bad commits. When you notice that
you cannot find any safe commit anymore you will perhaps
find the state that causes this or... you if you have
bad luck you will hit a commit that resets the state.
And in this case, again, you get commits that are good
that have been considered bad before, mostly after
one full run that turned up nothing valueable.
Analyze the character of the bug. This is not always easy and sometimes not possible, but think about it if the bug can be non-deterministic or stateful.
When you notice something strange about a
bisect run, try
to look at what you search for and compare it to the log entries.
Also go directly for the extremum and check from time to time
if you would end up at the top or bottom commit. It saves
time in these weird cases.
And finally, stop looking for such kind of bugs with
as soon as possible, because it is a huge waste of time. Look
for other ways how to reproduce them more reliably.