It left a rook unprotected and did not even calculate for it being captured. After the rook move he also claimed being ahead 500 points (equal to a rook) while in fact he was going to lose it.
I sat for for 5(!) hours yesterday and 5 hours today trying to figure out what had happened.
At first I thought the quiescent search was the problem, since I could not reproduce the error without it. But after playing a number of games with a 2-ply non quiescent search, the bug appeared in a game again.
After putting in a few trace fen-outputs I noticed the engine was searching positions like the one to the left. With both kings and rooks appearing from nowhere.
I tried cutting off games every time more than two rooks appeared on the board, in unmakeMove(), makeMove(), quiescentSearch(), alphaBeta(), and main(). But every time the game found more than two rooks the problem was nowhere to be found, they had just appeared out of nowhere, and I could not trace it back to the source.
When I reached the position in the diagram I decided to replay the game by hand (writing the moves manually instead of playing them in winboard). It was then I realized the engine did not update the castling rights correctly.
The solution? Change the line:
if(white_castle != CASTLE_NONE && black_castle != CASTLE_NONE)to this:
if(white_castle != CASTLE_NONE || black_castle != CASTLE_NONE)In the makeMove()-method in the Board-class.
That line is supposed to determine if a castling rights check is nescessary. If noone can castle, we do not need to check for castling right changes. But in the old line we only check for castling right changes if both sides can castle. Obviously we need to check for changes even if only one side can castle.
If we do not update the castling rights, the engine thinks he can castle and calculates the moves, but since the makeMove()-method simply puts a new king and rook on the board, without any further checks, it is imperative that the castling is correctly updated. Or else kings and rooks will be popping up out of nowhere.
Afterwards it feels like I should have realized this much earlier since there were rooks and kings appearing from nowhere, but no other pieces.
Anyway it is fixed now and it will be changed in the next download.