Everything that isn't a completely obvious bugfix (and even some of them) seem to hurt the performance.
So I'm trying to convince myself that I should rewrite the board representation and start using bitboards.
One of the ways (to convince myself) was to shut down everything that had something to do with attacks in the evaluation (which is extremely expensive without bitboards). This means most of the king safety, hanging pieces and mobility, and big parts of pawn evaluation.
Basically a huge part of the evaluation just removed. And the result versus my latest (non-handicapped) version was:
Score of M1B vs M1-1: 529 - 316 - 179 [0.60] 1024
Where M1B was the non-handicapped.
So basically +70 elo points (and probably less if you account for the self-play inflation), for 50% of the evaluation.
I'm sure this isn't completely true, the speed gain is probably worth more in self-play than against other engines (that are already faster).
However, I'll spend some time playing around with bitboards and see where I end up.