FOSDEM 2026 The creators of security software have encountered an unlikely foe in their attempts to protect us: modern compilers.
Today’s compilers boil down code into its most efficient form, but in doing so they can undo safety precautions.
“Modern software compilers are breaking our code,” said René Meusel, sharing his concerns in a FOSDEM talk on February 1.
Meusel manages the Botan cryptography library and is also a senior software engineer at Rohde & Schwarz Cybersecurity.
As the maintainer of Botan, Meusel is cognizant of all the different ways encryption can be foiled. It’s not enough to get the math right. Your software also needs to encrypt and decrypt safely.
Writing code to execute this task can be trickier than some might imagine. And the compilers aren’t helping.
Blocking the side channel
Meusel offered an example of the kind of problem he deals with implementing a simple login system.
The user types in a password, which gets checked against a database, character by character. Once the first character doesn’t match, an error message is returned.
For a close observer trying to break in, the time it takes the system to return that error indicates how many letters of the guessed password the user has already entered correctly. A longer response time indicates more of the password has been guessed.
This side-channel leak has been used in the past to facilitate brute-force break-ins. It just requires a high-resolution clock that can tell the minuscule differences in response times.
Good thing cryptographers are a congenitally paranoid sort. They have already created preventive functions to equalize these response times to the user so they are not so revealing. These constant-time implementations “make the run time independent of the password,” Meusel said.
Problem solved? Not if the compiler has its way
The GNU C compiler is excellent with reasoning about Boolean values. It may be too clever. Like Microsoft Clippy-level clever.
Meusel ran a constant-time implementation through GCC 15.2 (with -std=c++23 -O3).
The loop to check the character exits early when the character is correct, so GCC assumed the rest of the function wasn’t needed. But the rest of the code that actually fixed the timing was jettisoned, and the side-channel vulnerability was exposed once again. Thanks, GCC.
Meusel didn’t get into why C optimizers have it in for Boolean comparisons, but good C programmers know to fear the aggressive optimization of Boolean logic, which can be hazardous to their finished products.
Boolean decisions mean branching, which is expensive for the hardware, so the compiler would just rather turn your branchful code into branchless control-flow logic anyway, and that’s cool, right?
Is that some Boolean logic you got there?
The trick is to hide the semantics of this little program from the compiler, Meusel advised.
The first step is to replace the Boolean value that the loop is given with a two-bit integer, and use some bit shift or bitwise operations to mask the input (Meusel supplies the requisite code in his talk, so check out the slides for all the geeky goodness).
You would think this would do the job.
But GCC is smarter than that. It can see when you are trying to make a sneaky Boolean comparison.
So you need to apply an obfuscation function to both the input and the output. But not for any benefit to the program itself, but just because these are other values that the compiler could use “to screw us over,” Meusel said.
And, finally, you need to throw the value through some inline assembly code that does absolutely nothing but return the same value. In effect it warns the compiler, which doesn’t understand assembly, to not mess with these values however Boolean they may appear.
“That’s how these things are done nowadays,” he added.
The lot of the encryption coder
“Now you might say, ‘Well, this is becoming kind of like intricate and very easy to do wrong’ … and you wouldn’t be wrong.” Meusel said.
Can it be fair to require the average programmer to understand inline assembly, or any of these other inherently obtuse obfuscation techniques? Much less the maintainers of this software down the road? And how many more tokens would it cost for AI to wiggle through all these subterfuges?
But this is the lot of the cryptography library-coder, finding ways to shore up side-channel attacks while hiding the solutions from a ruthless compiler.
Other compilers no doubt have their own quirks as well. Who knows what horrors hide in the Intel C++ Compiler or Clang. And with each new generation of compilers come a new set of optimizations to contend with, Meusel lamented.
Know the terrain, travel in packs
There are a number of takeaways from Meusel’s talk, the chief one being: maybe just switch off the optimization button on GCC?
Nonetheless, compiler builders may want to consider factors other than code efficiency.
“They want to make your code fast, and they’re really good at it, but they don’t put any other qualitative requirements of your implementation into this consideration,” Meusel said.
As one audience member suggested, perhaps one day a compiler could accept prompts that specify what areas of the code not to tinker with.
Until then, security software designers will need to keep in mind all the parts of the system they are designing, including the development tools that are used, and how they work.
For them, Meusel recommended valgrind, an open source memory debugging tool. It can warn you, for instance, if your program is dependent on an undefined value.
Lastly, implementing security is hard, and not just because of the math involved. It’s too hard to go alone. Don’t roll your own; join a project instead, Meusel advised. ®