Calling conventions are hard - Fuzz them!

I am busy implementing the C AMD64 calling conventions in my C compiler suite and have a topic worthy of a post. It is about testing the C ABI (How C programs layout structs and perform function calls).

The old Linux C x86 ABI was relatively simple, to call a function you pushed arguments onto the stack in reverse order and you are done with it. Unfortunately for me, most people now use AMD64 processors, so that is what I need to target first. The AMD64 ABI designers apparently didn't like simple or well specified things (presumably because it would make software engineering too easy), so they created this document to describe the way C structs/arguments are laid out in memory and registers among other things.

I have a few problems with the document, such as a lack of examples, lack of pseudo code for the classification algorithm, and underspecified edge cases. However, regardless of whether my complaints are valid or not, I still need to implement the thing correctly before my compiler can self host. I need a good way to test my implementation...

Enter ABIFUZZ

We have a few C compilers like gcc and clang we can test against, but hand writing interesting test cases is a chore, so I decided to automate it. The general steps are quite simple:

• Decide how many arguments you want.
• Decide the types of those arguments.
• Generate values for the arguments.
• Decide the return type.
• Generate a return values.
• Generate code to do the call and check the values.

The tool is located here here and took an afternoon to write. Here's the end result:

The final step is to write a script to split the caller and callee into two files to test interop when each is compiled by different C compiler.

Bugs found:

http://savannah.nongnu.org/bugs/index.php?45950
https://github.com/andrewchambers/c/issues/14
https://github.com/andrewchambers/c/issues/13