Case 64: Calling Convention Changed¶

Field	Value
Verdict	🔴 BREAKING
Category	Breaking
Platforms	Linux
Flags	ABI break
Detected `ChangeKind`s	`calling_convention_changed`
Source files	browse on GitHub

Category: Function ABI | Verdict: BREAKING

What breaks¶

The functions vector_dot and vector_scale change from the default System V AMD64 calling convention to __attribute__((ms_abi)) (Microsoft x64 convention). This changes which CPU registers carry function parameters and return values:

	System V ABI (v1)	Microsoft x64 ABI (v2)
Integer args	`rdi`, `rsi`, `rdx`, `rcx`, `r8`, `r9`	`rcx`, `rdx`, `r8`, `r9`
Float args	`xmm0`–`xmm7`	`xmm0`–`xmm3`
Callee-saved	`rbx`, `rbp`, `r12`–`r15`	`rbx`, `rbp`, `rdi`, `rsi`, `r12`–`r15`

A caller compiled against v1 passes a in rdi, b in rsi, len in edx. The v2 function reads a from rcx, b from rdx, len from r8d — getting whatever stale values those registers happen to hold.

Why this matters¶

Calling conventions define the fundamental contract between caller and callee: - Which registers hold which parameters - How the return value is passed back - Who saves/restores which registers (caller-saved vs callee-saved) - Stack alignment requirements

When this contract is violated, the function operates on completely wrong data. The result is either garbage output, a segfault (if a pointer parameter is wrong), or silent data corruption.

This is particularly dangerous in math/compute libraries where: - Performance-sensitive code may switch conventions for platform interop - The library "works fine" when rebuilt from source — the bug only manifests with pre-compiled consumers

Code diff¶

/* v1: default System V AMD64 calling convention */
double vector_dot(const double *a, const double *b, int len);

/* v2: Microsoft x64 calling convention — different register assignment */
__attribute__((ms_abi))
double vector_dot(const double *a, const double *b, int len);

Real Failure Demo¶

Severity: CRITICAL

Scenario: compile app against v1, swap in v2 .so without recompile.

# Build old library + app
gcc -shared -fPIC -g v1.c -o libfoo.so
gcc -g app.c -L. -lfoo -Wl,-rpath,. -o app -lm
./app
# → dot product = 32.0 (expected 32.0)
# → scaled = {2.0, 4.0, 6.0} (expected {2.0, 4.0, 6.0})

# Swap in new library (no recompile)
gcc -shared -fPIC -g v2.c -o libfoo.so
./app
# → dot product = 0.0 (expected 32.0)
# → scaled = {0.0, 0.0, 0.0} (expected {2.0, 4.0, 6.0})
# → WRONG RESULT: calling convention mismatch — parameters passed in wrong registers!

Why CRITICAL: The v2 function expects pointer parameters in rcx/rdx (ms_abi) but receives them in rdi/rsi (sysv_abi). The function reads from registers that hold stale values (typically zero), producing garbage results. With different register contents this could also cause a segfault if the stale value is dereferenced as a pointer.

How to fix¶

Never change the calling convention of an exported function. If a new convention is needed:

Add a new symbol: vector_dot_fast() with the new convention
Use a wrapper: keep the old entry point as a thin wrapper that forwards to the optimized implementation
Version the API: provide libmath_v2.so with the new convention and a migration guide

Real-world example¶

The Windows ecosystem has dealt with this extensively: __stdcall vs __cdecl vs __fastcall vs __vectorcall have caused countless DLL compatibility issues. The Win32 API froze on __stdcall specifically to prevent this class of break.

Wine (the Windows compatibility layer) must carefully match calling conventions between Linux System V and Windows ms_abi for every thunked function — getting even one wrong causes the exact crash demonstrated here.

Intel's Math Kernel Library (MKL) uses __cdecl for its public API but internally uses __vectorcall for SIMD-heavy routines — the public entry points are thin wrappers that preserve the calling convention contract.

abicheck detection¶

abicheck detects this as calling_convention_changed (BREAKING) by comparing DWARF DW_AT_calling_convention attributes between the two binary versions.