GCD: Garbled, Corrected, Demonstrandum -- Fixing and Proving Go's Extended GCD Implementation

2026-06-04

AI Summary

This work identifies two critical deviations in the extendedGCD implementation used for RSA key generation in Go’s standard library: an erroneous coefficient update and out-of-spec input domains, both of which violate the algorithm’s invariants. The authors present the first complete formal verification of this implementation by combining deductive verification of Go code using Gobra with machine-checked proofs of key nonlinear arithmetic lemmas developed in Lean and imported into Gobra. This integrated approach not only uncovers subtle implementation flaws but also enables a corrected version that achieves an average 24% performance improvement. The repaired implementation is formally verified for correctness and termination over an extended input domain, ensuring robustness beyond the original specification.

Abstract

We verify the 'extendedGCD' implementation in Go's standard library ('crypto/internal/fips140/bigmod'), which plays a crucial role in the generation of RSA key pairs. Even though the Go implementation is supposedly a direct port from BoringSSL's implementation, we uncovered two deviations that each break the algorithm's invariants: (1) the Go implementation deviates in the way coefficients are updated, and (2) it permits a larger input domain. We address both deviations; the first by fixing the Go implementation, which results in an on average 24% speedup, and the second deviation by porting an existing proof for BoringSSL and extending it to cover the larger input domain. We prove correctness and termination of the fixed Go implementation using Gobra, a deductive program verifier for Go. Where necessary, we used Lean to prove key lemmata on non-linear arithmetic, which we import into Gobra. Our verification effort reveals three key insights: subtle bugs can slip into even well-reviewed code with surprising ease; formal verification is a powerful tool for uncovering them; and AI agents can facilitate the verification process by iteratively refining invariants and lemmata based on Gobra's error messages.

Problem

• extendedGCD
• formal verification
• Go implementation
• algorithm invariants
• input domain

Innovation

• formal verification
• extended GCD
• Go programming language
• Gobra
• AI-assisted verification