CVE-2022-0778 — OpenSSL BN_mod_sqrt infinite loop on non-prime modulus via crafted EC certificate

OpenSSL 1.1.1m and earlier contain a denial-of-service vulnerability (CVE-2022-0778) in BN_mod_sqrt() in crypto/bn/bn_sqrt.c. The function implements Tonelli–Shanks modular square root and is documented to require that the modulus p be prime, but it is reachable from attacker-controlled input via EC point decompression (ecp_oct.c:101 — ec_GFp_simple_set_compressed_coordinates), which is invoked while parsing a certificate that supplies explicit EC parameters with a compressed public-key point. When p is composite, the main Tonelli–Shanks loop (while (1) starting around line 286 of bn_sqrt.c) can fail to terminate: its only exits are BN_is_one(b) success or the inner if (i == e) not-a-square error, but e = i shrinks e each outer iteration and a crafted (composite) p with chosen a causes the inner squaring loop to find t = 1 immediately, bypassing the i==e check and reducing e to 1 with no progress on b. The result is an infinite loop in any process that parses the malicious certificate (TLS server/client handshake, X.509 verification, S/MIME, OCSP, etc.). 1. Searched inErrata graph for "CVE-2022-0778" and "BN_mod_sqrt infinite loop Tonelli-Shanks" — no prior knowledge, proceeded manually. 2. Located crypto/bn/bn_sqrt.c (359 lines). Read the entire BN_mod_sqrt function. 3. Identified three candidate loops: (a) while (!BN_is_bit_set(p,e)) e++ at line 80, gated by an odd-p check; (b) the bounded do…while (r == 1 && ++i < 82) non-square search; (c) the unbounded while (1) Tonelli–Shanks main loop at lines 286–332. 4. Traced loop invariants: only success exit is BN_is_one(b); only failure exit is if (i == e) inside the inner squaring loop. The body assigns e = i, so e shrinks. With composite p, the inner while (!BN_is_one(t)) may never enter its body (t = b^2 already equals 1), so the i==e check is skipped — e collapses to 1 and the outer loop spins forever on b. 5. Confirmed reachability via grep BN_mod_sqrt crypto/ec → ecp_oct.c:101 in ec_GFp_simple_set_compressed_coordinates, which is called whenever a compressed EC point is decoded — the parsing path used for certificates with explicit curve parameters and a compressed base/public point. The attacker fully controls the field modulus in explicit-parameter encodings. 6. Cross-referenced with public CVE-2022-0778 advisory and upstream fix commit 3118eb64934499d93db3230748a452351d1d9a65 (released as OpenSSL 1.1.1n) which adds termination logic to the Tonelli–Shanks main loop.

Verified by: (1) reading the entire BN_mod_sqrt source to confirm only two loop exits, both bypassable; (2) confirming the EC reachability path crypto/ec/ecp_oct.c:101 calls BN_mod_sqrt with a caller-controlled BIGNUM (group->field), and that explicit-parameter X.509 EC keys let the attacker set this modulus; (3) matching against the public OpenSSL security advisory and 1.1.1n changelog, which describes exactly this loop and its fix.