we found an error in the GF(p)-arithmetics of SunEC, while adding
support for brainpool-curves in ECDHE for TLS connections as
suggested in RFC 7027. BrainpoolP256r1 and brainpoolP512r1 worked
out of the box, but brainpoolP384r1 did not. The calculated public keys were
not on that curve and thus we got handshake failures.
After debugging the key generation during an TLS handshake, we came to
the conclusion, that there must be a flaw in ec_GFp_sub_6 from ecl_gf.c.
Using ec_GFp_sub instead worked with brainpoolP384r1.
> The functions ec_GFp_sub_5 and ec_GFp_sub_6 are missing an additional:
> MP_ADD_CARRY(b4, r4, r4, borrow, borrow)
> MP_ADD_CARRY(b5, r5, r5, borrow, borrow)
> in the /* Do quick 'add' if we've gone under 0
> * (subtract the 2's complement of the curve field) *
> we found an error in the GF(p)-arithmetics of SunEC, while adding
> support for brainpool-curves in ECDHE for TLS connections as
> suggested in RFC 7027.
Thanks! I created JDK-8189594 to track this issue. My understanding
is that this error doesn't cause any bugs in the existing JDK code, but
it may cause bugs if we add new curves that use this optimization. If I
am wrong about this, please let me know.