NID_sha
designates SHA-0, the direct ancestor of SHA-1. SHA-0 was originally called "SHA" but soon a modified version, called SHA-1, was published; the previous "SHA" was then declared obsolete, and is now traditionally called "SHA-0". The actual reason for the change was not made official, but it was generally understood that SHA-0 had some sort of weakness -- which was found by independent researchers a few years later, and used to produce collisions. Thus, SHA-0 is "broken" (much more than SHA-1) and you should not use it.
For RSA signing with SHA-256 you should use NID_sha256
(for SHA-512, NID_sha512
). If you use NID_sha
, then you get a wrong signature. Internally, the signing process includes a transform where the hash value (the m
/m_len
parameters to RSA_sign()
) is padded with a header which identifies the hash function (which is why RSA_sign()
must have access to this information, through its type
parameter). If you use NID_sha
then this header will say "this is a SHA-0 hashed value, of length 32 bytes" which is doubly wrong: a SHA-0 hash value has length 20 bytes, not 32; and this is not a SHA-0 hashed value, but a SHA-256 hashed value.
So basically your signer produces a signature which departs from the official RSA signature standard (PKCS#1). So your signature will not be verifiable with a compliant verifier. However, your verifier also departs from the standard, in the same way: by using NID_sha
as parameter for RSA_verify()
, you instruct the verifier to expect a "SHA-0" header (and a 32-byte hash value). This explains why things appear to work with your code: your verifier does the exact same mistake than your signer, and the two errors cancel out.
In other words, you are not signing with the true RSA, but a variant thereof, which, at first glance, is no less secure than the genuine thing, but still different and therefore not interoperable. Maybe this is not an issue in your specific situation, but, generally speaking, for cryptographic operations, you should stick to the letter of the standard (because security weaknesses are subtle things). If you use NID_sha256
when you sign a SHA-256 hash value, then you get a standard-conforming PKCS#1 v1.5 RSA signature, and that is much better.
To sum up: forget NID_sha
, use NID_sha256
if the hash value comes from SHA-256 (NID_sha512
for SHA-512), both for RSA_sign()
and RSA_verify()
.