ML-DSA-44 Wycheproof verification test failures

[tarcieri]

I added Wycheproof verification tests to ml-dsa in #1176, where we discovered the following tests are failing for ML-DSA-44 only:

• Test #18: signature with a repeated hint (Invalid)
• Test #68: public key with t1 component set to zero (Valid)

[tarcieri]

cc @bifurcation @orenyomtov

[tarcieri]

Regarding this failure: Test #18: signature with a repeated hint (Invalid)

I believe this is the relevant part of FIPS 204, from Section 7, Algorithm 21 HintBitUnpack:

If I'm reading that right, the if y[Index-1] ≥ y[Index] then return ⊥ is the relevant check and hints should be strictly monotonically increasing, which is what "signature with a repeated hint" is testing for.

So I agree with @orenyomtov's suggestion to change the check.

Regarding the signature round trip test failure it seems to be causing, that would suggest our signing implementation is producing signatures with repeat hints, but I still need to confirm that.

[tarcieri]

Note: full repro in #1186

If I change the following:

fn monotonic(a: &[usize]) -> bool {
    a.iter().enumerate().all(|(i, x)| i == 0 || a[i - 1] <= *x)
}

to:

fn monotonic(a: &[usize]) -> bool {
    a.iter().enumerate().all(|(i, x)| i == 0 || a[i - 1] < *x)
}

Then occasionally it will produce a signature with hint cuts that look like the following, containing a repeat:

[3, 3, 11, 19, 29, 30]

...which is rejected upon decoding by the new monotonic check.

Here is the test case that reproduces the issue:

const SEED: [u8; 32] = hex!("fa9f6c4d8fc0722c0dce5dbdf304da290c90eb9bb8b7d14bbb30870c78fef8d8");
const MSG: &[u8] = &hex!("f0a31bf33dd70cc121d13ad8f080f4b2ba1339112ce465eb8066c1b9d417ac17ee67c7314904486afe667ab369cfd88785b382413f4990271bfd4afddeebe239d697a55f5d109cdea407e107ed3ab05de590f6cce0091b78612d3a80f8e0d8c564c6eab78432cf6338ec8fe7cb11ad54f324d7eb223b29f6fe5ab1265569a66da73789236c4a32dabe9a1c6e9adba23ba383e70d3b04bb03c81e1ebfa63c79ba95c3e219a817ddf418d78f31388d1afaf194fca716a125d2928a4e65c187a48f7a5787dad01ae597c6dc801084b472a18161a6c40426b9d5cc845e52a9fd64b6893b");
const RND: [u8; 32] = hex!("6f00571d19e63f29170c3024b59520a09b12dd0314d61807d59a3875e5d8998b");

[tarcieri]

I think this might be the fix required: #1187

[tarcieri]

Fixed in #1187