Skip to content

aes-gcm/x86-64: Tweak AVX2 VAES-VCLMUL. #2477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
briansmith wants to merge 1 commit into from
Closed

aes-gcm/x86-64: Tweak AVX2 VAES-VCLMUL. #2477

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 9 additions & 15 deletions crypto/fipsmodule/aes/asm/aes-gcm-avx2-x86_64.pl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -446,19 +446,16 @@ sub _ghash_4x {
my ( $GHASH_ACC_PTR, $HTABLE, $AAD, $AADLEN ) = @argregs[ 0 .. 3 ];

# Additional local variables
my ( $TMP0, $TMP0_XMM ) = ( "%ymm0", "%xmm0" );
my ( $TMP1, $TMP1_XMM ) = ( "%ymm1", "%xmm1" );
my ( $TMP2, $TMP2_XMM ) = ( "%ymm2", "%xmm2" );
my ( $LO, $LO_XMM ) = ( "%ymm3", "%xmm3" );
my ( $MI, $MI_XMM ) = ( "%ymm4", "%xmm4" );
my ( $GHASH_ACC, $GHASH_ACC_XMM ) = ( "%ymm5", "%xmm5" );
my ( $BSWAP_MASK, $BSWAP_MASK_XMM ) = ( "%ymm6", "%xmm6" );
my ( $GFPOLY, $GFPOLY_XMM ) = ( "%ymm7", "%xmm7" );
my $H_POW2_XORED = "%ymm8";
my $H_POW1_XORED = "%ymm9";
my $TMP0_XMM = "%xmm0";
my $TMP1_XMM = "%xmm1";
my $TMP2_XMM = "%xmm2";
my $LO_XMM = "%xmm3";
my $GHASH_ACC_XMM = "%xmm4"; # Different than upsream
my $BSWAP_MASK_XMM = "%xmm5"; # Different than upsream
my $GFPOLY_XMM = "%xmm6"; # Different than upsream

@briansmith briansmith Mar 12, 2025

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: If we can avoid xmm6 then we can avoid a memory store in the prologue and a memory read in the epilogue on Windows.


$code .= <<___;
@{[ _save_xmmregs (6 .. 9) ]}
@{[ _save_xmmregs (6) ]} # Less than upstream
.seh_endprologue

# Load the bswap_mask and gfpoly constants. Since AADLEN is usually small,
Expand All @@ -471,8 +468,7 @@ sub _ghash_4x {
vmovdqu ($GHASH_ACC_PTR), $GHASH_ACC_XMM
vpshufb $BSWAP_MASK_XMM, $GHASH_ACC_XMM, $GHASH_ACC_XMM


# Update GHASH with the remaining 16-byte block if any.
# Update GHASH with the sinle 16-byte block.
.Lghash_lastblock:
vmovdqu ($AAD), $TMP0_XMM
vpshufb $BSWAP_MASK_XMM, $TMP0_XMM, $TMP0_XMM
Expand All @@ -485,8 +481,6 @@ sub _ghash_4x {
# Store the updated GHASH accumulator back to memory.
vpshufb $BSWAP_MASK_XMM, $GHASH_ACC_XMM, $GHASH_ACC_XMM
vmovdqu $GHASH_ACC_XMM, ($GHASH_ACC_PTR)

vzeroupper
___
}
$code .= _end_func;
Expand Down