gh-131298: eliminate HACL* static libraries for cryptographic modules

[picnixz]

In #130157, I actually statically linked HACL* modules but this has issues with freeze.py. Instead, I'm reverting back to dynamic linking when possible. .However, I don't know how we can make a dynamic linking for BLAKE2 and for HMAC (considering the underlying objects need to be compiled with possible SIMD instructions, I don't think it's possible to provide a shared library, or am I wrong here? honestly, I'm not really good at linking issues :D)

This supersedes #119320 but this does NOT simplify the build process (configure & co are still messy IMO but I've explained the whys in #132438 (comment))

cc @msprotz

• Issue: Undefined references to HACL symbols when statically linking Python 3.12 #131298

[picnixz]

I'm leaving my dev session for now so I'll be back tomorrow if needs arise.

[picnixz]

Mmh, I'll need some configure hacks when only BLAKE2 is present and other hash functions are disabled (otherwise the build bot will never build and it's not very useful).

[msprotz]

I don't think there's any issue with shared objects containing simd instructions, as long as the files are compiled with compiler flags consistent with their suffix (e.g. -mavx2 only for Hacl_*_Simd256) prior to linking into a shared object (which we already do properly).

Thanks for looking into this!

[picnixz]

Ok, it's not really a simplification of the build, but I managed to eliminate static libraries.

[picnixz]

!buildbot FIPS

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @picnixz for commit 8d86ff6 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F132438%2Fmerge

The command will test the builders whose names match following regular expression: FIPS

The builders matched are:

• AMD64 CentOS9 FIPS No Builtin Hashes PR
• AMD64 CentOS9 FIPS Only Blake2 Builtin Hash PR
• AMD64 RHEL8 FIPS No Builtin Hashes PR
• AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR

[picnixz]

Ok, so for WASI I don't know what to do :D

[picnixz]

Ok, so nothing was fixed. I give up. I don't know how to do it for WASI. Maybe static linking is the only way to do it?

[picnixz]

Ok, so here is how I eventually fix:

• HACL* can be built statically or dynamically. In other words, it's not possible to just use Setup.stdlib.in. Why? if I were to specify a .o file, it would actually create rules automatically and I won't be able to customize the HACL* build itself.
• So, what I do is the following: I just say "ok build this extension module but don't bother about the dependencies, I'll tell you which ones are dependencies".

For regular builds, there are two kinds of dependencies:

• the Makefile rule prequisites.
• the actual files to pass to the linker when building the final *.so or *.a file.

For instance, to build the HACL*-based MD5 module I first need to build the HACL* MD5 part and then build md5module.o itself, and finally link _md5.so.

We have:

Modules/_hacl/Hacl_Hash_MD5.o: $(srcdir)/Modules/_hacl/Hacl_Hash_MD5.c $(LIBHACL_MD5_HEADERS)
	$(CC) -c $(LIBHACL_CFLAGS) -o $@ $(srcdir)/Modules/_hacl/Hacl_Hash_MD5.c
$(LIBHACL_MD5_LIB_STATIC): $(LIBHACL_MD5_OBJS)
	-rm -f $@
	$(AR) $(ARFLAGS) $@ $(LIBHACL_MD5_OBJS)

This step builds the HACL* part. The $(LIBHACL_MD5_LIB_STATIC) rule may not necessarily be used later but it's the rule needed to build statically the HACL* MD5 part. It's as if I was building a separate project.

Then, I have explicitly defined:

MODULE__MD5_DEPS=$(srcdir)/Modules/hashlib.h $(LIBHACL_MD5_HEADERS) $(LIBHACL_MD5_LIB_SHARED)
MODULE__MD5_LDEPS=$(LIBHACL_MD5_LIB_SHARED)

The first variable is for everything related to rule dependencies. In this example, to build the md5 (cpython) module, I need the HACL* library and some other stuff. When calling the linker, I only need to pass the .o files but I also need them to be built separately, which is why they are marked as LDEPS. The LDEPS variable is only used when I want to indicate additional linker rule prerequisites. Example:

Modules/md5module.o: $(srcdir)/Modules/md5module.c $(MODULE__MD5_DEPS) $(MODULE_DEPS_SHARED) $(PYTHON_HEADERS)
	$(CC) $(MODULE__MD5_CFLAGS) $(PY_STDMODULE_CFLAGS) $(CCSHARED) -c $(srcdir)/Modules/md5module.c -o Modules/md5module.o
Modules/_md5$(EXT_SUFFIX):  Modules/md5module.o $(MODULE__MD5_LDEPS)
	$(BLDSHARED)  Modules/md5module.o $(MODULE__MD5_LDFLAGS) $(LIBPYTHON) -o Modules/_md5$(EXT_SUFFIX)

Note that the rules for Modules/_md5$(EXT_SUFFIX) are automatically created by makesetup and I cannot just change them. I need to tell makesetup that I have an extra prerequisite for that rule but I cannot tell it via Setup.stdlib.in. So I introduced the LDEPS variable instead. Note that it can be different from the LDFLAGS variable (in this case, it's not).

---

Now what was the issue with WASI? well... before, I didn't add the .a file as a rule prequisite (namely, I forgot to also include it in the DEPS variable). Because of that, when executing the libpython rule, there was a missing prequisite, which is why the static HACL* library was never built, leading to a missing file error. It's because LIBPYTHON was built before the extension modules were built.

[picnixz]

It works!!

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @picnixz for commit c87bb27 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F132438%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[picnixz]

All failures are unrelated or already known. So I think we're good with this change. I'll now actually check if freeze.py works (I actually didn't test this...).

The following works:

$ ./configure -q --prefix="$(pwd)/build" --libdir="$(pwd)/build/lib" --exec-prefix="$(pwd)/build" && make -j12 && make install
$ echo 'print("Hello world")' >> hello.py
$ ./python ./Tools/freeze/freeze.py -o frozenhello hello.py
$ make -C frozenhello

[asottile]

looks like Modules/Setup was missed as part of this patch -- I opened #133012 to update that as well!