BLD: fix building against numpy dev by neutrinoceros · Pull Request #140 · liberfa/pyerfa

neutrinoceros · 2024-03-08T08:22:42Z

This should fix the error seen in lastest nightly builds

neutrinoceros · 2024-03-08T09:17:36Z

If I understand the upstream change correctly, there is no C API that works both in numpy 1 and numpy 2 here. In other words, my patch fixes building against numpy 2 but breaks building against numpy 1. In the long game, this isn't necessarily an issue since building against numpy 2 is supposed to be the one way to produce backward compatible artifacts, so we should soon require numpy>=2.0 at build time anyway, but in the interval between now and the actual release (candidate), there's the question of how we should handle it in CI.

It should be possible (and relatively easy) to use pre-processor directives to preserve build time compat with numpy 1, so I'll try that.

neutrinoceros · 2024-03-08T10:08:41Z

I've made a couple unsuccessful attempts at keeping build-time compat with all both versions but now I feel I don't actually understand how it's supposed to work (if at all).
From my standpoint, it looks like the API that's available at runtime depends on the installed version of numpy, but we also can't build a universal binary, regardless the version of numpy we use at compile-time, since it would require invoking undefined symbols (i.e., dtype->elsize is undefined in numpy 2, and PyDataType_ELSIZE is undefined in numpy 1).
I must be missing something here, since the natural conclusion from here would be that we can't build backward compatible binaries, in direct contradiction to the big promise in numpy 1.25 and 2.0.

neutrinoceros · 2024-03-08T10:16:05Z

@seberg your input would be invaluable here 😅

seberg · 2024-03-08T10:22:03Z

You should add:

// Backported NumPy 2 API (can be removed if numpy 2 is required)
#if NPY_ABI_VERSION < 0x02000000
#define PyDataType_ELSIZE(descr) ((descr)->elsize)
#endif

since you don't want to break compiling with NumPy 1.x, but otherwise looks good.

seberg · 2024-03-08T10:23:49Z

        if (types[i] == NPY_VOID && dtypes != NULL) {
            int op_descr_type_num = op_descr->type_num;
-            int dtype_elsize = dtypes[i]->elsize;
+            int dtype_elsize = PyDataType_ELSIZE(dtypes[i]);


Suggested change

int dtype_elsize = PyDataType_ELSIZE(dtypes[i]);

npy_intp dtype_elsize = PyDataType_ELSIZE(dtypes[i]);

Probably also still wrong in NumPy :). Not that it matters in practice, I don't think NumPy actually allows that large voids yet. Just setting the stage.

of course ! I lost this bit while cleaning the branch, thanks for catching my mistake !

neutrinoceros · 2024-03-08T10:32:50Z

Thanks a lot @seberg !

neutrinoceros · 2024-03-08T10:52:39Z

    "setuptools_scm>=6.2",
    "jinja2>=2.10.3",
-    "numpy"
+    "numpy>=1.25",


see #122 (comment)

I think it's now an actual requirement since my patch is using a constant that's not defined in earlier versions.

I do think that NPY_ABI_VERSION was always defined (if it's not I should change the pattern).

FWIW, I never quite understood what is the advantage/disavantage, but overall it shouldn't matter much? For wheel builds you want it, but it's not technically a strict requirement for building. But if someone builds with --no-build-isolation it is ignored anyway...

Happy to remove it if that's what @mhvk prefers

Let's keep it out - I'd like to introduce a minimum version only if it is actually needed.

pllim · 2024-03-08T14:14:30Z

Thanks, all! Would be nice if you could trigger nightly build after merged via workflow_dispatch. 🙏

pyerfa/.github/workflows/publish.yml

Line 7 in 7847a2f

workflow_dispatch:

mhvk

Thanks, @neutrinoceros! This looks good and happy that the solution from @seberg works! Only, let's not set a minimum version unless it causes failures otherwise.

mhvk · 2024-03-08T16:08:01Z

    "setuptools_scm>=6.2",
    "jinja2>=2.10.3",
-    "numpy"
+    "numpy>=1.25",


Let's keep it out - I'd like to introduce a minimum version only if it is actually needed.

neutrinoceros · 2024-03-08T16:14:13Z

done !

mhvk

Super, thanks!

neutrinoceros · 2024-03-08T16:59:06Z

@mhvk can you re-trigger the publish workflow manually so that it actually uploads and hopefully fix astropy devdeps too ?

mhvk · 2024-03-08T17:53:05Z

@mhvk can you re-trigger the publish workflow manually so that it actually uploads and hopefully fix astropy devdeps too ?

Not sure I know how to do that, cc @astrofrog...

(in the worst case, we make a new point release...)

pllim · 2024-03-08T18:25:49Z

Please go to https://github.com/liberfa/pyerfa/actions/workflows/publish.yml and you should see an option (I don't see it here because I have no write access). It looks like this:

pllim · 2024-03-08T18:27:21Z

Once you have done that, a new entry would appear here:

https://github.com/liberfa/pyerfa/actions/workflows/publish.yml?query=event%3Aworkflow_dispatch

mhvk · 2024-03-08T18:48:47Z

OK, thanks, I've done that and something is afoot!

pllim · 2024-03-08T19:09:17Z

Upload step is green. Thanks!

pllim · 2024-03-08T19:22:38Z

For completeness, astropy devdeps job is green again. Thanks, all!

https://github.com/astropy/astropy/actions/runs/8205309742

mhvk · 2024-03-08T21:29:52Z

Thanks, @neutrinoceros and @pllim! So nice if all I really have to do is to press a button (and even get explained which button to press).

neutrinoceros force-pushed the bld/numpy_2_compat/drop_dtype_elsize branch 4 times, most recently from f0da0c6 to 5250b4b Compare March 8, 2024 09:01

neutrinoceros force-pushed the bld/numpy_2_compat/drop_dtype_elsize branch from f7356d7 to 5250b4b Compare March 8, 2024 10:01

seberg reviewed Mar 8, 2024

View reviewed changes

neutrinoceros force-pushed the bld/numpy_2_compat/drop_dtype_elsize branch from 99ae2fe to ecae457 Compare March 8, 2024 10:33

neutrinoceros marked this pull request as ready for review March 8, 2024 10:33

neutrinoceros commented Mar 8, 2024

View reviewed changes

avalentino approved these changes Mar 8, 2024

View reviewed changes

mhvk reviewed Mar 8, 2024

View reviewed changes

BLD: fix building against numpy dev

bff2fee

neutrinoceros force-pushed the bld/numpy_2_compat/drop_dtype_elsize branch from ecae457 to bff2fee Compare March 8, 2024 16:13

mhvk approved these changes Mar 8, 2024

View reviewed changes

mhvk merged commit 4a637c2 into liberfa:main Mar 8, 2024

neutrinoceros deleted the bld/numpy_2_compat/drop_dtype_elsize branch March 8, 2024 16:51

This was referenced Apr 11, 2024

TST: (weekly cron) install pyerfa from wheels astropy/astropy#16284

Closed

BLD: reintroduce compatibility shim for compiling against numpy 1.x #147

Merged

	int dtype_elsize = PyDataType_ELSIZE(dtypes[i]);
	npy_intp dtype_elsize = PyDataType_ELSIZE(dtypes[i]);

Conversation

neutrinoceros commented Mar 8, 2024

Uh oh!

neutrinoceros commented Mar 8, 2024

Uh oh!

neutrinoceros commented Mar 8, 2024

Uh oh!

neutrinoceros commented Mar 8, 2024

Uh oh!

seberg commented Mar 8, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

neutrinoceros commented Mar 8, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pllim commented Mar 8, 2024

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

neutrinoceros commented Mar 8, 2024

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

neutrinoceros commented Mar 8, 2024

Uh oh!

mhvk commented Mar 8, 2024

Uh oh!

pllim commented Mar 8, 2024

Uh oh!

pllim commented Mar 8, 2024

Uh oh!

mhvk commented Mar 8, 2024

Uh oh!

pllim commented Mar 8, 2024

Uh oh!

pllim commented Mar 8, 2024

Uh oh!

mhvk commented Mar 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants