Expose ABI information outside headers

[encukou]

For non-C languages, it would be great to expose information in a way that doesn't require a C preprocessor/compiler. It should be a Python dict in some module. (Or even a JSON file distributed with each build of Python -- but if code to locate that might be too complex.)

We definitely need:

• Values of #defined constants (e.g. slot IDs, feature flags, bitfield masks)

We might want to add:

• Sizes of types used in the API
• Struct field offsets

(These could live in ctypes.pythonapi, to make sure its struct support is correct.)

See non-C language document.
(PyPy is fine with headers, as they already went through the trouble of writing a C parser & FFI.)

[gvanrossum]

This looks like a combination of tooling and guidelines; ideally the tooling would enforce (this aspect of) the guidelines. My understanding is that the tooling would read and parse (some of) the header files, being careful not to require the C preprocessor.

Someone should probably create a prototype of the tooling and see how much the header files would need to change in order to support everything that's in the headers. If we can't get anyone to work on tooling this probably isn't actionable.

---

(UPDATE: We have a pretty decent C tokenizer written in Python, in Tools/cases_generator/lexer.py, and some rudimentary bits of C parsing infrastructure on top of that, in plexer.py and parsing.py in the same directory. I once wrote a parser for C expressions and statements on top of that. I still have it somewhere.)

[erlend-aasland]

There's also a C pre-processor parser in Argument Clinic, FWIW: Tools/clinic/cpp.py

[zooba]

Or even a JSON file distributed with each build of Python -- but if code to locate that might be too complex

There's currently some work going on to define a not-quite-sysconfig.json file to include with each build of Python, so that it's possible to determine build commands/options/paths without having to actually run Python (crucial for cross-compiling; convenient for other reasons). We can probably put this file in the same place, wherever it ends up being. python/cpython#107956

[FFY00]

I have started spec-ing out a static file (JSON) to be shipped with Python installations, which could include the information here.

python/peps#3599

@encukou would you be available to collaborate on defining the API and ABI sections?

[davidhewitt]

@FFY00 you might be interested in this script which PyO3 uses to query the Python interpreter to configure Rust extension builds:

https://github.com/PyO3/pyo3/blob/6776b90e15317283c733d8ac48d4f335f1ef7412/pyo3-build-config/src/impl_.rs#L191

If there's a static JSON file which we could locate to get the same information, this would be particularly helpful in cross-compile situations where we currently resort to different fallbacks.

We could also potentially consume the ABI information proposed here at the same time.

[FFY00]

Thanks!

If there's a static JSON file which we could locate to get the same information, this would be particularly helpful in cross-compile situations where we currently resort to different fallbacks.

Yes, this is one of the main target use-cases of the proposal.

[encukou]

@encukou would you be available to collaborate on defining the API and ABI sections?

Yes, next week or so.
But the info I'm talking about here is probably too big for the file you're adding in PEP 739. Let's discuss in a PEP discussion thread, rather than here?

[encukou]

Looking at it more closely, the static description file is something different than what I propose in this issue. The description file will need info like:

• Where are the header files?
• What libraries should I link to? (for an extension? for embedding?)
• What's the proper filename extension for extensions?

And so on. For a low-effort, *nix-specifix first approximation, you want the info exposed by python-config script:

click to expand

$ python-config --prefix
/usr
$ python-config --exec-prefix
/usr
$ python-config --includes
-I/usr/include/python3.12 -I/usr/include/python3.12
$ python-config --libs
 -ldl  -lm 
$ python-config --libs --embed
-lpython3.12 -ldl  -lm 
$ python-config --cflags
-I/usr/include/python3.12 -I/usr/include/python3.12  -fno-strict-overflow -Wsign-compare -fexceptions -fcf-protection   -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions -fcf-protection 
$ python-config --ldflags
 -L/usr/lib64  -ldl  -lm 
$ python-config --ldflags --embed
 -L/usr/lib64 -lpython3.12 -ldl  -lm 
$ python-config --extension-suffix
.cpython-312-x86_64-linux-gnu.so
$ python-config --abiflags

$ python-config --configdir
/usr/lib64/python3.12/config-3.12-x86_64-linux-gnu

This issue asks for info like:

• HAVE_FORK is defined
• Py_nb_add is 7
• long int is 8 bytes
• offset of flags in PyType_Spec is 16

(Again: ctypes.pythonapi might make a good home for most of it)

[zooba]

Looking at it more closely, the static description file is something different than what I propose in this issue.

Yes, I linked the two together because of the similarities of distributing a standalone file that has to live in a directory that is somehow discoverable by tools that cannot launch the runtime. I always expected two separate files, I just hoped that they could be next to each other and not in totally unrelated locations :)