Add PyLong_GetNumBits() function

[vstinner]

In Python 3.13 alpha 1, I removed the private _PyLong_NumBits() function, but it's used by 3 big/popular projects: pywin32, MariaDB and panda3d.

I propose adding a public function to replace this private API:

Py_ssize_t _PyLong_GetNumBits(PyObject *obj);

• Return the number of bits on success: greater than or equal to zero.
• Set an exception and return -1 on error.
• Set an OverflowError exception, and return -1 if the number of bits doesn't fit into Py_ssize_t.

The C function is similar to the Python int.bit_length() method. On overflow, it's recommended to call the int.bit_length() method which is not limited to Py_ssize_t.

• Issue: [C API] Add PyLong_GetNumBits() function python/cpython#119714
• PR: gh-119714: Add PyLong_GetNumBits() function python/cpython#119715

In the meanwhile, the private _PyLong_NumBits() function was restored in the 3.13 and main branches.

See also the proposed PyLong_GetSign() function.

[gvanrossum]

On overflow, it's recommended to call the int.bit_length() method which is not limited to Py_ssize_t.

Actually, if the number has more than ssize_t one bits, it occupies at least 1/8th of the total available memory space (18 exabytes), so maybe it's time to wave the white flag at that point. :-)

[vstinner]

Actually, if the number has more than ssize_t one bits, it occupies at least 1/8th of the total available memory space (18 exabytes), so maybe it's time to wave the white flag at that point. :-)

Right for 64-bit platforms (most common platforms). But on 32-bit platforms, the OverflowError "only" requires a single number of 273 MiB which is quick to create. Well, IMO Python is not designed to manage such "big integers", so we should not bother much about it.

My first proposition was int _PyLong_GetNumBits(PyObject *obj, size_t *numbits) API (unsigned size), but @encukou and @serhiy-storchaka prefer this API (if I understood correctly): python/cpython#119714 (comment)

[serhiy-storchaka]

I propose to return int64_t or uint64_t to avoid the problem of integer overflow on 32-bit platfoms. This will allow to use this C API in int.bitlength() (currently it reimplements the algorithm to support all integer objects).

[davidhewitt]

In updating to Python 3.13 in PyO3 I replaced _PyLong_NumBits with PyLong_AsNativeBytes. It returns the number of bytes required to represent the integer and so might satisfy the related use cases, for example it met our needs in PyO3. Just an observation as a possible reason to not need this at all.

[vstinner]

pywin32 uses _PyLong_Sign() and _PyLong_NumBits() to decide how to encode a Python int to Windows COM ABI: https://github.com/mhammond/pywin32/blob/ad5779b23b42653c9fa5dfbb18dd2a8fe5691d0d/com/win32com/src/oleargs.cpp#L139-L209. IMO _PyLong_NumBits() remains relevant even with PyLong_AsNativeBytes() addition.

panda3d uses _PyLong_NumBits() with _PyLong_AsByteArray(). The number of bits is used to allocate or resize a buffer to call _PyLong_AsByteArray():

• https://github.com/panda3d/panda3d/blob/adb92428856853019583f398d27c5d34e44a9343/panda/src/putil/doubleBitMask_ext.I#L25
• https://github.com/panda3d/panda3d/blob/adb92428856853019583f398d27c5d34e44a9343/panda/src/putil/bitArray_ext.cxx#L28

[encukou]

Seems to me like pywin32 could use PyLong_AsNativeBytes too.

[vstinner]

What is the C API Working Group call on this API? Should we add it, or should we guide 3rd party C extensions towards PyLong_AsNativeBytes()?

See also #31: "Add public function PyLong_GetDigits()".

[encukou]

I'd say guide them toward PyLong_AsNativeBytes.
If anyone sees a use case where a possibly over-estimated number of bytes rather than bits isn't enough, let's reopen?

[vstinner]

Ok, let's do that. I close the issue.