Alternatives to using private API for conversion to/from CPython int's

[skirpichev]

Currently, we are using a bunch of private CPython functions to provide fast mpz <-> int conversion: see GMPy_MPZ_From_PyLong, GMPy_PyLong_From_MPZ and

gmpy/src/gmpy2_convert.h

Lines 135 to 155 in 141eb88

	/* Compatibility macros (to work with PyLongObject internals).
	*/
	#if PY_VERSION_HEX >= 0x030C0000
	# define TAG_FROM_SIGN_AND_SIZE(is_neg, size) ((is_neg?2:(size==0)) | (((size_t)size) << 3))
	# define _PyLong_SetSignAndDigitCount(obj, is_neg, size) (obj->long_value.lv_tag = TAG_FROM_SIGN_AND_SIZE(is_neg, size))
	#elif PY_VERSION_HEX >= 0x030900A4
	# define _PyLong_SetSignAndDigitCount(obj, is_neg, size) (Py_SET_SIZE(obj, (is_neg?-1:1)*size))
	#else
	# define _PyLong_SetSignAndDigitCount(obj, is_neg, size) (Py_SIZE(obj) = (is_neg?-1:1)*size)
	#endif
	#if PY_VERSION_HEX >= 0x030C0000
	# define GET_OB_DIGIT(obj) obj->long_value.ob_digit
	# define _PyLong_IsNegative(obj) ((obj->long_value.lv_tag & 3) == 2)
	# define _PyLong_DigitCount(obj) (obj->long_value.lv_tag >> 3)
	#else
	# define GET_OB_DIGIT(obj) obj->ob_digit
	# define _PyLong_IsNegative(obj) (Py_SIZE(obj) < 0)
	# define _PyLong_DigitCount(obj) (_PyLong_IsNegative(obj)? -Py_SIZE(obj):Py_SIZE(obj))
	#endif

There should be a better way!

I see several variants to deal with this problem.

1. The CPython 3.13 got PyLong_AsNativeBytes() and PyLong_FromNativeBytes() function to import/export arbitrary int's. We can use these functions and handle small integers case separately with PyLong_AsLong()/PyLong_FromLong(). In general, this should be much slower than the current solution. On another hand, I think that for real world scenarios - small integers case should be most important.

2. There is a proposal to add mpz_import/export-like functions for PyLong's C API, but it seems that CPython core developers aren't very enthusiastic on this idea (see C API: Consider adding public PyLong_AsByteArray() and PyLong_FromByteArray() functions python/cpython#111140 (comment)).

3. Another possibility is the mpz_limbs_write()-like API to access unsigned bigint as an array of "digits", see this. With that kind of API on the CPython side - the gmpy2 could do fast conversion to/from int's using mpz_import/export functions.

I'm planning to open a discussion thread on the d.p.o, that will propose the 3rd option, but first I would appreciate any feedback here. CC @oscarbenjamin: perhaps this does make sense for the python-flint.

[skirpichev]

See capi-workgroup/decisions#31

[skirpichev]

@casevh, unfortunately it seems we aren't going to have the public API on CPython side.

See https://discuss.python.org/t/pep-757-c-api-to-import-export-python-integers/63895/66

Is it does make sense for you if the PyLong_Export API will fail on some integers? How we can recover from such failures? I think such export API doesn't make sense.