Workaround symengine serialization payload incompatibility

In QPY we rely on symengine's internal serialization to represent the
internal symbolic expression stored inside a ParameterExpression object.
However, this format is nominally symengine version specific and will
raise an error if there is a mismatch between the version used to
generate the payload and what is trying to read it. This became an issue
in the recent symengine 0.13 release which started to raise an error
when people installed it and tried to load QPY payloads across the
versions. This makes the symengine serialization unsuitable for use in
QPY because it's supposed to be independent of these kind of concerns,
especially when QPY is used in a server-client model where you don't
necessarily control the installed environment of symengine.

To correctly address this issue we'll need a new version of the QPY
format that owns the serialization format of ParameterExpressions directly
instead of relying on symengine which doesn't offer a compatibility
guarantee on the format. However this won't be quick solution and users
are encountering issues since the release of 0.13. This commit
introduces a workaround for this specific instance of the mismatch. It
turns out the payload format between 0.11 and 0.13 is completely
unchanged except for the version number. So before passing the parameter
expression payload to symengine for deserialization this commit checks
the versions numbers are the same, if they're not it checks that we're
dealing with 0.11 or 0.13, and if so it changes the version number in
the payloads header appropriately. If the version number is outside
those bounds it raises an exception because while this hack is known to
be safe for translating between symengine 0.11 and 0.13, it's not
possible to know for a future version whether the payload format changed
or not.

Longer term we will need a proper fix in qpy version 13 that introduces
a qiskit native serialization format for parameter expression instead of
relying on symengine or sympy to do it for us.

qiskit/qpy/binary_io/value.py

@@ -290,6 +290,29 @@ def _read_parameter_expression_v3(file_obj, vectors, use_symengine):
 
     payload = file_obj.read(data.expr_size)
     if use_symengine:
+        # This is a horrible hack to workaround the symengine version checking
+        # it's deserialization does. There were no changes to the serialization
+        # format between 0.11 and 0.13 but the deserializer checks that it can't
+        # load across a major or minor version boundary. This works around it
+        # by just lying about the generating version.
+        symengine_version = symengine.__version__.split(".")
+        major = payload[2]
+        minor = payload[3]
+        if int(symengine_version[1]) != minor:
+            if minor not in (11, 13):
+                raise exceptions.QpyError(
+                    f"Incompatible symengine version {major}.{minor} used to generate the QPY "
+                    "payload"
+                )
+            minor_version = int(symengine_version[1])
+            if minor_version not in (11, 13):
+                raise exceptions.QpyError(
+                    f"Incompatible installed symengine version {symengine.__version__} to load "
+                    "this QPY payload"
+                )
+            payload = bytearray(payload)
+            payload[3] = minor_version
+            payload = bytes(payload)
         expr_ = load_basic(payload)
     else:
         from sympy.parsing.sympy_parser import parse_expr

releasenotes/notes/fix-qpy-symengine-compat-858970a9a1d6bc14.yaml

@@ -0,0 +1,13 @@
+---
+fixes:
+  - |
+    Fixed an issue with :func:`.qpy.load` when loading a QPY payload that
+    contains a :class:`.ParameterExpression` that was generated when symengine
+    0.11 was installed and trying to load it in an environment where symengine
+    0.13 (the latest symengine release as of this qiskit release) or vice versa.
+    Previously, an error would have been raised by symengine around this version
+    mismatch. This has been worked around for these two specific symengine versions
+    but if you're trying to use different versions of symengine and there is a
+    mismatch with this version of Qiskit this might not work. You will need to
+    install Qiskit >1.3.0 to fix this mismatch issue more broadly for any potential
+    future version of symengine.