Skip to content

Commit

Permalink
Workaround symengine serialization payload incompatibility (#13251)
Browse files Browse the repository at this point in the history
* Workaround symengine serialization payload incompatibility

In QPY we rely on symengine's internal serialization to represent the
internal symbolic expression stored inside a ParameterExpression object.
However, this format is nominally symengine version specific and will
raise an error if there is a mismatch between the version used to
generate the payload and what is trying to read it. This became an issue
in the recent symengine 0.13 release which started to raise an error
when people installed it and tried to load QPY payloads across the
versions. This makes the symengine serialization unsuitable for use in
QPY because it's supposed to be independent of these kind of concerns,
especially when QPY is used in a server-client model where you don't
necessarily control the installed environment of symengine.

To correctly address this issue we'll need a new version of the QPY
format that owns the serialization format of ParameterExpressions directly
instead of relying on symengine which doesn't offer a compatibility
guarantee on the format. However this won't be quick solution and users
are encountering issues since the release of 0.13. This commit
introduces a workaround for this specific instance of the mismatch. It
turns out the payload format between 0.11 and 0.13 is completely
unchanged except for the version number. So before passing the parameter
expression payload to symengine for deserialization this commit checks
the versions numbers are the same, if they're not it checks that we're
dealing with 0.11 or 0.13, and if so it changes the version number in
the payloads header appropriately. If the version number is outside
those bounds it raises an exception because while this hack is known to
be safe for translating between symengine 0.11 and 0.13, it's not
possible to know for a future version whether the payload format changed
or not.

Longer term we will need a proper fix in qpy version 13 that introduces
a qiskit native serialization format for parameter expression instead of
relying on symengine or sympy to do it for us.

* Handle schedules too

This commit updates the schedule serialization path too, as it was also
directly loading symengine expressions. The code handling the workaround
is extracted to a standalone function which is used in both spots now
instead of calling symengine directly.

* Remove unused imports

* Gracefully handle failure to parse historical symengine files

During the discovery on the fix in this PR we discovered that setting
the ``use_symengine`` flag from Qiskit 0.45.x and 0.46.x would result in
newer versions of Qiskit being unable to parse the QPY file for the same
issue as being addressed here. This commit expands the logic to account
for this and raise a useful warning. It also updates the release notes
and documentation to document these limitations.

* Cap upper version of symengine in requirements list

Out of an abundance of caution this commit places a cap on the allowed
version of symengine users can install to be compatible with Qiskit.
Due to the symengine version dependence discovered in QPY around
serializing ParameterExpressions, we'll likely have a similar issue
when symengine 0.14.0 releases. Pre-emptively capping this means we
aren't going to be in this situation until we can confirm compatibility
with QPY serialization. The real solution for this will come in #13252,
although as this behavior is embedded in QPY formats 10, 11, and 12 at
this point we'll have to handle this edge case moving forward regardless
of whether we introduce a better solution in 1.3.0 or not. Although
realistically in that case we will likely need to just document this as
a limitation when exporting QPY payloads with Qiskit 0.45.0 through
1.2.3 (and with the ``version`` flag set to >= 10 and < 13) and have
explicit error checking around the symengine version (which this PR
adds) when in that code path.

* Fix release note upgrade section label

* Rewrite support documentation

* Fix mistakes in release note

Co-authored-by: Matthew Treinish <mtreinish@kortar.org>

---------

Co-authored-by: Jake Lishman <jake.lishman@ibm.com>
Co-authored-by: Jake Lishman <jake@binhbar.com>
(cherry picked from commit fee9f77)
  • Loading branch information
mtreinish authored and mergify[bot] committed Oct 2, 2024
1 parent 631abab commit 5179fad
Show file tree
Hide file tree
Showing 7 changed files with 145 additions and 10 deletions.
24 changes: 24 additions & 0 deletions qiskit/qpy/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,30 @@
.. autoexception:: QPYLoadingDeprecatedFeatureWarning
.. note::
With versions of Qiskit before 1.2.3, the ``use_symengine=True`` argument to :func:`.qpy.dump`
could cause problems with backwards compatibility if there were :class:`.ParameterExpression`
objects to serialize. In particular:
* When the loading version of Qiskit is 1.2.3 or greater, QPY files generated with any version
of Qiskit >= 0.46.0 can be loaded. If a version of Qiskit between 0.45.0 and 0.45.3 was used
to generate the files, and the non-default argument ``use_symengine=True`` was given to
:func:`.qpy.dump`, the file can only be read if the version of ``symengine`` used in the
generating environment was in the 0.11 or 0.13 series, but if the environment was created
during the support window of Qiskit 0.45, it is likely that ``symengine==0.9.2`` was used.
* When the loading version of Qiskit is between 0.46.0 and 1.2.2 inclusive, the file can only be
read if the installed version of ``symengine`` in the loading environment matches the version
used in the generating environment.
To recover a QPY file that fails with ``symengine`` version-related errors during a call to
:func:`.qpy.load`, first attempt to use Qiskit >= 1.2.3 to load the file. If this still fails,
it is likely because Qiskit 0.45.x was used to generate the file with ``use_symengine=True``.
In this case, use Qiskit 0.45.3 with ``symengine==0.9.2`` to load the file, and then re-export
it to QPY setting ``use_symengine=False``. The resulting file can then be loaded by any later
version of Qiskit.
QPY format version history
--------------------------
Expand Down
5 changes: 1 addition & 4 deletions qiskit/qpy/binary_io/schedules.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,6 @@

import numpy as np
import symengine as sym
from symengine.lib.symengine_wrapper import ( # pylint: disable = no-name-in-module
load_basic,
)

from qiskit.exceptions import QiskitError
from qiskit.pulse import library, channels, instructions
Expand Down Expand Up @@ -106,7 +103,7 @@ def _loads_symbolic_expr(expr_bytes, use_symengine=False):
return None
expr_bytes = zlib.decompress(expr_bytes)
if use_symengine:
return load_basic(expr_bytes)
return common.load_symengine_payload(expr_bytes)
else:
from sympy import parse_expr

Expand Down
5 changes: 1 addition & 4 deletions qiskit/qpy/binary_io/value.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,6 @@

import numpy as np
import symengine
from symengine.lib.symengine_wrapper import ( # pylint: disable = no-name-in-module
load_basic,
)


from qiskit.circuit import CASE_DEFAULT, Clbit, ClassicalRegister
Expand Down Expand Up @@ -290,7 +287,7 @@ def _read_parameter_expression_v3(file_obj, vectors, use_symengine):

payload = file_obj.read(data.expr_size)
if use_symengine:
expr_ = load_basic(payload)
expr_ = common.load_symengine_payload(payload)
else:
from sympy.parsing.sympy_parser import parse_expr

Expand Down
46 changes: 45 additions & 1 deletion qiskit/qpy/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,12 @@
import io
import struct

from qiskit.qpy import formats
import symengine
from symengine.lib.symengine_wrapper import ( # pylint: disable = no-name-in-module
load_basic,
)

from qiskit.qpy import formats, exceptions

QPY_VERSION = 12
QPY_COMPATIBILITY_VERSION = 10
Expand Down Expand Up @@ -304,3 +309,42 @@ def mapping_from_binary(binary_data, deserializer, **kwargs):
mapping = read_mapping(container, deserializer, **kwargs)

return mapping


def load_symengine_payload(payload: bytes) -> symengine.Expr:
"""Load a symengine expression from it's serialized cereal payload."""
# This is a horrible hack to workaround the symengine version checking
# it's deserialization does. There were no changes to the serialization
# format between 0.11 and 0.13 but the deserializer checks that it can't
# load across a major or minor version boundary. This works around it
# by just lying about the generating version.
symengine_version = symengine.__version__.split(".")
major = payload[2]
minor = payload[3]
if int(symengine_version[1]) != minor:
if major != "0":
raise exceptions.QpyError(
"Qiskit doesn't support loading a symengine payload generated with symengine >= 1.0"
)
if minor == 9:
raise exceptions.QpyError(
"Qiskit doesn't support loading a historical QPY file with `use_symengine=True` "
"generated in an environment using symengine 0.9.0. If you need to load this file "
"you can do so with Qiskit 0.45.x or 0.46.x and re-export the QPY file using "
"`use_symengine=False`."
)
if minor not in (11, 13):
raise exceptions.QpyError(
f"Incompatible symengine version {major}.{minor} used to generate the QPY "
"payload"
)
minor_version = int(symengine_version[1])
if minor_version not in (11, 13):
raise exceptions.QpyError(
f"Incompatible installed symengine version {symengine.__version__} to load "
"this QPY payload"
)
payload = bytearray(payload)
payload[3] = minor_version
payload = bytes(payload)
return load_basic(payload)
11 changes: 11 additions & 0 deletions qiskit/qpy/interface.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,17 @@ def dump(
from the QPY format at that version will persist. This should only be used if
compatibility with loading the payload with an older version of Qiskit is necessary.
.. note::
If serializing a :class:`.QuantumCircuit` or :class:`.ScheduleBlock` that contain
:class:`.ParameterExpression` objects with ``version`` set low with the intent to
load the payload using a historical release of Qiskit, it is safest to set the
``use_symengine`` flag to ``False``. Versions of Qiskit prior to 1.2.3 cannot load
QPY files containing ``symengine``-serialized :class:`.ParameterExpression` objects
unless the version of ``symengine`` used between the loading and generating
environments matches.
Raises:
QpyError: When multiple data format is mixed in the output.
TypeError: When invalid data type is input.
Expand Down
62 changes: 62 additions & 0 deletions releasenotes/notes/fix-qpy-symengine-compat-858970a9a1d6bc14.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
fixes:
- |
Fixed an issue with :func:`.qpy.load` when loading a QPY file containing
a :class:`.ParameterExpression`, if the versions of ``symengine`` installed
in the generating and loading environments were not the same. For example,
if a QPY file containing :class:`.ParameterExpression`\ s was generated
using Qiskit 1.2.2 with ``symengine==0.11.0`` installed, Qiskit 1.2.2 with
``syengine==0.13.0`` installed would be unable to load it.
Previously, an error would have been raised by ``symengine`` around this
version mismatch. This has been worked around for ``symengine`` 0.11 and
0.13 (there was no 0.12), but if you're trying to use different versions of
``symengine`` and there is a mismatch, this version of Qiskit still might not
work.
issues:
- |
Versions of Qiskit before 1.2.3 will not be able to load QPY files dumped
using :func:`.qpy.dump`, even with ``version`` set appropriately, if:
* there are unbound :class:`.ParameterExpression`\ s in the QPY file,
* the ``use_symengine=True`` flag was set (which is the default in Qiskit >=
1.0.0) in :func:`.qpy.dump`,
* the version of ``symengine`` installed in the generating and loading
environments are not within the same minor version.
This applies regardless of the version of Qiskit used in the generation (at
least up to Qiskit 1.2.3 inclusive).
If you want to maximize compatibility with older versions of Qiskit, you
should set ``use_symengine=False``. Newer versions of Qiskit should not
require this.
- |
QPY files from the Qiskit 0.45 series can, under a very specific and unlikely
set of circumstances, fail to load with any newer version of Qiskit,
including Qiskit 1.2.3. The criteria are:
* the :class:`.QuantumCircuit` or :class:`.ScheduleBlock` to be dumped
contained unbound :class:`.ParameterExpression` objects,
* the installed version of ``symengine`` was in the 0.9 series (which was the
most recent release during the support window of Qiskit 0.45),
* the ``use_symengine=True`` flag was set (which was *not* the default).
Later versions of Qiskit used during generation are not affected, because
they required newer versions than ``symengine`` 0.9.
In this case, you can recover the QPY file by reloading it with an environment
with Qiskit 0.45.3 and ``symengine`` 0.9.2 installed. Then, use
:func:`.qpy.dump` with ``use_symengine=False`` to re-export the file. This
will then be readable by any newer version of Qiskit.
upgrade:
- |
The supported versions of `symengine <https://pypi.org/project/symengine/>`__
have been pre-emptively capped at < 0.14.0 (which is expected to be the next
minor version, as of this release of Qiskit). This has been done to protect
against a potential incompatibility in :mod:`.qpy` when serializing
:class:`.ParameterExpression` objects. The serialization used in
:ref:`qpy_format` versions 10, 11, and 12 for :class:`.ParameterExpression`
objects is tied to the symengine version used to generate it, and there is the potential
for a future symengine release to not be compatible. This upper version cap is to prevent
a future release of symengine causing incompatibilities when trying to load QPY files
using :class:`.qpy.load`.
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ dill>=0.3
python-dateutil>=2.8.0
stevedore>=3.0.0
typing-extensions
symengine>=0.11
symengine>=0.11,<0.14

0 comments on commit 5179fad

Please sign in to comment.