Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metatest for swap mappers #1571

Merged
merged 86 commits into from
Feb 19, 2019
Merged

Metatest for swap mappers #1571

merged 86 commits into from
Feb 19, 2019

Conversation

1ucian0
Copy link
Member

@1ucian0 1ucian0 commented Dec 20, 2018

Fixes #1461.

This PR is a second attempt to have a simple meta test for all the "swappers".

The test checks the output of the swapper to a ground truth DAG (one for each test/swapper) saved in as a pickle (in test/python/pickles/). If they need to be regenerated, the DAG candidate is compiled and run in a simulator and the count is checked before being saved.

How to add a swapper

Add the following class in test/python/transpiler/test_mappers.py

class TestsSomeSwap(CommonTestCases, QiskitTestCase):
    """ Test CommonTestCases using SomeSwap """
    pass_class = SomeSwap           # <- The pass class
    additional_args = {'seed': 42}  # <- In case SomeSwap.__init__ requieres additional arguments

How to add a test for all the swappers

Add the following method in the class CommonTestCases:

    def test_a_common_test(self):
        self.count = {'000': 512, '110': 512}  # <- The expected count for this circuit
        self.delta = 5      # <- This is delta for the AlmostEqual during the count check
        coupling_map = [[0, 1], [0, 2]]  # <- The coupling map for this specific test

        qr = QuantumRegister(3, 'q')                       #
        cr = ClassicalRegister(3, 'c')                     #  Set the circuit to test
        circuit = QuantumCircuit(qr, cr, name='some_name') #  and dont't forget to put a name
        circuit.h(qr[1])                                   #  (it will be used to save the pickle
        circuit.cx(qr[1], qr[2])                           #
        circuit.measure(qr, cr)                            #

        result = transpile(circuit, self.create_backend(), coupling_map=coupling_map,
                           pass_manager=self.create_passmanager(coupling_map))
        self.assertResult(result, circuit.name)

@1ucian0 1ucian0 mentioned this pull request Dec 20, 2018
Copy link
Contributor

@delapuente delapuente left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the changes, @1ucian0. I think we are almost there. How do you plan to remove the old pickles in case the name of the circuit changes for some reason?

test/python/transpiler/test_mappers.py Outdated Show resolved Hide resolved
test/python/transpiler/test_mappers.py Outdated Show resolved Hide resolved
test/python/transpiler/test_mappers.py Outdated Show resolved Hide resolved
@delapuente
Copy link
Contributor

Hey, @1ucian0, what about removing the pickles that are no longer needed after regenerating?

@1ucian0
Copy link
Member Author

1ucian0 commented Feb 18, 2019

done in 0d65f25

Copy link
Contributor

@delapuente delapuente left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. My only concern is when to regenerate, I know there are plans for including a developer guide about the transpiler and passes and we can include information about it there (#83). Another option is to add some information to the CONTRIBUTING.rst guidelines, in the same way we did when we updated the test options (#1832).

@1ucian0 1ucian0 merged commit 3b7d26f into Qiskit:master Feb 19, 2019
@1ucian0 1ucian0 deleted the metatest_mapper branch February 19, 2019 19:51
lia-approves pushed a commit to edasgupta/qiskit-terra that referenced this pull request Jul 30, 2019
* Issue Qiskit#1186: Add test file for circuit drawers

This commit adds a new file `test_circuit_drawer`. It is created to test
all the circuit drawers: `circuit_drawer` as well as the underlying
circuit drawers.

This very commit contains a test case to test only behavior of
`circuit_drawer`. In particular, it checks how `circuit_drawer` handles
different provided values of `output` parameter.

* Add line breaks to follow the codestyle

* Add test case for underlying circuit drawers

This commit adds a test case for underlying circuit drawers. At the
moment, it tests only how circuit drawers behave while drawing a small
circuit. However, further tests of behaviour on greater circuits will be
added.

The test case works as follows: there are presaved references of how
each circuit drawer shall draw a particular small circuit which is
obtained upon calling of `_small_circuit` funtion. The test case makes
all circuit drawer to produce their versions of the small circuit and then
compares these versions with the corresponding references.

Since latex and matplotlib drawers produce pictures, the test case
compares the produced images while for text and latex source drawers it
compares bytes of the files.

* Add referencesfor medium circuit

* Remove redundant  variable

* Issue Qiskit#1886: Improve tests of circuit drawers

This commit completes the test case for different circuit drawers.
In particular, it adds tests and references for all circuit types. Also
it adds comments and structurizes the code.

* Force to skip testing of deprecated circuit_drawer fallback

* According to Qiskit#1628, import of QiskitTestCase was changed from relative
to absolute one

* Docstrings of functions were refactored in accordance with the project
codestyle. In particular, format of return string was changed.

* This commit changes defining the path to references locally in the test
file in favor of defining it qiskit-wide and the using
_get_resource_path of QiskitTestCase

* This commit moves common part of tests into a single function. Thus, it
significantly reduces code repetition.

* Update medium and large references for mpl

* As `assert_called_once` is available only for Python 3.6, the
corresponding usages of mock asserts were updated so that they can be
supported with Python 3.5. In particular, the were replaced to
`assert_called_once_with`.

* Fix style bug

* Add readable assert fail messages

* As different Python version (at least second digit matters,
probably versions of packages matters too, it is unknown up to now),
this commit adds mechanism to handle current Python version and to look
for references in the corresponding folder.

* Fix style bugs

* There were still some `assert_called_once` left after previous commits,
so I have now replaced them with `assert_called_once_with` and added a
dict of correspinding calling arguments

* Fix style bug

* Up to this moment all registers had their names generated automatically.
Hence, after consequent runs of several tests, their names could have
changed randomly. There was a function `_reset_indices` to cope with
this, but it didn't take into account any intermediate runs of other
tests. Now, each register is assigned with a constant name and thus no
fails should happen

* This commit changes the name to be set for all created registers and
changes the references correspondingly. In addition, it fixes a small
style bug.

* It seems that Travis has no pdflatex installed and thus, testing of
LaTeX drawer will obviously fail. So, this commit add handling of an
OSError.

* Add references for Python 3.7

* During previous commits it was discovered that `text_drawer` produces
different outputs for different versions of Python while drawing the
`deep_circuit`. In particular, outputs for Python 3.5, and 3.6 with 3.7
differ. A notorious fact is that they are not different in sense of some
minor things like spacing, they differs like there is a bug. So, until
this issue is managed, there should be no `deep_circuit` testing. Consequently, since all other cases produce similar outputs, they can be evaluated similarly without paying attention to the dependence on version.

* Thanks to @diego-plan9, this commits just replaces manual creation of
temporary directory with standard Python function.

* Update references for TestVisualizationImplementation

* Thanks to @diego-plan9, this commit moves functions common for
TestDrawingMethods and TestVisualizationImplementation to a common base
class, DrawingTestCase. Consequently, these test cases are inherited
from this class and still possess their previous functinalty.

* Fix style bug

* Fix style bugs

* Fix style bugs

* Fix style bug

* Following the @1ucian0 comment, this commit removes test for deprecated
fallback

* Fix style errors

* This commit at first, of course, update references to keep them in touch
with the underlying repository updates. But, as well, it removes
dependency on version of Python from the commented part of a setUp
function

* This commit follows the principle Qiskit#1571 and introduces an automatic
regeneration of references as soon as test file is called as a
standalone script. It is done as follows: as soon as this script is a
main one, it calls a static method of TestDrawingMethods class, which
regenerates the references, just before calling of the unittest
facilities.

In addition, this commit moves decalaration of all variables to the body
of class, not to the setUp methods, which turned out to be redundant.
Finally, it introduces a `setUpClass` for a TestDrawingMethods class to
create a ONE temporary folder for ALL the tests, not for each one
separately.

* Fix style bug
lia-approves pushed a commit to edasgupta/qiskit-terra that referenced this pull request Jul 30, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants