Whole API rework and reorganization + doc WIP

Webservices are all in webservices package and stuff that was in common package has been split either in core package for internal API or in products packages for public API Added support for doctest and started to rework documentation for better separation between user and dev doc. Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
SciQLop · jeandet · Oct 7, 2021 · Aug 4, 2021 · Aug 5, 2021 · Aug 5, 2021
commit 80bfdbddd3cf45bd1165392de828fc64376baf37
diff --git a/.coveragerc b/.coveragerc
@@ -6,3 +6,5 @@ omit =
     setup.py
     *cdf*
     .venv/*
+    docs/*
+    examples/*
diff --git a/.github/workflows/PRs.yml b/.github/workflows/PRs.yml
@@ -34,9 +34,9 @@ jobs:
           SPEASY_AMDA_USERNAME: ${{ secrets.SPEASY_AMDA_USERNAME }}
           SPEASY_AMDA_PASSWORD: ${{ secrets.SPEASY_AMDA_PASSWORD }}
       run: |
-        pip install pytest
-        pip install pytest-cov
+        pip install pytest pytest-cov sphinx doctest sphinx-rtd-theme
         SPEASY_AMDA_PASSWORD="${{ secrets.SPEASY_AMDA_PASSWORD }}" SPEASY_AMDA_USERNAME="${{ secrets.SPEASY_AMDA_USERNAME }}" pytest --cov=./ --cov-report=xml
+        make doctest
     - name: Upload coverage to Codecov
       if: matrix.python-version == '3.8'
       uses: codecov/codecov-action@v1

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -34,9 +34,9 @@ jobs:
           SPEASY_AMDA_USERNAME: ${{ secrets.SPEASY_AMDA_USERNAME }}
           SPEASY_AMDA_PASSWORD: ${{ secrets.SPEASY_AMDA_PASSWORD }}
       run: |
-        pip install pytest
-        pip install pytest-cov
+        pip install pytest pytest-cov sphinx doctest sphinx-rtd-theme
         SPEASY_AMDA_PASSWORD="${{ secrets.SPEASY_AMDA_PASSWORD }}" SPEASY_AMDA_USERNAME="${{ secrets.SPEASY_AMDA_USERNAME }}" pytest --cov=./ --cov-report=xml
+        make doctest
     - name: Upload coverage to Codecov
       if: matrix.python-version == '3.8'
       uses: codecov/codecov-action@v1

diff --git a/AUTHORS.rst b/AUTHORS.rst
@@ -10,7 +10,7 @@ Development Lead
 Contributors
 ------------
 
-AMDA
+AMDA_Webservice
 ^^^^
 
 * Alexandre Schulz <alexandre.schulz@irap.omp.eu>

diff --git a/LICENSE b/LICENSE
@@ -1,7 +1,7 @@
 GNU GENERAL PUBLIC LICENSE
                       Version 3, 29 June 2007
 
-    A simple Python package to deal with main Space Physics WebServices (CDA,CSA,AMDA,..) mainly written to ease development of SciQLop.
+    A simple Python package to deal with main Space Physics WebServices (CDA,CSA,AMDA_Webservice,..) mainly written to ease development of SciQLop.
     Copyright (C) 2018  Alexis Jeandet
 
     This program is free software: you can redistribute it and/or modify

diff --git a/Makefile b/Makefile
@@ -53,10 +53,13 @@ clean-test: ## remove test and coverage artifacts
 lint: ## check style with flake8
 	flake8 speasy tests --count --select=E9,F63,F7,F82 --show-source --statistics
 
+doctest:
+	$(MAKE) -C docs doctest
+
 test: ## run tests quickly with the default Python
-	py.test
+	PYTHONPATH=. py.test
 
-test-all: ## run tests on every Python version with tox
+test-all: doctest ## run tests on every Python version with tox
 	tox
 
 coverage: ## check code coverage quickly with the default Python
@@ -68,7 +71,7 @@ coverage: ## check code coverage quickly with the default Python
 docs: ## generate Sphinx HTML documentation, including API docs
 	rm -f docs/speasy.rst
 	rm -f docs/modules.rst
-	sphinx-apidoc -o docs/ speasy
+	sphinx-apidoc -M -o docs/ speasy
 	$(MAKE) -C docs clean
 	$(MAKE) -C docs html
 	$(BROWSER) docs/_build/html/index.html

diff --git a/README.rst b/README.rst
@@ -37,7 +37,7 @@ Space Physics made EASY
     :target: https://mybinder.org/v2/gh/SciQLop/spwc/HEAD?filepath=examples%2Fdemo.ipynb
     :alt: Discover on MyBinder
 
-A simple Python package to deal with main Space Physics WebServices (CDA,CSA,AMDA,..).
+A simple Python package to deal with main Space Physics WebServices (CDA,CSA,AMDA_Webservice,..).
 This package was initially written to ease development of `SciQLop <https://github.com/SciQLop/SciQLop>`_ , but
 now offers an intuitive and efficient access to any scientist or student who just want get spacecraft data.
 

diff --git a/docs/amda.rst b/docs/amda.rst
@@ -1,4 +1,4 @@
-AMDA
+AMDA_Webservice
 ====
 
 .. toctree::
@@ -7,17 +7,17 @@ AMDA
    amda_products
    amda_examples
    amda_notebooks
-
 
-Getting AMDA dataset and parameters
+
+Getting AMDA_Webservice dataset and parameters
 -----------------------------------
 
-First create a connexion to AMDA::
+First create a connexion to AMDA_Webservice::
 
-    >>> from speasy.amda import AMDA
-    >>> amda = AMDA()
+    >>> from speasy.amda import AMDA_Webservice
+    >>> amda = AMDA_Webservice()
 
-Downloading the data is done by using the :meth:`speasy.amda.amda.AMDA.get_data()` or :meth:`speasy.amda.amda.AMDA.get_parameter()` methods. For example
+Downloading the data is done by using the :meth:`speasy.amda.amda.AMDA_Webservice.get_data()` or :meth:`speasy.amda.amda.AMDA_Webservice.get_parameter()` methods. For example
 getting `imf` data between 2000-01-01 and 2000-02-01::
 
     >>> parameter = amda.get_data("imf", datetime.datetime(2000,1,1), datetime.datetime(2000,2,1))
@@ -39,7 +39,7 @@ The parameters data is stored as a `numpy.ndarray` object::
            [-4.413,  0.141, -0.247],
            [-4.335,  0.087, -0.323]])
 
-It is also possible to get all the parameters contained in a given dataset using the :meth:`speasy.amda.amda.AMDA.get_dataset()` method which returns a list of :class:`speasy.common.variable.SpeasyVariable` objects::
+It is also possible to get all the parameters contained in a given dataset using the :meth:`speasy.amda.amda.AMDA_Webservice.get_dataset()` method which returns a list of :class:`speasy.common.variable.SpeasyVariable` objects::
 
     >>> amda.get_dataset("ace-imf-all", datetime.datetime(2000,1,1), datetime.datetime(2000,2,1))
     [<speasy.common.variable.SpeasyVariable object at 0x7f79fa8f3720>, <speasy.common.variable.SpeasyVariable object at 0x7f79fa8fb950>, <speasy.common.variable.SpeasyVariable object at 0x7f79fa859540>]
@@ -48,16 +48,16 @@ It is also possible to get all the parameters contained in a given dataset using
 Parameter time range
 --------------------
 
-Downloading data requires the user to know the desired start and end date of the data. The start and 
-end dates for all available datasets and parameters are available through use of the 
-:meth:`speasy.amda.amda.AMDA.parameter_range()` method::
+Downloading data requires the user to know the desired start and end date of the data. The start and
+end dates for all available datasets and parameters are available through use of the
+:meth:`speasy.amda.amda.AMDA_Webservice.parameter_range()` method::
 
     >>> t_range = amda.parameter_range("imf")
     >>> type(t_range)
     <class 'speasy.common.datetime_range.DateTimeRange'>
     >>> t_range
     1997-09-02T00:00:12->2021-07-17T23:59:55
-    
+
 
 Listing available products
 --------------------------
@@ -68,23 +68,23 @@ Users can access the list of available parameters::
    >>> dataset_list = amda.list_datasets()
 
 You can get the list of parameters contained in a dataset with::
- 
+
    >>> amda.get_dataset_parameters("ace-imf-all")
    ['imf_mag', 'imf', 'imf_gsm']
 
 
 User parameters
 ---------------
 
-Users with an account on AMDA can access their private parameters. First store the users credentials
+Users with an account on AMDA_Webservice can access their private parameters. First store the users credentials
 using the :class:`~speasy.config.ConfigEntry` class::
-   
+
    >>> from speasy.config import ConfigEntry
-   >>> ConfigEntry("AMDA", "username").set("your_username")
-   >>> ConfigEntry("AMDA", "password").set("your_password")
+   >>> ConfigEntry("AMDA_Webservice", "username").set("your_username")
+   >>> ConfigEntry("AMDA_Webservice", "password").set("your_password")
 
 The login credentials are stored locally, you only need to execute the previous lines of code once
-to save the credentials. The configuration file can be found at :data:`/<user_config_dir>/speasy/config.ini`. 
+to save the credentials. The configuration file can be found at :data:`/<user_config_dir>/speasy/config.ini`.
 
 Parameters defined on your account can now be listed like follows::
 
@@ -96,7 +96,7 @@ Parameters defined on your account can now be listed like follows::
    ...
    ws_n your_nth_parameter
 
-Getting the parameter is then done with :meth:`~speasy.amda.amda.AMDA.get_user_parameter()`::
+Getting the parameter is then done with :meth:`~speasy.amda.amda.AMDA_Webservice.get_user_parameter()`::
 
    >>> from datetime import datetime
    >>> start, stop = datetime(2000,1,1), datetime(2000,1,2)
@@ -115,19 +115,19 @@ Getting the parameter is then done with :meth:`~speasy.amda.amda.AMDA.get_user_p
    array(['2001-01-01T00:00:00.000', '2001-01-01T00:00:16.000',
           '2001-01-01T00:00:32.000', ..., '2001-01-31T23:59:28.000',
           '2001-01-31T23:59:44.000', '2001-02-01T00:00:00.000'], dtype=object)
-
-See :meth:`~speasy.amda.amda.AMDA.list_user_parameters()` for a list of user parameter attributes.
 
-Calling :meth:`~speasy.amda.amda.AMDA.list_user_parameters()` or :meth:`~speasy.amda.amda.AMDA.get_user_parameter()` will raise an :class:`~speasy.config.exception.UndefinedConfigEntry` exception if the 
+See :meth:`~speasy.amda.amda.AMDA_Webservice.list_user_parameters()` for a list of user parameter attributes.
+
+Calling :meth:`~speasy.amda.amda.AMDA_Webservice.list_user_parameters()` or :meth:`~speasy.amda.amda.AMDA_Webservice.get_user_parameter()` will raise an :class:`~speasy.config.exception.UndefinedConfigEntry` exception if the
 credentials could not be found::
 
    raise UndefinedConfigEntry(key1=self.key1, key2=self.key2, default=self.default)
 
 Timetables and catalogs
 -----------------------
 
-The methods :meth:`~speasy.amda.amda.AMDA.get_timetable` and :meth:`~speasy.amda.amda.AMDA.get_catalog` allow you to download one of many timetables and catalogs available on AMDA. Listing the publicly
-available products is achieved through using the :meth:`~speasy.amda.amda.AMDA.list_timetables` and :meth:`~speasy.amda.amda.AMDA.list_catalogs` to list products by ID::
+The methods :meth:`~speasy.amda.amda.AMDA_Webservice.get_timetable` and :meth:`~speasy.amda.amda.AMDA_Webservice.get_catalog` allow you to download one of many timetables and catalogs available on AMDA_Webservice. Listing the publicly
+available products is achieved through using the :meth:`~speasy.amda.amda.AMDA_Webservice.list_timetables` and :meth:`~speasy.amda.amda.AMDA_Webservice.list_catalogs` to list products by ID::
 
    >>> for ttid in amda.list_timetables():
    >>>     print(ttid)
@@ -140,10 +140,10 @@ respective products::
 
     >>> for ttid in amda.timetable:
     >>>     print(amda.timetable[ttid])
-    {'xml:id': 'sharedtimeTable_139', 'id': 'sharedtimeTable_139', 'name': 'MMS_Burst_Mode_2021July', 'created': '2020-08-26T00:00:55', 'modified': '1970-01-01T00:00:00', 'surveyStart': '2021-07-01T00:03:43', 'surveyStop': '2021-07-25T13:51:53', 'contact': 'MMS_Burst_Mode_2021July', 'description': 'Time intervals for which MMS burst mode data are available. source: http://mmsburst.sr.unh.edu/', 'history': '', 'nbIntervals': '441', 'sharedBy': 'AMDA', 'sharedDate': '2021-08-02T03:25:05+00:00', 'folder': 'MMS_BURST_MODE_timeTable', 'timeTable': 'sharedtimeTable_139'}
+    {'xml:id': 'sharedtimeTable_139', 'id': 'sharedtimeTable_139', 'name': 'MMS_Burst_Mode_2021July', 'created': '2020-08-26T00:00:55', 'modified': '1970-01-01T00:00:00', 'surveyStart': '2021-07-01T00:03:43', 'surveyStop': '2021-07-25T13:51:53', 'contact': 'MMS_Burst_Mode_2021July', 'description': 'Time intervals for which MMS burst mode data are available. source: http://mmsburst.sr.unh.edu/', 'history': '', 'nbIntervals': '441', 'sharedBy': 'AMDA_Webservice', 'sharedDate': '2021-08-02T03:25:05+00:00', 'folder': 'MMS_BURST_MODE_timeTable', 'timeTable': 'sharedtimeTable_139'}
     ...
 
-For a description of the :data:`timetable` and :data:`catalog` attributes see the :class:`~speasy.amda.amda.AMDA` class documentation.
+For a description of the :data:`timetable` and :data:`catalog` attributes see the :class:`~speasy.amda.amda.AMDA_Webservice` class documentation.
 
 
 
diff --git a/docs/amda_catalogs.rst b/docs/amda_catalogs.rst
@@ -1,6 +1,6 @@
 .. _amda-catalogs-label:
 
-AMDA Catalogs
+AMDA_Webservice Catalogs
 -------------
 
 Description of AMDAs catalog objects and their representation in :mod:`speasy`. In :mod:`speasy`
@@ -18,6 +18,6 @@ catalog metadata is stored as dictionary objects. The catalog's attributes are :
 * :data:`surveyStart` : begining of survey
 * :data:`surveyStop` : end of survey
 * :data:`folder` : parent folder ID
-* :data:`catalog` 
+* :data:`catalog`
 
 
diff --git a/docs/amda_datasets.rst b/docs/amda_datasets.rst
@@ -1,4 +1,4 @@
-AMDA Datasets
+AMDA_Webservice Datasets
 -------------
 
 Description of AMDAs dataset objects and their representation in :mod:`speasy`. In :mod:`speasy`

diff --git a/docs/amda_example_1.rst b/docs/amda_example_1.rst
@@ -1,6 +1,6 @@
-AMDA Example 1: getting public dataset and parameter
+AMDA_Webservice Example 1: getting public dataset and parameter
 ----------------------------------------------------
 
-In this script we download the :data:`tao-ura-sw` dataset and the :data:`imf` parameter on AMDA.
+In this script we download the :data:`tao-ura-sw` dataset and the :data:`imf` parameter on AMDA_Webservice.
 
 .. literalinclude:: ../examples/amda_public_data.py
diff --git a/docs/amda_example_2.rst b/docs/amda_example_2.rst
@@ -1,6 +1,6 @@
-AMDA Example 2: getting user defined parameter
+AMDA_Webservice Example 2: getting user defined parameter
 ----------------------------------------------
 
-The following example illustrates how to access parameters defined on your user account within AMDA.
+The following example illustrates how to access parameters defined on your user account within AMDA_Webservice.
 
 .. literalinclude:: ../examples/amda_user_parameter.py
diff --git a/docs/amda_example_3.rst b/docs/amda_example_3.rst
@@ -1,14 +1,14 @@
-AMDA Example 3: listing available parameters
+AMDA_Webservice Example 3: listing available parameters
 --------------------------------------------
 
 Simple example illustrating how to list available parameters, datasets and timetables.
 
 .. literalinclude:: ../examples/amda_list_public_products.py
 
 Output ::
-   
+
    $ python amda_list_public_products.py
-   AMDA products
+   AMDA_Webservice products
         Datasets   : 935
         Parameters : 4696
         Timetables : 131
diff --git a/docs/amda_example_4.rst b/docs/amda_example_4.rst
@@ -1,4 +1,4 @@
-AMDA Example 4: getting timetables
+AMDA_Webservice Example 4: getting timetables
 ----------------------------------
 
 Download a public timetable.
@@ -7,8 +7,8 @@ Download a public timetable.
 
 Output ::
 
-   $ python amda_get_public_timetable.py 
+   $ python amda_get_public_timetable.py
    <speasy.common.variable.SpeasyVariable object at 0x7f23cde66f90>
    timetable id : sharedtimeTable_0
    time.shape   : (782,)
-   values.shape : (782, 2) 
+   values.shape : (782, 2)
diff --git a/docs/amda_example_get_catalog.rst b/docs/amda_example_get_catalog.rst
@@ -1,4 +1,4 @@
-AMDA Example 7: getting catalogs
+AMDA_Webservice Example 7: getting catalogs
 --------------------------------
 
 Download a public catalogs.
@@ -7,7 +7,7 @@ Download a public catalogs.
 
 Output ::
 
-   $ python amda_get_public_catalog.py 
+   $ python amda_get_public_catalog.py
    [['2007-02-25 02:47:01' '2007-02-25 02:49:49' '3.0']
    ...
    ['2007-02-26 00:22:41' '2007-02-26 00:28:46' '3.0']]

diff --git a/docs/amda_example_get_user_catalog.rst b/docs/amda_example_get_user_catalog.rst
@@ -1,4 +1,4 @@
-AMDA Example 11: getting user catalogs
+AMDA_Webservice Example 11: getting user catalogs
 --------------------------------------
 
 Downloading private catalogs.
@@ -7,7 +7,7 @@ Downloading private catalogs.
 
 Output ::
 
-   $ python amda_get_user_catalog.py 
+   $ python amda_get_user_catalog.py
    <speasy.common.variable.SpeasyVariable object at 0x7f76472b76d0>
 
 

diff --git a/docs/amda_example_get_user_timetable.rst b/docs/amda_example_get_user_timetable.rst
@@ -1,4 +1,4 @@
-AMDA Example 10: getting user timetables
+AMDA_Webservice Example 10: getting user timetables
 ---------------------------------------
 
 Downloading private timetables.
@@ -7,7 +7,7 @@ Downloading private timetables.
 
 Output ::
 
-   $ python amda_get_user_timetable.py 
+   $ python amda_get_user_timetable.py
    <speasy.common.variable.SpeasyVariable object at 0x7f4e8175bef0>
    ...
    <speasy.common.variable.SpeasyVariable object at 0x7f4e80977f40>

diff --git a/docs/amda_example_list_catalogs.rst b/docs/amda_example_list_catalogs.rst
@@ -1,4 +1,4 @@
-AMDA Example 6: listing catalogs
+AMDA_Webservice Example 6: listing catalogs
 --------------------------------
 
 Listing public catalogs.
@@ -7,11 +7,11 @@ Listing public catalogs.
 
 Output ::
 
-   $ python amda_list_public_catalogs.py 
+   $ python amda_list_public_catalogs.py
    sharedcatalog_0
    ...
    sharedcatalog_130
-   {'xml:id': 'sharedcatalog_16', 'id': 'sharedcatalog_16', 'name': 'vex', 'created': '2020-06-04T14:00:59', 'description': "Venus Express is a satellite optimised for studying the atmosphere of Venus, from the surface right up to the ionosphere. Launched on November 9th 2005, it arrived at Venus in April 2006 and continued operating for more than eight years. The mission ended on December 16th 2014. This catalog has been created using spice kernels provided by spdf and the JPL's orbnum tool (https://naif.jpl.nasa.gov/pub/naif/utilities/PC_Linux_64bit/orbnum.ug)", 'history': '', 'parameters': '', 'nbIntervals': '3162', 'sharedBy': 'AMDA', 'sharedDate': '2020-09-17T17:12:52+00:00', 'surveyStart': '2006-05-18T13:35:45', 'surveyStop': '2014-12-31T12:23:54', 'folder': 'SPACECRAFT_ORBIT_NUMBERS_catalog', 'catalog': 'sharedcatalog_16'}
+   {'xml:id': 'sharedcatalog_16', 'id': 'sharedcatalog_16', 'name': 'vex', 'created': '2020-06-04T14:00:59', 'description': "Venus Express is a satellite optimised for studying the atmosphere of Venus, from the surface right up to the ionosphere. Launched on November 9th 2005, it arrived at Venus in April 2006 and continued operating for more than eight years. The mission ended on December 16th 2014. This catalog has been created using spice kernels provided by spdf and the JPL's orbnum tool (https://naif.jpl.nasa.gov/pub/naif/utilities/PC_Linux_64bit/orbnum.ug)", 'history': '', 'parameters': '', 'nbIntervals': '3162', 'sharedBy': 'AMDA_Webservice', 'sharedDate': '2020-09-17T17:12:52+00:00', 'surveyStart': '2006-05-18T13:35:45', 'surveyStop': '2014-12-31T12:23:54', 'folder': 'SPACECRAFT_ORBIT_NUMBERS_catalog', 'catalog': 'sharedcatalog_16'}
    ...
 
 See :ref:`amda-catalogs-label` for a description of the :data:`catalog` fields.
diff --git a/docs/amda_example_list_timetables.rst b/docs/amda_example_list_timetables.rst
@@ -1,4 +1,4 @@
-AMDA Example 5: listing timetables
+AMDA_Webservice Example 5: listing timetables
 ----------------------------------
 
 Listing public timetables.
@@ -7,11 +7,11 @@ Listing public timetables.
 
 Output ::
 
-   $ python amda_list_public_timetables.py 
+   $ python amda_list_public_timetables.py
    sharedtimeTable_0
    ...
    sharedtimeTable_130
-   {'xml:id': 'sharedtimeTable_139', 'id': 'sharedtimeTable_139', 'name': 'MMS_Burst_Mode_2021July', 'created': '2020-08-26T00:00:55', 'modified': '1970-01-01T00:00:00', 'surveyStart': '2021-07-01T00:03:43', 'surveyStop': '2021-07-25T13:51:53', 'contact': 'MMS_Burst_Mode_2021July', 'description': 'Time intervals for which MMS burst mode data are available. source: http://mmsburst.sr.unh.edu/', 'history': '', 'nbIntervals': '441', 'sharedBy': 'AMDA', 'sharedDate': '2021-08-02T03:25:05+00:00', 'folder': 'MMS_BURST_MODE_timeTable', 'timeTable': 'sharedtimeTable_139'}
+   {'xml:id': 'sharedtimeTable_139', 'id': 'sharedtimeTable_139', 'name': 'MMS_Burst_Mode_2021July', 'created': '2020-08-26T00:00:55', 'modified': '1970-01-01T00:00:00', 'surveyStart': '2021-07-01T00:03:43', 'surveyStop': '2021-07-25T13:51:53', 'contact': 'MMS_Burst_Mode_2021July', 'description': 'Time intervals for which MMS burst mode data are available. source: http://mmsburst.sr.unh.edu/', 'history': '', 'nbIntervals': '441', 'sharedBy': 'AMDA_Webservice', 'sharedDate': '2021-08-02T03:25:05+00:00', 'folder': 'MMS_BURST_MODE_timeTable', 'timeTable': 'sharedtimeTable_139'}
    ...
 
 See :ref:`amda-timetables-label` for a description of the :data:`timetable` fields.
diff --git a/docs/amda_example_list_user_catalogs.rst b/docs/amda_example_list_user_catalogs.rst
@@ -1,4 +1,4 @@
-AMDA Example 9: listing user catalogs
+AMDA_Webservice Example 9: listing user catalogs
 -------------------------------------
 
 Listing private catalogs.
@@ -7,7 +7,7 @@ Listing private catalogs.
 
 Output ::
 
-   $ python amda_list_user_catalogs.py 
+   $ python amda_list_user_catalogs.py
    {'name': 'mycata', 'intervals': '1457', 'id': 'cat_0'}