LIN-290 documentation fixes (LineaLabs#629)

* add docstring to get_code, just that nothing else absolutely. Fixes LIN-300 * move docker test example from howto to usage/airflow. Also reorg other items in howto to the appropriate locations. Should fix LIN-299 * related to LIN-300, the parameter is used in two places. Also fix the parameter passed to GetCodeEvent - was hardcoded before. * fix a small indentation issue * late night spell checks * Reorganize contents under User Guide section * Use consistent doc style Co-authored-by: Sangyoon Park <datumvitae@gmail.com>
daavoo · May 5, 2022 · c0a6a59 · c0a6a59
1 parent 2c09587
commit c0a6a59
Show file tree

Hide file tree

Showing 27 changed files with 273 additions and 144 deletions.
diff --git a/.cspell/custom-dictionary-workspace.txt b/.cspell/custom-dictionary-workspace.txt
@@ -0,0 +1,30 @@
+__getitem__
+# Custom Dictionary Words
+astpretty
+asttokens
+builtins
+getattr
+graphviz
+ipython
+isinstance
+jupyterlab
+kwargs
+linea
+lineabuiltins
+lineapy
+maxdepth
+nbconvert
+nbformat
+nbsphinx
+nbval
+NBVAL_IGNORE_OUTPUT
+networkx
+psycopg2
+Pydantic
+scipy
+sklearn
+sqlalchemy
+templating
+toctree
+traceback
+xdist
diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -14,5 +14,29 @@
   "python.formatting.blackArgs": [
     "--line-length",
     "79"
+  ],
+  "cSpell.customDictionaries": {
+    "custom-dictionary-workspace": {
+      "name": "custom-dictionary-workspace",
+      "path": "${workspaceFolder:lineapy}/.cspell/custom-dictionary-workspace.txt",
+      "addWords": true,
+      "scope": "workspace"
+    }
+  },
+  "cSpell.ignoreWords": [
+    "BITOR",
+    "ISNOT",
+    "MULT",
+    "NONAME",
+    "NOTEQ",
+    "PICKLER",
+    "excepthook",
+    "ints",
+    "ormobject",
+    "postgresdb",
+    "postgrespwd",
+    "postgresuser",
+    "quickstart",
+    "reqs"
   ]
 }
diff --git a/Makefile b/Makefile
@@ -50,7 +50,7 @@ bash-airflow:
 	docker-compose run --rm ${service_name}-airflow /bin/bash
 
 build-docs:
-	docker-compose run --rm ${service_name} /bin/bash -c "cd docs && rm -rf source/build source/autogen && SPHINX_APIDOC_OPTIONS=members sphinx-apidoc -d 2 -f -o ./source/autogen ../lineapy/ && make html"
+	docker-compose run --rm ${service_name} /bin/bash -c "cd docs && rm -rf build source/build source/_build source/autogen && SPHINX_APIDOC_OPTIONS=members sphinx-apidoc -d 2 -f -o ./source/autogen ../lineapy/ && make html"
 
 test:
 	make deps

diff --git a/docs/source/autogen/lineapy.plugins.rst b/docs/source/autogen/lineapy.plugins.rst
@@ -16,6 +16,18 @@ lineapy.plugins.base module
 .. automodule:: lineapy.plugins.base
    :members:
 
+lineapy.plugins.script module
+-----------------------------
+
+.. automodule:: lineapy.plugins.script
+   :members:
+
+lineapy.plugins.task module
+---------------------------
+
+.. automodule:: lineapy.plugins.task
+   :members:
+
 lineapy.plugins.utils module
 ----------------------------
 

diff --git a/docs/source/autogen/lineapy.utils.analytics.rst b/docs/source/autogen/lineapy.utils.analytics.rst
@@ -0,0 +1,23 @@
+lineapy.utils.analytics package
+===============================
+
+Submodules
+----------
+
+lineapy.utils.analytics.event\_schemas module
+---------------------------------------------
+
+.. automodule:: lineapy.utils.analytics.event_schemas
+   :members:
+
+lineapy.utils.analytics.usage\_tracking module
+----------------------------------------------
+
+.. automodule:: lineapy.utils.analytics.usage_tracking
+   :members:
+
+Module contents
+---------------
+
+.. automodule:: lineapy.utils.analytics
+   :members:
diff --git a/docs/source/autogen/lineapy.utils.rst b/docs/source/autogen/lineapy.utils.rst
@@ -1,9 +1,23 @@
 lineapy.utils package
 =====================
 
+Subpackages
+-----------
+
+.. toctree::
+   :maxdepth: 2
+
+   lineapy.utils.analytics
+
 Submodules
 ----------
 
+lineapy.utils.benchmarks module
+-------------------------------
+
+.. automodule:: lineapy.utils.benchmarks
+   :members:
+
 lineapy.utils.config module
 ---------------------------
 
@@ -34,6 +48,12 @@ lineapy.utils.logging\_config module
 .. automodule:: lineapy.utils.logging_config
    :members:
 
+lineapy.utils.tree\_logger module
+---------------------------------
+
+.. automodule:: lineapy.utils.tree_logger
+   :members:
+
 lineapy.utils.utils module
 --------------------------
 

diff --git a/docs/source/examples.rst b/docs/source/examples.rst
diff --git a/docs/source/features/index.rst b/docs/source/features/index.rst
diff --git a/docs/source/guide/build_pipelines/airflow.rst b/docs/source/guide/build_pipelines/airflow.rst
@@ -0,0 +1,19 @@
+.. _testingairflow:
+
+Running Airflow DAGs
+--------------------
+
+Every time :func:`lineapy.to_pipeline` is called, a Dockerfile and a ``requirements.txt`` with the dag name as prefix will be generated in the same folder.
+When using ``framework="AIRFLOW"``, we can use the dockerfile that is output to set up an image with a test airflow instance. 
+This standalone instance can be used to test your dag. To build an image, run the following command:
+
+.. code:: bash
+
+    docker build -t <image_name> . -f <dagname>_Dockerfile
+
+To then stand up an airflow instance with the dag in it, run the following command:
+
+.. code:: bash
+
+    docker run -it -p 8080:8080 <image_name>
+
diff --git a/docs/source/guide/build_pipelines/index.rst b/docs/source/guide/build_pipelines/index.rst
@@ -0,0 +1,8 @@
+Building Pipelines
+==================
+
+.. toctree::
+   :maxdepth: 1
+
+   ../../tutorials/02_build_pipelines
+   airflow
diff --git a/docs/source/guide/manage_artifacts/database/index.rst b/docs/source/guide/manage_artifacts/database/index.rst
@@ -0,0 +1,7 @@
+Databases for Artifact Store
+============================
+
+.. toctree::
+   :maxdepth: 1
+
+   postgres
diff --git a/docs/source/features/postgres.rst → ...de/manage_artifacts/database/postgres.rst b/docs/source/features/postgres.rst → ...de/manage_artifacts/database/postgres.rst
@@ -1,7 +1,7 @@
 .. _postgres:
 
-Using PostgreSQL with LineaPy
-=============================
+PostgreSQL
+==========
 
 By default, LineaPy uses SQLite for artifact store, which keeps the package light and simple.
 Given the limitations of SQLite (e.g., single write access to a database at a time), however,
@@ -91,3 +91,16 @@ if successful. Otherwise, it will default back to SQLite and print:
 .. code:: none
 
     sqlite:///.linea/db.sqlite
+
+Known Issues
+------------
+
+If you are using PostgreSQL as your database, you might encounter the following error:
+
+.. code:: none
+
+    NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgres
+
+
+This is caused by a change in SQLAlchemy where they dropped support for DB URLs of the form ``postgres://``.
+Using ``postgresql://`` instead should fix this error.
diff --git a/docs/source/guide/manage_artifacts/index.rst b/docs/source/guide/manage_artifacts/index.rst
@@ -0,0 +1,7 @@
+Managing Artifacts
+==================
+
+.. toctree::
+   :maxdepth: 1
+
+   database/index
diff --git a/docs/source/guide/refactor_code/index.rst b/docs/source/guide/refactor_code/index.rst
@@ -0,0 +1,7 @@
+Refactoring Code
+================
+
+.. toctree::
+   :maxdepth: 1
+
+   ../../tutorials/01_refactor_code
diff --git a/docs/source/howto.rst b/docs/source/howto.rst
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -16,9 +16,10 @@ Welcome to LineaPy!
    :caption: User Guide
 
    tutorials/00_api_basics
-   examples
-   features/index
-
+   guide/refactor_code/index
+   guide/build_pipelines/index
+   guide/manage_artifacts/index
+
 
 .. toctree::
    :maxdepth: 1
@@ -34,12 +35,3 @@ Welcome to LineaPy!
    references/development
    references/internals
    references/api_reference
-
-
-..
-   Indices and tables
-   ==================
-
-   * :ref:`genindex`
-   * :ref:`modindex`
-   * :ref:`search`
diff --git a/docs/source/references/internals.rst b/docs/source/references/internals.rst
@@ -187,8 +187,8 @@ That goes through a number of steps, which we outline below, from outside in:
 ~~~~~~~~~~~~~
 
 We currently support two ways to start tracing from LineaPy. The CLI,
-which is used to trace Python scripts, and our Juptyer integration which is
-used in Juptyer Notebooks and IPython. Both of them go from source, to AST,
+which is used to trace Python scripts, and our Jupyter integration which is
+used in Jupyter Notebooks and IPython. Both of them go from source, to AST,
 to a graph.
 
 CLI