unit test / code coverage improvements (langchain-ai#322)

This PR has two contributions: 1. Add test for when stop token is found in middle of text 2. Add code coverage tooling and instructions - Add pytest-cov via poetry - Add necessary config files - Add new make instruction for `coverage` - Update README with coverage guidance - Update minor README formatting/spelling Co-authored-by: Hunter Gerlach <hunter@huntergerlach.com>
rgbkrk · Dec 13, 2022 · 482611f · 482611f
1 parent 8861770
commit 482611f
Show file tree

Hide file tree

Showing 9 changed files with 1,921 additions and 1,644 deletions.
diff --git a/.coveragerc b/.coveragerc
@@ -0,0 +1,2 @@
+[run]
+omit = tests/*
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
@@ -6,31 +6,31 @@ on:
   pull_request:
 
 env:
-  POETRY_VERSION: "1.2.0"
+  POETRY_VERSION: "1.3.1"
 
 jobs:
   build:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: 
-         - "3.8"
-         - "3.9"
-         - "3.10"
-         - "3.11"
+        python-version:
+          - "3.8"
+          - "3.9"
+          - "3.10"
+          - "3.11"
     steps:
-    - uses: actions/checkout@v3
-    - name: Install poetry
-      run: |
-        pipx install poetry==$POETRY_VERSION
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v4
-      with:
-        python-version: ${{ matrix.python-version }}
-        cache: poetry
-    - name: Install dependencies
-      run: |
-        poetry install
-    - name: Analysing the code with our lint
-      run: |
-        make lint
+      - uses: actions/checkout@v3
+      - name: Install poetry
+        run: |
+          pipx install poetry==$POETRY_VERSION
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: poetry
+      - name: Install dependencies
+        run: |
+          poetry install
+      - name: Analysing the code with our lint
+        run: |
+          make lint
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -6,29 +6,29 @@ on:
   pull_request:
 
 env:
-  POETRY_VERSION: "1.2.0"
+  POETRY_VERSION: "1.3.1"
 
 jobs:
   build:
     runs-on: ubuntu-latest
     strategy:
       matrix:
         python-version:
-         - "3.8"
-         - "3.9"
-         - "3.10"
-         - "3.11"
+          - "3.8"
+          - "3.9"
+          - "3.10"
+          - "3.11"
     steps:
-    - uses: actions/checkout@v3
-    - name: Install poetry
-      run: pipx install poetry==$POETRY_VERSION
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v4
-      with:
-        python-version: ${{ matrix.python-version }}
-        cache: 'poetry'
-    - name: Install dependencies
-      run: poetry install
-    - name: Run unit tests
-      run: |
-        make tests
+      - uses: actions/checkout@v3
+      - name: Install poetry
+        run: pipx install poetry==$POETRY_VERSION
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: "poetry"
+      - name: Install dependencies
+        run: poetry install
+      - name: Run unit tests
+        run: |
+          make tests
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -18,15 +18,15 @@ with sorting and discovery of issues of interest. These include:
 - prompts: related to prompt tooling/infra.
 - llms: related to LLM wrappers/tooling/infra.
 - chains
-- utilities: related to different types of utilities to integrate with (Python, SQL, etc).
+- utilities: related to different types of utilities to integrate with (Python, SQL, etc.).
 - agents
 - memory
 - applications: related to example applications to build
 
 If you start working on an issue, please assign it to yourself.
 
 If you are adding an issue, please try to keep it focused on a single modular bug/improvement/feature.
-If two issues are related, or blocking, please link them rather than keep them as one single one.
+If the two issues are related, or blocking, please link them rather than keep them as one single one.
 
 We will try to keep these issues as up to date as possible, though
 with the rapid rate of develop in this field some may get out of date.
@@ -55,28 +55,27 @@ even patch releases may contain [non-backwards-compatible changes](https://semve
 If your contribution has made its way into a release, we will want to give you credit on Twitter (only if you want though)!
 If you have a Twitter account you would like us to mention, please let us know in the PR or in another manner.
 
-
 ## 🤖Developer Setup
 
 ### 🚀Quick Start
 
-This project uses [Poetry](https://python-poetry.org/) as a dependency manager. Check out Poetry's own [documentation on how to install it](https://python-poetry.org/docs/#installation) on your system before proceeding.
+This project uses [Poetry](https://python-poetry.org/) as a dependency manager. Check out Poetry's [documentation on how to install it](https://python-poetry.org/docs/#installation) on your system before proceeding.
 
 To install requirements:
 
 ```bash
 poetry install -E all
 ```
 
-This will install all requirements for running the package, examples, linting, formatting, and tests. Note the `-E all` flag will install all optional dependencies necessary for integration testing.
+This will install all requirements for running the package, examples, linting, formatting, tests, and coverage. Note the `-E all` flag will install all optional dependencies necessary for integration testing.
 
 Now, you should be able to run the common tasks in the following section.
 
 ### ✅Common Tasks
 
 #### Code Formatting
 
-Formatting for this project is a combination of [Black](https://black.readthedocs.io/en/stable/) and [isort](https://pycqa.github.io/isort/).
+Formatting for this project is done via a combination of [Black](https://black.readthedocs.io/en/stable/) and [isort](https://pycqa.github.io/isort/).
 
 To run formatting for this project:
 
@@ -86,19 +85,29 @@ make format
 
 #### Linting
 
-Linting for this project is a combination of [Black](https://black.readthedocs.io/en/stable/), [isort](https://pycqa.github.io/isort/), [flake8](https://flake8.pycqa.org/en/latest/), and [mypy](http://mypy-lang.org/).
+Linting for this project is done via a combination of [Black](https://black.readthedocs.io/en/stable/), [isort](https://pycqa.github.io/isort/), [flake8](https://flake8.pycqa.org/en/latest/), and [mypy](http://mypy-lang.org/).
 
 To run linting for this project:
 
 ```bash
 make lint
 ```
 
-We recognize linting can be annoying - if you do not want to do it, please contact a project maintainer and they can help you with it. We do not want this to be a blocker for good code getting contributed.
+We recognize linting can be annoying - if you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed.
+
+#### Coverage
+
+Code coverage (i.e. the amount of code that is covered by unit tests) helps identify areas of the code that are potentially more or less brittle.
+
+To get a report of current coverage, run the following:
+
+```bash
+make coverage
+```
 
 #### Testing
 
-Unit tests cover modular logic that does not require calls to outside apis.
+Unit tests cover modular logic that does not require calls to outside APIs.
 
 To run unit tests:
 
@@ -142,4 +151,4 @@ Docs are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/
 
 For that reason, we ask that you add good documentation to all classes and methods.
 
-Similar to linting, we recognize documentation can be annoying - if you do not want to do it, please contact a project maintainer and they can help you with it. We do not want this to be a blocker for good code getting contributed.
+Similar to linting, we recognize documentation can be annoying. If you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed.
diff --git a/Makefile b/Makefile
@@ -1,5 +1,11 @@
 .PHONY: format lint tests integration_tests
 
+coverage:
+	poetry run pytest --cov \
+		--cov-config=.coveragerc \
+		--cov-report xml \
+		--cov-report term-missing:skip-covered
+
 format:
 	poetry run black .
 	poetry run isort .

diff --git a/README.md b/README.md
@@ -13,23 +13,24 @@
 Large language models (LLMs) are emerging as a transformative technology, enabling
 developers to build applications that they previously could not.
 But using these LLMs in isolation is often not enough to
-create a truly powerful app - the real power comes when you are able to
-combine them with other sources of computation or knowledge.
+create a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge.
 
 This library is aimed at assisting in the development of those types of applications.
 
 ## 📖 Documentation
 
 Please see [here](https://langchain.readthedocs.io/en/latest/?) for full documentation on:
-- Getting started (installation, setting up environment, simple examples)
+
+- Getting started (installation, setting up the environment, simple examples)
 - How-To examples (demos, integrations, helper functions)
 - Reference (full API docs)
-- Resources (high level explanation of core concepts)
+  Resources (high-level explanation of core concepts)
 
 ## 🚀 What can this help with?
 
 There are four main areas that LangChain is designed to help with.
 These are, in increasing order of complexity:
+
 1. LLM and Prompts
 2. Chains
 3. Agents