Add support for piping to mfa g2p

.github/workflows/main.yml

@@ -48,6 +48,10 @@ jobs:
           extra-specs: |
             python=3.9
 
+      - name: Configure mfa
+        shell: bash -l {0}
+        run: python -m montreal_forced_aligner configure --disable_auto_server
+
       - name: Initialize database
         shell: bash -l {0}
         run: python -m montreal_forced_aligner server init -p test

docs/source/changelog/changelog_2.2.rst

@@ -9,6 +9,11 @@
 =====
 
 - Fixes an issue where some directories in Common Voice Japanese were causing FileNotFound errors for sound files
+- Changes PostgreSQL database connections to use socket directories rather than ports
+- Added the ability to manage MFA database servers (:ref:`server`), along with the configuration flag to disable automatic starting/stopping of databases
+- Disabled starting servers for subcommands like ``configure``, ``version``, ``history`` or ``--help`` invocations
+- Added the ability to pipe words via stdin/stdout when :ref:`g2p_dictionary_generating`
+- Added a first pass at providing estimations of alignment quality through the ``alignment_analysis.csv`` file exported with alignments, see :doc:`alignment_analysis` for more details.
 
 2.2.3
 =====

docs/source/reference/helper/helper.rst

@@ -3,7 +3,6 @@
    .. autosummary::
       :toctree: generated/
 
-       TerminalPrinter
        comma_join
        make_safe
        make_scp_safe
@@ -15,4 +14,3 @@
        compare_labels
        overlap_scoring
        align_phones
-       CustomFormatter

docs/source/reference/index.rst

@@ -23,4 +23,5 @@ This should (hopefully) make it easier to extend MFA for your own purposes if yo
 
    core_index
    top_level_index
+   server/index
    helper/index

docs/source/reference/server/index.rst

@@ -0,0 +1,20 @@
+
+.. _server_api:
+
+Managing MFA servers
+====================
+
+Functions
+---------
+
+.. currentmodule:: montreal_forced_aligner.command_line.utils
+
+.. autosummary::
+   :toctree: generated/
+
+   configure_pg
+   initialize_server
+   check_databases
+   start_server
+   stop_server
+   delete_server

docs/source/user_guide/implementations/alignment_analysis.md

@@ -0,0 +1,2 @@
+
+# Analyzing alignment quality

docs/source/user_guide/index.rst

@@ -135,6 +135,7 @@ We acknowledge funding from Social Sciences and Humanities Research Council (SSH
    workflows/index
    corpus_creation/index
    configuration/index
+   server/index
    models/index
    implementations/index
    concepts/index

docs/source/user_guide/server/index.rst

@@ -0,0 +1,54 @@
+
+.. _server:
+
+***********
+MFA Servers
+***********
+
+MFA database servers
+====================
+
+By default, MFA starts or creates a PostgreSQL servers when a command is invoked, and stops the server at the end of processing.  The goal here is to have as unobtrusive of a database server as possible, however there are use cases that you may require more control. To turn off the automatic management of PostgreSQL servers, run :code:`mfa configure --disable_auto_server`.
+
+You can have multiple PostgreSQL servers by using the :code:`--profile` flag, if necessary.  By default the "global" profile is used.  The profile flags are used in :ref:`configure_cli`, as the default options set with :code:`configure` are done on a per-profile basis.
+
+
+PostgreSQL configuration
+------------------------
+
+MFA overrides some default configuration values for its PostgreSQL servers when they are initialized.
+
+.. code-block::
+
+   log_min_duration_statement = 5000
+   enable_partitionwise_join = on
+   enable_partitionwise_aggregate = on
+   unix_socket_directories = '/path/to/current/profile/socket_directory'
+   listen_addresses = ''
+
+   maintenance_work_mem = 500MB
+   work_mem = 128MB
+   shared_buffers = 256MB
+   max_connections = 1000
+
+The goal for MFA is to run on local desktops at reasonable performance on moderate sized corpora (<3k hours).  Depending on your use case, you may need to tune the :code:`postgres.conf` file further to suit your set up and corpus (see `PostgreSQL's documentation <https://www.postgresql.org/docs/15/runtime-config.html>`_ and `postgresqltuner utility script <https://github.com/jfcoz/postgresqltuner>`_.  Additionally, note that any port listening is turned off by default and connections are handled via socket directories.
+
+.. warning::
+
+   MFA PostgreSQL databases are meant to be on the expendable side. Though they can persist across use cases, it's not really recommended.  Use of :code:`--clean` drops all data in the database to ensure a fresh start state, as various commands perform destructive commands.  As an example :ref:`create_segments` deletes and recreates :class:`~montreal_forced_aligner.db.Utterance` objects, so the original text transcripts are absent in the database following its run.
+
+.. _server_cli:
+
+Managing MFA database servers
+=============================
+
+MFA PostgreSQL servers can be managed via the subcommands in `mfa server`, allowing you to initialize new servers, and start, stop, and delete existing servers.
+
+.. click:: montreal_forced_aligner.command_line.server:server_cli
+   :prog: mfa server
+   :nested: full
+
+API reference
+-------------
+
+- :ref:`server_api`

docs/source/user_guide/workflows/alignment.rst

@@ -8,9 +8,10 @@ This is the primary workflow of MFA, where you can use pretrained :term:`acousti
 
 .. seealso::
 
-   * :ref:`alignment_evaluation` for details on how to evaluate alignments against a gold standard.
-   * :ref:`fine_tune_alignments` for implementation details on how alignments are fine tuned.
-   * :ref:`phone_models` for implementation details on using phone bigram models for generating alignments.
+   * :doc:`alignment_evaluation` for details on how to evaluate alignments against a gold standard.
+   * :doc:`fine_tune_alignments` for implementation details on how alignments are fine tuned.
+   * :doc:`phone_models` for implementation details on using phone bigram models for generating alignments.
+   * :doc:`alignment_analysis` for details on the fields generated in the ``alignment_analysis.csv`` file in the output folder
 
 Command reference
 -----------------

docs/source/user_guide/workflows/dictionary_generating.rst

@@ -30,6 +30,17 @@ See :ref:`dict_generating_example` for an example of how to use G2P functionalit
 
    As of version 2.0.6, users on Windows can run this command natively without requiring :xref:`wsl`, see :ref:`installation` for more details.
 
+Piping stdin/stdout
+-------------------
+
+If you specify the input path as ``-`` instead of a file path, the g2p command will run through each line in the stdin and G2P each word with minimal processing.  Words will be lower cased and any graphemes that were not in the model's training data will be removed.
+
+If you specify the output path as ``-`` instead of a file path, the g2p command will send pronunciations as stdout rather than writing to a file.
+
+.. note::
+
+   Using stdin will also bypass database set up (though the database server will still be started and stopped, so be sure to run :code:`mfa configure --no_auto_server` if speed is of necessity.
+
 Command reference
 -----------------
 

environment.yml

@@ -32,6 +32,7 @@ dependencies:
   - pytest
   - pytest-mypy
   - pytest-cov
+  - pytest-timeout
   - mock
   - coverage
   - coveralls

montreal_forced_aligner/alignment/base.py

@@ -8,6 +8,7 @@
 import logging
 import multiprocessing as mp
 import os
+import shutil
 import subprocess
 import time
 import typing
@@ -24,6 +25,8 @@
 from montreal_forced_aligner.alignment.multiprocessing import (
     AlignmentExtractionArguments,
     AlignmentExtractionFunction,
+    CalculateSpeechPostArguments,
+    CalculateSpeechPostFunction,
     ExportTextGridArguments,
     ExportTextGridProcessWorker,
     FineTuneArguments,
@@ -46,6 +49,7 @@
     CorpusWorkflow,
     Dictionary,
     File,
+    Phone,
     PhoneInterval,
     PhonologicalRule,
     Pronunciation,
@@ -148,6 +152,80 @@ def score_options(self) -> MetaDict:
             "word_insertion_penalty": getattr(self, "word_insertion_penalty", 0.5),
         }
 
+    def calculate_speech_post_arguments(self) -> List[CalculateSpeechPostArguments]:
+        return [
+            CalculateSpeechPostArguments(
+                j.id,
+                getattr(self, "db_string", ""),
+                self.working_log_directory.joinpath(f"calculate_speech_post.{j.id}.log"),
+                self.model_path,
+                self.align_options,
+            )
+            for j in self.jobs
+        ]
+
+    def analyze_alignments(self):
+        with self.session() as session:
+            update_mappings = []
+            query = session.query(
+                PhoneInterval.phone_id,
+                sqlalchemy.func.avg(PhoneInterval.duration),
+                sqlalchemy.func.stddev_samp(PhoneInterval.duration),
+            ).group_by(PhoneInterval.phone_id)
+            for p_id, mean_duration, sd_duration in query:
+                update_mappings.append(
+                    {"id": p_id, "mean_duration": mean_duration, "sd_duration": sd_duration}
+                )
+            bulk_update(session, Phone, update_mappings)
+            session.commit()
+
+            arguments = self.calculate_speech_post_arguments()
+            update_mappings = []
+            with tqdm(total=self.num_current_utterances, disable=GLOBAL_CONFIG.quiet) as pbar:
+                for utt_id, speech_log_likelihood, duration_deviation in run_kaldi_function(
+                    CalculateSpeechPostFunction, arguments, pbar.update
+                ):
+                    update_mappings.append(
+                        {
+                            "id": utt_id,
+                            "speech_log_likelihood": speech_log_likelihood,
+                            "duration_deviation": duration_deviation,
+                        }
+                    )
+
+            bulk_update(session, Utterance, update_mappings)
+            session.commit()
+
+            csv_path = self.working_directory.joinpath("alignment_analysis.csv")
+            with mfa_open(csv_path, "w") as f:
+                writer = csv.writer(f)
+                writer.writerow(
+                    [
+                        "file",
+                        "begin",
+                        "end",
+                        "speaker",
+                        "overall_log_likelihood",
+                        "speech_log_likelihood",
+                        "phone_duration_deviation",
+                    ]
+                )
+                utterances = (
+                    session.query(
+                        File.name,
+                        Utterance.begin,
+                        Utterance.end,
+                        Speaker.name,
+                        Utterance.alignment_log_likelihood,
+                        Utterance.speech_log_likelihood,
+                        Utterance.duration_deviation,
+                    )
+                    .join(Utterance.file)
+                    .join(Utterance.speaker)
+                )
+                for row in utterances:
+                    writer.writerow([*row])
+
     def alignment_extraction_arguments(self) -> List[AlignmentExtractionArguments]:
         """
         Generate Job arguments for
@@ -1162,6 +1240,11 @@ def export_files(
             f"Exporting {self.current_workflow.name} TextGrids to {self.export_output_directory}..."
         )
         self.export_output_directory.mkdir(parents=True, exist_ok=True)
+        analysis_csv = self.working_directory.joinpath("alignment_analysis.csv")
+        if analysis_csv.exists():
+            shutil.copyfile(
+                analysis_csv, self.export_output_directory.joinpath("alignment_analysis.csv")
+            )
         self.export_textgrids(output_format, include_original_text)
 
     def evaluate_alignments(