Faceswap 3 beta (#1392)

* Replace TF entry point code * Update extract pipeline * Update requirements + setup for nvidia * Remove allow-growth option * tf.keras to keras updates * Basic training functionality (original) * lib.model.losses - Port + fix all loss functions for Keras3 * lib.model - port initializers, layers. normalization to Keras3 * training - Fix previews and LR_Finder * lib.logger - Prevent crash on custom loglevels when launching scripts external to FS * lib.model.autoclip to Keras 3 * Update mixed precision layer storage * model file to .keras format * typo fix * Set loss debug logging to trace * Restructure nn_blocks to initialize layers in __init__ * lib.model - standardise class init logging * Tensorboard - Trainer: Add Torch compatible Tensorboard callbacks - GUI event reader remove TF dependency * Loss logging - Flush TB logs on save - Replace TB live iterator for GUI * Backup models on total loss drop rather than per side * Update all models to Keras3 Compat * Clean up console output * Fix learn mask and multiscale preview outputs * linting * pylint: reduce min-public-methods to 1 + linting * lib.logger/utils - remove TF specific log handling + linting * Remove lib.model.session * Extract - explicitly pass batch_size to predict * Exclude GPU for CUDA backends * plugins.model - remove exclude_gpu * Add exclude_gpus properties to ROCm and DirectML * Enable CPU mode for compatible extractors * Fix optimizer saving * bug/typo fixes * Update clip ViT to Keras 3 * lib.cli - Deprecate multi-character switches * tools cli - deprecate multi character switches * convert - remove trainer cli-arg * Remove colab references from project * Extract: Supress Keras compile warnings for FAN + Unet * plugins.extract.mask.unet-dfl - Fix for Keras3/Torch backend * plugins.extract - linting + typing * Port AdaBelief to Keras 3 * linting: Remove pylint hint from logger * typofix * Fix GUI for refactored lib.cli.args * Use saving API for load_model * setup.py: - Add --dev flag for dev tool install - Group pytorch items together for install * extract: Fix keras.device to call keras.backend.device_scope * setup.py add more dev requirements * training: Linting, fixes and docstrings setup: Update requirements files * Bugfixes for video file alignments storage: - extract/convert: Load images with correct video extension - Manual tool: Cache thumbnails with correct extension - Mask tool + Preview tool:: Update legacy alignment keys for pre-video extension storage * Fix Keras 3 syntax * Fix LR Finder for Keras 3 * lib: Linting, typing and minor clean up/fixes * extract: Fix device selection for plugins * Fix mixed precision switching for Keras 3 * Add more optimizers + open up config setting * train: Remove updating FS1 weights to FS2 models * Alignments: Remove support for legacy .json files * train: Fix Keras imports in model plugins * Interim fix: Disable loading of legacy models * tools.model: - Remove TF Saved Format saving - Fix Backup/Restore + Nan-Scan * Update deprecated LeakyReLu arg * Fix inference model creation for Keras 3 * Preview tool: Fix for Keras3 * setup.py: Configure keras backend requirements: Bump keras version * - Bump pytorch max version - Fix LossWrapper deserialization * bugfix: Lion optimizer * Bugfix: Select .keras file for models in GUI * bugfix: Model - prevent ConvAware from re-calculating on model load * train: Migration of FS2 models to FS3 * Training: Default coverage to 100% * Bugfix: GUI Stats: - Fix 'learn mask' output tracking - Reset data cache if a model folder has been deleted between reading stats and starting training * bugfix: Gui stats. Reset session if log files have been replaced * numexpr: Default to use 2/3rds of available cores * Prevent model summaries from becoming truncated * fix mixed precision layer storage to be compatible with ConvNext * Bugfix: ClipV unnest model outputs * Bugfix: Create keras config folder if it does not exist * ClipV: Don't load weights by name to supress Keras warning message * clip: Supress weights mismatch warning * Bugfix: Mean output on FFL to prevent NaNs * Remove DirectML backend * setup: update for Linux * bugfix: Legacy model loading with unnested inbound nodes * Support Python up to 3.12 * Lower max python version to 3.11 * Bugfix: Porting Keras 2 models that use ClipV to Keras 3 * Update setup for MacOS * Allow python up to 3.12 + Fix locale deprecation * GUI: Force line reading to UTF-8 * Remove redundant Tensorflow references * Disable distribution + remove redumdant tf references * pytest fixes * Remove redundant code * pytest: Set KERAS_BACKEND EnvVar * lib.model.normalization: bugfixes * lib.model.loss.LaplacianPyramidLoss: Shape dtype bugfix * lib.model.layers: Minor bugfixes * scripts.conver.ConvertItem: default item bugfix * PyTest fixes * bugfix: Don't error on TestTensorBoardLogs teardown * bugfix: Mutable types in DataClass * github workflow: Fix macOS * setup.py bugfix: Don't error on cuda key lookup * Github workflow: All loss calculations on CPU * tests: Place ML ops explicitly on CPU * Legacy model loading: Fix TFLamdaOp scalar ops and DepthwiseConv2D * Add vertical offset option for training * Typing and linting * Github actions: Add more python versions * Add python version to workflow names * Github workflow: Exclude Python 3.12 for macOS * Implement custom training loop * Docs update and pin keras
deepfakes · Jun 7, 2024 · 56dd410 · 56dd410
1 parent c8652ec
commit 56dd410
Show file tree

Hide file tree

Showing 123 changed files with 7,081 additions and 6,286 deletions.
diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml
@@ -9,7 +9,7 @@ on:
 
 jobs:
   build_conda:
-    name: conda (${{ matrix.os }}, ${{ matrix.backend }})
+    name: conda (${{ matrix.os }}, ${{ matrix.backend }} ${{ matrix.python-version }})
     runs-on: ${{ matrix.os }}
     defaults:
       run:
@@ -18,12 +18,14 @@ jobs:
       fail-fast: false
       matrix:
         os: ["ubuntu-latest", "macos-latest", "windows-latest"]
+        python-version: ["3.10", "3.11", "3.12"]
         backend: ["nvidia", "cpu"]
         include:
           - os: "ubuntu-latest"
             backend: "rocm"
-          - os: "windows-latest"
-            backend: "directml"
+        exclude:
+          - os: "macos-latest"
+            python-version: "3.12"
     steps:
       - uses: actions/checkout@v3
       - name: Set cache date
@@ -40,7 +42,8 @@ jobs:
       - name: Set up Conda
         uses: conda-incubator/setup-miniconda@v2
         with:
-          python-version: "3.10"
+          python-version: ${{ matrix.python-version }}
+          miniconda-version: "latest"
           auto-update-conda: true
           activate-environment: faceswap
       - name: Conda info
@@ -63,23 +66,23 @@ jobs:
         run: python -c "from lib.sysinfo import sysinfo ;  print(sysinfo)"
       - name: Simple Tests
         # These backends will fail as GPU drivers not available
-        if: matrix.backend != 'rocm' && matrix.backend != 'nvidia' && matrix.backend != 'directml'
+        if: matrix.backend != 'rocm' && matrix.backend != 'nvidia'
         run: |
-          FACESWAP_BACKEND="${{ matrix.backend }}" py.test -v tests/;
+         KERAS_BACKEND=torch FACESWAP_BACKEND="${{ matrix.backend }}" py.test -v tests/;
       - name: End to End Tests
         # These backends will fail as GPU drivers not available
         # macOS fails on first extract test with 'died with <Signals.SIGSEGV: 11>'
-        if: matrix.backend != 'rocm' && matrix.backend != 'nvidia' && matrix.backend != 'directml' && matrix.os != 'macos-latest'
+        if: matrix.backend != 'rocm' && matrix.backend != 'nvidia' && matrix.os != 'macos-latest'
         run: |
-          FACESWAP_BACKEND="${{ matrix.backend }}" python tests/simple_tests.py;
+          KERAS_BACKEND=torch FACESWAP_BACKEND="${{ matrix.backend }}" python tests/simple_tests.py;
 
   build_linux:
-    name: "pip (ubuntu-latest, ${{ matrix.backend }})"
+    name: "pip (ubuntu-latest, ${{ matrix.backend }} ${{ matrix.python-version }})"
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10"]
+        python-version: ["3.10", "3.11", "3.12"]
         backend: ["cpu"]
         include:
           - backend: "cpu"
@@ -111,22 +114,21 @@ jobs:
           mypy .
       - name: Simple Tests
         run: |
-          FACESWAP_BACKEND="${{ matrix.backend }}" py.test -v tests/;
+          KERAS_BACKEND=torch FACESWAP_BACKEND="${{ matrix.backend }}" py.test -v tests/;
       - name: End to End Tests
         run: |
-          FACESWAP_BACKEND="${{ matrix.backend }}" python tests/simple_tests.py;
+          KERAS_BACKEND=torch FACESWAP_BACKEND="${{ matrix.backend }}" python tests/simple_tests.py;
 
   build_windows:
-    name: "pip (windows-latest, ${{ matrix.backend }})"
+    name: "pip (windows-latest, ${{ matrix.backend }} ${{ matrix.python-version }})"
     runs-on: windows-latest
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10"]
-        backend: ["cpu", "directml"]
+        python-version: ["3.10", "3.11", "3.12"]
+        backend: ["cpu"]
         include:
           - backend: "cpu"
-          - backend: "directml"
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -143,8 +145,10 @@ jobs:
           pip install -r ./requirements/requirements_${{ matrix.backend }}.txt
       - name: List installed packages
         run: pip freeze
-      - name: Set Backend EnvVar
+      - name: Set Faceswap Backend EnvVar
         run: echo "FACESWAP_BACKEND=${{ matrix.backend }}" | Out-File -FilePath $env:GITHUB_ENV -Append
+      - name: Set Keras Backend EnvVar
+        run: echo "KERAS_BACKEND=torch" | Out-File -FilePath $env:GITHUB_ENV -Append
       - name: Lint with flake8
         run: |
           # stop the build if there are Python syntax errors or undefined names

diff --git a/INSTALL.md b/INSTALL.md
@@ -56,9 +56,8 @@ The type of computations that the process does are well suited for graphics card
     - Laptop CPUs can often run the software, but will not be fast enough to train at reasonable speeds
 - **A powerful GPU**
     - Currently, Nvidia GPUs are fully supported
-    - DirectX 12 AMD GPUs are supported on Windows through DirectML.
     - More modern AMD GPUs are supported on Linux through ROCm.
-    - M-series Macs are supported through Tensorflow-Metal
+    - M-series Macs are supported using Metal
     - If using an Nvidia GPU, then it needs to support at least CUDA Compute Capability 3.5. (Release 1.0 will work on Compute Capability 3.0)
       To see which version your GPU supports, consult this list: https://developer.nvidia.com/cuda-gpus
       Desktop cards later than the 7xx series are most likely supported.
@@ -67,14 +66,13 @@ The type of computations that the process does are well suited for graphics card
 ## Supported operating systems
 - **Windows 10/11**
   Windows 7 and 8 might work for Nvidia. Your mileage may vary.
-  DirectML support is only available in Windows 10 onwards.
   Windows has an installer which will set up everything you need. See: https://github.com/deepfakes/faceswap/releases
 - **Linux**
   Most Ubuntu/Debian or CentOS based Linux distributions will work. There is a Linux install script that will install and set up everything you need. See: https://github.com/deepfakes/faceswap/releases
 - **macOS**
   Experimental support for GPU-accelerated, native Apple Silicon processing (e.g. Apple M1 chips). Installation instructions can be found [further down this page](#macos-apple-silicon-install-guide).
   Intel based macOS systems should work, but you will need to follow the [Manual Install](#manual-install) instructions.
-- All operating systems must be 64-bit for Tensorflow to run.
+- All operating systems must be 64-bit.
 
 Alternatively, there is a docker image that is based on Debian.
 
@@ -134,7 +132,7 @@ To enter the virtual environment:
 
 #### Manual install
 Do not follow these steps if the Easy Install above completed succesfully.
-If you are using an Nvidia card make sure you have the correct versions of Cuda/cuDNN installed for the required version of Tensorflow
+If you are using an Nvidia card make sure you have the correct versions of Cuda/cuDNN installed for the required version of Torch
 - Install tkinter (required for the GUI) by typing: `conda install tk`
 - Install requirements:
   - For Nvidia GPU users: `pip install -r ./requirements/requirements_nvidia.txt`
@@ -224,7 +222,7 @@ Obtain git for your distribution from the [git website](https://git-scm.com/down
 The recommended install method is to use a Conda3 Environment as this will handle the installation of Nvidia's CUDA and cuDNN straight into your Conda Environment. This is by far the easiest and most reliable way to setup the project.
   - MiniConda3 is recommended: [MiniConda3](https://docs.conda.io/en/latest/miniconda.html)
 
-Alternatively you can install Python (3.10 64-bit) for your distribution (links below.) If you go down this route and are using an Nvidia GPU you should install CUDA (https://developer.nvidia.com/cuda-zone) and cuDNN (https://developer.nvidia.com/cudnn). for your system. If you do not plan to build Tensorflow yourself, make sure you install the correct Cuda and cuDNN package for the currently installed version of Tensorflow (Current release: Tensorflow 2.9. Release v1.0: Tensorflow 1.15). You can check for the compatible versions here: (https://www.tensorflow.org/install/source#gpu).
+Alternatively you can install Python (3.12 64-bit) for your distribution (links below.) If you go down this route and are using an Nvidia GPU you should install CUDA (https://developer.nvidia.com/cuda-zone) and cuDNN (https://developer.nvidia.com/cudnn). for your system. If you do not plan to build Torch yourself, make sure you install the correct Cuda and cuDNN package for the currently installed version of Torch.
   - Python distributions:
     - apt/yum install python3 (Linux)
     - [Installer](https://www.python.org/downloads/release/python-368/) (Windows)

diff --git a/README.md b/README.md
@@ -25,7 +25,7 @@
 
 Make sure you check out [INSTALL.md](INSTALL.md) before getting started.
 
-- [deepfakes_faceswap](#deepfakes_faceswap)
+- [deepfakes\_faceswap](#deepfakes_faceswap)
 - [Manifesto](#manifesto)
   - [FaceSwap has ethical uses.](#faceswap-has-ethical-uses)
 - [How To setup and run the project](#how-to-setup-and-run-the-project)
@@ -79,7 +79,7 @@ We are very troubled by the fact that FaceSwap can be used for unethical and dis
 # How To setup and run the project
 FaceSwap is a Python program that will run on multiple Operating Systems including Windows, Linux, and MacOS.
 
-See [INSTALL.md](INSTALL.md) for full installation instructions. You will need a modern GPU with CUDA support for best performance. Many AMD GPUs are supported through DirectML (Windows) and ROCm (Linux).
+See [INSTALL.md](INSTALL.md) for full installation instructions. You will need a modern GPU with CUDA support for best performance. Many AMD GPUs are supported through ROCm (Linux).
 
 # Overview
 The project has multiple entry points. You will have to:

diff --git a/docs/full/lib/gpu_stats.rst b/docs/full/lib/gpu_stats.rst
@@ -38,14 +38,6 @@ gpu_stats.cpu module
    :undoc-members:
    :show-inheritance:
 
-gpu_stats.directml module
--------------------------
-
-.. automodule:: lib.gpu_stats.directml
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
 gpu_stats.nvidia_apple module
 -----------------------------
 

diff --git a/docs/full/lib/model.rst b/docs/full/lib/model.rst
@@ -63,29 +63,29 @@ model.losses module
 .. autosummary::
    :nosignatures:
 
-   ~lib.model.loss.loss_tf.FocalFrequencyLoss
-   ~lib.model.loss.loss_tf.GeneralizedLoss
-   ~lib.model.loss.loss_tf.GradientLoss
-   ~lib.model.loss.loss_tf.LaplacianPyramidLoss
-   ~lib.model.loss.loss_tf.LInfNorm
-   ~lib.model.loss.loss_tf.LossWrapper
-   ~lib.model.loss.feature_loss_tf.LPIPSLoss
-   ~lib.model.loss.perceptual_loss_tf.DSSIMObjective
-   ~lib.model.loss.perceptual_loss_tf.GMSDLoss
-   ~lib.model.loss.perceptual_loss_tf.LDRFLIPLoss
-   ~lib.model.loss.perceptual_loss_tf.MSSIMLoss
-
-.. automodule:: lib.model.loss.loss_tf
+   ~lib.model.losses.loss.FocalFrequencyLoss
+   ~lib.model.losses.loss.GeneralizedLoss
+   ~lib.model.losses.loss.GradientLoss
+   ~lib.model.losses.loss.LaplacianPyramidLoss
+   ~lib.model.losses.loss.LInfNorm
+   ~lib.model.losses.loss.LossWrapper
+   ~lib.model.losses.feature_loss.LPIPSLoss
+   ~lib.model.losses.loss.perceptual_loss.DSSIMObjective
+   ~lib.model.losses.loss.perceptual_loss.GMSDLoss
+   ~lib.model.losses.loss.perceptual_loss.LDRFLIPLoss
+   ~lib.model.losses.loss.perceptual_loss.MSSIMLoss
+
+.. automodule:: lib.model.loss.losses.loss
    :members:
    :undoc-members:
    :show-inheritance:
 
-.. automodule:: lib.model.loss.feature_loss_tf
+.. automodule:: lib.model.loss.feature_loss
    :members:
    :undoc-members:
    :show-inheritance:
 
-.. automodule:: lib.model.loss.perceptual_loss_tf
+.. automodule:: lib.model.loss.perceptual_loss
    :members:
    :undoc-members:
    :show-inheritance:
@@ -152,9 +152,9 @@ model.optimizers module
 .. autosummary::
    :nosignatures:
 
-   ~lib.model.optimizers_tf.AdaBelief
+   ~lib.model.optimizers.AdaBelief
 
-.. automodule:: lib.model.optimizers_tf
+.. automodule:: lib.model.optimizers
    :members:
    :undoc-members:
    :show-inheritance:

diff --git a/docs/full/plugins/train.rst b/docs/full/plugins/train.rst
@@ -57,10 +57,37 @@ model.original module
 trainer package
 ===============
 
+This package contains the training loop for Faceswap
+
+.. rubric:: Module Summary
+
+.. autosummary::
+   :nosignatures:
+
+   ~plugins.train.trainer._base
+   ~plugins.train.model._display
+   ~plugins.train.model._tensorboard
+
 trainer._base module
 ----------------------
 
 .. automodule:: plugins.train.trainer._base
    :members:
    :undoc-members:
    :show-inheritance:
+
+trainer._display module
+-----------------------
+
+.. automodule:: plugins.train.trainer._display
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+trainer._tensorboard module
+---------------------------
+
+.. automodule:: plugins.train.trainer._tensorboard
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/faceswap.py b/faceswap.py
@@ -7,7 +7,9 @@
 
 # Translations don't work by default in Windows, so hack in environment variable
 if sys.platform.startswith("win"):
-    os.environ["LANG"], _ = locale.getdefaultlocale()
+    import ctypes
+    windll = ctypes.windll.kernel32
+    os.environ["LANG"] = locale.windows_locale[windll.GetUserDefaultUILanguage()]
 
 from lib.cli import args as cli_args  # pylint:disable=wrong-import-position
 from lib.cli.args_train import TrainArgs  # pylint:disable=wrong-import-position

diff --git a/lib/align/aligned_face.py b/lib/align/aligned_face.py
@@ -85,7 +85,8 @@ def transform_image(image: np.ndarray,
 def get_adjusted_center(image_size: int,
                         source_offset: np.ndarray,
                         target_offset: np.ndarray,
-                        source_centering: CenteringType) -> np.ndarray:
+                        source_centering: CenteringType,
+                        y_offset: float) -> np.ndarray:
     """ Obtain the correct center of a face extracted image to translate between two different
     extract centerings.
 
@@ -99,20 +100,22 @@ def get_adjusted_center(image_size: int,
         The pose offset to translate a base extracted face to target centering
     source_centering: ["face", "head", "legacy"]
         The centering of the source image
+    y_offset: float
+        Amount to additionally offset the center of the image along the y-axis
 
     Returns
     -------
     :class:`numpy.ndarray`
         The center point of the image at the given size for the target centering
     """
     source_size = image_size - (image_size * EXTRACT_RATIOS[source_centering])
-    offset = target_offset - source_offset
+    offset = target_offset - source_offset - [0., y_offset]
     offset *= source_size
     center = np.rint(offset + image_size / 2).astype("int32")
     logger.trace(  # type:ignore[attr-defined]
         "image_size: %s, source_offset: %s, target_offset: %s, source_centering: '%s', "
-        "adjusted_offset: %s, center: %s",
-        image_size, source_offset, target_offset, source_centering, offset, center)
+        "y_offset: %s, adjusted_offset: %s, center: %s",
+        image_size, source_offset, target_offset, source_centering, y_offset, offset, center)
     return center
 
 
@@ -154,6 +157,7 @@ def get_centered_size(source_centering: CenteringType,
         ratio
     """
     if source_centering == target_centering and coverage_ratio == 1.0:
+        src_size: float | int = size
         retval = size
     else:
         src_size = size - (size * EXTRACT_RATIOS[source_centering])
@@ -263,6 +267,8 @@ class AlignedFace():
         The amount of the aligned image to return. A ratio of 1.0 will return the full contents of
         the aligned image. A ratio of 0.5 will return an image of the given size, but will crop to
         the central 50%% of the image.
+    y_offset: float, optional
+        Amount to adjust the aligned face along the y-axis in the range -1. to 1. Default: 0.0
     dtype: str, optional
         Set a data type for the final face to be returned as. Passing ``None`` will return a face
         with the same data type as the original :attr:`image`. Default: ``None``
@@ -279,6 +285,7 @@ def __init__(self,
                  centering: CenteringType = "face",
                  size: int = 64,
                  coverage_ratio: float = 1.0,
+                 y_offset: float = 0.0,
                  dtype: str | None = None,
                  is_aligned: bool = False,
                  is_legacy: bool = False) -> None:
@@ -288,6 +295,7 @@ def __init__(self,
         self._centering = centering
         self._size = size
         self._coverage_ratio = coverage_ratio
+        self._y_offset = y_offset
         self._dtype = dtype
         self._is_aligned = is_aligned
         self._source_centering: CenteringType = "legacy" if is_legacy and is_aligned else "head"
@@ -320,6 +328,11 @@ def padding(self) -> int:
         extracted face image for the selected extract type. """
         return self._padding[self._centering]
 
+    @property
+    def y_offset(self) -> float:
+        """ float: Additional offset applied to the face along the y-axis in -1. to 1. range """
+        return self._y_offset
+
     @property
     def matrix(self) -> np.ndarray:
         """ :class:`numpy.ndarray`: The 3x2 transformation matrix for extracting and aligning the
@@ -532,8 +545,7 @@ def extract_face(self, image: np.ndarray | None) -> np.ndarray | None:
                          "image. Returning empty face.")
             return None
 
-        if self._is_aligned and (self._centering != self._source_centering or
-                                 self._coverage_ratio != 1.0):
+        if self._is_aligned:
             # Crop out the sub face from full head
             image = self._convert_centering(image)
 
@@ -648,7 +660,8 @@ def get_cropped_roi(self,
                 center = get_adjusted_center(image_size,
                                              self.pose.offset[self._source_centering],
                                              self.pose.offset[centering],
-                                             self._source_centering)
+                                             self._source_centering,
+                                             self.y_offset)
                 padding = target_size // 2
                 roi = np.array([center - padding, center + padding]).ravel()
                 logger.trace(  # type:ignore[attr-defined]