error: cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- failed with code 101

[himanshushukla12]

Problem

getting error message, after installing the following dependencies:
pip install -q accelerate==0.21.0 peft==0.4.0 bitsandbytes==0.40.2 transformers==4.31.0 trl==0.4.7
I'm using the windows 10, Python --version: Python 3.12.3

the error log is given below:

error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
     --> tokenizers-lib\src\models\bpe\trainer.rs:526:47
      |
  522 |                     let w = &words[*i] as *const _ as *mut _;
      |                             -------------------------------- casting happend here
  ...
  526 |                         let word: &mut Word = &mut (*w);
      |                                               ^^^^^^^^^
      |
      = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
      = note: `#[deny(invalid_reference_casting)]` on by default
  
  warning: `tokenizers` (lib) generated 3 warnings
  error: could not compile `tokenizers` (lib) due to 1 previous error; 3 warnings emitted
  
  Caused by:
    process didn't exit successfully: `D:\Users\z004x2xz\.rustup\toolchains\stable-x86_64-pc-windows-msvc\bin\rustc.exe --crate-name tokenizers --edition=2018 tokenizers-lib\src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"cached-path\"" --cfg "feature=\"clap\"" --cfg "feature=\"cli\"" --cfg "feature=\"default\"" --cfg "feature=\"dirs\"" --cfg "feature=\"esaxx_fast\"" --cfg "feature=\"http\"" --cfg "feature=\"indicatif\"" --cfg "feature=\"onig\"" --cfg "feature=\"progressbar\"" --cfg "feature=\"reqwest\"" -C metadata=52fdcaa13bbc08d3 -C extra-filename=-52fdcaa13bbc08d3 --out-dir D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps -L dependency=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps --extern aho_corasick=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libaho_corasick-31d38de356baac92.rmeta --extern cached_path=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libcached_path-14bc83a2040b050a.rmeta --extern clap=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libclap-454a5398859662e6.rmeta --extern derive_builder=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libderive_builder-71f03773f7c9a29d.rmeta --extern dirs=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libdirs-cd4d3f06cf0cb9be.rmeta --extern esaxx_rs=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libesaxx_rs-58b8a51cf6f8cfc2.rmeta --extern getrandom=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libgetrandom-54a122f8dbd8d573.rmeta --extern indicatif=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libindicatif-d82e96c444a94e25.rmeta --extern itertools=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libitertools-2e961c6c4b29c202.rmeta --extern lazy_static=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\liblazy_static-80afce55129fc996.rmeta --extern log=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\liblog-6cb77d9aa7592a89.rmeta --extern macro_rules_attribute=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libmacro_rules_attribute-de4dbb0819fd6892.rmeta --extern monostate=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libmonostate-6139f95c2c7f40f0.rmeta --extern onig=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libonig-defe33180c8a5417.rmeta --extern paste=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\paste-46846aa10bacaa7d.dll --extern rand=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\librand-05f7aaeca9ac0882.rmeta --extern rayon=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\librayon-e7ed10ab28edc36e.rmeta --extern rayon_cond=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\librayon_cond-9d6177d31c663e3c.rmeta --extern regex=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libregex-23751f74122cb7a5.rmeta --extern regex_syntax=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libregex_syntax-fbdd93e4f1660d3c.rmeta --extern reqwest=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libreqwest-796974923851d8cf.rmeta --extern serde=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libserde-1df1d457891c0e3e.rmeta --extern serde_json=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libserde_json-1ed47b6a51fd4260.rmeta --extern spm_precompiled=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libspm_precompiled-72ca5f2f7663caf9.rmeta --extern thiserror=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libthiserror-7af0579bece1f9c0.rmeta --extern unicode_normalization_alignments=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libunicode_normalization_alignments-cca31768f716dbdd.rmeta --extern unicode_segmentation=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libunicode_segmentation-c99f69c10847e8a7.rmeta --extern unicode_categories=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\deps\libunicode_categories-9b6dc46228f07f55.rmeta -L native=D:\Users\z004x2xz\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.52.5\lib -L native=D:\Users\z004x2xz\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.48.5\lib -L "native=E:\ExternalApps\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.40.33807\atlmfc\lib\x64" -L native=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\build\bzip2-sys-a3e3249c185e78f0\out\lib -L "native=E:\ExternalApps\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.40.33807\atlmfc\lib\x64" -L native=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\build\zstd-sys-03d6b9e60b2fc6b3\out -L "native=E:\ExternalApps\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.40.33807\atlmfc\lib\x64" -L native=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\build\esaxx-rs-7d634e0f3c55e5e3\out -L "native=E:\ExternalApps\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.40.33807\atlmfc\lib\x64" -L native=D:\Users\z004x2xz\AppData\Local\Temp\pip-install-4vjhi7ii\tokenizers_1a8365565e424c5abb31cc2f0e15b4bb\target\release\build\onig_sys-f9447075092ba5a1\out` (exit code: 1)
  error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
  [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects`

Steps

Run the dependencies that's it and the error will come.

Possible Solution(s)

No response

Notes

No response

Version

cargo 1.79.0 (ffa9cf99a 2024-06-03)
release: 1.79.0
commit-hash: ffa9cf99a594e59032757403d4c780b46dc2c43a
commit-date: 2024-06-03
host: x86_64-pc-windows-msvc
libgit2: 1.7.2 (sys:0.18.3 vendored)
libcurl: 8.6.0-DEV (sys:0.4.72+curl-8.6.0 vendored ssl:Schannel)
os: Windows 10.0.19045 (Windows 10 Enterprise) [64-bit]

[ehuss]

The error message looks to be coming from building the tokenizers library. This does not appear to be a cargo issue. You might want to head over to https://github.com/huggingface/tokenizers to see if they can help.