An efficient GPU support for LLM inference with 6-bit quantization (FP6).

A fast inference library for running LLMs locally on modern consumer-class GPUs

4 bits quantization of LLaMA using GPTQ

Reorder-based post-training quantization for large language model

llama/alpaca onnx models, quantization and testcase

Aligning pretrained language models with instruction data generated by themselves.

Transformer related optimization, including BERT, GPT

👻

[ICML'21] I-BERT: Integer-only BERT Quantization

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

TNN：由腾讯优图实验室打造，移动端高性能、轻量级推理框架，同时拥有跨平台、高性能、模型压缩、代码裁剪等众多突出优势。TNN框架在原有Rapidnet、ncnn框架的基础上进一步加强了移动端设备的支持以及性能优化，同时也借鉴了业界主流开源框架高性能和良好拓展性的优点。目前TNN已经在手Q、微视、P图等应用中落地，欢迎大家参与协同共建，促进TNN推理框架进一步完善。

An open-source C++ library developed and used at Facebook.

MegEngine 是一个快速、可拓展、易于使用且支持自动求导的数值计算框架

Assembler for NVIDIA Volta and Turing GPUs

A simple tool to profile performance of multiple combinations of GEMM of cuBLAS

TensorRT is a C++ library for high performance inference on NVIDIA GPUs and deep learning accelerators.

flexible-gemm conv of deepcore

Compiler for Neural Network hardware accelerators

A CPU tool for benchmarking the peak of floating points

PlaidML is a framework for making deep learning work everywhere.

Visualizer for neural network, deep learning and machine learning models

"Multi-Level Intermediate Representation" Compiler Infrastructure

SLING - A natural language frame semantics parser

Code for: "And the bit goes down: Revisiting the quantization of neural networks"

MNN is a lightweight deep neural network inference engine.

Fast & Simple Resource-Constrained Learning of Deep Network Structure

Efficient neural speech synthesis

Catamount is a compute graph analysis tool to load, construct, and modify deep learning models and to symbolically analyze their compute requirements

Quantized Neural Network PACKage - mobile-optimized implementation of quantized neural network operators