High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

Code for Palu: Compressing KV-Cache with Low-Rank Projection

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

Running large language models on a single GPU for throughput-oriented scenarios.

Downloads videos and playlists from YouTube

Abstraction layer over Khronos Vulkan API

The open-source project for "Mandheling: Mixed-Precision On-Device DNN Training with DSP Offloading"[MobiCom'2022]

MACE is a deep learning inference framework optimized for mobile heterogeneous computing platforms.

[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

SHMT for MICRO 2023

Lilu plugin for AMD Vega iGPUs

The AMD Vega iGPU support patch kext. No commercial use.

Reformer, the efficient Transformer, in Pytorch

🍰 Desktop utility to download images/videos/music/text from various websites, and more.

[MLSys 2021] IOS: Inter-Operator Scheduler for CNN Acceleration

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficie…

A list of awesome compiler projects and papers for tensor computation and deep learning.

The official gpt4free repository | various collection of powerful language models

直播源相关资源汇总 📺 💯 IPTV、M3U —— 勤洗手、戴口罩，祝愿所有人百毒不侵

A curated list of practical guide resources of LLMs (LLMs Tree, Examples, Papers)

Stable Diffusion and Flux in pure C/C++

ChatGLM-6B: An Open Bilingual Dialogue Language Model | 开源双语对话语言模型

Bash script for installing V2Ray in operating systems such as Debian / CentOS / Fedora / openSUSE that support systemd