shelfry.app/p/mlc-llm

mlc-llm

MLC LLM 是一个面向大语言模型的编译与高性能部署引擎，核心是 MLCEngine——一套统一的推理运行时，可把模型编译后原生运行在 AMD、NVIDIA、Apple、Intel 的 GPU，以及 Linux、Windows、macOS、浏览器、iOS、Android 等平台上。它对外提供 OpenAI 兼容的 API，可通过 REST 服务、Python、JavaScript、Swift、Kotlin 等接口调用，且各端共用同一套引擎和编译器。项目基于 TVM 的机器学习编译技术，支持 Vulkan、CUDA、Metal、ROCm、WebGPU、OpenCL 等后端，适合需要在本地或边缘设备上运行大模型、又不想为每个平台单独适配的开发者。

WebAI 工具 Apache-2.0

访问官网 GitHub

$pip install mlc-llm-nightly

Stars

23.3K

Forks

2.3K

语言

Python

MLC LLM

Universal LLM Deployment Engine with ML Compilation

Get Started | Documentation | Blog

About

MLC LLM is a machine learning compiler and high-performance deployment engine for large language models. The mission of this project is to enable everyone to develop, optimize, and deploy AI models natively on everyone's platforms.

AMD GPU

NVIDIA GPU

Apple GPU

Intel GPU

Linux / Win

✅ Vulkan, ROCm

✅ Vulkan, CUDA

N/A

✅ Vulkan

macOS

✅ Metal (dGPU)

N/A

✅ Metal

✅ Metal (iGPU)

Web Browser

✅ WebGPU and WASM

iOS / iPadOS

✅ Metal on Apple A-series GPU

Android

✅ OpenCL on Adreno GPU

✅ OpenCL on Mali GPU

MLC LLM compiles and runs code on MLCEngine -- a unified high-performance LLM inference engine across the above platforms. MLCEngine provides OpenAI-compatible API available through REST server, python, javascript, iOS, Android, all backed by the same engine and compiler that we keep improving with the community.

@inproceedings{tensorir,
    author = {Feng, Siyuan and Hou, Bohan and Jin, Hongyi and Lin, Wuwei and Shao, Junru and Lai, Ruihang and Ye, Zihao and Zheng, Lianmin and Yu, Cody Hao and Yu, Yong and Chen, Tianqi},
    title = {TensorIR: An Abstraction for Automatic Tensorized Program Optimization},
    year = {2023},
    isbn = {9781450399166},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3575693.3576933},
    doi = {10.1145/3575693.3576933},
    booktitle = {Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2},
    pages = {804–817},
    numpages = {14},
    keywords = {Tensor Computation, Machine Learning Compiler, Deep Neural Network},
    location = {Vancouver, BC, Canada},
    series = {ASPLOS 2023}
}

@inproceedings{metaschedule,
    author = {Shao, Junru and Zhou, Xiyou and Feng, Siyuan and Hou, Bohan and Lai, Ruihang and Jin, Hongyi and Lin, Wuwei and Masuda, Masahiro and Yu, Cody Hao and Chen, Tianqi},
    booktitle = {Advances in Neural Information Processing Systems},
    editor = {S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho and A. Oh},
    pages = {35783--35796},
    publisher = {Curran Associates, Inc.},
    title = {Tensor Program Optimization with Probabilistic Programs},
    url = {https://proceedings.neurips.cc/paper_files/paper/2022/file/e894eafae43e68b4c8dfdacf742bcbf3-Paper-Conference.pdf},
    volume = {35},
    year = {2022}
}

@inproceedings{tvm,
    author = {Tianqi Chen and Thierry Moreau and Ziheng Jiang and Lianmin Zheng and Eddie Yan and Haichen Shen and Meghan Cowan and Leyuan Wang and Yuwei Hu and Luis Ceze and Carlos Guestrin and Arvind Krishnamurthy},
    title = {{TVM}: An Automated {End-to-End} Optimizing Compiler for Deep Learning},
    booktitle = {13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)},
    year = {2018},
    isbn = {978-1-939133-08-3},
    address = {Carlsbad, CA},
    pages = {578--594},
    url = {https://www.usenix.org/conference/osdi18/presentation/chen},
    publisher = {USENIX Association},
    month = oct,
}

mlc-llm

MLC LLM

About

Get Started

Citation

核心功能