Best Open Source AI Inference Projects

LLM inference in C/C++

🪨 why use many token when few token do trick — Claude Code skill that cuts 65% of tokens by talking like caveman

A high-throughput and memory-efficient inference and serving engine for LLMs

Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks.

AI agent toolkit: unified LLM API, agent loop, TUI, coding agent CLI

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

⭐AI-driven public opinion & trend monitor with multi-platform aggregation, RSS, and smart alerts.🎯 告别信息过载，你的 AI 舆情监控助手与热点筛选工具！聚合多平台热点 + RSS 订阅，支持关键词精准筛选。AI 智能筛选新闻 + AI 翻译 + AI 分析简报直推手机，也支持接入 MCP 架构，赋能 AI 自然语言对话分析、情感洞察与趋势预测等。支持 Docker ，数据本地/云端自持。集成微信/飞书/钉钉/Telegram/邮件/ntfy/bark/slack 等渠道智能推送。

Context7 Platform -- Up-to-date code documentation for LLMs and AI code editors

The best-benchmarked open-source AI memory system. And it's free.

LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.

Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search

🦍 The API and AI Gateway

A unified AI model hub for aggregation & distribution. It supports cross-converting various LLMs into OpenAI-compatible, Claude-compatible, or Gemini-compatible formats. A centralized gateway for personal and enterprise model management. 🍥

Jan is an open source alternative to ChatGPT that runs 100% offline on your computer.

[EMNLP2025] "LightRAG: Simple and Fast Retrieval-Augmented Generation"

A modular graph-based Retrieval-Augmented Generation (RAG) system

《开源大模型食用指南》针对中国宝宝量身打造的基于Linux环境快速微调（全参数/Lora）、部署国内外开源大模型（LLM）/多模态大模型（MLLM）教程

SGLang is a high-performance serving framework for large language models and multimodal models.

Hundreds of models & providers. One command to find what runs on your hardware.

Description not yet verified.

Python scraper based on AI

A one stop repository for generative AI research updates, interview resources, notebooks and much more!

Find secrets with Gitleaks 🔑

A list of free LLM inference resources accessible via API.

Official code repo for the O'Reilly Book - "Hands-On Large Language Models"

Fully automatic censorship removal for language models

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

Distribute and run LLMs with a single file.

整理开源的中文大语言模型，以规模较小、可私有化部署、训练成本较低的模型为主，包括底座模型，垂直领域微调及应用，数据集与教程等。

Universal LLM Deployment Engine with ML Compilation

Automate browser based workflows with AI

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools

The official repo of Qwen (通义千问) chat & pretrained large language model proposed by Alibaba Cloud.

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

Local, open-source AI app builder for power users ✨ v0 / Lovable / Replit / Bolt alternative 🌟 Star if you like it!

Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model family and using them on various provider services

High-performance In-browser LLM Inference Engine

Machine Learning Engineering Open Book

AirLLM 70B inference with single 4GB GPU

[🔥updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. 📃 online docs: https://ufund-me.github.io/Qbot ✨ :news: qbot-mini: https://github.com/Charmve/iQuant

Local knowledge graph for Claude Code. Builds a persistent map of your codebase so Claude reads only what matters — 6.8× fewer tokens on reviews and up to 49× on daily coding tasks.

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

A powerful tool for creating datasets for LLM fine-tuning 、RAG and Eval

Structured Outputs

A vector index built on TurboQuant, written in Rust with Python bindings

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

A curated list of modern Generative Artificial Intelligence projects and services