Best Open Source AI Inference Projects

Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

f.k.a. Awesome ChatGPT Prompts. Share, discover, and collect prompts from the community. Free and open source — self-host for your organization with complete privacy.

LLM inference in C/C++

🪨 why use many token when few token do trick. Viral skill + proxy for coding agents that cuts 65% of tokens by talking like a caveman.

A high-throughput and memory-efficient inference and serving engine for LLMs

Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks.

AI agent toolkit: unified LLM API, agent loop, TUI, coding agent CLI

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

⭐AI-driven public opinion & trend monitor with multi-platform aggregation, RSS, and smart alerts.🎯 告别信息过载，你的 AI 舆情监控助手与热点筛选工具！聚合多平台热点 + RSS 订阅，支持关键词精准筛选。AI 智能筛选新闻 + AI 翻译 + AI 分析简报直推手机，也支持接入 MCP 架构，赋能 AI 自然语言对话分析、情感洞察与趋势预测等。支持 Docker ，数据本地/云端自持。集成微信/飞书/钉钉/Telegram/邮件/ntfy/bark/slack 等渠道智能推送。

Context7 Platform -- Up-to-date code documentation for LLMs and AI code editors

The best-benchmarked open-source AI memory system. And it's free.

LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.

Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search

🦍 The API and AI Gateway

A unified AI model hub for aggregation & distribution. It supports cross-converting various LLMs into OpenAI-compatible, Claude-compatible, or Gemini-compatible formats. A centralized gateway for personal and enterprise model management. 🍥

Jan is an open source alternative to ChatGPT that runs 100% offline on your computer.

[EMNLP2025] "LightRAG: Simple and Fast Retrieval-Augmented Generation"

A modular graph-based Retrieval-Augmented Generation (RAG) system

《开源大模型食用指南》针对中国宝宝量身打造的基于Linux环境快速微调（全参数/Lora）、部署国内外开源大模型（LLM）/多模态大模型（MLLM）教程

SGLang is a high-performance serving framework for large language models and multimodal models.

Hundreds of models & providers. One command to find what runs on your hardware.

Community plugin to control Blender 3D with any LLM of your choice. Not affiliated with the official Blender Foundation.

Description not yet verified.

Python scraper based on AI

A one stop repository for generative AI research updates, interview resources, notebooks and much more!

Find secrets with Gitleaks 🔑

A list of free LLM inference resources accessible via API.

Official code repo for the O'Reilly Book - "Hands-On Large Language Models"

Local-first code intelligence graph for MCP and CLI. Builds a persistent map of your codebase so AI coding tools read only what matters, with benchmarked context reductions on reviews and large-repo workflows.

Fully automatic censorship removal for language models

Community plugin to control Blender 3D with any LLM of your choice

Faster Whisper transcription with CTranslate2

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

Distribute and run LLMs with a single file.

AirLLM 70B inference with single 4GB GPU

整理开源的中文大语言模型，以规模较小、可私有化部署、训练成本较低的模型为主，包括底座模型，垂直领域微调及应用，数据集与教程等。

Universal LLM Deployment Engine with ML Compilation

Automate browser based workflows with AI

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

Run agents like Hermes, LangChain Deep Agents, and OpenClaw more securely inside NVIDIA OpenShell with managed inference

The official repo of Qwen (通义千问) chat & pretrained large language model proposed by Alibaba Cloud.

🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

Local, open-source AI app builder for power users ✨ v0 / Lovable / Replit / Bolt alternative 🌟 Star if you like it!

[🔥updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. 📃 online docs: https://ufund-me.github.io/Qbot ✨ :news: qbot-mini: https://github.com/Charmve/iQuant

Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model family and using them on various provider services

Machine Learning Engineering Open Book

High-performance In-browser LLM Inference Engine

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

LLM Wiki is a cross-platform desktop application that turns your documents into an organized, interlinked knowledge base — automatically. Instead of traditional RAG (retrieve-and-answer from scratch every time), the LLM incrementally builds and maintains a persistent wiki from your sources。