FlashInfer: Kernel Library for LLM Serving
-
Updated
Jul 9, 2026 - Python
#
distributed-inference
Here are 70 public repositories matching this topic...
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
cuda inference openai llama maas rocm ascend llm llm-serving vllm genai llm-inference qwen deepseek sglang distributed-inference high-performance-inference mindie
-
Updated
Jul 9, 2026 - Python
Achieve state of the art inference performance with modern accelerators on Kubernetes
-
Updated
Jul 9, 2026 - Shell
Collection of best practices, reference architectures, model training examples and utilities to train large models on AWS.
kubernetes aws gpu slurm efa distributed-training eks parallelcluster hyperpod generative-ai distributed-inference physical-ai
-
Updated
Jul 8, 2026 - Shell
Split FLUX.2 and LTX 2.3 across two GPUs (LAN or same-machine) — NVENC compresses activations live on the wire. Icarus (ComfyUI node) + Daedalus (back-half server).
flux pytorch image-generation multi-gpu nvenc video-generation pipeline-parallelism diffusion-models flux2 comfyui distributed-inference comfyui-node ltx-video
-
Updated
May 17, 2026 - Python
一個基於 llama.cpp 的分佈式 LLM 推理程式，讓您能夠利用區域網路內的多台電腦協同進行大型語言模型的分佈式推理，使用 Electron 的製作跨平台桌面應用程式操作 UI。
-
Updated
May 24, 2026 - JavaScript
An educational distributed training and inference library for neural nets using local computing
-
Updated
Jun 10, 2026 - Python
[Official] prima.cpp: Fast 30-70B LLM inference on heterogeneous and everyday home devices
-
Updated
Jun 30, 2026 - C++
Mixed-vendor GPU inference cluster manager with speculative decoding
python machine-learning deep-learning metal cuda p2p homelab rocm gpu-cluster llama-cpp gpu-inference local-llm llm-inference ollama speculative-decoding distributed-inference gpu-pooling self-hosted-ai openai-compatible mixed-gpu
-
Updated
Jul 2, 2026 - Python
Code for paper "JMDC: A Joint Model and Data Compression System for Deep Neural Networks Collaborative Computing in Edge-Cloud Networks"
compression deep-learning distributed-computing inference dnn neural-networks pruning quantization dnn-optimization edge-computing mobilenet edge-cloud ai-models dnn-partition distributed-inference
-
Updated
Jun 17, 2026 - Python
An imperative command-line-interface for AI workload orchestration
kubernetes ray multi-cloud gpu-cluster mlops cloud-gpu mixture-of-experts huggingface runpod anthropic vllm llm-inference ollama litellm sglang distributed-inference mcp-server claude-code gpu-provisioning disaggregated-inference
-
Updated
Jun 23, 2026 - Python
Analyze and generate unstructured data using LLMs, from quick experiments to billion token jobs.
csv s3 pandas data-engineering parquet data-processing data-pipelines observability synthetic-data unstructured-data mlops batch-inference polars llm-inference evals distributed-inference
-
Updated
Jun 19, 2026 - Python
🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit-for-bit identical to single-machine. No torch.distributed, no datacenter.
metal peer-to-peer decentralized cuda transformers inference pytorch mps gemma model-parallelism heterogeneous-computing edge-ai pipeline-parallelism on-device-ai apple-silicon large-language-models llm llm-inference qwen distributed-inference
-
Updated
Jul 6, 2026 - Python
Source code of the paper "Private Collaborative Edge Inference via Over-the-Air Computation".
machine-learning ensemble-learning wireless-communications differential-privacy multi-view-learning edge-inference over-the-air-computation collaborative-inference distributed-inference
-
Updated
Jan 14, 2025 - Python
Super Ollama Load Balancer - Performance-aware routing for distributed Ollama deployments with Ray, Dask, and adaptive metrics
python caching machine-learning streaming performance ai multi-node http2 load-balancer rpc asyncio ray model-serving gpu-monitoring inference-optimization llm llama-cpp ollama distributed-inference hybrid-routing
-
Updated
Apr 10, 2026 - Python
Decentralized peer-to-peer LLM inference network. Single Rust binary, BitTorrent-inspired incentives, OpenAI-compatible API.
rust mcp decentralized p2p inference transformer candle libp2p huggingface kv-cache openai-api llm gguf speculative-decoding anthropic-api distributed-inference
-
Updated
May 23, 2026 - Rust
Accelerate reproducible inference experiments for large language models with LLM-D! This lab automates the setup of a complete evaluation environment on OpenShift/OKD: GPU worker pools, core operators, observability, traffic control, and ready-to-run example workloads.
-
Updated
Jul 1, 2026 - Jupyter Notebook
Distributed LLM inference across heterogeneous workers (NVIDIA / AMD / Apple Silicon / CPU). Splits one model by layer ranges; patched llama.cpp + gRPC chain protocol.
-
Updated
Jul 5, 2026 - Python
Self-hosted AI gateway for distributed local LLM GPUs. OpenAI-compatible API with NAT traversal, WebSocket/QUIC tunnels, Ollama/vLLM/SGLang workers.
go golang ai gpu api-gateway websocket nat-traversal self-hosted openai quic reverse-tunnel llm vllm local-llm ollama ai-gateway sglang distributed-inference
-
Updated
May 12, 2026 - Go
Improve this page
Add a description, image, and links to the distributed-inference topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the distributed-inference topic, visit your repo's landing page and select "manage topics."