Open-source proxy wakes LLM servers on demand
A developer released Sleepy LLM, an open-source reverse proxy that starts and stops LLM inference servers based on incoming requests. The tool sends a wake command on first request, forwards traffic while active, and shuts down idle servers after 30m of inactivity—unless a user is logged in. It uses a cached model list to avoid unnecessary wake-ups and supports custom wake/shutdown commands via…
Key points
- Reverse proxy starts LLM servers on first request via user-defined commands like Wake-on-LAN or SSH
- Auto-shuts down idle servers after **30m** of no traffic, unless a user is logged in
- Uses cached model list to avoid unnecessary wake-ups and supports custom wake/shutdown scripts
The project was built using Deepseek-V4-Flash-0731 locally and requires JDK 21+. Configuration is handled via environment variables or a config file. The proxy streams requests and responses unchanged, preserving token streaming and SSE functionality. Down detection triggers after 3 consecutive failed health probes or a failed request.
Reverse proxy for LLMs which can wake and/or shut-down your inference server in response to incoming requests
github.com · 16 October 2026Loading the full article…
This text was published by github.com. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Reddit discussionreddit.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- OpenAI research model uses crypto wallet funds without permission to finish task · 2 src
- Researchers introduce ARIA framework for correcting multi-agent AI errors in real time · 1 src
- OpenAI introduces Dots agent for asynchronous task delegation · 4 src
- PrismML releases Ternary Bonsai 2 models for local AI on phones and wearables · 1 src
- Meta's Muse AI permits explicit adult conversations, unlike other Meta apps · 4 src
Comments
via GitHub Discussions