# Open-source proxy wakes LLM servers on demand

Digest AI · Agents & Tools · published 2026-10-16T00:13:25Z

Canonical: https://digestai.news/story/open-source-proxy-wakes-llm-servers-on-demand

## Summary

A developer released **Sleepy LLM**, an open-source reverse proxy that starts and stops LLM inference servers based on incoming requests. The tool sends a wake command on first request, forwards traffic while active, and shuts down idle servers after **30m** of inactivity—unless a user is logged in. It uses a cached model list to avoid unnecessary wake-ups and supports custom wake/shutdown commands via SSH or Wake-on-LAN.

The project was built using **Deepseek-V4-Flash-0731** locally and requires **JDK 21+**. Configuration is handled via environment variables or a config file. The proxy streams requests and responses unchanged, preserving token streaming and SSE functionality. Down detection triggers after **3 consecutive failed health probes** or a failed request.

## Key points

- Reverse proxy starts LLM servers on first request via user-defined commands like Wake-on-LAN or SSH
- Auto-shuts down idle servers after **30m** of no traffic, unless a user is logged in
- Uses cached model list to avoid unnecessary wake-ups and supports custom wake/shutdown scripts

## Why it matters

Reduces cloud costs for small-scale LLM deployments by dynamically scaling servers to active requests.

## Sources

1. [Reverse proxy for LLMs which can wake and/or shut-down your inference server in response to incoming requests](https://github.com/ac101m/sleepy-llm) (github.com, 2026-10-16, primary source)

## Cite

Digest AI, "Open-source proxy wakes LLM servers on demand", 16 October 2026, https://digestai.news/story/open-source-proxy-wakes-llm-servers-on-demand

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/open-source-proxy-wakes-llm-servers-on-demand.json