{"version":1,"type":"story","url":"https://digestai.news/story/open-source-proxy-wakes-llm-servers-on-demand","json":"https://digestai.news/story/open-source-proxy-wakes-llm-servers-on-demand.json","markdown":"https://digestai.news/story/open-source-proxy-wakes-llm-servers-on-demand.md","slug":"open-source-proxy-wakes-llm-servers-on-demand","headline":"Open-source proxy wakes LLM servers on demand","summary":"A developer released **Sleepy LLM**, an open-source reverse proxy that starts and stops LLM inference servers based on incoming requests. The tool sends a wake command on first request, forwards traffic while active, and shuts down idle servers after **30m** of inactivity—unless a user is logged in. It uses a cached model list to avoid unnecessary wake-ups and supports custom wake/shutdown commands via SSH or Wake-on-LAN.\n\nThe project was built using **Deepseek-V4-Flash-0731** locally and requires **JDK 21+**. Configuration is handled via environment variables or a config file. The proxy streams requests and responses unchanged, preserving token streaming and SSE functionality. Down detection triggers after **3 consecutive failed health probes** or a failed request.","keyPoints":["Reverse proxy starts LLM servers on first request via user-defined commands like Wake-on-LAN or SSH","Auto-shuts down idle servers after **30m** of no traffic, unless a user is logged in","Uses cached model list to avoid unnecessary wake-ups and supports custom wake/shutdown scripts"],"whyItMatters":"Reduces cloud costs for small-scale LLM deployments by dynamically scaling servers to active requests.","category":{"slug":"agents","name":"Agents & Tools","url":"https://digestai.news/category/agents"},"entities":{"companies":[],"models":["Deepseek-V4-Flash-0731"],"people":[]},"firstPublishedAt":"2026-10-16T00:13:25Z","updatedAt":"2026-10-16T00:13:25Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"github.com","title":"Reverse proxy for LLMs which can wake and/or shut-down your inference server in response to incoming requests","url":"https://github.com/ac101m/sleepy-llm","publishedAt":"2026-10-16T00:13:25Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[{"site":"Reddit","url":"https://www.reddit.com/r/LocalLLaMA/comments/1x71uju/reverse_proxy_for_llms_which_can_wake_andor/","points":null}],"thread":null,"cite":{"text":"Digest AI, \"Open-source proxy wakes LLM servers on demand\", 16 October 2026, https://digestai.news/story/open-source-proxy-wakes-llm-servers-on-demand","publisher":"Digest AI","title":"Open-source proxy wakes LLM servers on demand","datePublished":"2026-10-16T00:13:25Z","url":"https://digestai.news/story/open-source-proxy-wakes-llm-servers-on-demand"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}