Article September 8, 2026 Home Assistant Hermes Agent · Telegram · LLM

Home Assistant, Overhauled from a Telegram Chat: an LLM and Hermes Agent

No laptop, no configuration files, no dashboard designer. I sent instructions from my phone over Telegram; an always-on agent carried them out, and the LLM found the broken automations, repaired them and rebuilt the whole dashboard. A weekend of tinkering became a single conversation.

LLM Hermes Agent · Nous Research Telegram gateway Home Assistant on RPi CM5 Hermes on ZimaBoard 2 · 24/7 320B / 18B activated MoE 1M-token context Native multimodal Reasoning always on
The redesigned Home Assistant dashboard, rebuilt end-to-end by the LLM through Hermes Agent and a Telegram chat
Cover. The redesigned Home Assistant dashboard: every view rebuilt end-to-end by the LLM, driven by a Telegram conversation from my phone.

My Home Assistant server, the same Raspberry Pi CM5 I wrote about earlier, had collected the usual clutter: automations that died quietly after an update, sensors with broken templates, and a dashboard that had grown into an unusable wall of cards. Rather than lose a weekend to configuration files, I pointed Hermes Agent, Nous Research’s self-improving agent, at the problem from Telegram, with an LLM doing the thinking.

In one long conversation the agent audited the setup, fixed what was broken and redesigned the whole dashboard. The biggest surprise was not the speed. It was how accurate and precise LLMs have become: frontier-class behaviour is no longer the preserve of the largest flagship models.

A note on safety

The agent had shell access to my home’s configuration, which is exactly as dangerous as it sounds. So it worked under guardrails: the configuration lives in a git repository, a backup was taken before every edit, every change passed Home Assistant’s own checker before reloading, and anything destructive needed my explicit approval in the chat. That discipline is the difference between an agent and a hazard.

Three pieces, no glue

The whole system is strikingly simple: three off-the-shelf parts and almost no custom code. Telegram is the interface, the app already on my phone, carrying messages, screenshots, voice notes and approvals in a single thread. Hermes Agent is the worker, a persistent agent with a real terminal, memory and a built-in Telegram connection, running around the clock on a ZimaBoard 2. And The LLM is the brain. The target was the home server that already runs my sensing, automation, analytics and media.

A request travels a short path. I type a message; Hermes picks it up; the model decides what to do; Hermes carries it out on the ZimaBoard, reading files, searching logs, editing configuration and reloading the system; and the result, often a screenshot, comes back to the same chat. Connecting the model took a single command.

An agent that lives where you do

I had tried “chat with your server” setups before, and they always failed in the same three ways: the assistant was tied to a laptop session, forgot everything overnight, and fumbled its tools as if meeting them for the first time. Hermes is the first framework I have used that solves all three.

It lives in the messaging apps people already use, Telegram among them, and it transcribes voice notes, so I can talk to my house from my phone. It works with any model provider, which is how I could adopt the new LLM the day it was released and switch to a heavier model for a hard debugging session without touching anything else. It learns: after the first evening it had written itself a Home Assistant skill that remembered my naming conventions, my file layout and how I like changes staged, and the second evening was noticeably faster. It runs anywhere, from a small cloud server to the ZimaBoard on my shelf, always on and ready the moment a message arrives. It can split work between helpers, one auditing automations while another drafts dashboard layouts. And it keeps a schedule; it now sends me a weekly report, in plain English, of anything that has drifted.

The underrated advantage

The agent streams its terminal activity into the chat, so you can watch it search, read, edit and validate, live, on your phone. That transforms the question of trust: nothing happens out of sight, and you can interrupt and redirect it mid-task rather than discovering problems afterwards.

The model

The most interesting character in this story is the model. The LLM is a natively multimodal model and, according to its developer, the first open-source frontier model to combine two forms of efficient attention, which sharply reduces the memory and computation it needs without losing its grip on long contexts.

18B
parameters active at a time, out of 320B: a model that thinks big and runs light
1M
tokens of context: the whole smart home fits in one conversation
Always
reasoning before it acts, never skipped

Four of its qualities mattered most here. It can see, so it not only writes the dashboard but looks at the result and critiques its own layout. It can hold everything at once: configuration, automations, scripts, dashboards and recent logs, all in view for the whole session. It always reasons before it acts, which is the only kind of model worth trusting with work that can fail silently. And its tool calls are precise, well formed and correctly typed first time, which sounds unremarkable until you have supervised a model that fumbles them.

One evening, one thread

The overhaul took a single evening. It began with an audit: the agent pulled the configuration, the device registry and the logs, and came back with a diagnosis. Seventeen dashboard cards pointed at sensors renamed months earlier, two automations had been dead since an update, and several template sensors were throwing errors.

Then came the fixes, each one a clean, reviewable change on its own branch rather than a blind edit, checked by Home Assistant before being reloaded in stages. Dead references were repaired or removed, and the broken automations were traced to their root cause.

Next, the redesign. The brief was an overview that shows what matters at a glance, dedicated views for climate, energy, media and security, consistent cards and nothing decorative, and the model turned it into a complete dashboard.

Finally, the part that felt new: the visual loop. After rebuilding each view, the agent took a screenshot, sent it into the chat and, being able to see, inspected its own work, catching uneven spacing and a mis-sized gauge before I ever noticed them.

The result

The dashboard finally looks designed rather than accumulated: five purpose-built views instead of one endless scroll, every card working, every graph on the right sensor, and the whole thing under version control, so future changes are tidy differences rather than archaeology.

Rebuilt Home Assistant dashboard view: overview rebuilt by the LLM via Hermes Agent and Telegram
The rebuilt dashboard: designed by prompt, checked by screenshot.
A second rebuilt Home Assistant dashboard view produced by the LLM session
A second view from the same session, refined by the model after looking at it.

What stays with me is how the matters of taste, what belongs on the overview and how much is too much for one card, were settled in the chat in seconds. A layout complaint made in passing was fixed and re-photographed within the minute. That loop, from complaint to finished result in under a minute, is the real product.

The surprise: how accurate LLMs have become

I had tried this before with a smaller, lighter model, and it went badly: plausible-sounding sensor names that never existed, invented card syntax that failed validation, and automations that read well but did the wrong thing. I spent longer checking and fixing than I would have spent doing the work myself. That has always been the bargain with lightweight models: fast, imprecise and in constant need of supervision.

The LLM broke the bargain. Before writing anything, it searched the registry and used only names that existed; across dozens of files and hundreds of references, I did not see a single invented one. Its configuration passed validation first time, every time, and it is that last ten percent of subtle errors that used to eat the hours. When a reload did fail, it read the error, found the line and fixed it on the first attempt, and its visual check caught problems no validator ever would. And it stayed consistent for hours, holding to naming conventions and decisions I had stated at the start, finishing the session as disciplined as it began.

The headline

Frontier performance used to belong only to the biggest flagship models. This LLM shows that frontier behaviour, the accuracy, the precision and the self-correction, now comes in the compact model of the series. For always-on, personal-scale work like running a home, that changes what is practical: frontier-grade judgement on tap, around the clock.

Where a human still matters

I still reviewed every change, and approvals stayed switched on. For anything touching door locks, alarms or routines that run when nobody is home, that will not change; not because the model failed, but because that kind of failure is unacceptable. Taste is still mine, too: the model executes a design brief very well, but it only knows I prefer sparse dashboards because I told it. A session without a single invented name is encouraging, but it is a data point rather than a guarantee. And trust was earned in stages, from read-only audits, to fixes on separate branches, to the full redesign, which is exactly how it should be done.

The distance between “I wish my dashboard did this” and “it does this” is now one Telegram message. The weekend of configuration is optional.

The bigger picture

Home servers are quietly becoming the natural home for frontier AI: an always-on agent, a capable model on tap, and a chat app already in your pocket. Hermes supplies the persistence, the memory and the hands; the LLM supplies the judgement; Telegram supplies an interface everyone already knows.

The smart home was supposed to be about automation. What this adds is maintenance, the unglamorous half that eats weekends: the automation that dies after an update, the broken template, the dashboard nobody tidies because tidying it is a chore. That work can now be delegated, in plain language, from anywhere. The interesting question is no longer whether an agent can look after a home, but what else, around the house and the office, gets the same treatment next.


If you enjoyed this, you may also like the story of the RPi CM5 that runs all of this, LLM-orchestrated CFD, and where LLMs still need a human.