Home Assistant, Overhauled from a Telegram Chat: GLM-5.3-Flash Meets Hermes Agent

The stack: three pieces, zero glue

The entire system is remarkably simple: three off-the-shelf components, connected with almost no custom code.

A command travels a short path: I type a message in Telegram → the Hermes gateway picks it up → GLM-5.3-Flash reasons about it and emits tool calls → Hermes executes them on the ZimaBoard 2 (reading configs, grepping logs, editing YAML, reloading Home Assistant) → the result, often a screenshot, comes back to the same chat. The model was wired in with a single hermes model selection pointing at Z.AI's OpenAI-compatible endpoint. That was the entire integration effort.

Why Hermes is the right chassis

I had tried "chat with your server" setups before, and they always failed in the same way: the assistant was tied to a laptop session, lost all context overnight, and executed tool calls as though encountering the interface for the first time. Hermes is the first agent framework to resolve all three of those defects, and its advantages compound in a home-automation context:

• It lives where you do. Telegram is not a bolt-on here; it is a first-class gateway, alongside Discord, Slack, WhatsApp and Signal, all from one process. Voice-memo transcription means I can issue spoken instructions to my home from my phone. Conversations continue across platforms without me having to repeat context.
• It is model-agnostic. Any provider, any endpoint: switch with one command, no code changes, no lock-in. This is precisely what allowed me to adopt GLM-5.3-Flash the moment Z.AI released it, and to switch back to a heavier model for difficult debugging without touching the setup.
• It has a closed learning loop. Hermes creates skills from experience, refines them during use, and searches its own past sessions. After the first evening it had written itself a Home Assistant skill that remembered my entity naming conventions, my configuration layout, and how I prefer reloads to be staged. The second evening was noticeably faster than the first.
• It runs anywhere, not just on your laptop. Seven terminal backends are supported, from local to Docker, SSH, Modal, Daytona and Vercel Sandbox. An automation agent needs to be always-on, and Hermes runs reliably on a small VPS or, in my case, on a ZimaBoard 2 that stays active around the clock and is ready the instant a message arrives.
• It delegates and parallelizes. While one subagent audited automations, another drafted dashboard YAML, and the results merged in the main thread. For a job that is mostly independent files, that parallelism is effectively free.
• Scheduled automations are built in. A cron scheduler with delivery to any platform. The agent now sends me a weekly config-drift report, in natural language, unattended.

GLM-5.3-Flash: frontier performance, efficient by design

The interesting character in this story is the model. GLM-5.3-Flash is the first natively multimodal model in Z.AI's GLM-5 series, and Z.AI's own framing is blunt: stronger intelligence than GLM-5.2. It is, per their documentation, the first open-source frontier model to combine sparse and linear attention, cutting attention computation by 3.01× and KV-cache size by 4.44× versus the flagship GLM-5.3, while keeping long-context quality.

For this job, four capabilities mattered more than the rest:

• Native multimodality. Vision is built into the coding loop: the model doesn't just write the dashboard, it looks at the rendered result and critiques its own layout. More on that below; it changed the workflow more than anything else.
• A 1M-token context. My configuration, automations, scripts, dashboard YAML and recent logs were all in context simultaneously. No retrieval gymnastics and no re-reading files between steps; the model held the entire smart home in its head for the whole session.
• Reasoning that is always on. GLM-5.3-Flash cannot skip its thinking pass: low, high or max effort, but never off. For work that silently breaks things (automations, templates), a model that reasons before every edit is the only kind worth trusting.
• First-class function calling and structured output. Tool calls came back well-formed, arguments typed correctly, JSON valid on the first pass. That description sounds unremarkable, until you have supervised a model that fumbles them.

The session: how the overhaul actually went

One evening, one thread. The sequence went like this:

1. The audit. The agent started by pulling the config, the entity registry and the logs, and came back with a diagnosis: 17 dashboard cards referencing entity IDs that no longer existed after a sensor rename months ago, two automations silently dead since an update, and a handful of template sensors throwing errors.

2. The fixes. Each fix landed as a clean diff on a git branch, never a blind edit. Every change was run through Home Assistant's config checker, validated, then reloaded in stages. Dead references were fixed or removed; the broken automations were repaired and traced back to their root cause.

3. The redesign. The target was an overview that shows what matters at a glance, dedicated views for climate, energy, media and security, consistent cards, nothing decorative. GLM-5.3-Flash turned those goals into a complete Lovelace dashboard: custom cards via HACS, sensible defaults, no lorem-ipsum placeholders.

4. The visual loop. This is the part that felt new: after each view was rebuilt, the agent screenshotted it, sent the image into the chat and, because GLM-5.3-Flash is natively multimodal, inspected its own work, catching spacing problems and a mis-sized gauge before I ever saw them.

What the update produced

The result is a dashboard that finally looks designed rather than accreted. Five purpose-built views instead of one endless scroll. Every card resolves. The graphs use the right sensors. And the whole thing is version-controlled, so future changes are diffs instead of archaeology.

What I keep coming back to is that the taste-level decisions, what belongs on the overview and how much is too much on one card, were negotiated in the chat, in seconds. A layout nitpick raised in passing was fixed and re-screenshotted before the minute was out. That iteration loop, from complaint to rendered result inside a minute, is the actual product here.

The part that surprised me: how accurate and precise GLM has become

To be candid about my prior assumptions, I had attempted this before, deploying a smaller, lightweight model against the same Home Assistant configuration. The result was poor: hallucinated entity IDs that sounded plausible but never existed, invented card syntax that failed validation, and automation logic that read correctly yet did the wrong thing. I spent more time reviewing and repairing than I would have spent writing the configuration myself. That has historically been the trade-off with the lightweight model class: fast, imprecise, and in constant need of supervision.

GLM-5.3-Flash broke that deal. Concretely:

• It grounded itself before writing. Before generating a single card, it grepped the entity registry and used only IDs that actually exist. Across the entire session, dozens of YAML files and hundreds of references, I did not encounter a single hallucinated entity.
• First-pass validity became the norm. Every YAML block it produced parsed and passed Home Assistant's config check on the first attempt. Earlier models would hand back something ninety percent correct and ten percent subtly invalid; that residual ten percent is what consumed the hours.
• Tool calls were precise. Well-formed function calls, correct argument types, JSON that validated on the first attempt, consistently. That precision is what allows an agent to chain twenty steps without human intervention.
• It caught its own mistakes. When a reload did return an error, it read the traceback, identified the line, and corrected it on the first attempt. And the vision pass found layout issues no config checker ever would.
• It stayed consistent across a long session. Naming conventions, design decisions, constraints I'd stated hours earlier, all held. With a 1M-token context there is no "forgetting" the plan halfway through; the model ended the session as disciplined as it began.

An honest assessment

• I still reviewed every diff. Approval for each change stayed on. For anything that touches door locks, alarms or scenes that run when nobody is home, that stays non-negotiable, not because the model failed, but because the failure mode is unacceptable.
• Taste is still mine. The model executes a design brief very well; it does not know that I like sparse dashboards unless I say so. The visual loop helps it respond to feedback fast, but the feedback has to come from a human eye.
• Hallucination is suppressed, not abolished. Zero occurrences in this session is a data point, not a theorem. The greps-before-writes behavior makes it rare; nothing makes it impossible.
• The stack earned trust gradually. It started with read-only audit tasks, graduated to fix-on-branch, and only then to full redesign. That onboarding curve is how it should be done, for the human's sake as much as the model's.

The distance between "I wish my dashboard did this" and "it does this" is now one Telegram message. The weekend of YAML is optional.

The bigger picture

Home servers are quietly becoming the most natural home for frontier AI: an always-on agent runtime, an always-available frontier model, and a chat app already in your pocket. Hermes supplies the persistence, the memory and the hands from its ZimaBoard 2; GLM-5.3-Flash supplies the judgment; Telegram supplies the interface everyone already knows.

The smart home was supposed to be about automation. What this stack adds is maintenance: the unglamorous half that eats the weekends: the dead automation after an update, the broken template, the dashboard nobody reorganizes because reorganizing it is a chore. That work is now delegable, in natural language, from anywhere. The interesting question is no longer whether an agent can run a home; it is what else around the house, and the office, gets this treatment next.

Technical summary

If you found this interesting, you may also enjoy the post about the RPi CM5 that runs all of this, LLM-orchestrated CFD, and where LLMs still need a human.