Building a Local AI Agent (Part 2): Six UX and UI Design Challenges

Source: Dev.to

Part 2 – Design Decisions

In Part 1 I covered the six technical problems behind Reiseki’s ReAct loop — iteration caps, context management, persistent memory, and security. If you haven’t read it, start there.

This part is about the design decisions. Even though I had an initial concept in mind, the current layout actually developed over time while testing. I asked the following questions:

• Who is this for?
• What should be adjustable by the user and what be automated?
• For an agent that has access to your files and remembers your conversations, how can I make this as transparent as possible?

Reiseki is open source:

Two Goals That Pulled in Different Directions

Reiseki was designed with two principles in mind that don’t always sit comfortably together.

1. Usability without technical knowledge – No terminal, no Python, no config files. If you can install an app, you can run Reiseki.
2. Full transparency – The user should always know what the agent knows about them. Memories, conversation history, file access — all visible and deletable at any time.

The tension is that transparency often means complexity. The challenge is surfacing the right information without overwhelming the user.

Here’s how I approached it.

The Six UX/UI Challenges and How I Solved Them

1. No setup required — just download and start

Challenge – If you can install an app, you should be able to run Reiseki. Everything else (model configuration, database setup, …) should happen invisibly in the background.

Solution – Reiseki ships as a Windows installer. You pick a workspace folder during installation, and that’s it. No Python environment to configure, no config files to edit, no command line.
Given you have Ollama installed and a model downloaded, the agent opens with a setup screen asking for a name and a short description of your goal — then it’s ready.

The workspace‑setup idea and the drop‑down model selector were added in versions 0.1.3 and 0.1.4 while testing.

2. Live tool trace

Challenge – Show what the model is doing in real time to create transparency.

Solution – Every time the agent calls a tool, the UI displays the call instantly: which tool, the arguments, and the returned result.

3. Memory‑management panel

Challenge – Make memories visible in the UI for full transparency.

Solution – Early versions tried to automate memory saving, but smaller models often ignore the automation even when it’s in the system prompt.
I added a “Save‑Memory” button that triggers the save_memory tool call, letting the user decide when a conversation is worth preserving.

A memory panel at the top lists every stored memory with a one‑click delete button. No need to ask the agent to forget something and hope it complies — you delete it directly. The agent’s knowledge about you is a list you can edit.

4. Conversation‑history modal

Challenge – Provide transparent, editable chat logs.

Solution – The chat log persists across sessions in SQLite, giving the agent a record of past conversations. The history modal makes that visible and includes a delete button. The entire log can be cleared at any time.

5. Smartphone access via QR toggle

Challenge – Enable phone use without any extra app and with minimal friction.

Solution – The agent runs as a local web server, reachable from other devices on the same network only if you explicitly enable it.
A QR modal contains a toggle: when on, a QR code appears for scanning with your phone; when off, the server blocks all non‑localhost requests at the middleware level.

Key decisions: give the user control over network exposure and use a QR code for easy access.

6. Model switcher

Ollama supports multiple models, and switching between them is a common workflow (e.g., a fast model for quick questions, a more capable one for complex tasks). Requiring a server restart to change the model creates unnecessary friction.

Challenge – Make model switching as easy as possible.

Solution – The model switcher lets you change the model directly in the UI. The change takes effect on the next message. No need to restart Ollama or Reiseki, provided the models are already downloaded in Ollama.

Additional Thoughts

The tension between “simple for everyday users,” “using as little RAM as possible,” and “keeping it as functional as possible” never fully resolves — it just gets managed. A few things I’d approach differently next time:

• Data storage – The SQLite database is simple and reliable, but I like Claude’s Markdown‑file approach for flexibility. (I used Claude.md and other .md files during the Claude Code sessions and it worked great!)
• First‑time guidance – An automated tutorial session or an introductory video could help users discover the various functionalities more intuitively.
• Model trade‑offs – During development I switched from Qwen 2.5‑coder:7B to Gemma 4:e2b, which dramatically improved the agent’s tool calls. I hope we’ll see even more capable small models soon. Larger models would also eliminate many of the coding challenges described in Part 1, but we can’t have it all right now.

This project was built entirely with Claude.

ode. The technical decisions and design goals are mine; Claude handled the implementation.

What UX/UI problems have you run into when building tools for non-technical users? Very curious about your experience.