How I taught Claude to see inside Unreal Engine

Source: Dev.to

Familiar Beginnings

A few months ago I decided to finally start making my dream game in Unreal Engine 5, a 3D action RPG. With some purchased Udemy lessons I was off to the races and began the learning process. I’m a software developer by trade, so at least the coding shouldn’t be a problem. Everything went fine for a week or two, until I hit an issue where my game did not work like the one in the tutorial video despite following all of the steps exactly.

Solving even the smallest problems at this stage of learning is immensely difficult. You are completely new to a massive set of complex, interconnected systems and have no idea how they work. My gut tells me it’s not a fundamental issue—I followed the tutorial step‑by‑step, so it’s likely something small: a typo, a version mismatch, or a tweak that’s needed. I thought I would have my good friend Claude help me, assuming it would be something simple.

The problem is that inside Unreal Engine everything is serialized binary data stored as a .uasset file. When I fire up Claude Code to have him review my blueprints, he can’t read them. A quick search revealed that this isn’t a well‑solved problem—there isn’t a full solution for an LLM to understand the contents of a .uasset file.

I started with the UnrealClaude plugin (https://github.com/Natfii/UnrealClaude) as the base (shoutout to Natfii, who created much of the core capabilities). It can create blueprints but has limited ability to parse existing ones. I leveraged the existing code and extended it so it could read blueprints in Unreal. After that, Claude caught a node value that should have been 10 but was 1, and the roadblock was removed. I was back on track!

Teaching Claude to Read Blueprints

As the tutorial progressed in complexity, the “I did it exactly as the tutorial yet it doesn’t work” scenario kept popping up. Unreal Engine is very complex, and version updates sometimes break things. Documentation, Google, forums, and tutorials all offered mixed advice about systems I didn’t yet understand.

I turned back to Claude—not to fix the problem directly, but to help me navigate the complex systems so I could solve it myself. However, Claude still couldn’t read animation montages or animation graphs, which prevented him from helping with those issues.

I went back to the plugin code and taught Claude how to parse and dump these assets into a format he could work with. Once Claude could see the animation data, he could correlate information from code, blueprints, and montages, identify root causes, and the tutorial could finally be completed.

Extending the Plugin for Creation

When I started a new project for the actual game, the workflow felt slow and clunky. I knew what I needed to click and enter to set up a new feature, but doing it manually would take forever. It would be faster if I taught Claude to write things instead of just read them.

I added capabilities for Claude to generate blueprints, animation montages, and animation graphs. My implementation pace began to speed up: I could playtest or tune while Claude planned or implemented the next feature based on my very specific prompts.

Each new gap became a target for the plugin:

• Missing blendspaces → back to the plugin code.
• Modifying Control Rigs → back to the plugin code.
• Debugging with image captures → wrote a low‑latency frame grabber (a simple “take a screenshot” was too heavy).

Now I could capture per‑frame images for granular analysis, solve many bugs, improve combat, and keep moving forward.

Optimizing Token Usage

Claude could now see everything, but the token cost was growing quickly. Each request involved sending full JSON payloads and parsing entire blueprint graphs just to inspect a single node.

To reduce token consumption I built a thin interface layer—a script that abstracts raw curl commands and JSON parsing. This cut token usage by more than half.

Further improvements:

• Added node‑querying capabilities so Claude no longer needed to parse an entire blueprint each time.
• Created a domain system to compartmentalize Unreal‑Engine‑specific knowledge.

These changes halved the token usage again and made the interaction far more efficient.

Open‑Source Release

At this point I realized I had built something that could be valuable to others. I’m releasing it as the Unreal Engine LLM Toolkit:

https://github.com/ColtonWilley/ue-llm-toolkit

I will continue to develop it as I need new capabilities for my game, so the repository will receive ongoing updates.