Landing AI in Game Production · Form 1 «Within Easy Reach»: Making AI Understand a Project’s Existing Assets

Series intro: “The Kung-Fu Manual for Landing AI in Game Production” is a hands-on series for game technical artists (TAs) and tool developers — a record of the road I walked, all of it written in blood and tears.

1. Origin: Don’t Leave AI in the Dark

Our project is a large open-world game with an asset library on the order of 100k items, spanning a dozen-plus major categories — trees, rocks, shrubs, buildings, and more — and each asset also has mobile variants, LODs, Impostors, and sub-resources.

For AI to take part in game production, it first has to “know” what’s in the project. But the reality is —

1
2
3

"Place a low Jiangnan-style shrub"
  → What the AI sees is shrub_a1_01a, shrub_a1_02b
  → Which one is the low one? Which one belongs in this scene? No idea.

Humans aren’t much better than AI here. Artists find assets the same way — digging through folders, guessing at filenames, asking colleagues, retrieving from memory. A 2024 master’s thesis from Aalto University in Finland[^1] documents the same predicament: in HypeHype’s 5,000+ asset library, searching “forest” fails to find an asset whose description says “woods.”

So making assets understandable is the prerequisite for AI to participate in production. But for AI to understand human things, humans have to teach it (annotation). Annotation is time-consuming, and artists have no obligation to help you with it. So I also need an SOP that lets artists annotate assets while they search — trying to spin a virtuous cycle out of that.

2. The Distance Between Assets and Understanding

In fact, whether you’re a human or an AI, understanding an asset involves three different kinds of distance.

1. What is this asset called? (semantic)
2. What does it look like? (visual)
3. What is it for? Where and how is it used? (logical)

BTW: most good search tools can retrieve every character related to an article — even a string that’s just a stray note buried in a document. But you know how the brain works: you might only remember some “punctum” (Roland Barthes, Camera Lucida) tucked in a corner that once moved you, yet you just can’t recall the title of the thing. Every game-engine search box I know of is exactly like this — it can only find filenames, not some blueprint note inside a file, and it doesn’t support more complex search logic.
That’s a product-design problem, and it isn’t “distance” yet. A great search box should be like Everything (yes, I’m praising it by name) — able to search everything in the system! So our search capability should, in theory, be able to fuzzily retrieve an asset from a mere impression. That’s exactly what I’m out to build.

Layer 1: Semantic distance. Literal matching across filenames, paths, and tags — searching “rock” finds rock_granite_01, but searching “石头” or “岩石” (Chinese for “stone”/“rock”) finds nothing, even though the words mean almost the same thing, or differ only by language.

Layer 2: Visual distance. You want to describe a building of a particular form, but — alas, my literature teacher must have passed away too early — you can’t precisely name terms like “庑殿 / 闇栔 / 甍 / 甓” (obscure classical Chinese architectural terms), yet you know it’s that exact thing! The poverty of language turns out to be painfully real.

Layer 3: Logical distance. Searching “a worn-out object you could put in a corner” — this isn’t describing appearance, it’s describing purpose and state. Visual retrieval can’t help here. There’s reasoning baked in: how do you know you can put a broom in the corner of a home, rather than a pig-slaughtering knife? That requires understanding the need and reasoning from it.

Each of the three layers has its blind spots and its strengths, and each suits a different scenario.

3. The Framework: Inner Method

First, an “inner method” of tool design: users are willing to wait for a tool, but the longer they wait, the higher their expectations climb. Waiting a full minute only to get a terrible result is the most unacceptable outcome. So the whole framework is built around layering by response speed — fast results first, slow ones backfilled, never leaving the user hanging.

Following the three “distances” from Section 2, I split retrieval into three layers, each one reaching for a different distance:

• Layer 1 · Base retrieval (fastest, < 100ms): optimizations on top of UE’s native search, doing literal and rule-based matching over filenames / paths / tags. It’s the “safety net” — it works without any AI model.
• Layer 2 · Vector retrieval (instant, < 100ms): this layer actually runs two parallel paths — visual vectors reach for the «visual distance» (search-by-image), and text / description vectors reach for the «semantic distance» (synonyms, Chinese↔English, paraphrases all become searchable, closing the gap that Layer 1’s literal matching can’t cross). Each path recalls independently, then the results are fused and re-ranked.
• Layer 3 · Logical retrieval (re-rank, async in the background): CLI and MCP bring an external LLM into the loop to reach for the hardest «logical distance» — understanding purpose, state, and intent, the kind of query that needs reasoning.

One line to untangle the naming: the three-layer framework maps to the three distances, but the «semantic distance» is closed by a relay between Layer 1 (literal matching) and Layer 2 (text vectors) — which is why, despite being called a “three-layer framework,” it technically runs four paths: «SQL + Visual + Description + LLM». The NeuroBrowser in Section 5 is exactly this inner method put into practice.

4. Sword Summit at Mount Hua · Model Selection

The framework is designed; now for the most critical question: which models?

This isn’t a question you can answer by intuition or by paper leaderboards. There’s a domain gap between game render images and natural images — our thumbnails are rendered in-engine (sky sphere + multi-directional lights + stylized materials), and look nothing like the Flickr / LAION datasets CLIP[^2] was trained on. Leaderboard scores definitely don’t transfer directly.

So I ran two benchmarks: visual-encoder selection %% (maps to subsection 1) %% and description / text-embedding selection %% (maps to subsection 3) %%. The dataset is 100 real project thumbnails + 5 sets of artist-annotated Ground Truth (each set = 1 Chinese description + 3 correct images).

Also, description embedding has no initial data, and making artists annotate it from scratch isn’t quite appropriate either — so I directly used an LLM to generate the initial descriptions for all assets, then ran text-semantic embedding over those descriptions (for future asset ingestion this approach can be reused too, with artists then editing the LLM-generated descriptions). That’s why I also benchmarked a batch of different multimodal LLMs %% (maps to subsection 2) %%.

1. Visual Encoder: 14 Models, Two Rounds of Head-to-Head

Image → semantic vector

1
2
3
4
5
6
7
8
9

flowchart LR
    A["🖼️ Asset thumbnail"] --> B["Visual model"]
    B --> C["1024-dim vector"]
    C --> D["FAISS search"]

    style A fill:#4a90d9,color:#fff
    style B fill:#e8793a,color:#fff
    style C fill:#50b86c,color:#fff
    style D fill:#9b59b6,color:#fff

Round 1 started from the 4 models recommended by the Aalto thesis (a minimal test). The core finding: CLIP without Chinese support completely fails on Chinese queries — Recall@10 of only 13.3%, basically random. The version with an added xlm-roberta multilingual text tower led by a wide margin at 66.7% — a 5–10× gap.

Round 2 expanded along this lead to 10 new models, covering 4 tiers: 2025 SOTA (FG-CLIP 2 family), Chinese-native training (Chinese-CLIP family), multilingual up-and-comers (MEXMA / NLLB / AltCLIP), and the scaled-up family of Round 1’s strongest approach.

The result was unexpected: Qihoo 360’s FG-CLIP 2 large (0.9B params, natively bilingual) took a double crown at P@3=73.3% and R@10=93.3%. In 4 of the 5 queries it nailed all correct answers within the Top-2 — that’s an industrially usable level. A 26.6-point improvement over Round 1’s strongest.

Even more surprising, FG-CLIP 2 large actually beats the larger so400m (1.0B) by 20 points on P@3 — bigger isn’t always better; training steps and adaptation to thumbnail size may matter more.

On bang-for-buck, Chinese-CLIP ViT-L/14@336 ties FG-CLIP 2 large for first on R@10 (93.3%), with a smaller dimension (768 vs 1024), the simplest loading, and the most mature ecosystem — a solid backup.

So I ultimately chose FG-CLIP 2 large as the encoder for image semantics.

⚠️ First, a splash of cold water: all these percentages rest on 5 GT sets × 3 correct images = 15 data points, where a single misplaced result is a ±6.7pp swing. So phrases like “73.3% double crown” and “+26.6pp over Round 1” point in the right direction, but the size of the lead probably has luck baked in — it can’t be treated as settled until GT is expanded to 30+ sets. This warning applies equally to the text benchmark in subsection 3 below — don’t let any single high score go to your head.

2. Benchmarking LLM Generation

Thumbnail → Chinese description

As described above, I need a first wave of initialization text for text semantics, so I planned to use an LLM to generate the initial descriptions for all assets, with artists then editing on top of that text to better fit the “production content” scenario. But understanding an asset and generating a description is no simple matter — different LLMs vary in generation time, quality, and stability. So I ran a test:

1
2
3
4
5
6
7
8
9

flowchart LR
    A["🖼️ Asset thumbnail"] --> B["Multimodal LLM"]
    B --> C["Chinese description text"]
    C --> D["Text embedding"]

    style A fill:#4a90d9,color:#fff
    style B fill:#e8793a,color:#fff
    style C fill:#50b86c,color:#fff
    style D fill:#9b59b6,color:#fff

I pulled 13 mainstream multimodal LLMs (late May 2026) to generate descriptions for the same batch of 100 asset thumbnails (the thumbnail itself takes some care — ideally high-resolution, and if you can, even multi-angle thumbnails; I used a single 2048×2048 thumbnail, and it’s best to avoid over- or under-exposure), benchmarking across four dimensions: cost, speed, quality, stability:

• The benchmark times are for reference only and depend on the API provider too. This is just my environment.
• Also, LLM generation cost depends on the prompt provided. Reproduction costs will differ.
• In testing I used a fixed prompt and ran each image in its own independent context.

A few key findings:

1. The cost gap is staggering — cheapest to priciest is a 20× spread. gemini-3.1-flash-lite ¥0.026/img vs gpt-5.4 ¥0.567/img. For a full 100k-image run, the former is ¥2,607, the latter ¥56,700 — a difference of a dozen-plus RTX 4060 Ti’s worth of money. More surprising, claude-opus is actually cheaper than sonnet (¥0.221 vs ¥0.258) — sonnet got overtaken by opus.

2. Description length varies hugely, but longer isn’t better. Claude-haiku averages the longest at 298 chars with the smallest variance (min 186 ~ max 435), and the best structure (the four-part “shape & structure / material & texture / color & tone / fine details”); qwen3.5-plus is the shortest at just 154 chars. But the later embedding retrieval validated a counterintuitive conclusion — the most human-readable description ≠ the most embedding-friendly description (see next section).

3. Qwen — a dagger hidden in a smile: tag leakage directly pollutes the embedding. qwen3.5-plus leaks <skill> tags (one image output 38 chars of garbage skill\nname visual verdict\n/name\n/skill), and qwen3.6-plus leaks <think chain-of-thought (up to 1,347 chars of reasoning written straight into the output), with stability of only ⭐⭐. These residual markers go straight into the description text and become noise in downstream embedding retrieval. A cleaning step is mandatory before full production (regex tag-stripping + truncation to 800 chars).

4. Kimi-k2.6 is the top domestic pick — 0 anomalies, fastest. 10.2s pure query time, the fastest of all, 0 anomalous outputs, full-star stability. If you’re wary of trust_remote_code or need a fully domestic solution, kimi-k2.6 is the most stable choice. Its discounted version kimi-k2.6-yd occasionally has 1 image with a 2,448-char suspected think leak, usable after adding a max-length filter.

5. The free model is usable but weak. Mimo-v2.5-free is zero-cost but occasionally times out (120s) and has token-loop anomalies (30K–44K stuck in a loop), with some misidentification on water-body types; stability only ⭐⭐⭐. Good for PoC validation, not for full production.

6. Description quality correlates positively with render quality. The higher the render quality, the better the description.

3. Text Embedding

Visual encoding solved the “search-image-find-image” problem, but that search capability is still too shallow. To make AI deeply understand this game’s assets, the more important thing is to break past surface-level image semantics — to grasp who this asset belongs to, and what it means.

And this content is what truly needs artists to teach the AI bit by bit, and it’s also the most “grounded.” So how do we teach it?

My answer is — have the AI first write a description for each asset, then run semantic retrieval over the descriptions (later, artists can edit that description and re-embed it into the vector store). So I invited the previous round’s 4 players to do another cross-benchmark: 7 text-embedding models × 4 description sources = 28 combos, plus an FG-CLIP2 visual baseline + RRF fusion, for 53 experiments total.

Seven embedding candidates (all runnable in bf16 on my 8GB 4060Ti; Qwen3-Embedding-4B/8B excluded because their bf16 weights are ≥8GB):

The four description sources were picked from the previous section’s 13 models: claude-haiku-4-5 (highest human-readability), kimi-k2.6 (production pick), glm-5v-turbo (domestic & stable), gemini-3.1-flash-lite (the value king).

⚠️ gte-Qwen2-1.5B results are unusable, excluded from ranking. Its 2024 custom modeling_qwen.py (bidirectional attention + old KV-cache API) is incompatible with transformers 5.9.0 in three places. I loaded it with native Qwen2Model (causal attention) as a stopgap, but gte’s core is bidirectional — under causal attention, last-token pooling degrades severely (R@10 only 20–60%), which doesn’t represent its true quality. Its C-MTEB of 67.7 suggests its real level should be upper-middle.

The unorthodox blade: the most human-readable description ≠ the most embedding-friendly one

6 fair models × 4 description sources = 24 single-path text retrievals, sorted by R@10 (gte-qwen2 excluded):

⭐ e5-large + gemini single-path R@10=93.3%, tying the FG-CLIP2 image single-path ceiling — the biggest surprise (e5’s C-MTEB is only 58.8, the weakest of the candidates). But this is a single combo over 15 data points, fully within the ±6.7pp noise band — most likely partly luck, and it can’t be taken at face value until GT is expanded.

The most jarring number is in the last row — bge-m3 + claude-haiku has a P@3 of only 26.7%, the worst of all. Yet in the previous section, claude-haiku-4-5 was the “highest human-readability” description model — avg 298 chars, smallest variance, most perfectly structured (the four-part “shape & structure / material & texture / color & tone / fine details”).

Pulling out the average performance of the four description sources across the six embedding models makes it clearer:

Human-Readable Doesn’t Mean Embedding-Friendly

The description ranked #1 for human readability ranks #4 (dead last) for embedding-retrieval P@3; the shortest description, ranked #4 for human readability, ranks #1 for embedding retrieval.

Three likely reasons:

1. Too long → key tokens get truncated. bge-large-zh’s context is only 512 tokens; a 298-char Chinese description likely gets truncated, with the key info lost at the tail.
2. Boilerplate dilutes the semantic keywords. Template words like “shape & structure / material & texture / color & tone / fine details” appear in every description — to the embedding they’re noise: they don’t distinguish asset A from asset B, yet they occupy vector space.
3. CLS pooling is sensitive to long text → the gist vector gets averaged out. The longer the text, the more the CLS token has to encode, which actually dilutes the most critical semantic features (“inn,” “worn-out,” “moss”). gemini-3.1-flash-lite’s descriptions are the shortest (avg 165 chars), letting the embedding focus on the core semantic keywords — less is more.

The Choice of Description Source Matters More Than the Embedding Model

This finding has an important corollary: swapping the description source on the same embedding model can shift P@3 by 20–40 points; swapping the embedding model on the same description usually differs by only ±7pp.

In other words, the description source is the lever, the embedding model is the fine-tune. Investment should go first into iterating the description source (prompt engineering, description cleaning, mixing multiple models’ descriptions); which embedding model you pick barely matters — look at the per-model averages:

None of the 7 Models Pulls Clearly Ahead on Average

All fall within the GT noise band (±7pp). Note that the C-MTEB ranking (stella 68.6 ≫ e5 58.8) is badly inconsistent with the in-domain measurements (e5 ≈ stella ≈ 70%), once again proving “a general benchmark ≠ your domain” — selection must be based on your own in-domain GT.

Fusion: Twin Swords United, 1+1 > 2

The highest single-path text-embedding R@10 reached 93.3% (e5-large + gemini, tying FG-CLIP2), but that’s a single combo over 15 data points, within the noise band — you can’t conclude from it that text has caught up with vision. The more robust judgment: on average, single-path text R@10 sits in the 68–73% range; text can’t replace vision, but it can supplement it. After fusing the visual and text paths with RRF (Reciprocal Rank Fusion, k=60):

Selection Conclusion

In one sentence: Chinese capability is a hard gate, not a bonus; the description model and the retrieval model must be selected separately, with the embedding benchmark as the standard; academic leaderboards are reference-only — you must benchmark on your own data.

📒 While we’re at it, let’s tally the one-time landing costs. Beyond the description generation that keeps coming up (gemini ¥0.026/img, ~¥2,600 for 100k, ~8+ hours), don’t forget a few one-time costs when you actually roll out — full thumbnail export ≈ 33 hours (CPU single-thread, a one-time overnight run, incremental afterward), visual encoding ≈ 3.3 hours, text encoding ≈ 20 minutes (all on a single 4060 Ti). Plus the artists’ later incremental annotation, but that’s optional labor amortized “as you use, you annotate.” All told, the only real money is the bit of API cost for description generation; local encoding is all just electricity — which is why I dared to roll out across 100k-scale assets in one shot.

5. Forging a Handy Weapon: NeuroBrowser (for Carbon-Based Lifeforms)

This weapon is NeuroBrowser — an asset-search panel of mine that runs inside the UE4 editor. It embeds directly as a Slate plugin panel, sitting side by side with the Content Browser (mimicking the native Content Browser experience). An out-of-the-box experience for artists.

Design Philosophy: Three-Layer Retrieval, Progressive Presentation

Back to the three distances raised in [[#2. The Distance Between Assets and Understanding]], NeuroBrowser’s retrieval architecture is also three-layered:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

flowchart TB
    Q["👤 User query"] --> SQL["📋 SQL search<br/>FTS5 + tag filter<br/>&lt; 5ms"]
    Q --> VIS["👁️ Visual search<br/>FG-CLIP-2 large<br/>~30ms"]
    Q --> DESC["📝 Description search<br/>Qwen3-Embedding-0.6B<br/>~30ms"]
    SQL --> RRF["🔄 RRF fusion ranking"]
    VIS --> RRF
    DESC --> RRF
    RRF --> RESULT["⚡ Instant results &lt; 100ms"]
    RRF --> LLM["🤖 LLM Re-rank<br/>300-600ms"]
    LLM --> REFRESH["🔄 Re-rank refresh"]

    style Q fill:#4a90d9,color:#fff
    style RRF fill:#e8793a,color:#fff
    style RESULT fill:#50b86c,color:#fff
    style LLM fill:#9b59b6,color:#fff

Layer 1: SQL search (< 5ms). FTS5 full-text + tag filtering + numeric ranges (tri-count, LOD, material count) + logical combinations (AND/OR/NOT). This layer depends on no AI model and is the safety net — even if the Python backend hasn’t started and no model is loaded, keyword search still works, and it’s 50×+ faster than UE4 native.

Layer 2: Vector search (< 30ms). Visual (FG-CLIP-2) and description (Qwen3-Embedding-0.6B) retrieve independently, each returning Top-K, then fused and ranked via RRF. The three path weights are user-adjustable — for “mossy rock” the visual weight is high, for “worn-out object in a corner” the description weight is high, for “big rock with tri-count < 5000” the SQL weight is high.

Layer 3: LLM re-rank (300–600ms, async in the background). See the next section.

About This Plugin’s Design

1. Heavy local, light server. All indexes (SQLite + FAISS) and models run on the artist’s local machine. Search works fully offline, even with no network. The Python backend (FastAPI) runs on local localhost, depending on no cloud service. The only thing that goes through the cloud is team sync of descriptions/tags — when collaborating, the descriptions you edit are visible to others. Pure-local operation is also supported, so if a user has local data updates, the database can be refreshed quickly.

2. No engine source modification. The whole plugin is built on UE 4.27.2’s public Slate APIs — thumbnails via FAssetThumbnailPool, drag-and-drop via FAssetDragDropOp, asset queries via IAssetRegistry. It never touches the engine’s Private directory and never forks the engine, so future upgrades are painless. It asks little of the engine interface, so it can support more engine versions.

3. Asset-family folding — 100k files collapse to tens of thousands. The same asset often has a PC version + Mobile version (_ios) + LOD variants + material variants (_01a/_01b). If everything is shown flat, the same logical asset appears 5–8 times in the results — that’s not “found it,” that’s “blew up the search.” NeuroBrowser aggregates by “asset family”: _01a/_01b material variants fold into sub-items, _01/_02 different meshes show separately, _ios mobile hides behind the PC version. ~100k asset files → tens of thousands of asset families, results an order of magnitude cleaner.

4. Tag semantics are config-driven, easy to modify. Filenames encode a lot of project conventions — region/scene markers, technical-variant suffixes, and so on — dozens of scene markers plus a dozen-plus technical-variant suffixes, hundreds of rules in total. This logic is itself semantic, and once aligned with the artists it becomes part of the searchable content. Moreover, all these rules are managed via the tag_rules.json config file, not hard-coded. Artists/TAs can add rules themselves, without changing code and recompiling.

5. Descriptions are alive, not dead. The LLM-generated description is only an initial value — artists can edit the description and tags directly in the asset detail panel. After editing, the old embedding is automatically marked stale, and a triple background-update mechanism (lazy + scheduled + manual trigger) rebuilds it asynchronously. Descriptions/tags sync across the team via the cloud — the description you edit is available to a colleague on their next search.

6. A usage-analytics loop. NeuroBrowser has a built-in NeuroBrowserAnalytics module — tab focus, drag-to-viewport, viewport-placement attribution, all logged asynchronously to JSONL. A companion Plotly dashboard shows “which assets are searched most,” “which searches return nothing,” “which assets users drag into scenes most.” This data is the basis for later tuning of search weights and filling in description blind spots.

7. Multiple search modes. Users can adjust each path’s retrieval weight, and refine with logic (filter by tags, filter by model poly-count, etc.).

8. The virtuous cycle — back to the pit we dug at the start. Remember what I said at the start: for AI to understand assets, someone has to teach it; but artists have no obligation to annotate just for you. My solution isn’t “force artists to annotate,” but to embed annotation into the search actions they were going to perform anyway — while searching for and using assets, an artist can casually fix a description or add a tag. These edits sync in real time to the whole team via the cloud shared database: the description you correct today directly benefits a colleague’s search tomorrow. The more it’s used → the more accurate the descriptions/tags → the more accurate the search → the more people want to use it — that’s the “virtuous cycle” promised at the start, and now it’s truly spinning.

Feature Demo

6. Forging a Less-Handy Weapon: MCP & CLI (for Silicon-Based Lifeforms)

Remember why we set out to do this?
We want to teach AI to recognize and understand assets. Now we have image semantics, text descriptions, and a big pile of semantic tags — enough to let AI understand the specific purpose of each asset. We even expose thumbnails to multimodal models, so if it wants to carefully compare two models, it can even spin up a sub-agent to look at the images and compare two assets in detail. (Of course, more exposed interfaces isn’t always better for the LLM — but that’s another story.)

Afterword

To master this kung-fu manual, you must reach the state of “person and weapon as one” with your weapon (AI) — so please download this document and show it to your artificial “idiot” too. Say “get me one of these too,” and you might just get the same tool.

Coming next: Form Two will focus on «how AI understands scenes» (a pit I dug myself — if I can’t pull it off, I might just switch to a different move).

[^1]: FatemiJahromi, S. A. (2024). Enhancing 3D Asset Retrieval with Semantic Search. Aalto University. link
[^2]: Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. OpenAI. CLIP uses contrastive learning to put images and text in a shared vector space — the foundational capability of this approach.