RAG Beginner Learning Notes

Starting point: the asset semantic-search project from [[20260522.SearchToolDesign（Private）]]
Goal: understand the principles of RAG → run a minimal demo against local image data

1. What Is RAG?

RAG = Retrieval-Augmented Generation

In one sentence: first find relevant content from a knowledge base, then use that content to help the AI answer the question.

1.1 Why Do We Need RAG?

2. The Complete RAG Pipeline

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

╔══════════════════════════════════════════════════════════╗
║              Indexing Stage (offline, one-time build)     ║
║                                                          ║
║  Raw data (docs / images / asset descriptions)            ║
║       ↓                                                  ║
║  Embedding model  →  content → fixed-length vectors       ║
║       ↓                                                  ║
║  Vector database (FAISS / ChromaDB) stores vectors,       ║
║  enabling fast retrieval                                 ║
╚══════════════════════════════════════════════════════════╝
                          ↕
╔══════════════════════════════════════════════════════════╗
║              Retrieval Stage (online, per query)          ║
║                                                          ║
║  User question  →  Embedding model  →  question vector    ║
║       ↓                                                  ║
║  Vector DB: compute similarity, find Top-K nearest        ║
║       ↓                                                  ║
║  Return corresponding raw content (doc chunks /           ║
║  image paths / asset info)                               ║
╚══════════════════════════════════════════════════════════╝
                          ↕ (optional)
╔══════════════════════════════════════════════════════════╗
║              Generation Stage (optional, classical RAG)   ║
║                                                          ║
║  [User question] + [retrieved content]  →  LLM  →         ║
║  grounded answer                                         ║
╚══════════════════════════════════════════════════════════╝

3. Core Concept Explanations

3.1 Embedding (Vectorization)

Compress any content (text, images) into a fixed-length list of numbers (a vector), such that:

• Semantically similar content → vectors that are close in space
• Semantically different content → vectors that are far apart in space

1
2
3

"a willow tree"     → [0.12, -0.34, 0.87, ..., 0.05]  (768-dim)
"willows by the river" → [0.11, -0.31, 0.89, ..., 0.06] ← very close!
"a rock"             → [-0.45, 0.72, -0.13, ..., 0.91] ← far away

Common Embedding Models:

3.2 FAISS (Vector Similarity Search Library)

FAISS = Facebook AI Similarity Search (open-sourced by Meta)

Why do we need it?
When you have 20,000 vectors and want to find the Top-10 most similar to a query vector, comparing them one by one needs 20,000 computations. FAISS uses special data structures to speed this up by tens to thousands of times.

An analogy:

• Brute-force search = flip through every book in a library to find the one closest to your interests
• FAISS = the library is partitioned by topic, and you go straight to the “Botany” shelf

Three Main Index Types:

3.3 Similarity Metrics

4. Image RAG (The Special Power of CLIP)

The core innovation of CLIP: make images and text share the same vector space.

1
2
3
4

"a willow tree" (text)  → CLIP text encoder  → vector A
 🌳 (willow image)       → CLIP image encoder → vector B

Vector A ≈ Vector B  ← this is CLIP's training objective!

This means you can:

• text → image: input “give me a flowchart” and find the most similar images in the library
• image → image: upload a reference image to find assets with similar style
• image → text: input an image to find the most relevant descriptive documents

This is exactly the principle behind the “text-to-image search” feature of the asset-search tool.

5. Local Demo Plan

5.1 Goal

Run a minimal image semantic-retrieval demo using the blog images (PNG/JPG) in D:\Project\UGit\MyPicGo\Images\:

• Input: a piece of text description (e.g. “code screenshot”, “flowchart”, “UI interface”)
• Output: paths to the Top-5 most similar images in the library

5.2 Tech Stack

1
2
3
4
5
6
7

D:\Project\UGit\MyPicGo\Images\  (blog images, ~100+ PNGs/JPGs)
    ↓ batch read
CLIP ViT-B/32  →  visual encoder, each image → 512-dim vector
    ↓ build index
FAISS IndexFlatL2  →  in-memory index (no GPU, no server needed)
    ↓ query
user text → CLIP text encoder → 512-dim vector → Top-K retrieval

5.3 Hands-on Process

1. Create project directory D:\Project\UGit\PicGoRAGDemo\ + Python venv
2. Install deps: torch / transformers / faiss-cpu / modelscope / Pillow / numpy
3. Write build_index.py (indexing stage) + search.py (query stage)
4. Run python build_index.py: encode 225 images into 225 × 512-dim vectors, build FAISS index and dump to disk (~15s on CPU)
5. Run python search.py "query" to validate retrieval (single query < 100ms)

Three key pitfalls:

• transformers 5.x changed the CLIP API: get_image_features(...) now returns BaseModelOutputWithPooling (not a bare Tensor anymore); you must call .pooler_output to get the 512-dim vector.
• HuggingFace is unreachable from mainland China: direct connection times out and mirrors are unstable; switch to ModelScope (Alibaba’s ModelScope — domestic servers, no obstacles).
• ModelScope uses different naming: HF’s OFA-Sys/chinese-clip-vit-base-patch16 corresponds to AI-ModelScope/chinese-clip-vit-base-patch16 on ModelScope (AI-ModelScope is the HF-format mirror namespace on ModelScope; the file structure is identical to HF and transformers.AutoModel can load it directly).

5.4 Test Findings: Comparing Chinese vs English Query Effectiveness

Real testing found that English queries are slightly more accurate:

Possible reason: the blog images are mostly English-language technical screenshots (code, terminal, UI text are mostly in English). OpenAI CLIP simultaneously recognizes English text + visual features, so its hits are more reliable; Chinese-CLIP’s training data leans toward Chinese everyday/news content and isn’t sharp enough on the boundaries of technical visual concepts like “flowchart”.

This validates the core point from §6.1: the performance of an embedding model on your domain depends on the training data distribution — choose models based on the scenario.

5.5 Complete Code

build_index.py (indexing stage)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

"""build_index.py — encode N images into N 512-dim vectors and build the FAISS index

Outputs:
  index.faiss  FAISS index (row i = 512-dim vector of the i-th image)
  paths.json   row number → original image path
"""
from __future__ import annotations
import json
from pathlib import Path
import faiss, numpy as np, torch
from PIL import Image
from modelscope import snapshot_download
from transformers import AutoModel, AutoProcessor

# ===== Config =====
IMAGE_DIR  = Path(r"D:\Project\UGit\MyPicGo\Images")
INDEX_PATH = Path(__file__).parent / "index.faiss"
PATHS_PATH = Path(__file__).parent / "paths.json"
# AI-ModelScope is the HF-format mirror namespace on ModelScope, directly reachable in China
MODEL_ID   = "AI-ModelScope/chinese-clip-vit-base-patch16"
BATCH_SIZE = 16
IMG_EXTS   = {".png", ".jpg", ".jpeg", ".webp", ".bmp"}


def collect_images(root: Path) -> list[Path]:
    """Recursively scan all images."""
    return sorted(p for p in root.rglob("*") if p.suffix.lower() in IMG_EXTS)


def main() -> None:
    device = "cuda" if torch.cuda.is_available() else "cpu"

    # === 1. Load model ===
    # snapshot_download downloads ~700MB on first run to ~/.cache/modelscope/,
    # then returns the local path directly
    model_dir = snapshot_download(MODEL_ID)
    model     = AutoModel.from_pretrained(model_dir).to(device).eval()
    processor = AutoProcessor.from_pretrained(model_dir)

    # === 2. Scan image library ===
    paths = collect_images(IMAGE_DIR)
    if not paths:
        raise SystemExit(f"no images found under {IMAGE_DIR}")

    # === 3. Batch encoding ===
    embeddings, ok_paths = [], []
    for i in range(0, len(paths), BATCH_SIZE):
        batch = paths[i:i + BATCH_SIZE]
        imgs, ok = [], []
        for p in batch:
            try:
                imgs.append(Image.open(p).convert("RGB"))
                ok.append(p)
            except Exception as e:
                print(f"  skip {p.name}: {e}")
        if not imgs:
            continue
        inputs = processor(images=imgs, return_tensors="pt").to(device)
        with torch.no_grad():
            # transformers 5.x: get_image_features returns BaseModelOutputWithPooling;
            # the 512-dim projection vector is in the .pooler_output field
            feats = model.get_image_features(**inputs).pooler_output
        # L2 normalize: after normalization, L2 distance ranking ≡ cosine similarity ranking
        feats = feats / feats.norm(dim=-1, keepdim=True)
        embeddings.append(feats.cpu().numpy().astype("float32"))
        ok_paths.extend(ok)

    # === 4. Build FAISS index and dump to disk ===
    mat = np.vstack(embeddings)
    # IndexFlatL2: exact brute-force retrieval; sufficient for < 100K vectors,
    # single-query latency < 1ms
    index = faiss.IndexFlatL2(mat.shape[1])
    index.add(mat)
    faiss.write_index(index, str(INDEX_PATH))
    PATHS_PATH.write_text(
        json.dumps([str(p) for p in ok_paths], ensure_ascii=False, indent=2),
        encoding="utf-8",
    )


if __name__ == "__main__":
    main()

search.py (query stage)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

"""search.py — text query → CLIP text encoding → FAISS Top-K → print paths

Usage:
  python search.py "代码截图"
  python search.py "flowchart" --topk 10
  python search.py "blue UI" --open    # also open the Top-1 result
"""
from __future__ import annotations
import argparse, json, os
from pathlib import Path
import faiss, torch
from modelscope import snapshot_download
from transformers import AutoModel, AutoProcessor

INDEX_PATH = Path(__file__).parent / "index.faiss"
PATHS_PATH = Path(__file__).parent / "paths.json"
# Must match build_index.py — different models' vector spaces are incompatible
MODEL_ID   = "AI-ModelScope/chinese-clip-vit-base-patch16"


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("query", type=str, help="Natural-language query")
    parser.add_argument("--topk", type=int, default=5)
    parser.add_argument("--open", action="store_true", help="Open the Top-1 result")
    args = parser.parse_args()

    if not INDEX_PATH.exists():
        raise SystemExit(f"Cannot find {INDEX_PATH.name}; run python build_index.py first")

    # === 1. Load model + index ===
    device = "cuda" if torch.cuda.is_available() else "cpu"
    model_dir = snapshot_download(MODEL_ID)
    model     = AutoModel.from_pretrained(model_dir).to(device).eval()
    processor = AutoProcessor.from_pretrained(model_dir)
    index = faiss.read_index(str(INDEX_PATH))
    paths = json.loads(PATHS_PATH.read_text(encoding="utf-8"))

    # === 2. Encode text into a vector ===
    inputs = processor(text=[args.query], return_tensors="pt", padding=True).to(device)
    with torch.no_grad():
        # Also take .pooler_output to get the 512-dim projection vector
        feat = model.get_text_features(**inputs).pooler_output
    feat = feat / feat.norm(dim=-1, keepdim=True)
    query_vec = feat.cpu().numpy().astype("float32")

    # === 3. FAISS Top-K retrieval ===
    distances, indices = index.search(query_vec, args.topk)

    # === 4. Print results ===
    for rank, (idx, dist) in enumerate(zip(indices[0], distances[0]), 1):
        # For L2-normalized vectors: cos_sim = 1 - L2_dist^2 / 2
        sim = 1.0 - dist / 2.0
        print(f"  #{rank}  sim={sim:.3f}  {paths[idx]}")

    if args.open and len(indices[0]) > 0:
        os.startfile(paths[indices[0][0]])


if __name__ == "__main__":
    main()

6. Learning Resources

6.1 Introductory Articles (Easy → Deep)

6.2 Must-Read Papers (Optional)

6.3 Chinese Resources

• BAAI BGE series documentation: official docs for Chinese embedding models; directly guides model selection
• Zhihu “RAG Practice”: notes on pitfalls by Chinese engineers; very practical
• Bilibili @跟李沐学AI: deep-learning fundamentals; necessary background for understanding embeddings

7. Notes

Module 1: RAG Overview

1. RAG Architecture

Inside a RAG system, there is first a retriever that has access to the database and fires off a query (similar to a database). The retriever then receives the result of the query (the information deemed most relevant). It then uses this written, possibly most relevant information to generate an augmented prompt.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

# ============================================================
# Stage 1: Define the user question (the original prompt)
# ============================================================
# This is the user's original question, containing time-sensitive
# information ("this weekend") that the large model's own training
# data cannot cover in real time.
prompt = "Why are hotel prices in Vancouver super expensive this weekend?"


# ============================================================
# Stage 2: Naive Generation (baseline for comparison)
# ============================================================
# Throw the original prompt directly to the LLM to generate an answer.
# Downside: the model doesn't know what events are happening in
# Vancouver "this weekend", so it can only offer generic guesses
# (tourist season, exchange rate, etc.) and may hallucinate.
generate(prompt)


# ============================================================
# Stage 3: Retrieve relevant documents
# ============================================================
# Use the prompt to query external knowledge bases (vector DB,
# search engine, internal docs, etc.) and retrieve real-time /
# authoritative information related to the question, such as:
#   - concerts, conventions, sports events happening this weekend
#   - hotel-industry supply/demand news
# These documents are key material for the next "augmentation" step.
retrieved_documents = retrieve(prompt)
print(retrieved_documents)


# ============================================================
# Stage 4: Construct the augmented prompt
# ============================================================
# Stitch "original question" + "retrieved documents" into a new
# prompt, explicitly feeding external knowledge to the model so
# it answers with something to rely on.
augmented_prompt = f"""Respond to the following prompt: {prompt}

using the following retrieved information to help you answer {retrieved_documents}"""


# ============================================================
# Stage 5: Regenerate based on the augmented prompt
# ============================================================
# The LLM now receives "question + contextual evidence"; the
# output is more accurate, more timely, and reduces hallucination.
# This is the complete closed loop of RAG
# (Retrieval-Augmented Generation):
#   Retrieve → Augment → Generate
generate(augmented_prompt)

2. Understanding RAG Through LLMs

LLMs are essentially constantly predicting the probability distribution of the next value to appear; RAG changes that distribution.

3. Information Retrieval: RAG Retrieve vs Search Engine vs Database Query

All three are “looking for things”, but the underlying matching logic is completely different:

A one-line distinction of essence:

1
2
3

Database query — "Are they exactly equal?"   (exact)
Search engine — "Does it have this word?"   (frequency)
RAG retrieve   — "Are they alike in meaning?" (semantic)

Module 2: Information Retrieval & Search Techniques

1. Retriever Architecture Overview

Metadata Filtering

Before or after vector retrieval, use structured fields to hard-filter the candidate set, shrinking the retrieval scope.

• Principle: documents are ingested with metadata fields (e.g. source, date, category); during query, filter first, then rank by vector similarity.
• Advantage: dramatically reduces interference from irrelevant documents, improves both precision and retrieval efficiency.
• Limitation: relies on metadata quality at ingestion time; missing or misclassified fields can wrongly filter out effective documents. It doesn’t understand content and is almost never used alone.

Keyword Searching

Searches documents by exact-word matching; the most classical search method. Mainly includes TF-IDF and BM25.

• Principle: split the query into tokens, compute term frequency (e.g. BM25), return documents containing those words.
• Advantage: fast, results explainable, accurate hits on proper nouns (codes, IDs, model numbers).
• Limitation: doesn’t understand synonyms or paraphrases — say it differently and you might not find it.

Semantic Searching

Encode text into vectors and compute vector similarity to match “similarly-meaningful” content.

• Principle: use an Embedding model to map both query and documents into a high-dimensional vector space; measure distance using cosine similarity, etc.
• Advantage: captures semantic generalization; finds related content even when phrased differently.
• Limitation: higher compute cost; less reliable than keyword search for exact terms (e.g. specific IDs).

2. Two Common Keyword-Retrieval Algorithms in Detail

TF-IDF

Core formula: Score = TF(term, doc) × log(total docs / docs containing term)

• TF (term frequency): the more a term appears in the current doc, the more relevant
• IDF (inverse document frequency): the more docs contain a term, the more “filler” it is — weight drops; rarer terms get higher weight
• Result: high-frequency function words like “the”, “is” score near zero; proper nouns like “quantum entanglement” or “BM25” score high

BM25 (the industry-grade improvement of TF-IDF; Elasticsearch’s default algorithm)

Core improvements:

k1: controls how fast term frequency saturates; higher means “saying it more makes it more important”; lower means “3 times and 100 times are nearly the same”.
b: controls length penalty strength; b=0 ignores length, b=1 strictly scores by density; 0.75 is an industry empirical default.

3. Similarities and Differences Between Semantic Search and Keyword Search

1. Common ground: Digitization

• Prompt and documents each get a vector:
  No matter the search type, the computer can’t directly “read” text. The first step is always turning both the user’s question and the documents in the database into a list of numbers, called a vector.
• Vectors compared to generate scores:
  Once converted into numbers, the computer can use math formulas (e.g. cosine similarity) to compute the “distance” between two vectors. The closer, the higher the score, the more relevant.

2. Core difference: How is the vector generated?

Keyword Search: Count word occurrences

• Principle: the result is called a sparse vector.
• Logic: each position of the vector represents a specific word. If the document contains that word, the corresponding position gets a score (based on term frequency, e.g. BM25).
• Characteristic: literal matching; only recognizes “identical-looking” words.
  Limitation: searching for “doctor” won’t find a document containing “physician” but not “doctor”, because it doesn’t understand word meaning.

Semantic Search: Use an embedding model

• Principle: the result is called a dense vector.
• Logic: instead of counting words directly, text is fed into a pre-trained deep learning model (Embedding Model, e.g. BERT). The model “maps” the text into a multidimensional semantic space.
• Characteristic: understands meaning; the numbers in the vector represent abstract “features” or “concepts”.
  Advantage: recognizes synonyms; even if words don’t match, as long as the “meaning” is close (e.g. “cat” and “kitten”, “doctor” and “physician”), their vectors are mathematically close in space.

3. RRF Algorithm (Balancing Keyword and Semantic)

It is the core technique in hybrid retrieval. When you run both “keyword retrieval” and “semantic retrieval” in your RAG system, you get two completely different scoring lists. RRF (Reciprocal Rank Fusion) merges these two lists fairly into one final ranking list.

1. Core problem: The “apples and oranges” comparison dilemma

Hybrid retrieval faces a fundamental difficulty:

• Keyword (BM25) scores may be 15.4, 12.8, etc.
• Semantic (vector-search) scores (cosine similarity) typically range between 0.8 and 0.9.
• Problem: these two scores have different units and cannot be directly summed or compared.

RRF’s solution: completely ignore raw scores; only look at documents’ positions (ranks) in the lists.

2. Core mechanism

• Reward “consensus” documents: if a document ranks high in both keyword search and semantic search, RRF gives it an extremely high final score.
• Normalize weights across searches: provides a fair comparison across strategies; neither algorithm can dominate just because its score range is larger.
• Score = reciprocal of rank (the origin of the algorithm’s name):
  • Rank 1 → 1/1 = 1.0 score
  • Rank 2 → 1/2 = 0.5 score
  • Rank 10 → 1/10 = 0.1 score
  • Logic: higher rank = higher score; the score drops faster as rank drops.
• Aggregate scores: sum each document’s reciprocal score across all lists; the one with the highest total wins the final ranking.

3. Formula

$$RRF(d) = \sum_{i=1}^{n} \frac{1}{k + rank_i(d)}$$

• rank_i: document d‘s ranking in the i-th retrieval list (starting from 1).
• k: smoothing constant, the industry default is usually 60.

Why is k needed?

Without k, the gap between rank 1 (1.0 score) and rank 100 (0.01 score) is enormous, giving the first-ranked document overwhelming power. k acts as a “shock absorber” — compressing extreme differences and weakening the influence of noisy documents that happen to land at rank 1.

4. RRF’s core advantage: Only cares about rank

• No score normalization needed: no need to convert BM25’s 20 score and vector search’s 0.9 score — compare positions directly.
• Seamless cross-strategy merging: with 2 or 5 different retrieval techniques, as long as each gives a rank list, RRF can fuse fairly.

5. Parameters for tuning semantic vs keyword weights

Standard RRF itself has no direct “semantic vs keyword” weight parameter — k is just a smoothing constant, not a control over their ratio.
But Weighted RRF introduces an independent weight w for each retrieval strategy in the formula:

$$RRF(d) = \sum_{i=1}^{n} \frac{w_i}{k + rank_i(d)}$$

• Increase w_semantic → results skew toward semantic understanding (synonyms, conceptual relevance)
• Increase w_keyword → results skew toward exact literal matching

Where do you tune this weight in an actual implementation?
Take LangChain‘s EnsembleRetriever as the most intuitive example:

1
2
3
4
5
6
7
8
9
10
11
12

from langchain.retrievers import BM25Retriever, EnsembleRetriever
from langchain_community.vectorstores import FAISS

# Two retrievers
bm25_retriever = BM25Retriever.from_documents(docs)
vector_retriever = FAISS.from_documents(docs, embedding_model).as_retriever()

# weights correspond to w_i in the formula; the two values usually sum to 1.0
ensemble_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, vector_retriever],
    weights=[0.3, 0.7]  # keyword 30%, semantic 70%
)

weights=[0.3, 0.7] are w_keyword and w_semantic from the formula. Increase semantic → better at understanding intent; increase keyword → better at exact proper-noun matching.

4. Evaluation Metrics

Quantifying how good a retriever is with numbers is the scientific basis of system tuning.

Three core metrics

A concrete example to understand Precision and Recall

Scenario: the knowledge base has 100 docs, of which 10 are truly relevant (Ground Truth). The retriever returns 8, of which 6 are truly relevant.

$$Precision = \frac{\text{returned and relevant}}{\text{total returned}} = \frac{6}{8} = 75%$$

$$Recall = \frac{\text{returned and relevant}}{\text{all relevant docs in the library}} = \frac{6}{10} = 60%$$

Precision: from the perspective of “the docs you returned” — how many are real, how much is noise?
Recall: from the perspective of “the knowledge base” — of the 10 correct answers, how many did you find?

The intrinsic tension between Precision and Recall

The two naturally pull against each other:

Practical uses of metrics

• Establish baseline performance: score your current system and make clear where you are
• Validate optimization impact: when changing Embedding model, adjusting Chunk size, or modifying hybrid-retrieval weights, compare before/after metrics to confirm whether the change actually worked

The most critical prerequisite: Ground Truth

All metrics depend on a “ground truth” dataset

• What is Ground Truth? A human-annotated dataset. For example, for question A, pre-annotate that “doc 1” and “doc 5” are the only correct answers in the knowledge base.
• Why does it matter? Recall, Precision, MAP, MRR all require comparing “system’s answers” against “standard answers”. Without Ground Truth, scientific tuning is impossible.

5. Embedding Model In-Depth

Contrastive Training Process

Training objective: pull vectors of similar content closer, push vectors of dissimilar content apart.

Positive Samples: From “Natural Pairings on the Internet”

Humans’ natural behavior on the internet inherently produces massive paired data:

Core idea: the internet itself is a huge “implicit-annotation dataset”. When humans create content in daily life, they are unconsciously labeling data for AI.

Negative Samples: Automatically Generated by Algorithms

Rare Scenarios That Need Human Annotation

Only when you need a high-precision Benchmark (used to test how good a model is) do you bring in human annotation:

1
2
3

STS (Semantic Textual Similarity) dataset:
  Humans score the similarity of two sentences from 1 to 5
  But this is used to "evaluate" the model, not the main data used to "train" it

Training process: forward pass generates vectors → build similarity matrix → InfoNCE loss penalizes cases where “positive samples don’t rank high” → backprop updates weights.

CLIP’s 400M training pairs are almost entirely auto-acquired. The internet itself is the implicit-annotation dataset; human annotation is only used for evaluation benchmarks, not as the main training data.

I also feel that this training process is quite similar to the node graph on my personal website — the node graph has repulsive and attractive forces, the entire node-graph system needs to maintain a stable state. Therefore, to maintain the balance of repulsion and attraction, it needs an adjustment process. And I think this adjustment process may be exactly what embedding is.

A common misconception — LLM parameters vs Embedding dimensions: parameters are the total weights inside the neural network (CLIP has ~150M), i.e. the knowledge learned during training; the dimension is the length of the output vector (e.g. 512), a fixed spec at design time. Parameters are the total knowledge the model learned (the more, the “smarter”); the dimension is the length of the output vector (it bounds the upper limit of expressiveness). They are not the same thing.

Module 3: Information Retrieval with Vector Data

1. From Brute-Force Search to ANN

All vector-retrieval algorithms share one key variable — K: the number of vectors most similar to the query (Top-K) to return. Around “how to efficiently find these K vectors”, two major schools have evolved: exact retrieval and approximate retrieval (ANN).

The most basic algorithm: brute-force search (Brute Force / Flat Search / Linear Scan)

The simplest direct approach — compute the distance (cosine / Euclidean) between the query vector and every vector in the database, then sort and take the Top-K.

In FAISS it’s called IndexFlatL2; in academia it’s called Exact KNN; in engineering it’s often called Brute Force or Linear Scan. They’re all the same thing: build no index, brute force to the end.

• Pros: trivial to implement (one line of numpy); Recall = 100%, the accuracy ceiling and evaluation baseline of all ANN algorithms
• Fatal problem: complexity is O(N×D) (N = number of vectors, D = dimension). The bigger the database, the slower a single query — millions of vectors is already hard; billions are outright unusable
• Suitable for: small datasets (< 100K), offline evaluation, providing ground truth for ANN

The solution: ANN (Approximate Nearest Neighbor) search

Core idea: pre-build an index, skip most vectors that can’t possibly be relevant, only do fine-grained comparison in a small range — sacrificing a tiny bit of accuracy in exchange for orders-of-magnitude faster queries.

Mainstream implementations:

• HNSW (Hierarchical Navigable Small World graphs): based on “skip-list + small-world networks”; complexity drops to O(log N); high accuracy and fast queries — industry mainstream
• IVF (Inverted File Index): first K-means clustering; at query time only scan the nearest few clusters — suited for very large scale
• PQ (Product Quantization): splits high-dim vectors into segments then replaces with cluster codes — compresses vectors 64× or more, dramatically saving memory

In engineering you must trade off between accuracy (Recall) ↔ speed (QPS) ↔ memory; common combinations are IVF+PQ, HNSW+PQ.

2. Vector Databases

A vector database is a database system designed specifically for storing, managing, and retrieving high-dimensional vectors. Ordinary databases store structured rows and columns; vector databases store Embedding vectors — and have ANN indexes built in, making semantic retrieval a first-class citizen.

Core differences from traditional databases

Vector databases usually store vectors + metadata together, supporting “first hard-filter by metadata, then do vector retrieval” — i.e. the combination of Metadata Filtering and semantic retrieval mentioned earlier.

Mainstream Vector Database Comparison

• Qdrant: written in Rust; strong performance; REST/gRPC API; supports payload filtering; open source, self-hostable
• Milvus: designed for very large scale (billions); cloud-native architecture; suited for production distributed scenarios
• ChromaDB: lightest, starts in a few lines of code; first pick for dev/debugging, not suited for large-scale production
• pgvector: PostgreSQL extension; add vector retrieval to an existing PG database with no new system
• FAISS: strictly speaking a library, not a database — no persistence, no CRUD, but the source of all vector DB algorithms underneath

Basic Process for Creating a Vector Database

Step 1 — Database Setup
Create a collection and define its schema — specifying which fields to store, the vector dimension, and which distance metric to use (cosine / L2 / inner product). This is the container for all subsequent operations, equivalent to creating a table.

Step 2 — Loading Documents
Read raw data (text, PDFs, images, etc.) into memory, chunking as needed — splitting long documents into small chunks suitable for embedding, to avoid semantic dilution when a chunk is too long.

Step 3 — Sparse Vectors (for keyword retrieval)
Use algorithms like BM25 to generate sparse vectors for each document chunk — most positions are zero; only positions for terms that appear have weights. Designed for exact keyword matching; this is the keyword side of hybrid retrieval.

Step 4 — Dense Vectors (for semantic retrieval)
Pass document chunks through an Embedding model (e.g. BGE, CLIP), outputting a dense vector for each chunk — every dimension has a value, carrying semantic information. The core of semantic search; understands synonyms and intent.

Step 5 — Build the HNSW Index
Build an HNSW (Hierarchical Navigable Small World) index on top of the dense vectors. Pre-weave a “navigation network” among the vectors; at query time you follow graph jumps to locate, reducing complexity from O(N) to O(log N) and achieving millisecond-level ANN search. This step can be skipped; if skipped, you have brute-force retrieval.

3. Chunk (Chunking Techniques)

Splitting long documents into small chunks suitable for embedding is the key pre-processing step of the RAG indexing stage. Too-large a chunk dilutes semantics; too-small a chunk loses context; whether you split well directly affects retrieval quality.

Why do we need chunking?

Embedding models have a token limit (e.g. BERT series 512 tokens, BGE-M3 8192 tokens). Stuffing an entire book in only yields one fuzzy “average semantic”, and retrieval will struggle to hit specific passages. After chunking, each chunk has its own independent vector, and retrieval precision improves dramatically.

Main chunking strategies

• Fixed-size Chunking: hard-cut by character or token count; crude. Downside: may cut mid-sentence and lose context. Usually paired with Overlap — adjacent chunks share some tokens — to mitigate the truncation issue.
• Semantic Chunking: cuts along sentence boundaries, paragraphs, heading hierarchy; preserves natural semantic integrity. Suited for structured documents (Markdown, PDF).
• Recursive Character Splitting: LangChain’s default strategy; tries priorities like \n\n → \n → period → space in order, cutting on natural boundaries as much as possible; balances simplicity with semantic completeness.

4. Some More Advanced Chunking Techniques

Using LLMs for Semantic Chunking

Traditional chunking relies on rules (paragraphs, headings, fixed size); “LLM semantic chunking” lets the model truly understand the semantic boundaries of content before deciding how to cut — higher quality but also higher cost.

① Embedding-similarity-based Semantic Chunking (SemanticChunker)

Principle: split text into sentences first → compute embedding for each sentence → compute cosine similarity of adjacent sentences → when similarity drops sharply, the topic has shifted; cut there.

1
2
3

sentence1 → sentence2 → sentence3 ↘↘sharp drop↘↘ sentence4 → sentence5
                                          ↑
                                      cut point (semantic break)

• Tool: LangChain’s SemanticChunker directly wraps this logic
• Pros: no LLM inference needed; low cost; truly cuts by semantic breakpoints, not by character count
• Cons: needs a similarity threshold; a badly-chosen threshold cuts too finely or too coarsely

② Proposition Chunking

Principle: use an LLM to further distill each text segment into several atomic propositions — each proposition is an independent, complete, self-contained minimal fact unit.

Example: original text “Einstein published the special theory of relativity in 1905 and won the Nobel Prize for the photoelectric effect”
→ Proposition 1: “Einstein published the special theory of relativity in 1905”
→ Proposition 2: “Einstein won the Nobel Prize for his research on the photoelectric effect”

• Pros: at retrieval time, query and proposition map one-to-one; precision is extremely high; each proposition is self-contained, understandable without context
• Cons: every chunk must invoke an LLM to distill — token-heavy and slow; indexing is expensive
• Suited for: scenarios that demand the highest knowledge-base retrieval quality (medical, legal, precision Q&A)

③ Agentic Chunking

Hand the whole document directly to the LLM, letting it autonomously decide semantic boundaries, output split points, or output the chunks directly. Most flexible; can handle complex unstructured documents (e.g. dialogue records, mixed-format docs) but highest cost; generally only used in offline preprocessing pipelines.

Comparison of the three approaches

The most common pragmatic compromise: first use recursive character splitting for rough cuts, then use embedding similarity for semantic-boundary correction; only enable proposition chunking on core knowledge-base passages.

5. Query Parsing

User questions are often colloquial, vague, and contain multiple sub-intents — using them directly for retrieval, vector similarity will drift, missing genuinely relevant documents. The goal of “query parsing” is to use an LLM to make the question more retrieval-friendly before retrieval.

Core idea: before Retrieval, use one (or more) LLM calls to transform the Query, then retrieve.

Query Rewriting

Principle: directly use an LLM to rewrite the user’s original question into one (or more) new queries that are more precise and closer to the language of the knowledge-base documents.

Example: user asks “how to fix this bug” → rewritten: “how to fix an IndexError array-out-of-bounds exception?”

Why it works: Embedding models use symmetric similarity — the more alike the user’s words and the document’s words, the closer the vectors. Rewriting bridges the “colloquial ↔ document language” vocabulary gap.

1
2
3
4

# Rewrite prompt sketch
system = "You are a professional search assistant. Rewrite the user's question into a precise query suited for document retrieval. Output only the rewritten sentence."
rewritten = llm.chat(system, user_query)
results = vector_db.search(embed(rewritten), top_k=5)

• Pros: simple to implement; one LLM call; significantly improves vocabulary-mismatch
• Cons: the rewrite may drift from the original meaning; adds one LLM’s latency

Query Decomposition

Principle: faced with a complex query containing multiple sub-questions, let the LLM break it into several independent sub-questions; each sub-question retrieves separately; finally aggregate the answers.

Example: user asks “What are the pros and cons of Python vs JavaScript in web development, and which should I choose?”
→ sub-question 1: “What are the pros of Python for web development?”
→ sub-question 2: “What are the cons of Python for web development?”
→ sub-question 3: “What are the pros of JavaScript for web development?”
→ sub-question 4: “What are the cons of JavaScript for web development?”

Each sub-question retrieves separately; the per-sub-question retrieval results are merged and handed to the LLM for the final answer.

Why it works: a complex question often corresponds to multiple scattered pieces of information in the knowledge base. Without decomposition, it’s hard for a single query to hit all relevant passages at once; with decomposition each query is more focused, and retrieval precision improves dramatically.

• Pros: especially suited for multi-hop reasoning and comparison questions
• Cons: number of sub-questions is uncontrolled; multiple retrievals multiply cost; aggregation logic is complex

HyDE — Hypothetical Document Embeddings

Principle: instead of embedding the Query directly, first let the LLM generate a hypothetical “ideal answer”, then embed that hypothetical answer and use it for retrieval.

1

User Query → LLM generates "hypothetical answer" → Embed(hypothetical answer) → vector retrieval

Example: user asks “Why is HNSW fast to query?”
→ LLM generates a hypothetical answer: “HNSW is based on a hierarchical graph structure; at query time, it quickly locates a general area from the high-level sparse graph, then does fine-grained search layer by layer…”
→ Embedding this hypothetical answer → compared to embedding the question directly, the vector is closer to the real documents in the knowledge base

Intuition: question embeddings and answer embeddings are not in the same place in semantic space. A hypothetical answer is more like a “real document”; therefore its vector is closer to documents in the knowledge base, so retrieval Recall is higher.

• Pros: very effective for “knowledge-intensive Q&A”; doesn’t depend on keywords; purely semantically driven
• Cons: the hypothetical answer may contain hallucinations, but the retrieval stage doesn’t rely on the answer’s correctness, only its vector, so hallucinations don’t directly affect results
• Suited for: specialized-domain Q&A; scenarios where document phrasing differs greatly from question phrasing

Multi-Query

Principle: let an LLM generate N variants of the same question from multiple angles, retrieve each separately, then deduplicate and merge the results (often combined with RRF).

Example: original question “the limitations of RAG”
→ variant 1: “In what scenarios does RAG perform poorly?”
→ variant 2: “What are the drawbacks of Retrieval-Augmented Generation?”
→ variant 3: “Failure cases of RAG systems”

Each of the three queries retrieves separately; after merging, re-rank with RRF; final Top-K coverage far exceeds any single query.

• Pros: fills coverage blind spots of a single query; naturally pairs with RRF
• Cons: N embeddings + LLM call; latency and cost scale linearly

Summary / Comparison of Approaches

In engineering practice: the most common lightweight recipe is Query Rewriting + Multi-Query (N=3) + RRF — only one extra LLM call, and retrieval quality improves noticeably. HyDE and Decomposition are reserved for scenarios with extreme quality requirements (e.g. legal-doc Q&A).

6. Reranker & Re-ranking Strategy

Background: vector retrieval (the regular two-encoding kind called Bi-Encoder) is fast but coarse — it compresses the Query and Document into one vector each and takes the dot product, losing lots of fine-grained interaction information. A Reranker is the second-stage module that does fine-grained scoring and re-ranking on Top-N candidates after coarse retrieval.

Two-stage retrieval (Bi-Encoder) pipeline

1
2
3
4
5

User Query
    ↓
[Stage 1] Vector retrieval (Bi-Encoder + FAISS) → Top-100 candidates   ← fast but coarse
    ↓
[Stage 2] (Cross-Encoder) Reranker fine-grained scoring → Top-10 final results   ← slow but accurate

Cross-Encoder

Principle: concatenate the Query and Document and input them to the same Transformer, letting all tokens of the two text segments attend to each other — directly output a relevance score.

1
2

Input:  [CLS] Query tokens [SEP] Document tokens [SEP]
Output: a relevance score between 0 and 1

• Pros: full attention; most accurate scoring; the precision ceiling
• Cons: Document vectors can’t be precomputed — every query has to rerun each candidate document with the Query; 100 candidates = 100 model inferences; high latency
• Suited for: small candidate sets (within Top-100), scenarios that demand the highest precision

ColBERT (Contextualized Late Interaction over BERT)

As shown in the image, each word in the question will resonate with every word in the article, yielding more precise results.

Principle: Query and Document are still encoded separately (so Documents can be precomputed), but instead of compressing to a single vector, the vector of each token is preserved. The relevance score is computed via MaxSim: for each token of the Query (split by token), find the most similar token across all Document tokens, sum across.

1
2
3
4

Query    → [q₁, q₂, q₃, ...]     one vector per token
Document → [d₁, d₂, d₃, ...]    one vector per token (can be stored offline)

Score = Σ max_j sim(qᵢ, dⱼ)     ← "Late Interaction"

• Pros: Documents can be precomputed offline and stored; interaction is much richer than a single-vector Bi-Encoder
• Cons: large storage (one vector per token, longer docs cost more); slower than Bi-Encoder, faster than Cross-Encoder
• Position: the middle tier between precision and speed; suited for larger candidate sets (Top-1000)

Side-by-side comparison

7. ReRanking

Re-ranking is not a specific model; it’s a pipeline-position concept — re-scoring and re-ordering the candidate set after coarse retrieval.
There are multiple implementations; §6 already detailed Cross-Encoder and ColBERT; here are two more paths.

Three paths for re-ranking (overview)

① RRF (Reciprocal Rank Fusion)

Suited when: you’ve run multiple retrieval sources in parallel (e.g. sparse BM25 + dense CLIP) and need to merge two rank lists into one.

Core formula:

1
2

RRF_score(doc) = Σ  1 / (k + rank_i)       k usually taken as 60
                 i

For each retrieval source, compute each document’s contribution from its rank position (not its score), then sum up.

Concrete example:

1
2
3
4
5
6
7
8
9
10
11

Query: "stone covered with moss"

BM25 ranking (keyword):        CLIP ranking (visual):
  stone_moss_03  → rank 1        stone_moss_03  → rank 1
  stone_moss_01  → rank 2        rock_wet_01    → rank 2
  rock_wet_01    → rank 8        stone_moss_01  → rank 5

RRF fusion:
  stone_moss_03 = 1/(60+1) + 1/(60+1) = 0.0328  ← both sides agree ✅
  stone_moss_01 = 1/(60+2) + 1/(60+5) = 0.0315
  rock_wet_01   = 1/(60+8) + 1/(60+2) = 0.0308

Why not just sum the two raw scores?

BM25 scores (term-frequency stats) and CLIP cosine similarities have completely different units; summing directly is meaningless. RRF only cares about rank position, not absolute values, naturally sidestepping the unit-alignment problem.

② Direct LLM scoring

Feed the candidate result’s text description to an LLM, letting the model judge relevance to the Query directly:

1
2
3
4
5
6
7
8
9
10
11

Prompt example:
  query: "stone covered with moss"
  candidate asset: "grayish-brown stone, surface densely covered with green moss,
                    obvious weathered texture, suited to ancient-style scenes"

  Please score this asset's relevance to the query (0-10) and give a one-sentence reason.

LLM output:
  Score: 9
  Reason: the two core elements — stone and moss — match exactly; the weathered
  texture further reinforces ancient-style relevance

Quick selection guide

Module 4: LLMs & Text Generation

The first three modules all covered Retrieve — finding the most relevant content. This module enters the final RAG stage — Generate: hand the retrieved content to a large language model (LLM) and generate a grounded answer.

1
2

User question → [Retrieve] Top-K relevant chunks → [Assemble Prompt] → [LLM Generate] → answer
                     ↑ Module 1–3                          ↑ Module 4

Transformer Architecture Overview

In one sentence: Transformer is the common foundation of nearly all modern LLMs, Embedding models, and Rerankers. Throughout these notes: BERT, CLIP, BGE, Cross-Encoder, GPT/Claude — all are Transformer variants.

1. Why Transformer? (vs RNN)

First, the naming: why “Transformer” instead of “Attention”?

• Attention is a component, Transformer is the whole machine. Attention appeared as early as 2014 (Bahdanau) as an RNN accessory for machine translation; by 2017 it was a common technique — if the new architecture had also been called Attention, it would have collided with “RNN+attention”.
• The paper title is the argument, the architecture name is the product name. “Attention Is All You Need” reads between the lines: “previously it was RNN plus attention; drop RNN and keep only attention is enough.” The title shouts the slogan, the new architecture chose a new name — Transformer — to cut ties with the RNN era.
• “Transform” = transforming representation layer by layer. The model rewrites each token’s vector layer by layer through stacked layers, ultimately turning “isolated word vectors” into “context-rich semantic vectors”. Attention only describes an operation at one layer (Q·K→weighted V); Transformer describes the whole machine — different abstraction levels.

So why does it matter? Before “Attention Is All You Need” (2017), text was handled mainly by RNN/LSTM (RNN, Recurrent Neural Network), which reads one word at a time in sequence, with two hard flaws:

Transformer solves both at once with self-attention, the prerequisite for LLMs scaling to hundreds of billions of parameters.

2. Core mechanism: Self-Attention

Self-attention makes every word in a sentence “look at” every other word, absorbing information weighted by relevance.

The key triad Q / K / V:

Computation flow:

1
2
3
4

For each word:
  1. Dot-product the Query with all words' Keys → relevance scores
  2. softmax normalization → attention weights (sum = 1)
  3. Use the weights to take a weighted sum of all Values → the word's new representation

Concrete example — understanding what “it” refers to in a sentence:

1
2
3
4
5
6

Sentence: This stone is covered with moss because it is very damp
                                 ↑
The "it" word's Query matches all words' Keys:
  "stone"  → attention 0.7   ← highest weight; the model infers "it = stone"
  "moss"   → attention 0.2
  "damp"   → attention 0.1

3. Multi-Head Attention + Positional Encoding + Overall Structure

• Multi-Head Attention: run multiple Q/K/V groups in parallel; each “head” focuses on a different angle (one head looks at syntax, one at coreference, one at semantics…). Then concatenate. Analogy: multiple experts reviewing the same sentence from different dimensions simultaneously.
• Positional Encoding: self-attention itself doesn’t distinguish word order (“dog bites man” and “man bites dog” look the same), so position info must be injected for each word.
• Residual connections + Layer Normalization + Feed-Forward Networks (FFN): the standard accessories of every layer, ensuring that deep networks (tens to hundreds of layers) train stably.

4. Three Architecture Variants (Decide the model’s use)

LLM Sampling Strategies

Retrieval decides “what content to feed”; sampling strategy decides “how the model says the content” — with the same context and different parameters, the output can be rigorous or wide-ranging. For RAG this section directly affects hallucination rate.

1. How does the LLM “spit out” words?

At each step, the Decoder model does only one thing: predict the probability distribution of the next word.

1
2
3
4
5
6
7

Input: "Stones covered with moss usually appear in"
Model output (next-word probabilities):
  damp    0.45
  shade   0.25
  forest  0.15
  desert  0.02
  ... (tens of thousands of words in the vocab, each with a probability)

The “sampling strategy” is the rule for picking a word from this distribution. After picking, append it to the input, predict the next, and loop.

2. Deterministic decoding

3. Stochastic sampling (mainstream for chat/creative writing)

① Temperature — adjusts how “peaked” or “flat” the distribution is:

1
2
3
4
5

logits are divided by T before softmax:

T = 0 (extremely cold): only the top score remains; equivalent to greedy; most conservative
T = 0.7 (warm): high-score words are more likely, but occasional surprises  ← general default
T = 1.5 (hot): distribution flattens; low-score words also have a chance; more random / creative

② Top-k: sample from only the highest-probability k words (e.g. k=40); cut off the long tail.

③ Top-p / Nucleus sampling: accumulate probabilities from high to low; stop once p (e.g. 0.9) is reached; only sample within this dynamic set. Smarter than Top-k — fewer candidates when distribution is steep, more when flat.

1
2

Candidates (sorted by probability): damp 0.45  shade 0.25  forest 0.15  moss 0.08 ...
Top-p=0.9: accumulate 0.45+0.25+0.15+0.08=0.93 ≥ 0.9 → sample only from these 4

④ min-p (relatively new): uses the highest-probability word as the baseline; sets a dynamic minimum ratio; more robust than Top-p.

⑤ Repetition / frequency penalty: downweight words that have already appeared; avoid the “repeater” effect.

4. How to tune for RAG?

Choosing a Suitable LLM

There’s no “best” model — only “most suited to the current task + budget + compliance requirements”. Here’s a reusable selection framework.

1. Seven evaluation dimensions

2. Closed-source API vs open-source local

3. Mid-2026 mainline model quick view

Closed-source APIs (price = input/output, per million tokens):

Open-source / locally deployable (suited to private data):

4. Context window vs RAG

RAG must stuff [system prompt] + [Top-K retrieved chunks] + [conversation history] + [user question] into the context all at once. The larger the window, the more retrieval evidence you can include.

5. Quick selection guide

Prompt Engineering

Once the model is chosen and retrieval is done, the prompt is your only control lever at inference time — without retraining, you can dramatically change output quality just by organizing the input. For RAG, the core goal of prompt engineering is: make the model answer strictly based on retrieved content and refuse to hallucinate.

1. Basic structure of a prompt

1
2
3
4
5
6
7

┌── System: set role, rules, tone, boundaries
│      "You are an asset-library assistant; answer only from the provided asset info..."
├── Context: inject retrieved chunks
│      "[Asset 1] grayish-brown stone, covered with moss... [Asset 2] ..."
├── User: this turn's question
│      "Recommend stone assets suited for ancient-style scenes"
└── (optional) Few-shot examples / output-format constraints

2. General techniques

3. RAG-specific prompt template (key)

A qualified RAG prompt must include three “anti-hallucination” instructions: ① use only provided content ② if not found, say so ③ cite sources.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

You are the retrieval assistant for the game asset library. Strictly observe:
1. Answer only from the information in the [Retrieved Results] below; do not use your own knowledge.
2. If the retrieval results contain no answer, respond directly with "No matching assets found"; do not fabricate.
3. When answering, tag the asset you relied on with [asset ID].

[Retrieved Results]
[stone_moss_03] Grayish-brown stone, densely covered with green moss, with obvious weathered texture, suitable for ancient-style scenes
[rock_wet_01]   Damp rock, dark color, no moss

[User Question]
Any stone with moss, suitable for an ancient-style scene?

[Expected Output]
Recommend [stone_moss_03]: the grayish-brown stone densely covered with moss and weathered texture fits the ancient-style scene.
([rock_wet_01] has no moss, low relevance and not recommended.)

4. “Lost in the Middle” phenomenon

Research shows: when the context is very long, the model remembers the beginning and end most firmly, while the middle is most easily ignored — recall curves take a U-shape.

1
2
3
4
5

Model attention ▲
              │＼            ／
              │  ＼ ____ ／      ← middle sag
              └──────────────▶ chunk position
               beginning  middle  end

5. Good vs bad prompt comparison

Module 5: RAG Systems in Production

1. What challenges does production face?

When RAG leaves the demo, it hits a string of “walls” that demos don’t see:

2. Implement an RAG Evaluation Strategy

Evaluation = answering “how good is my RAG really right now?”. Three steps:

1
2
3

Offline eval set (gold set) → Automated scoring → Regression comparison
       ↑                              ↓
       └──── Collect online feedback, expand the set ──────┘

• Gold set: human-annotated question + expected answer / relevant chunks
• Metrics: retrieval (Recall@k, MRR), generation (faithfulness, answer relevance), end-to-end (hit rate)
• Tools: RAGAS, ARES, LangSmith Evaluation, DeepEval

3. Logging, Monitoring, and Observability

Production RAG is a black-box pipeline; no logging = no debuggability. Three layers:

4. Customized Evaluation

Generic metrics (e.g. “semantic similarity”) often don’t align with the business. Business says it’s good, then it’s actually good.

• Domain-expert scoring: legal / medical / customer-service leads score 1–5 by business criteria
• Rule-based validation: must include a certain field, must cite a certain type of source, must not contain certain words
• Behavioral proxy metrics: like rate, copy rate, follow-up rate (user keeps asking = wasn’t answered clearly)
• LLM-as-a-Judge: use a strong model as judge, but must calibrate against a gold set, otherwise the judge itself drifts.

5. Quantization

Swap a model’s/vector’s “high-precision numbers” for “low-precision representation” to save VRAM, save money, save latency.

6. Cost vs Response Quality

1
2
3
4
5
6
7

Quality ▲
       │      ╭──── Large model (expensive)
       │    ╱
       │  ╱
       │╱
       └──────────────▶ Cost
  Small model  Routing

Common cost-saving strategies:

• Model routing: simple questions → small model / rules; complex questions → large model
• Caching: same query hits the cache, return directly, save one LLM
• Token reduction: re-rank to Top-3, truncate long docs, use smaller context
• Batching / async: non-real-time tasks batch requests

7. Latency vs Response Quality

1
2
3
4
5

Time to first token ▲
                   │   ╭─ Long context + large model
                   │ ╱
                   │╱──── Streaming + small model + re-ranking
                   └──────────────▶ Quality

Hard production latency requirements:

Means: streaming output (SSE), parallel retrieval (vector + BM25 in parallel), re-ranking first, speculative decoding.

8. Security

RAG’s attack surface = retriever + LLM; defend at both ends:

9. Multimodal Retrieval-Augmented Generation

RAG isn’t limited to feeding text. The more modalities you can retrieve/generate, the wider your application scope:

1
2
3
4
5
6

text    ←─────► text   (most common; Q&A / search)
image   ←─────► text   (image-to-text search; CLIP)
text    ←─────► image  (text-to-image / text-to-image-search)
image   ←─────► image  (image-to-image search)
audio   ←─────► text   (meeting recordings → summaries)
video   ←─────► text   (long video → key frames + subtitle retrieval)

Key techniques:

• Unified embedding: CLIP / SigLIP / BGE-M3 map multiple modalities into one vector space
• Multimodal LLM: GPT-4o, Gemini, Qwen-VL — can see images, hear audio, read docs
• Structured parsing: PDF tables, scans, charts → specialized models convert to Markdown/JSON before feeding RAG

8. References

The local repo path is D:\Project\UGit\starting-ragchatbot-codebase