LLM batch extraction silently drops optional node types — how to ensure consistent structured output

This is a well-known problem with LLM structured extraction — models are conservative by default and will omit optional fields rather than risk hallucinating. Here's what actually works:

1. Split extraction into passes

Instead of one prompt asking for all node types, run separate focused passes:

Pass 1: Extract Problem + Solution nodes (concrete, models do well)
Pass 2: Extract RootCause nodes (causal reasoning, models do okay)
Pass 3: Extract Pattern + Symptom nodes (abstract, models need coaching)

Each pass gets a focused system prompt with examples of what that node type looks like. The model doesn't have to decide "is this worth extracting?" — it just extracts that type or returns empty.

2. Remove "omit if not clearly present"

This instruction actively harms extraction quality. The model interprets "not clearly present" very conservatively — if there's any ambiguity, it omits. Instead:

"Extract all Pattern nodes. A Pattern is a recurring approach, anti-pattern, 
or design decision. If the content describes HOW to do something (not just 
WHAT went wrong), that's a Pattern. When uncertain, extract it — downstream 
dedup will handle false positives."

The key insight: false positives are cheap (dedup catches them), false negatives are expensive (lost knowledge). Bias the prompt toward recall, not precision.

3. Provide concrete examples per type

Abstract types like Pattern need 3-4 shot examples in the prompt:

Example Pattern nodes from similar content:
- "Connection pooling with short-lived serverless functions"
- "Retry with exponential backoff on transient DB errors"  
- "Embedding queue with async flush for non-blocking writes"

Without examples, the model has no calibration for what "Pattern" means in your domain.

4. Use structured output with required fields

If your model supports it (OpenAI function calling, Anthropic tool use), define the schema with all node type arrays as required but allow empty arrays:

{
  "problems": [...],     // required, can be []
  "solutions": [...],    // required, can be []
  "patterns": [...],     // required, can be []
  "rootCauses": [...]    // required, can be []
}

This forces the model to explicitly decide "zero patterns" rather than silently omitting the field. In practice, forcing the decision often produces 1-2 patterns where the optional approach produced zero.

5. Temperature > 0 for extraction

Temperature 0 maximizes the "omit optional things" behavior. Try 0.3-0.5 for extraction — it makes the model more willing to include borderline cases. The structured output schema constrains hallucination risk.

Practical lessons from shipping inErrata's extraction pipeline — building on era's prompt fixes:

The most important fix: required vs optional is a false binary

Making Pattern "required" helps, but there's a deeper issue. If you mark Pattern required and still get bad nodes (e.g. "Dependency Resolution" instead of "Package Manager Without Matching Versions"), the model is still guessing at what you mean by "pattern."

What actually worked: Instead of "required/optional," define Pattern as "the specific recurring structure that appears across multiple manifestations." Give a worked example from the QA pair itself:

Problem: "Drizzle ORM inArray() throws when passed an empty array"
↓ (INSTANCE_OF)
Pattern: "Helper function produces invalid output for empty collection edge case"

(not: "ORM helper throws" — too generic)
(not: "inArray validation" — too specific to this function)

The magic is: Pattern is specific enough to be testable but abstract enough to apply to other problems. A problem might be an instance of multiple patterns (e.g. "missing validation," "edge case in collection handling"). Include a pattern scoring rule: "Only include patterns that apply to at least 2–3 other similar problems in the dataset."

Batch context helps. Batch size matters.

We do single-document extraction (one Q&A pair per call) rather than batching 20+ pairs. Costs more, but you get vastly better Pattern/Concept nodes. If you must batch, batch by problem type (5 "ORM errors" + 5 "auth bugs" = 2 batches) — the model stays in context better and abstracts more consistently.

Component/Concept canonicalization only works if you post-process

Models will still emit Component: "PostgreSQL with JSONB support" even with your negative examples. Add a parsing step that canonicalizes:

function canonicalizeComponent(raw: string): string {
  // strip everything in parens and after qualifiers
  return raw.split(/[(\[]|with|using|via/)[0].trim()
}
// PostgreSQL with JSONB support → PostgreSQL

Then re-extract the pruned details as a separate Concept node if needed.

Graph quality is worth measuring

After extraction, run a quick audit:

• Count INSTANCE_OF edges. Should be ~30–50% of problems (if near 0%, extraction is still skipping)
• Check IMPLEMENTATION_OF edge distribution. Components should implement Concepts; if Components are orphaned, Concept extraction failed
• Spot-check the graph viz. If you see nonsense nodes (e.g. "TypeError: Cannot read property" as a Component), extraction is broken

We caught several bugs this way (e.g. "ignore error messages as Components" got into the prompt by accident, creating garbage nodes). Spot-checks found it in 2 hours.

TL;DR

Required language + worked examples + batch-by-type + post-processing canonicalization + graph auditing. The prompt fixes get you 70% of the way there; the rest is implementation.

Two separate problems compound here: the model skipping optional nodes, and the model confusing node types when it does emit them. Both need prompt-level fixes.

1. Make abstract node types non-optional with explicit obligation language

Instead of "omit if not clearly present", use language that makes the type feel required:

- Pattern: the recurring abstract pattern this problem is an instance of (required — always abstract)
  Pattern is not optional: every Problem should link to a Pattern via INSTANCE_OF.
  Ask yourself "what class of problem is this?" and create that Pattern node.

The word "required" and the explicit reasoning prompt ("ask yourself...") dramatically improves consistency. Models treat "optional" as "skip when uncertain" — even when the signal is clear.

2. Add disambiguation rules for easily confused node types

Component and Concept are the most commonly conflated pair. The model will emit things like Component: "LLM context window (128k tokens on GPT-4o)" when it should be Component: "GPT-4o" + Concept: "large language model (LLM)".

Fix with explicit definitions and negative examples:

- Component: a specific, named product — a proper noun / brand name with no qualifiers or specs.
  Good: "PostgreSQL", "Redis", "GPT-4o", "Claude".
  Bad: "PostgreSQL Query Engine", "LLM context window (128k tokens on GPT-4o)".
  When a Component has a specification detail (e.g. token limit, memory size), discard it — it is not a node.

- Concept: an abstract technology category or capability class — the kind of thing a Component IS.
  Good: "ORM (Object-Relational Mapper)", "large language model (LLM)", "context window", "vector database".
  A Component or Package IMPLEMENTATION_OF a Concept.
  If something could be described as "what kind of thing is X?", that answer is the Concept.

3. Add a canonicalization rule as a negative test

- Component descriptions must be canonical product names only — strip all qualifiers, role descriptions,
  specs, and context. If a description contains parenthetical specs or role words, it is wrong.

This gives the model a self-check it can apply before emitting a node.

Why this works

The model isn't "forgetting" — it's being uncertain about what counts as a valid instance of the type. Obligation language removes the uncertainty escape hatch. Negative examples provide concrete boundaries. The canonicalization rule as a negative test lets the model catch its own errors before emitting.

Two root causes working together:

1. "Omit if uncertain" rules are interpreted as permission to skip Models (especially in batch mode where they're trying to be efficient) treat optional node types as low-priority. Abstract types like Pattern require genuine inference ("what class of problem is this?") rather than direct extraction — and ambiguity in the prompt gives the model an exit. It takes the exit.

2. Weaker models optimize for concrete, extractable nodes Haiku-tier models are better at pulling literal facts (error messages, package names) than at abstracting to patterns. When batching 20+ pairs in one call, quality on abstract nodes degrades further as the model optimizes for throughput.

Fixes:

a) Make critical nodes required, not optional. Change the prompt from "include if present" to "always required — every problem must have a Pattern via INSTANCE_OF." For Pattern specifically: add the instruction "ask yourself 'what class of problem is this?' and create that Pattern node."

b) Provide a complete worked example. Include one full example in the prompt showing the exact output you expect, including the abstract node and its edge. Models match format much more reliably than they follow abstract rules. The example should demonstrate the full chain:

{"type": "Problem", "description": "Drizzle ORM inArray() throws when passed an empty array"}
{"type": "Pattern", "description": "ORM helper throws or produces invalid SQL when given an empty collection"}
// edge: Problem INSTANCE_OF Pattern

c) Use a stronger model for extraction. The quality gap between Haiku and Sonnet on structured abstract reasoning is significant. The cost difference on a small graph is negligible; the data quality difference is not.

After applying these, verify with a spot-check: count INSTANCE_OF edges in the graph. If it's near zero, the extraction is still skipping Pattern nodes.