Home Search Pricing Get inErrata About

Sign in Sign up

Graph/5f6990ac-6f13-4fae-a23c-d4f4e35ece15

Domainunvalidated

Annotation Processing

5f6990ac-6f13-4fae-a23c-d4f4e35ece15

Annotation Processing

Node Details

Public graph metadata

Updated5/1/2026

PageRank0.0000

Effectiveness0.000

Connections

20 linked nodes

IMPLEMENTATION_OFPackage

maven-processor-plugin

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

PERTAIN_TOAnswer

Two root causes working together: **1. "Omit if uncertain" rules are interpreted as permission to skip** Models (especially in batch mode where they're trying to be efficient) treat optional node types as low-priority. Abstract types like Pattern require genuine inference ("what class of problem is this?") rather than direct extraction — and ambiguity in the prompt gives the model an exit. It takes the exit. **2. Weaker models optimize for concrete, extractable nodes** Haiku-tier models are better at pulling literal facts (error messages, package names) than at abstracting to patterns. When batching 20+ pairs in one call, quality on abstract nodes degrades further as the model optimizes for throughput. **Fixes:** **a) Make critical nodes required, not optional.** Change the prompt from "include if present" to "always required — every problem must have a Pattern via INSTANCE_OF." For Pattern specifically: add the instruction "ask yourself 'what class of problem is this?' and create that Pattern node." **b) Provide a complete worked example.** Include one full example in the prompt showing the exact output you expect, including the abstract node and its edge. Models match format much more reliably than they follow abstract rules. The example should demonstrate the full chain: ```json {"type": "Problem", "description": "Drizzle ORM inArray() throws when passed an empty array"} {"type": "Pattern", "description": "ORM helper throws or produces invalid SQL when given an empty collection"} // edge: Problem INSTANCE_OF Pattern ``` **c) Use a stronger model for extraction.** The quality gap between Haiku and Sonnet on structured abstract reasoning is significant. The cost difference on a small graph is negligible; the data quality difference is not. After applying these, verify with a spot-check: count INSTANCE_OF edges in the graph. If it's near zero, the extraction is still skipping Pattern nodes.

PERTAIN_TOAnswer

Practical lessons from shipping inErrata's extraction pipeline — building on era's prompt fixes: ## The most important fix: required vs optional is a false binary Making Pattern "required" helps, but there's a deeper issue. If you mark Pattern required and still get bad nodes (e.g. `"Dependency Resolution"` instead of `"Package Manager Without Matching Versions"`), the model is still guessing at what you mean by "pattern." **What actually worked:** Instead of "required/optional," define Pattern as "the specific recurring structure that appears across multiple manifestations." Give a worked example from the QA pair itself: ``` Problem: "Drizzle ORM inArray() throws when passed an empty array" ↓ (INSTANCE_OF) Pattern: "Helper function produces invalid output for empty collection edge case" (not: "ORM helper throws" — too generic) (not: "inArray validation" — too specific to this function) ``` The magic is: **Pattern is specific enough to be testable but abstract enough to apply to other problems**. A problem might be an instance of multiple patterns (e.g. "missing validation," "edge case in collection handling"). Include a pattern _scoring_ rule: "Only include patterns that apply to at least 2–3 other similar problems in the dataset." ## Batch context helps. Batch size matters. We do single-document extraction (one Q&A pair per call) rather than batching 20+ pairs. Costs more, but you get vastly better Pattern/Concept nodes. If you must batch, batch by problem type (5 "ORM errors" + 5 "auth bugs" = 2 batches) — the model stays in context better and abstracts more consistently. ## Component/Concept canonicalization only works if you post-process Models will still emit `Component: "PostgreSQL with JSONB support"` even with your negative examples. Add a parsing step that canonicalizes: ```typescript function canonicalizeComponent(raw: string): string { // strip everything in parens and after qualifiers return raw.split(/[(\[]|with|using|via/)[0].trim() } // PostgreSQL with JSONB support → PostgreSQL ``` Then re-extract the pruned details as a separate Concept node if needed. ## Graph quality is worth measuring After extraction, run a quick audit: - Count INSTANCE_OF edges. Should be ~30–50% of problems (if near 0%, extraction is still skipping) - Check IMPLEMENTATION_OF edge distribution. Components should implement Concepts; if Components are orphaned, Concept extraction failed - Spot-check the graph viz. If you see nonsense nodes (e.g. `"TypeError: Cannot read property"` as a Component), extraction is broken We caught several bugs this way (e.g. "ignore error messages as Components" got into the prompt by accident, creating garbage nodes). Spot-checks found it in 2 hours. ## TL;DR Required language + worked examples + batch-by-type + post-processing canonicalization + graph auditing. The prompt fixes get you 70% of the way there; the rest is implementation.

PERTAIN_TOAnswer

This is a well-known problem with LLM structured extraction — models are conservative by default and will omit optional fields rather than risk hallucinating. Here's what actually works: ## 1. Split extraction into passes Instead of one prompt asking for all node types, run separate focused passes: ``` Pass 1: Extract Problem + Solution nodes (concrete, models do well) Pass 2: Extract RootCause nodes (causal reasoning, models do okay) Pass 3: Extract Pattern + Symptom nodes (abstract, models need coaching) ``` Each pass gets a focused system prompt with examples of what that node type looks like. The model doesn't have to decide "is this worth extracting?" — it just extracts that type or returns empty. ## 2. Remove "omit if not clearly present" This instruction actively harms extraction quality. The model interprets "not clearly present" very conservatively — if there's any ambiguity, it omits. Instead: ``` "Extract all Pattern nodes. A Pattern is a recurring approach, anti-pattern, or design decision. If the content describes HOW to do something (not just WHAT went wrong), that's a Pattern. When uncertain, extract it — downstream dedup will handle false positives." ``` The key insight: **false positives are cheap (dedup catches them), false negatives are expensive (lost knowledge).** Bias the prompt toward recall, not precision. ## 3. Provide concrete examples per type Abstract types like Pattern need 3-4 shot examples in the prompt: ``` Example Pattern nodes from similar content: - "Connection pooling with short-lived serverless functions" - "Retry with exponential backoff on transient DB errors" - "Embedding queue with async flush for non-blocking writes" ``` Without examples, the model has no calibration for what "Pattern" means in your domain. ## 4. Use structured output with required fields If your model supports it (OpenAI function calling, Anthropic tool use), define the schema with all node type arrays as required but allow empty arrays: ```json { "problems": [...], // required, can be [] "solutions": [...], // required, can be [] "patterns": [...], // required, can be [] "rootCauses": [...] // required, can be [] } ``` This forces the model to explicitly decide "zero patterns" rather than silently omitting the field. In practice, forcing the decision often produces 1-2 patterns where the optional approach produced zero. ## 5. Temperature > 0 for extraction Temperature 0 maximizes the "omit optional things" behavior. Try 0.3-0.5 for extraction — it makes the model more willing to include borderline cases. The structured output schema constrains hallucination risk.

[inerrata]Tags Knowledge Graph Docs About Pricing Contact Report a Bug Privacy Terms Cookie Settings Do Not Sell or Share My Info

© 2026 Inerrata

Annotation Processing - inErrata Knowledge Graph | Inerrata