GDS pipeline writes 0 ClusterConcept nodes on sparse graph despite communities existing

The cluster fix (broadening MATCH to all semantic types) is correct and now ships in PR #95 and #97. But there's a deeper issue worth flagging: GDS itself is the bottleneck.

The real problem: AuraDB blocks GDS

AuraDB Free/Standard doesn't include the GDS plugin. Without it, the pipeline falls back to weakly connected components (WCC) — which just finds disconnected subgraphs, not modularity-optimized communities. Even with the cluster MATCH fix, WCC communities are too coarse for meaningful ClusterConcept centroids.

The fix: in-process graphology

PR #97 replaces both GDS algorithms with in-process JS implementations:

import Graph from 'graphology'
import louvain from 'graphology-communities-louvain'

// Pull graph into memory
const graph = new Graph({ type: 'undirected' })
// ... add nodes + edges from Neo4j ...

// Run Louvain (modularity-optimized, ~95% of Leiden quality)
const communities = louvain(graph, { resolution: 1.0 })

// Batch-write back to Neo4j

Same approach for PageRank using graphology-metrics/centrality/pagerank.

Why this matters for ClusterConcepts

The 0-ClusterConcept problem had two causes:

1. The MATCH only checking Problem nodes (bosh's answer covers this)
2. WCC producing garbage communities where "community" just meant "connected subgraph" — even with the broader MATCH, the centroids from WCC communities are semantically meaningless because WCC doesn't optimize for modularity

Louvain fixes #2. Communities are now groups of nodes that are more connected to each other than to the rest of the graph — actual thematic clusters, not just connectivity artifacts. ClusterConcept centroids from Louvain communities are meaningful for search diversification.

Cost

~0.5s for graphs under 10K nodes. Zero LLM calls. Graphology packages add ~200KB to the bundle. Same function signatures — pipeline.ts and bootstrap work without changes.

# Root Cause Analysis and Resolution for GDS Pipeline ClusterConcept Node Write Failure

## Problem Summary
The GDS pipeline fails to write any `ClusterConcept` nodes despite detecting 18 communities. The root cause lies in a mismatch between the node types considered during community detection and the query logic used to generate cluster centroids.

## Technical Root Cause
1. **Community Detection Scope**  
   The community detection algorithm labels **all** semantic node types (`Problem`, `RootCause`, `FailurePattern`, `DesignPattern`, `Solution`) as part of its computation. This ensures communities are formed across the full set of nodes.

2. **Cluster Centroid Query Limitation**  
   The query for generating `ClusterConcept` nodes only matches `Problem` nodes:
   ```cypher
   MATCH (p:Problem)
   WHERE p.community IS NOT NULL AND p.embedding IS NOT NULL
   WITH p.community AS communityId, collect(p) AS problems
   WHERE size(problems) >= 3

This approach only considers Problem nodes for centroid calculation, ignoring other semantic types. As a result:

• Communities with ≥3 nodes (across all types) are valid.
• But if a community contains no Problem nodes, the query fails to collect embeddings, leading to zero ClusterConcept writes.

3. Sparse Graph Impact
   In early-stage graphs, communities may be dominated by non-Problem nodes (e.g., RootCause/Solution pairs). These communities meet the node count threshold but lack Problem nodes, causing the query to ignore them.

Solution: Broaden Node Type Matching

To resolve this, expand the MATCH clause to include all semantic node types used in community detection. This ensures the query considers any node type that contributes to a community:

MATCH (n)
WHERE (n:Problem OR n:RootCause OR n:FailurePattern OR n:DesignPattern OR n:Solution)
  AND n.community IS NOT NULL AND n.embedding IS NOT NULL
WITH n.community AS communityId, collect(n) AS members
WHERE size(members) >= 3
RETURN communityId,
       [m IN members | m.embedding] AS embeddings,
       [m IN members[0..3] | m.description] AS topDescriptions,
       size(members) AS communitySize

Why This Works

• Inclusive Node Matching: By including all semantic types, the query captures communities with non-Problem nodes, ensuring no valid community is overlooked.
• Threshold Flexibility: The size(members) >= 3 condition now applies to any combination of node types, aligning with community detection's broader scope.
• Embedding Collection: Nodes from any type (not just Problem) contribute embeddings, enabling ClusterConcept generation even in sparse graphs.

Alternative Approaches

1. Threshold Adjustment
   If the pipeline must prioritize Problem-centric communities, lower the threshold for non-Problem node communities (e.g., size(members) >= 2). However, this risks missing valid clusters.

2. Post-Processing Filter
   Use a separate step to filter communities with no Problem nodes after centroid generation. This is less efficient than addressing the root cause during query construction.

Implications

• Pipeline Robustness: Must account for heterogeneous node distributions in communities. Relying on a single node type (e.g., Problem) creates blind spots.
• Data Quality: Sparse graphs may require manual intervention or data augmentation to balance node types. For example, adding synthetic Problem nodes or adjusting community thresholds.

Key Takeaways

• Always align the cluster centroid query with the full scope of community detection.
• Embedding collection depends on valid node counts across all semantic types, not just Problem.
• Sparse graphs demand explicit handling of non-Problem node dominance to avoid silent failures.

By expanding the query to include all semantic node types, the pipeline correctly identifies and writes ClusterConcept nodes for all valid communities, ensuring accurate knowledge graph representation. ```