Semantic Metadata Registry — Technical Specification

— — Important

Hard Prerequisite: No Gold-level table gets authored until the full schema in §5 is finalized. Gold tables are downstream of every registry and projection in this document. Schema first, Gold tables second.

1. The Core Tension & Fix

Three fundamental failure modes occur when a single monolithic schema serves every consumer:

STEP— Authoring Pain

Too many required fields per column when trying to satisfy every downstream consumer at once.

STEP— Context Overload

Full objects are shipped to LLM prompts and APIs, exceeding token limits and bloating latencies.

STEP— Rigidity & Coupling

Authoring schema == Runtime payload. A column rename breaks five separate consumer integrations.

DEC — Decision · ACCEPTEDaccepted

The Core Architectural Fix: Author a rich ontology once in a stable storage format, then compile thin, purpose-built projections (Prompt View, Planner View, Resolver View, Tool Registry) for each consumer instead of shipping raw objects anywhere.

2. Entity-First, Not Column-First

Analytics is about a dataset made of tables, relationships, entities, measures, dimensions, and hierarchies. Columns are temporary physical representations.

typescript

// Separation of Concept (what it means) from Mapping (where it lives)
interface SemanticEntity {
  semanticId: string;
  semantic: "measure" | "dimension" | "entity" | "date" | "currency" | "category" | "hierarchy";
  entityType?: "customer" | "vendor" | "product" | "order" | "invoice";
  businessMeaning: string; // e.g. "finance.revenue"
  synonyms: string[];
  description: string;
}

interface ColumnMapping {
  semanticEntityRef: string; // References SemanticEntity.semanticId
  tableId: string;
  columnName: string;
  dataType: "string" | "integer" | "decimal" | "boolean" | "datetime";
  isCanonicalSource: boolean;
  capabilities: CapabilityMetadata;
}

A schema migration or column rename creates a new ColumnMapping pointing to the existing SemanticEntity — zero ripple effects through downstream prompts or query planners.

3. The Four Registries

The ontology splits into four peer registries based on lifecycle and ownership:

Registry	Contains	Change Frequency	Ownership
Semantic Registry	`SemanticEntity` + `ColumnMapping` definitions	Rarely (data model changes)	Platform Team (CI-gated)
Relationship Registry	Table definitions, joins, cardinality, chasm-trap rules	Rarely (schema migrations)	Platform Team (CI-gated)
Entity Registry	Aliases, fuzzy match rules, embeddings	Continuously (new vendors/typos)	Semi-automated pipeline
Statistics Registry	Min/max, null %, top values, sample data	Automated refresh cycle	Automated pipeline

4. Authoring Shape: SemanticEntity & ColumnMapping

typescript

export type SemanticKind =
  | "measure"
  | "dimension"
  | "entity"
  | "date"
  | "datetime"
  | "currency"
  | "category"
  | "identifier"
  | "hierarchy";

export interface CapabilityMetadata {
  supportedAggregations: Array<"sum" | "avg" | "min" | "max" | "count" | "cardinality">;
  searchable: boolean;
  supportsRange: boolean;
  supportsAliases: boolean;
  filterPriority?: number;
}

5. Keyword Reference & Derivation Rules

Capability Derivation Matrix

Capabilities are derived automatically from SemanticEntity.semantic by default, eliminating manual authoring overhead:

`semantic`	Aggregations	`supportsRange`	`groupable`	`searchable`
`measure`	Required (`sum`, `avg`, `min`, `max`)	Yes	Rare	No
`dimension`	None	Rare	Yes	Yes
`entity`	None	No	Yes	Yes (`supportsAliases: true`)
`date` / `datetime`	`min`, `max`, `count`	Yes	Yes	Limited
`currency`	Treated as measure	Yes	Rare	No

6. Tool Registry: Automatic Codegen

Tools are auto-generated from the Semantic Registry based on concept capabilities. A concept with groupable: true automatically emits a groupBy tool.

STEP— Discover

Agent passes natural-language intent to Tool Registry (discover(intent, entityType?)); receives minified tool candidates.

STEP— Bind

Agent requests full input schema for the chosen tool (bind(toolName)).

STEP— Invoke

Agent executes tool with validated arguments (invoke(toolName, args)).

7. Projection Pipeline & Query Planner

typescript

// Planner View — structural slice consumed by Query Planner & Shared Runtime
interface PlannerView {
  semanticId: string;
  table: string;
  column: string;
  dataType: string;
  defaultAggregation?: string;
  joinPathToCanonical: string[];
}

OBS — Observation

Runtime Discipline: A user query first narrows to relevant concepts using vector embeddings. Only those shortlisted entities are serialized into the Prompt View. The full table schema is never dumped into an LLM prompt.

8. Versioning & Session Isolation

Independent Layer Versioning: semanticModelVersion (Semantic + Relationship), registryVersion (Entity / Statistics), and projectionVersion evolve independently to prevent global cache invalidation.
Strict Session Isolation: Multi-tenant concurrent sessions are strictly isolated. Session state (conversation history, resolved entity caches) lives 100% outside the stateless read-only registries.

TRD — Trade-off

Gain

Zero cross-tenant context leaks. Highly cached, high-throughput projection lookups.

Cost

Requires explicit sessionId passing across all resolution pipelines.