How can LLMs be applied to domain-driven design experiments?
Eric built an experimental pipeline using LLMs to look at code supposedly in a bounded context and extract its actually used language. This statistical approach helps overcome the brittleness of pre-LLM tools like stop-word lists and TF-IDF.
How does the domain language surveyor pipeline extract vocabulary?
The pipeline grabs random chunks of code, classifies them by domain, and uses an LLM to extract domain phrases. It then verifies terms against pure code, tokenizes them, and filters them with an LLM for relevance.
Why use random sampling instead of targeted code inspection?
Random sampling avoids human biases that lead us straight to where we think domain logic should be located. It ensures you find the domain logic wherever it actually resides in the codebase.
How do you prevent LLM hallucinations during vocabulary extraction?
The pipeline uses multiple verification steps with deterministic code to ensure every extracted term actually exists in the code sample. This prevents hallucinations from causing real problems downstream.
What similarity metrics are used to compare code vocabularies?
The system uses Jaccard similarity to measure overlap between term sets and document frequency to build term vectors. These metrics allow you to compare the consistency of vocabularies within and across repositories.
How can similarity score comparisons act as a system sanity check?
You can compare a codebase's similarity to itself versus its similarity to other systems to ensure valid relationships. If a system isn't more similar to parts of itself than to random external pieces, it indicates a major problem.
Can historical context maps be treated as testable hypotheses?
Yes, you can draw a context map based on a system's documented history and derive expected similarity scores for relationships. Comparing these predictions against actual empirical similarity scores provides hard evidence for or against your architectural assumptions.
How can a coding agent orchestrate the analysis workflow?
You can write an instruction file and build a skill so an agent like Claude Code can orchestrate the pipeline end-to-end. The agent handles loops like resampling when data is insufficient while keeping raw processing details out of its main context.
How do you balance token costs across different model tiers?
Expensive top-tier models like Opus are reserved for high-level orchestration, while cheaper models handle specific tasks. For example, classification can be done by Haiku and vocabulary extraction by Sonnet, minimizing overall token costs.
How do domain models change when incorporating statistical metrics?
Domain models do not have to be limited to traditional object-relation diagrams and can instead be statistical models of phenomena occurring in code. This allows you to capture actual usage patterns and characteristics through metrics like corpus definitions and cohort term frequencies.