Platform map · orientation · re-measured 2026-10-07
samfrons/messai.ai · development
One repo, four Vercel zones, nine open-source packages, one Postgres.
Every surface MESSAI ships, what feeds it, and what is still dark. Requests enter apps/web and are rewritten to site, lab and api by URL path. Only apps/api writes to Postgres. Everything to the left of the zones is a person or a script; everything to the right is state or a model.
Batch work has no worker host: scripts run on an operator's machine or in the weekly Claude routine. The BullMQ queue scaffold was archived on 2026-10-02 (archive/2026-10-bullmq-scaffold/); background work uses Next after() or Vercel Queues, and the Upstash store that remains holds rate-limit buckets. Blue cards are Vercel zones, green are shared libraries or live services, amber are scripts, dashed are dormant.
Every request enters through the messai-ai Vercel project (apps/web), which owns / and rewrites everything else to the other three zones.
- entry
- apps/web/next.config.js
- rewrite map
- apps/web/multi-zone-rewrites.json
- blocks
- api 76 · lab 11 · site 24 entries (2026-10-07)
One canonical RAG chat endpoint with the full tool registry (chat bifurcation resolved 2026-05-12). Tools read the DB and the computed JSON artifacts.
- lib
- @messai/ai-chat
- embeddings
- HF Inference Router, live
Local dev reads and writes the staging Supabase project since 2026-10-01: every .env.development.local points DATABASE_URL and DIRECT_URL at it, so pnpm dev:*, build:local, pre-deploy and db:migrate:local all touch staging. The Supabase CLI stack on :54322 (2026-05-07 → 2026-10-01) is retired. Docker compose for the ML engine on :8001. PDF store on disk is a cache of R2 since 2026-08-14 (pnpm papers:hydrate).
Marketing and docs surface, in English at bare URLs and in German under /de/ (Astro i18n fallback, 2026-09-29). Reads the DB only for the server-rendered /methodologies pages; no API routes.
- pages
- 32 .astro (measured 2026-10-07; 25 @ 2026-09-03)
- dev port
- 4321
- rewrite block
- site · 24 entries
Owns /, /research, /papers, /parameters, /protocols, /admin, /dashboard, /hunter, /insights, /datasets, /experiments, /runs, /survey. The only zone with rewrites. Every page lives under src/app/[lang]/ and is served in English at /x and in German at /de/x (2026-09-29). Reads the DB in server components; every write is a fetch to /api.
- pages
- 108 (measured 2026-10-07; 95 @ 2026-09-10, 81 @ 2026-07-07)
- bundler
- Turbopack — plain next build; the May 2026 wedge was the layout import graph, not the bundler
- build
- ~5 min
- dev port
- 3000
/lab/*, /lab/dac (electrochemical DAC workspace, 2026-08-25), /models/*, /predictions/*, /methodology, /parameters/sweep. Owns the heavy 3D tree; the seven wastewater unit-process models moved to @messai/process-3d on 2026-09-08 so other zones render the same geometry (apps/lab keeps re-export shims).
- pages
- 17 (measured 2026-10-07; docs said 10 @ 2026-07-06)
- dev port
- 3002
- rewrite block
- lab · 11 entries
All /api/* handlers. Heavy server deps live here: Sentry, AI SDK, AWS SDK (R2), pdf-parse. Only zone that writes to Postgres. Background work runs in Next after(); the BullMQ scaffold and its bullmq / ioredis deps were archived on 2026-10-02.
- routes
- 247 (measured 2026-10-07; 229 @ 2026-09-10, 206 @ 2026-05-20)
- caching
- force-dynamic default; 15 ISR routes (13 × revalidate=60, 2 × 3600) · 2026-10-07
- build
- ~3.2 min
- dev port
- 3003
- parameter wiki
- /api/wiki/* — entries, revisions, moderation; every edit stays PENDING until a MODERATOR or ADMIN approves (2026-09-25)
NextAuth v4 with the Prisma adapter (two adapters installed — a known P1 cleanup). Sessions, accounts and verification tokens are Prisma models.
Anything used by more than one zone lives here; apps never import each other's src/. Wired via tsconfig paths, transpilePackages, and workspace:* deps.
libs/shared/ui@messai/ui — design system + Tailwind preset, sharp corners onlylibs/shared/i18n@messai/i18n — English and German message bundles (German under /de/ since 2026-09-29) and the scientific-notation normalizerlibs/shared/ml@messai/ml — runFullPrediction, per-class routing, priors v2 readerlibs/shared/ai-chat— chat tools, live embed via HF routerlibs/shared/electrochemistry— analytical predictForSystem, 5 physics familieslibs/shared/dac— electrochemical DAC model (2026-08-25)libs/shared/electrode-3d,component-catalog— electrode recipes + catalog datalibs/shared/mes-3d@messai/mes-3d — shared MES R3F parts (reactor parts, particle flows, labels, studio lighting), promoted out of apps/site on 2026-08-10; used by apps/lab, apps/web and apps/sitelibs/shared/process-3d@messai/process-3d — the seven wastewater unit-process models and<ProcessModelScene>, promoted out of apps/lab on 2026-09-08 so non-lab zones render the same geometrylibs/shared/pid-schematic@messai/pid-schematic — ISA-5.1 P&ID sheets from a declarative PidSpec, checked by a lint pass; one sheet per catalog reactor model via getModelPidSpec, shown in the lab catalog, /lab/schematic and /demo/resource-recoverylibs/shared/wiki@messai/wiki — the parameter wiki: typed sections, section diff, three-way merge, role rules; every edit waits for a moderator (2026-09-25)libs/shared/platform-map@messai/platform-map — this map: the /decks/platform-map deck and the onboarding atlas render the same datalibs/feature/lab-cad@messai/lab-cad — parametric MES design graph, validators and preview compiler
Single Prisma client. Server contexts import @messai/database/server; there is no db export. Client-safe Zod schemas only outside server modules.
- models
- 126 · 88 enums · 69 migrations (2026-10-20)
- schema
- prisma/schema.prisma
Prebuilt outputs of the quality/ML scripts, committed to git and served statically. API routes under research/* and parameters/[slug]/* read DB-first with these as fallback (60s ISR).
hierarchical-priors-v2.json— served by /api/ml/predicthierarchical-priors-v1.json— parameter pages, wastewater overlayresearch/calibration.json(npe-health.jsonis expected here but absent)protocols/protocol-snapshot.json— 97 nodes / 130 edges
All corpus acquisition, extraction, quality refresh and model refits are standalone scripts run by hand or by the weekly Claude routine. There is no worker host.
- gate
- resolveDbTarget() · scripts/lib/db-target.ts
- targets
- staging (default for reads) | prod (--expect-ref) | local (stack retired 2026-10-01)
Replaced two GitHub Actions crons (npe-nightly, ml-retrain) that burned ~1,350 min/month. Runs in a Claude remote environment; commits only npe-health.json when it changed. No npe-health.json exists in the repo, so no committed run has landed yet; the 2026-09-03 commit that added step 5b was a dry-run.
- 1 install Bayes deps (PyMC, sbi, torch)
- 2 simulator Liu-Logan ±25% gate — hard fail
- 3 SBC audit N=500
- 4 compose npe-health.json
- 5 quality refresh + calibration + GP-SCM fit; 5b two-tier effects refit + OpenAlex gap-search dry-run (added 2026-09-03)
- 6 platform-health collector
- spec
- docs/routines/weekly-ml-audit.md
No lint/test/build CI. Validation is local via husky pre-commit/pre-push and pnpm pre-deploy. Three workflows remain, all hand-triggered.
changesets.yml— version PRssync-public-mirrors.yml— push open-source/* to Messai-io/MESS-*, publish npm/PyPI via OIDCfit-priors-v2.yml— OOM-safe priors refit
One schema, two live environments, staging and prod; local dev points at staging since 2026-10-01 (the :54322 local stack is retired). Pooled DATABASE_URL :6543 for the app, DIRECT_URL :5432 for migrations and bulk. Staging and prod share the pooler host — writes gate on the project ref.
- papers
- 23,568 local · 23,569 prod (2026-07-06/07)
- EPD rows
- 196,522 · 25,988 modelable (2026-07-07)
- vector cols
- ResearchPaper.abstract_embedding (bge-large 1024d) · PaperChunk.embedding
- dead URLs
- STAGING_DATABASE_URL, PRODUCTION_DATABASE_URL (db.prisma.io)
Local papers/pdf-storage/objects/ is only a cache. Hydrate before any job that reads PDFs off disk.
- hydrate
- scripts/papers/hydrate-pdfs-from-r2.ts
- verify
- scripts/storage/r2-verify-backups.ts
- column
- ResearchPaper.r2Key
Token buckets for every /api/* request in the apps/web proxy, and per-route budgets in apps/api (chat, sweep, MCP tools). The store is the Marketplace database messai-ratelimit, connected 2026-10-01 (libs/shared/rate-limit/src/limiter.ts); with no store configured the limiter lets every request through, as in local dev. Whether the prod env is set was not checked from this machine. This box used to be the BullMQ queue scaffold: it never enqueued a job, its processors were TODOs and no worker host existed. It was archived on 2026-10-02 (archive/2026-10-bullmq-scaffold/).
Chat and extraction go through the Vercel AI Gateway with provider fallback gateway → gemini → groq → ollama (local). Extraction costs ~$0.05–0.10 per paper.
- extractor
- scripts/extraction/simple_value_extractor.ts
- env
- AI_GATEWAY_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY, GROQ_API_KEY
Semantic search embeds the query live via router.huggingface.co, then pgvector cosine on abstract_embedding. Batch backfill uses the Docker TEI service instead.
- call site
- libs/shared/ai-chat/src/lib/embed.ts
- env
- HUGGINGFACE_API_KEY
Gaussian-process structural causal model over the named parameter DAG. Reached from /api/ml/predict behind the hybrid opt-in via GP_SCM_SERVICE_URL. A blank value darkened it for ~7 weeks in 2026-07; the client now names that case in its fallback reason. Whether the Fly app is up and the prod env is non-blank was not checked from this machine. Trains with scikit-learn, not torch. Served pickle carries zero non-zero physics means, so no redeploy was warranted after the 2026-07-31 fit.
- served pkl
- gp-scm-fitted-named-db-2026-06-07.pkl
- trainer
- services/ml-engine/train_gp_scm.py
FastAPI + Python: batch embeddings (HF TEI), Nougat OCR, pgmpy fitted posteriors (cached JSON, no live server), NPE simulator, hierarchical priors trainers, conformal calibration. Runs from docker-compose.ml.yml with its own postgres/redis/mlflow for experiments.
- dockerfiles
- Dockerfile · .gpscm · .nougat · .npe
Request and batch paths · 26 edges
| From | To | Path | Kind |
|---|---|---|---|
| Researcher browser | apps/web | HTTPS messai.io | live request path |
| apps/web | apps/site | rewrite /about /learn … | live request path |
| apps/web | apps/lab | rewrite /lab /models … | live request path |
| apps/web | apps/api | rewrite /api/* · all writes | live request path |
| AI chat & agents | apps/api | POST /api/chat | live request path |
| apps/web | @messai/database | server reads | server-component / build-time read |
| apps/lab | @messai/database | server reads | server-component / build-time read |
| apps/api | @messai/database | reads + writes | live request path |
| @messai/database | Supabase Postgres | Prisma · pooled :6543 | live request path |
| apps/api | Cloudflare R2 | PDF bytes | live request path |
| apps/web · api | Upstash Redis | rate-limit buckets | live request path |
| apps/api | AI Gateway | chat + tools | live request path |
| apps/api | HF Inference Router | embed query | live request path |
| apps/api | GP-SCM on Fly.io | ?hybrid=true | live request path |
| apps/api | Computed JSON artifacts | JSON fallback · ISR 60s | server-component / build-time read |
| apps/web · lab · api | @messai/* shared libs | imports @messai/* | server-component / build-time read |
| apps/api | next-auth | sessions | live request path |
| Operator scripts | Supabase Postgres | sync scripts · DIRECT_URL :5432 | batch or script |
| Operator scripts | Cloudflare R2 | hydrate / migrate | batch or script |
| Operator scripts | AI Gateway | extraction LLM calls | batch or script |
| Weekly Claude routine | Computed JSON artifacts | refits → commits JSON | batch or script |
| Weekly Claude routine | GP-SCM on Fly.io | GP-SCM fit | batch or script |
| GitHub Actions | Computed JSON artifacts | fit-priors-v2 | batch or script |
| GitHub Actions | ML engine | workflow_dispatch | batch or script |
| ML engine | Supabase Postgres | batch embed → pgvector | batch or script |
| Computed JSON artifacts | apps/web | static /data/* | server-component / build-time read |
Four lanes, read top to bottom. Every script is dry-run by default and needs --apply --target to write. The scheduled part is lanes 3 and 4: the weekly Claude routine refits priors, effects, calibration and GP-SCM. Discovery, download and LLM extraction are still hand-run; the GitHub job-DAG that would automate them is a design, not code.
The only stage nobody pushes to you, so it is poll-based. Weekly step 2 discovers new DOIs since the cursor; search-gaps-openalex.ts (2026-09-03) runs gap queries into papers/acquisition/gaps-<date>.json and the admin route records a GapSearchRun.
- scripts
- search_openalex_underrepresented.py · search-gaps-openalex.ts
- source tag
- ResearchPaper.source = openalex-weekly
Bounded to 2,000 per tier per run. curl_cffi impersonates Chrome to get past MDPI/Wiley/ACS/Elsevier 403s. 2,075 no-DOI papers remain unreachable by this path.
- scripts
- download_paperscraper.py · download_curl_cffi.py · download_unpaywall.py · xml_to_pmc_pdf.py
Content-addressed by pdfHash. Since 2026-08-14 the bytes live on R2 and the local store is a cache; migrate-local-to-r2.ts / backfill-r2-keys.ts keep ResearchPaper.r2Key populated.
Source of truth for PDF bytes.
Creates or links ResearchPaper rows. Mutation gate mirrors the whole pipeline: dry-run unless --apply --target {local|staging|prod}; prod requires --expect-ref.
Hub of the schema. Carries pdfStoragePath, r2Key, aiSummary, taxonomy columns (primarySystemType TEXT), demo content, and the pgvector abstract_embedding.
Pulls the needed PDFs from R2 into the local cache before any extraction cohort runs.
Nougat runs as an overnight batch via pnpm nougat-batch. 1,444 BioC-PMC XML files are parsed by an existing parser but not yet re-extracted (cost-gated ~$70–150).
- server
- services/ml-engine/nougat_server.py
- out
- papers/nougat/<doi>.mmd
The active extractor (v1.6 orchestrator deprecated 2026-05-08). Writes JSON under papers/extracted/<aa>/<sha>/; every run must end with its sync exiting 0 (Rule §7). 9,511 papers done as of the last full run.
- fallback
- docs/extraction/provider-fallback-2026-05-09.md
- smoke
- scripts/extraction/phase6-smoke-test.ts
Five sequential jobs against the local corpus, ending in job E which syncs all four artifact types to the DB. Ollama-backed with provider fallback; manifest survives SIGTERM.
- runbook
- docs/runbooks/overnight-pipeline.md
Still present and runnable; hard-code claude-sonnet-4-6 (P1 cleanup). Populate ExtractedTableData / ExtractedFigureData / ExperimentalContext.
Lands rows with full provenance. audit-disk-vs-db.ts fails pre-deploy on disk-only drift. Launch-blocker fields (derivationMethod, uncertainty±, catholyteBuffer) are proper columns, not JSONB.
Per-paper extracted values with conditions, canonical slug, SI-normalized numericValue, verifierPassed. Staging measured 55% of rows with a NULL canonical_slug (109,378 of 199,896) on 2026-09-07. The binding constraint on modelability is canonicalization coverage, not extraction.
Typed sidecar tables linked by paperId. ConditionSet carries phAnolyte/phCatholyte/reference electrodes as columns; ReactorGeometry has ~47 typed columns.
Idempotent, dry-run by default. Volumetric densities route to their own kinds so they never pool with areal values in priors. Propagation order is staging → prod (local dev reads staging since 2026-10-01), additive and ref-checked.
- normalizer
- scripts/quality/normalize-to-si.py
- canonicalize
- canonicalize-name.ts
Steps 6–9 of refresh-all. v2 is Student-t, per-class stratified, nutpie sampler; the full fit must use the OOM-safe batched runner. Never ship a FAST-mode artifact. Both v1 and v2 must be refit together.
- batched
- training/run_priors_v2_batched.py
- benchmark
- bes-benchmark-v1 (paper-disjoint, 2026-09-02)
Effects now fit from the DB, not just the table CSV, with gap records feeding the OpenAlex gap search. KG correlation matrix is slug-keyed Pearson + Spearman.
Batch backfill of the pgvector column. If the service is down the embedding is silently skipped and nothing retries.
12/27 MFC nodes fitted after the present-parent fix; binding constraint is joint-data coverage.
Committed to main by the routine only on substantive change; a commit deploys all four Vercel projects.
runFullPrediction reads priors v2, conformal calibration, OOD detection and runtime physics validation. Non-MFC classes route to the analytical predictor; MFC keeps the data-tuned path.
Semantic search: live query embedding → pgvector cosine → hydrate rows. New extractions appear instantly; prebuilt artifacts need a redeploy or pnpm regenerate.
The design in docs/corpus-refresh-architecture.md. Not implemented: the workflows directory holds only changesets, sync-public-mirrors and fit-priors-v2. Acquisition and extraction remain manual; only refits are scheduled (weekly Claude routine).
Pipeline edges · 26 edges
| From | To | Path | Kind |
|---|---|---|---|
| OpenAlex | Acquire PDFs | new DOIs | batch or script |
| Acquire PDFs | Promote to canonical store | downloaded/ | batch or script |
| Promote to canonical store | R2 | upload · r2Key | batch or script |
| R2 | Sync papers → DB | manifest | batch or script |
| Sync papers → DB | ResearchPaper | insert / link | batch or script |
| R2 | Hydrate PDFs | pull cache | batch or script |
| Hydrate PDFs | Parse | PDFs | batch or script |
| Parse | v2 value extractor · Overnight jobs | .mmd / text | batch or script |
| Parse | Legacy Claude/Gemini scripts | ad hoc | dormant |
| v2 value extractor | Sync extractions → DB | values_v2.json | batch or script |
| Overnight jobs A–E | Sync extractions → DB | job E | batch or script |
| Legacy scripts | Extracted{Table,Figure,Paper}Data | tables · figures · context | dormant |
| Sync extractions → DB | ExtractedParameterData · sidecar tables | rows + provenance | batch or script |
| ExtractedParameterData | refresh-all.sh | reads EPD · writes slug, SI value, verifierPassed | batch or script |
| refresh-all.sh | Priors & calibration refit | steps 6–9 | batch or script |
| Priors & calibration refit | Within-paper effects + KG | then | batch or script |
| Within-paper effects + KG | GP-SCM fit | then | batch or script |
| ResearchPaper | embed_papers.py | abstracts | batch or script |
| Priors & calibration refit | Computed artifacts (git) | priors · calibration | batch or script |
| Within-paper effects + KG | Computed artifacts (git) | effects · dag | batch or script |
| GP-SCM fit | apps/api routes | pkl → Fly | batch or script |
| Computed artifacts (git) | apps/api routes | JSON fallback | server-component / build-time read |
| ExtractedParameterData | apps/api routes | DB-first reads | live request path |
| apps/api routes | UI · apps/web · apps/lab | fetch /api/* | live request path |
| embed_papers.py | UI · apps/web · apps/lab | pgvector cosine | live request path |
| Within-paper effects + KG | OpenAlex | gap records → gap search (5b) | batch or script |
Since the 2026-04-25 consolidation every package lives in this repo; the public Messai-io repos are read-only mirrors pushed by hand at release time. mess-parameters is the heavyweight: the product reads its ontology at build time and the DB syncs extracted values back into it. Dataset blobs never enter git or npm: the catalog ships manifests, the bytes live on Hugging Face Hub.
open-source/ · 9 packages · workspace:*
| Package | Version · license · files | What it is | Mirror · refs |
|---|---|---|---|
| mess-parameters | v0.3.0 · CC-BY-4.0 · 1,444 | Standardized parameter ontology and analysis tools: 835 parameters across 15 categories. data/parameter-definitions-rich.json (pinned v0.2.0 for fixtures) plus SCIENTIFIC_INTEGRITY.md (power density CoV ≈1,285%).
| @messai-io/mess-parameters github.com/Messai-io/MESS-Parameters 349 refs in main repo |
| mess-materials | v0.2.0 · CC-BY-4.0 · 104 | DFT-computed material properties for electrodes, membranes, catalysts with Materials Project provenance, Pourbaix stability, elasticity.
| @messai-io/mess-materials github.com/Messai-io/MESS-Materials 69 refs in main repo |
| mess-microbes | v0.1.0 · MIT · 41 | Curated microorganisms relevant to MES: electrogens, electrotrophs, community partners and intentional negative controls. Catalog = 28 microbes (corrected 2026-06-30).
| @messai-io/mess-microbes github.com/Messai-io/MESS-Microbes 34 refs in main repo |
| mess-datasets-catalog | v0.1.0 · CC-BY-4.0 · 612 | Catalog and classifications for open MES datasets (Zenodo, Figshare), slug-keyed to parameters and materials. Metadata only; blobs on HF Hub.
| @messai-io/mess-datasets-catalog github.com/Messai-io/MESS-datasets 6 refs in main repo |
| mess-simulations | v0.1.0 · MIT · 31 | Physics-based simulation and modeling tools for MES. | @messai-io/mess-simulations github.com/Messai-io/MESS-Simulations 4 refs in main repo |
| mess-agents | v0.1.0 · MIT · 27 | Multi-agent research orchestration framework. | @messai-io/mess-agents github.com/Messai-io/MESS-Agents 1 refs in main repo |
| mess-hypotheses | v0.1.0 · MIT · 28 | Research-gap identification and hypothesis generation. No chat tool wraps it yet (P2 gap). | @messai-io/mess-hypotheses github.com/Messai-io/MESS-Hypotheses 1 refs in main repo |
| mess-learning | v0.1.0 · CC-BY-4.0 · 22 | Educational content and calculators. | @messai-io/mess-learning github.com/Messai-io/MESS-Learning 1 refs in main repo |
| mess-methods | python · pyproject · 32 | Python package of MES methods (tests/, src/). Published to PyPI by the mirror workflow. | PyPI mess-methods github.com/Messai-io/MESS-Methods 1 refs in main repo |
The product reads packages at build time (fixtures), at seed time (materials, microbes), and at runtime (datasets manifests → HF Hub).
ParameterDefinition is edited in the DB and synced back to mess-parameters; materials and microbes are seeded from their packages.
Changesets version PRs; canonical package shape enforced by the linter. Rollback = git revert in open-source/* then re-run the mirror.
Pushes open-source/<pkg> to its Messai-io repo with a tag, then publishes. Kept on GitHub because OIDC trusted publishing only works there. Auto-triggers were stripped to hold Actions spend at $0.
MESS-Parameters, MESS-Materials, MESS-datasets, MESS-Agents, MESS-Hypotheses, MESS-Learning, MESS-Microbes, MESS-Methods, MESS-Simulations. Contributions from the public flow back via docs/contributing-from-public.md.
Large dataset blobs (PDFs, CSVs, images) referenced by download_url in each manifest; consumers verify the md5 checksum after download.
Sync, publish and consume paths · 13 edges
| From | To | Path | Kind |
|---|---|---|---|
| mess-parameters | apps/* + libs/* | fixtures at build | server-component / build-time read |
| Supabase Postgres | mess-parameters | defs + values sync | batch or script |
| mess-materials | Supabase Postgres | seed Material | server-component / build-time read |
| mess-microbes | Supabase Postgres | seed Microbe | server-component / build-time read |
| mess-datasets-catalog | apps/* + libs/* | /datasets manifests | server-component / build-time read |
| Releases · pnpm changeset | sync-public-mirrors.yml | version PR merged | batch or script |
| sync-public-mirrors.yml | github.com/Messai-io/MESS-* | git push + tag | batch or script |
| sync-public-mirrors.yml | npm · @messai-io/* | pnpm publish | batch or script |
| sync-public-mirrors.yml | PyPI · mess-methods | mess-methods only | batch or script |
| mess-datasets-catalog | Hugging Face Hub | blobs · download_url | batch or script |
| apps/* + libs/* | Hugging Face Hub | fetch + checksum | live request path |
| github.com/Messai-io/MESS-* | External researchers | clone / issues | server-component / build-time read |
| npm · @messai-io/* | External researchers | install | server-component / build-time read |
ResearchPaper (86 relation fields) is the hub; User (46), Experiment (34), Microbe (26), ParameterDefinition and ConditionSet (22 each) follow (counted 2026-10-07). ResearchPaper.abstract_embedding and PaperChunk.embedding are the two pgvector columns. Schema rule §8: any field that a UI filter, ML feature or WHERE clause touches is a typed column, never JSONB.
| Domain | Models | Names |
|---|---|---|
| Papers & corpus | 13 | ResearchPaper · PaperRelationship · PaperMergeAudit · ResearchCluster · ResearchTrend · AnomalousPaper · ResearchPaperEquation · PaperParameter · PaperMaterial · PaperMicrobe · MetagenomicPaperLink · GapSearchRun · PaperChunk |
| Extraction | 15 | ExtractedParameterData · ExtractedTableData · ExtractedFigureData · ExtractedPaperData · ExtractionJob · ExtractionProvenance · ExperimentalContext · ConditionSet · ConditionSetMergeAudit · ParameterConditionLink · ParameterObservation · ParameterProvenance · ReactorGeometry · OperatingCondition · MFCDesign |
| Parameters & knowledge | 22 | ParameterDefinition · ParameterEdge · ParameterPrior · PriorStratumAxis · ParameterTemplate · ParameterClassification · CustomField · ElectrochemicalParameter · BiologicalParameter · EnvironmentalParameter · OperationalParameter · MaterialParameter · KnowledgeNode · KnowledgeEdge · LearnedDagEdge · SymbolicLaw · SubstrateClassification · DataClassificationRule · BufferChemistryProfile · WikiEntry · WikiRevision · WikiRevisionSource |
| Materials | 7 | Material · MaterialClassification · MaterialMicrobeAffinity · MaterialPaperCrossref · ElectrodeFormulation · AiMaterialDefinition · AiGeometryRecipe |
| Microbes & biofilm | 15 | Microbe · MicrobeCytochrome · MicrobeShuttle · MicrobePerformanceMeasurement · MicrobeEngineeringEvent · MicrobeOmicsStudy · MicrobeReference · MicrobeSubstrateProductPair · MicrobeKineticConstant · EETPathwayAssignment · MetagenomicAnalysis · AiMicrobeDefinition · BiofilmSample · BiofilmCreepCurve · MicroelectrodeProfile |
| Experiments & lab | 21 | Experiment · ExperimentCollaborator · ExperimentEvent · ExperimentPaper · Run · Measurement · LabConfiguration · LabConfigurationSnapshot · LabWastewaterQueryLog · MethodologyPreset · Workflow · WorkstreamArtifact · SimulationReplay · SimulationResult · Model · KineticCurve · PolarizationPoint · EISPoint · ElectrochemicalKinetic · MESProduct · MESProductYield |
| ML & agents | 10 | Prediction · PredictionOutcome · CrossSystemPrediction · CalibrationResult · TrainedModelLineage · EvalGateRun · EvalResult · AgentRun · Hypothesis · IntegrationMapping |
| Datasets | 5 | Dataset · DatasetMeasurement · DatasetCurvePoint · DatasetParameter · DatasetMaterial |
| Users & admin | 14 | User · Account · Session · VerificationToken · PasswordResetToken · Permission · Team · Project · BetaSignup · SurveyResponse · SharedView · AuditLog · ApiKey · PageComment |
| Feedback | 4 | Feedback · FeedbackAnalytics · FeedbackNotification · WastewaterFeedback |
What the diagrams draw dotted or red. Severity follows the platform docs; states were re-checked against CLAUDE.local.md and the working tree on 2026-09-03, and the BullMQ row was closed on 2026-10-07 after its 2026-10-02 archive. Resolved items (chat bifurcation, force-dynamic audit, priors JSON validity, the BullMQ scaffold) are listed separately.
| Sev | Gap | Current state | What unblocks it |
|---|---|---|---|
| P0 | Extraction & acquisition are not scheduled | weekly_pipeline.sh, simple_value_extractor.ts run by hand; corpus-refresh.yml is a design (2026-05-30) with no code. Only refits run weekly (Claude routine). | Land the GitHub job-DAG or a scheduled routine step for discover → acquire → extract with the budget cap. |
| P1 | Canonicalization coverage, not extraction, bounds modelability | 80.7% of EPD rows had a NULL canonical_slug on local (2026-06-30; 84,243 numeric rows recoverable); staging measured 55% (109,378 of 199,896) on 2026-09-07. Modelable 25,988 (2026-07-07) vs ceiling ≈40–50k. | Extend alias maps / canonicalize-name.ts; re-run refresh-all steps 2–3. |
| P1 | Prod FK under-population | ~33k ExtractedParameterData rows in prod lack parameterDefinitionId (affects FK/DAG read paths, not the modelable count). | Ref-gated backfill, staging first. |
| P1 | Papers ↔ Materials / Microbes orphaned | anodeMaterials / cathodeMaterials are strings; PaperMaterial / PaperMicrobe junctions mostly empty. | Backfill junctions from extraction; add /api/materials/[id]/papers. |
| P1 | Unread corpus | 1,444 BioC-PMC XML files parsed but not re-extracted (~$70–150); 2,075 no-DOI papers unreachable; MDPI/Wiley/ACS/Elsevier 403s. | Cost-gated v2 run over XML; curl_cffi path for no-DOI URLs. |
| P1 | Confidence and integrity caveats not surfaced | confidence columns exist; UI shows point estimates. SCIENTIFIC_INTEGRITY.md (power-density CoV ≈1,285%) has no UI banner. | Add confidence to /api/parameters/* responses; collapsible callout on /parameters/*. |
| P1 | GP-SCM fits are data-limited | 12/27 MFC nodes fitted; new fits are interpolation artifacts (energy_efficiency flat at 43.02%). toc_removal has 1 joint observation. | Targeted paid re-extraction for biofilm/biomass nodes; do not lower --min-samples. |
| P2 | No feedback / retraining loop | Prediction table exists; no /lab widget records {predicted, actual}; no drift-triggered refit. | Feedback widget → POST /api/predictions; weekly regenerate if drift > 5%. |
| P2 | Chat tools missing for 4 packages | query-methods, query-hypotheses, query-datasets, query-learning not wrapped. | One tool per package in @messai/ai-chat. |
| P1 | GP-SCM can be silently dark in prod | A blank GP_SCM_SERVICE_URL on Vercel disabled the pillar for ~7 weeks in 2026-07 while it looked wired (libs/shared/ml/src/gp-scm-client.ts). Whether prod has a non-blank value today was not verified from this machine. | Check the Vercel env on messai-api; the client now reports the blank-value case as its fallback reason. |
| P1 | Weekly routine output never landed | No npe-health.json exists anywhere in the tree, so either the routine has never run with --commit or its commit step has not fired. The routine script itself is present and gained step 5b on 2026-09-03. | Run bash scripts/routines/weekly-ml-audit.sh --commit once by hand; confirm the scheduled session exists in Claude Code on the web. |
| P2 | Dev hygiene | Two NextAuth Prisma adapters installed; react-query in devDependencies. (Legacy extractors already read ANTHROPIC_MODEL with a claude-sonnet-4-6 default — that item from the 2026-05 docs is resolved.) | Pick @auth/prisma-adapter; move dependency. |
- Chat bifurcation — one canonical
/api/chat(2026-05-12). - Priors JSON validity — v1 and v2 parse cleanly (commit 715b63e96).
- Per-class ML routing — 6 non-MFC classes route to the analytical predictor (commits 4778f8589, bef7acedd); lab UI still passes proxy inputs.
- Harmonization quick wins merged (PR #501): +434 modelable; propagated to staging and prod 2026-07-06/07.
- GP-SCM all-parents dropna — present-parent fit measured 9 → 12 nodes (2026-07-31, branch fix/gpscm-present-parent-fit, unmerged).
- BullMQ queue scaffold archived (2026-10-02,
archive/2026-10-bullmq-scaffold/): it never enqueued a job and had no worker host; background work uses Nextafter()or Vercel Queues.