Data Flow

Health data reaches the engine through three intake paths: a provider that Mirobody pulls or that pushes a webhook, a batch of samples the client app reads on the device and uploads, and a document you upload and an LLM reads. They differ in almost everything except the destination — every reading ends up in the same PostgreSQL tables, keyed by user and by indicator name. From there a scheduled pass turns raw readings into daily summaries, a second pass mines them for insights, and during a chat turn the agent reads them back through a single tool.

Provider pull or webhook scheduled polling, or a vendor POST via the normalized write

On-device batch import the client app POSTs samples read on the device via the normalized write

File extraction a report read by a vision LLM straight to th_series_data

write

series_data + th_series_data raw readings and summaries, one row per reading PostgreSQL, scoped by user_id

read + write back

Aggregation and insights scheduled passes that read the store and write back into it summaries every few minutes · insights every 6 h

queried by

The agent search, read and aggregate in one call query_health_indicators

Three intake paths, one store. Paths 1 and 2 share the same normalized write; path 3 writes extracted readings straight into the summary table.

Path 1 — provider pulls and webhooks

A source with a server API is polled on a schedule; a source that can push is left alone until its webhook arrives. Which route applies is the provider’s own declaration: one that needs polling gets a scheduled task at startup, one that only pushes gets none. The polling interval and the distributed-lock duration are configured per provider, defaulting to hourly with a thirty-minute lock — see Pulse Provider System for the per-provider values.

Webhooks enter at POST /api/v1/pulse/{platform}/webhook (or /{platform}/{provider}/webhook). Pulls and webhooks converge on the same normalized write, and the pull path takes no HTTP round trip — it hands the data in in-process.

On a poll, the provider loads every linked user’s credentials, fetches from the vendor per user, drops whatever it recognises as already processed, and hands the rest to the write path.

Path 2 — on-device batch import

An on-device health store has no server API to pull, so the client app reads the samples locally and POSTs them in batches. Apple Health is this channel’s main case, but the channel is not Apple-specific: the type vocabulary is the Flutter health plugin’s cross-platform names (the same names hold on iOS HealthKit and Android Health Connect), and it also carries twelve body-composition scale fields. Three endpoints take them, and both /apple/* and /api/v1/pulse/apple/* answer, because an uploader may point at either mount:

Endpoint	Body	What it does
POST /apple/health	{ request_id, metaInfo, healthData[] }	The main import. Accepts Content-Encoding: gzip — a gzipped JSON body, not Apple’s export archive.
POST /apple/statistics	{ metaInfo, statistics[] }	Client-side pre-aggregated sum / average / minimum / maximum / mostRecent per grouping, written straight to th_series_data.
POST /apple/cda	{ request_id, metaInfo, cdaData[] }	Clinical Document Architecture documents.

Samples are mapped by type onto standard indicator names. Two behaviours are silent, and a client author needs to know both:

• A sample whose type is not in the mapping is discarded, with a warning naming the type, the UUID and the value. Nothing is stored under a guessed name.
• A sleep-stage sample is duplicated into a second total-sleep record, so total sleep exists as a first-class reading rather than being recomputed by every reader.

A successful write triggers an incremental aggregation immediately, so the client sees fresh summaries without waiting for the scheduled pass; a failure there is only logged, because the scheduled pass is the safety net.

Note

The CDA endpoint is wired end to end, but it produces no readings yet: a CDA upload is accepted and nothing is written. Don’t build on it.

Path 3 — file extraction

Uploading a lab report is not a data-source integration; it is an LLM reading a document. A PDF or an image is turned into text, the text is handed to a model with a JSON schema, and the indicators the model returns are written as readings. The mechanics of the extraction are on File Processing; what matters here is where it lands.

Each extracted indicator is written straight into th_series_data, with source_table set to 'th_files' and source_table_id carrying the file key — which is how a reading stays traceable back to the page it came from. The unit and the reference range do not get their own columns: they are written as JSON into the encrypted comment. The worker then materialises any newly seen indicator names and backfills their embeddings, so semantic search becomes available after the write rather than during it.

Note

This path is the one that produces free-text indicator names — whatever the report called the row, in whatever language. They are stored as-is rather than forced into the registry, and made findable afterwards by semantic search. See Health Indicators.

One normalized model

Paths 1 and 2 converge on the same normalized write, which does five things per record before anything reaches the database:

1. 1
   Canonicalise the indicator name A case-insensitive match against the registry, so bloodglucoses becomes bloodGlucoses. An unrecognised name is passed through unchanged rather than dropped.
2. 2
   Convert the unit The value is converted into the indicator's standard unit. A conversion with no rule keeps the original value *and* the original unit — it never silently relabels.
3. 3
   Validate the range The normalised value is checked against the indicator's value range. A failure does not delete the row: its task_id is set to filtered_out_of_range so reads can hide it and an audit can still find it.
4. 4
   Resolve the wall clock The record's timezone, or the user's configured timezone, or UTC. Summary rows are converted so th_series_data.start_time is the user's local wall clock, not UTC.
5. 5
   Classify by data type A summary indicator becomes a summary row, a series indicator becomes a series row, and an indicator that is both becomes both.

That last step is a fork rather than an either/or: an indicator can be both a summary and a series indicator, in which case the same reading is written to both tables.

Two tables

Table	Holds	Uniqueness	Delete
series_data	Raw timestamped readings — the minute-by-minute stream	(user_id, indicator, source, time)	Physical. There is no deleted column.
th_series_data	Summary and mixed readings, plus everything aggregation and file extraction produce	(user_id, indicator, start_time, end_time)	Soft, via deleted = 1.

The two writes on paths 1 and 2 are upserts, so replaying the same batch is a no-op rather than a duplicate; path 3 skips on conflict, so re-uploading the same file neither duplicates nor updates the rows already there. th_series_data also carries fhir_id (looked up from the coding registry — see Health Indicators), fhir_mapping_info holding the unit as JSON, and a comment encrypted inside the database.

There is a third mode for the case an upsert cannot express: a client that re-uploads a corrected window. When a batch’s metaInfo.taskId looks like repair-<uuid> and carries windowFrom / windowTo, the engine sweeps in-window rows the batch did not re-confirm — hard-deleting from series_data, soft-deleting from th_series_data. Rows written by an earlier batch of the same repair are protected, so a repair split across several uploads converges instead of eating itself. An incomplete window skips the sweep and only the upsert applies.

Aggregation

Raw readings are not what a question like “how did I sleep last month” wants. A scheduled pass refreshes daily summaries incrementally every few minutes: it remembers where it stopped and aggregates only what changed since, so freshly written data becomes queryable by day almost immediately (with a cold-start fallback of the last 24 hours when there is no cursor).

The rules are not a hand-maintained list; they are generated from the registry. Every series indicator that declares aggregation methods yields one rule per method, naming the target daily{Method}{Indicator} — so heartRates with ['avg', 'max', 'min'] yields dailyAvgHeartRates, dailyMaxHeartRates, dailyMinHeartRates. Adding an aggregate means adding a method to an indicator definition. The supported methods are avg, max, min, total, count, last, first, stddev, variance, median and p95.

One rule breaks the day boundary, and it has to. Sleep does not fit inside 00:00–24:00, so sleep indicators are grouped on an 18:00-to-18:00 window in the user’s own timezone: a night that starts at 23:00 on the 1st and ends at 07:00 on the 2nd belongs to one day, not two halves. The grouping and the query use the same definition.

Derived indicators that need more than one source get a second, slower pass (every 6 hours). A bounded backfill is available at POST /api/v1/manage/aggregate/recalculate-range — capped at 30 days when no user_id is given, because an all-users range grows with the active user count.

Insights

The insight engine runs every 6 hours. Per user it computes a baseline, builds a profile of which indicator categories have enough density, picks the recipes that profile can support, and runs only those. Each recipe declares which indicator categories it needs, how many days of density and overlap, and how long it must wait before reporting again — so a user with two weeks of heart rate and no glucose never runs the glucose recipe.

Six recipes ship:

Recipe	Category	Requires	Density	Cooldown
multi_signal_deterioration	anomaly	heartRate	14 days, 10 overlapping	3 days
single_sustained_anomaly	anomaly	— (any of seven optional)	14 days	7 days
long_term_trend	trend	—	21 days	14 days
recovery_trend	recovery	heartRate	14 days	7 days
weekday_weekend_pattern	pattern	steps	21 days	30 days
glucose_control	anomaly	bloodGlucose	14 days	7 days

Results are persisted and exposed at GET /api/v1/pulse/user/insights, with feedback at POST /api/v1/pulse/user/insights/{insight_id}/feedback. The cooldown is what keeps the same observation from being reported every six hours.

Reading data back

During a chat turn the agent does not query SQL. It gets one tool, query_health_indicators, with search, read and aggregate collapsed into a single call:

1. 1
   keywords are matched semantically embedded first, then recalled by vector similarity scoped to that user's rows
2. 2
   indicators are matched exactly the names a previous call returned, so the second question does not re-run the search
3. 3
   Readings come back in the same response the newest rows per indicator, with soft-deleted rows skipped
4. 4
   aggregate answers trend questions server-side statistics or a day/week/month bucket, instead of pulling raw readings the model then has to add up
5. 5
   No match returns the catalog what this user actually has — so the next call picks from reality rather than re-guessing keywords

Rows that came from a file carry a file_key back in the result, so the agent can open the source document. Every result also carries its system / code identity: the same code means the same test, whatever the names — which is what makes a Garmin reading and a lab row comparable.

The tool runs as the injected caller identity and refuses to run without a user, which is what makes every health read user-scoped by construction rather than by remembering a WHERE clause.

Note

It is discovered at runtime like any other tool, so the allow/deny lists apply: DISALLOWED_TOOLS_DEEP: [query_health_indicators] removes the read path from that agent without touching the ingest side. On the MCP surface it is also data-gated — an account with no health rows is not offered it at all. See Built-in Tools.

Scope of the FHIR layer

Caution

FHIR is used here as a vocabulary layer, for coding and retrieval: a fhir_id column and an embedding index over LOINC / SNOMED CT / RxNorm concepts, described on Health Indicators. The engine exposes no FHIR REST store — there are no /fhir/* routes and no resource table. Readings are rows in series_data and th_series_data, and the only way in is the three intake paths above.

Mapping to Mirobody Cloud

Mirobody Cloud organises writes by data shape; self-hosting organises them by mechanism. The two sides are separate codebases and readings do not land in the same columns, so use the table rather than assuming a shared name:

Cloud /v1	Equivalent in the self-hosted engine
POST /v1/data (structured readings you already hold)	Paths 1 and 2 above. Cloud does not host device OAuth, so the provider-platform half has no Cloud equivalent: vendor authorisation and the webhook pipeline stay in your own product, and you write to /v1/data once you hold the samples.
POST /v1/files	The file parsing of path 3.
POST /v1/standardize	The library surface’s parse_file() and resolve(), or the mirobody parse / mirobody resolve CLI — see The Engine as a Library.
Cloud only	The four retention tiers, Subject (user) multi-tenancy, the 500-record request cap, and source values api / extract / upload / consolidation.
Self-host only	The minute-by-minute raw stream in series_data, the daily{Method}{Indicator} aggregates, the six insight recipes, and the repair window.

The field names differ too: a Cloud reading carries parsed_value, parsed_unit, loinc_code, canonical_name and fhir_resource_id, while this engine converts to a standard unit and carries fhir_id and fhir_mapping_info.

Note

One behaviour differs between the two sides. On Cloud, all four writers (/v1/data, /v1/files, /v1/standardize, and readings consolidated from stored conversations) converge on one standardization pipeline, so readings extracted from a file also carry a code and a UCUM unit. In this engine only paths 1 and 2 take the normalized write; path 3 does not: it keeps the report’s own free-text indicator names and relies on semantic search to reconcile them afterwards. Standardization is therefore identical on both sides for structured readings, and not identical for file extraction.

Next steps

Pulse Provider System

How path 1 is built: the base class, link types, scheduled pulls

→

File Processing

How path 3 turns a PDF into readings

→

Health Indicators

The registry, unit conversion, and semantic search

→

Architecture Overview

Where the Pulse data plane sits in the whole engine

→

The source for this step lives in mirobody/pulse/.