File Processing

Overview

Mirobody can ingest health documents — lab reports, spreadsheets, scanned records — and turn them into content a chat turn can carry. The document transcoder in src/transcode/ (document.hpp / document.cpp) does that work, the document counterpart to the image transcoder. Its strategy is text first, image (and OCR) fallback.

Note

The transcoder decomposes a document into text and image parts; a vision-capable LLM then reads them during the chat turn.

Supported formats & build gates

The container format is sniffed from the leading magic bytes, then routed to a handler. Three of the four handlers are behind a CMake gate, and the defaults are conservative:

Format	Library	CMake gate	Default	Output
PDF	PDFium	MIROBODY_ENABLE_PDF	off	Text layer per page; scanned pages → image
.xlsx	xlnt	(auto when xlnt is found)	on where xlnt is present	One Markdown table per worksheet
.xls (legacy BIFF)	vendored libxls	MIROBODY_ENABLE_XLS	off	One Markdown table per sheet
.csv	built in (RFC 4180)	—	always	A single Markdown table
OCR	Tesseract + Leptonica	MIROBODY_ENABLE_OCR	off	Recovered text for a scanned page (requires MIROBODY_ENABLE_PDF)

cmake -DMIROBODY_ENABLE_PDF=ON -DMIROBODY_ENABLE_OCR=ON -DMIROBODY_ENABLE_XLS=ON ...

With a gate off, the format is still detected but process() throws DocumentError("... not built") — nothing is silently mis-handled.

Note

An OCR build also needs Tesseract’s language data: one <lang>.traineddata per language on ocr_datapath (or TESSDATA_PREFIX) — so eng+chi_sim means two files.

Note

Images uploaded on their own are handled by the sibling image transcoder (src/transcode/image.hpp), which fits an image to a target vision model’s input limits. Audio and genetic files are not handled by this transcoder.

The pipeline

Sniff the format

detect_format() reads the magic bytes: %PDF- → PDF, PK\x03\x04 → xlsx (any OOXML zip), the OLE2 signature → legacy .xls, anything else non-empty → CSV.

PDF — text layer first

For each page, the embedded text layer is extracted as a text part. A page whose extractable text is below pdf_text_threshold (default 8 characters — i.e. a scan) is treated as an image instead.

Scanned page — rasterize → vision-compliant image

A scanned page is rasterized at raster_dpi (default 150), repacked to PNG, and run through image::Transcoder so it satisfies the target vision model’s limits (Qwen-VL by default; Gemini / GPT limits are also available). It is emitted as an image part the chat turn inlines for the vision model.

OCR (optional)

In a MIROBODY_ENABLE_OCR build, the same raster is also run through Tesseract and the recovered text appended as a text part — so a scan yields both the image and its OCR text. OCR language defaults to eng (e.g. eng+chi_sim for mixed docs).

Spreadsheets — always text

Each worksheet (xlsx / xls) or the whole CSV is rendered to a GitHub-flavored Markdown table; spreadsheet handling never produces image parts.

The result is a Document: an ordered list of Parts, each either Text (UTF-8 / Markdown) or Image (an already vision-compliant image), tagged with the 1-based source page / sheet.

The API

namespace mirobody { namespace document {

enum class Format { Unknown, Pdf, Xlsx, Xls, Csv };

struct Part {
    enum class Kind { Text, Image };
    Kind              kind = Kind::Text;
    std::string       text;    // Text: UTF-8 / Markdown
    image::Transcoded image;   // Image: already vision-compliant
    int               page = 0;// 1-based source page / sheet
};

struct Document { Format format; std::vector<Part> parts; };

struct Options {
    image::Limits image_limits = image::QwenLimits;  // Qwen / Gemini / Gpt presets
    int  raster_dpi        = 150;   // density for a rasterized scanned page
    int  pdf_text_threshold = 8;    // < this many chars on a page => treat as a scan
    bool ocr_enabled       = true;  // honored only in a MIROBODY_ENABLE_OCR build
    std::string ocr_lang   = "eng"; // e.g. "eng+chi_sim"
    std::string ocr_datapath;       // dir with <lang>.traineddata; "" => TESSDATA_PREFIX
    std::size_t max_pages  = 0;     // 0 = no cap
};

class Transcoder {
public:
    explicit Transcoder(Options opts = Options());
    Document process(const std::string& input) const;         // throws DocumentError
    static std::string to_markdown(const Document& doc);       // parts -> one Markdown blob
    static Format detect_format(const std::string& input);     // cheap magic-byte sniff
};

}}

to_markdown() flattens a Document for callers (and the CLI) that want a single string: text parts inline, image parts as a ![page N image](...) placeholder line.

Tip

max_pages caps pages/sheets; when it trips, a trailing “(remaining … omitted: max_pages reached)” note is appended so nothing is silently dropped.

Threading (PDF)

PDFium’s library init/teardown is process-global and not thread-safe, and a loaded document may not be touched concurrently. The PDF path serializes all PDFium work behind an internal mutex and owns the one-time global init, so Transcoder is safe to call from multiple threads — PDF work just isn’t parallel while one is in flight. CSV / xlsx have no such constraint.

Units get normalized to UCUM

Values pulled out of documents and charts are normalized by the terminology engine in src/fhir/units/: a free-text “value + unit” string becomes a canonical UCUM unit plus the LOINC PROPERTY family that disambiguates it. This is pure local computation — no DB, no embedding API.

"MG/DL"

"mg/dL"

"毫摩尔每升"

"mmol/L"

"mmHg"

"mm[Hg]"

"<5.6 mg/dL"

value 5.6 · unit "mg/dL" · comparator "<" A comparator is split off rather than dropped, so a bounded result stays bounded.

The pipeline: NFKC-lite (full-width forms, superscripts) → symbol fold → alias lookup → annotation strip → greedy morpheme tokenize-compose. Case is preserved (UCUM is case-sensitive). Multilingual input is supported: en, zh, ja, ko, ru, de, fr, es.

Try it — the document CLI

The cli/document.cpp tool (built when MIROBODY_BUILD_TOOLS is ON) exercises the transcoder end to end:

document extract <in> [<out.md>] [--target qwen|gemini|gpt] [--dpi N] \
                      [--no-ocr] [--ocr-lang L] [--ocr-data DIR] [--max-pages N]

# a scanned PDF, Gemini vision preset, English+Simplified-Chinese OCR
./build/document extract report.pdf out.md --target gemini --ocr-lang eng+chi_sim

Image parts (rasterized scanned pages) are written next to <out>; text parts land in the Markdown. With no <out.md>, the Markdown goes to stdout.

What happens to the data

The transcoded parts are read by the agent during a chat turn: uploaded files are stored per-user and the agent lists / reads them through the list_files and read_file MCP tools, feeding text parts and vision-compliant image parts to the LLM.

Caution

At least one LLM key (or on-device Gemma) is required to make sense of documents — uploading and storing a file works without one, but extracting meaning from PDFs / images needs a model. Text-only formats (CSV / xlsx) render to Markdown with no model.

Note

Writing extracted lab values back into the FHIR store as coded Observations (the document → indicator → FHIR pipeline) is in progress: UCUM unit normalization and the RESTful FHIR store are shipped; terminology resolve (SNOMED CT / LOINC / RxNorm) is still landing and automatic write-back is planned (src/fhir/README.md phases 3–4).

Tests

Covered by Catch2: tests/transcode/document_test.cpp, tests/transcode/image_test.cpp, tests/transcode/file_test.cpp, and tests/fhir/units_test.cpp. Run the transcoder subset:

./build/tests/mirobody_tests "[transcode]"

See Testing.

Next steps

Data Flow

How uploads fit alongside vendor and on-device data

→

Data Mapping

Domains, UCUM units, and FHIR output

→

Vendor System

Server-to-server health-data sources

→

API Reference

The OpenAI-compatible /v1 surface

→