File Processing Implementation

Overview

Document ingestion is handled by the document transcoder in src/transcode/ — document.hpp / document.cpp. It decomposes an uploaded PDF or spreadsheet into an ordered list of LLM-consumable parts (text and/or images), strategy “text first, image (and OCR) fallback.” For the conceptual walkthrough see File Processing.

The document transcoder decomposes an uploaded document into text and image parts that a vision-capable LLM reads during the chat turn. (verify upload wiring against the current HTTP layer.)

Supported formats & build gates

Format is sniffed from magic bytes (detect_format), then routed:

Format	Library	CMake gate	Default	Notes
PDF	PDFium	`MIROBODY_ENABLE_PDF`	off	text layer per page; scans → image
.xlsx	xlnt	(auto when xlnt found)	on where xlnt present	one Markdown table / sheet
.xls	vendored libxls	`MIROBODY_ENABLE_XLS`	off	legacy BIFF
.csv	built-in (RFC 4180)	—	always	one Markdown table
OCR	Tesseract + Leptonica	`MIROBODY_ENABLE_OCR`	off	requires `MIROBODY_ENABLE_PDF`

With a gate off, the format still sniffs but process() throws DocumentError("... not built").

cmake -DMIROBODY_ENABLE_PDF=ON -DMIROBODY_ENABLE_OCR=ON -DMIROBODY_ENABLE_XLS=ON ...

The API

namespace mirobody { namespace document {

enum class Format { Unknown, Pdf, Xlsx, Xls, Csv };

struct Part {
    enum class Kind { Text, Image };
    Kind              kind = Kind::Text;
    std::string       text;    // Text: UTF-8 / Markdown
    image::Transcoded image;   // Image: already vision-compliant
    int               page = 0;// 1-based source page / sheet
};

struct Document { Format format; std::vector<Part> parts; };

struct Options {
    image::Limits image_limits = image::QwenLimits;  // Qwen / Gemini / Gpt presets
    int  raster_dpi        = 150;   // density for a rasterized scanned page
    int  pdf_text_threshold = 8;    // < this many chars on a page => treat as a scan
    bool ocr_enabled       = true;  // honored only in a MIROBODY_ENABLE_OCR build
    std::string ocr_lang   = "eng"; // e.g. "eng+chi_sim"
    std::string ocr_datapath;       // dir with <lang>.traineddata; "" => TESSDATA_PREFIX
    std::size_t max_pages  = 0;     // 0 = no cap
};

class Transcoder {
public:
    explicit Transcoder(Options opts = Options());
    Document process(const std::string& input) const;         // throws DocumentError
    static std::string to_markdown(const Document& doc);       // parts -> one Markdown blob
    static Format detect_format(const std::string& input);     // cheap magic-byte sniff
};

}}

What each path emits

PDF, text page → a Text part with the extracted text layer.
PDF, scanned page (text below pdf_text_threshold) → rasterized at raster_dpi, PNG-encoded, run through image::Transcoder to fit image_limits, emitted as an Image part. In a MIROBODY_ENABLE_OCR build the same raster is also OCR’d and the recovered text appended as a Text part.
.xlsx / .xls → one Text part per worksheet, a GitHub-flavored Markdown table (## <sheet title> + table). Never an image.
.csv → a single Text part (RFC 4180 parse → Markdown table; UTF-8 BOM stripped).

to_markdown() flattens a Document: text parts inline, image parts as a ![page N image](...) placeholder line.

max_pages caps pages/sheets; when it trips, a trailing “(remaining … omitted: max_pages reached)” note is appended so nothing is silently dropped.

Threading (PDF)

PDFium’s library init/teardown is process-global and not thread-safe, and a loaded document may not be touched concurrently. The PDF path serializes all PDFium work behind an internal mutex and owns the one-time global init, so Transcoder is safe to call from multiple threads — PDF work simply isn’t parallel while a PDF is being processed. CSV / xlsx have no such constraint.

Try it — the `document` CLI

The cli/document.cpp tool (built when MIROBODY_BUILD_TOOLS is ON) exercises the transcoder end to end:

document extract <in> [<out.md>] [--target qwen|gemini|gpt] [--dpi N] \
                      [--no-ocr] [--ocr-lang L] [--ocr-data DIR] [--max-pages N]

# a scanned PDF, Gemini vision preset, English+Simplified-Chinese OCR
./build/cli/document extract report.pdf out.md --target gemini --ocr-lang eng+chi_sim

Image parts (rasterized scanned pages) are written next to <out>; text parts land in the Markdown. --target picks the vision preset for rasterized pages (default qwen).

Units

Values pulled from documents can be normalized to UCUM with the units engine (src/fhir/units/) — see Data Mapping.

Tests

tests/transcode/document_test.cpp, tests/transcode/file_test.cpp, tests/transcode/image_test.cpp, and tests/fhir/units_test.cpp (Catch2). Run the transcoder subset:

./build/tests/mirobody_tests "[transcode]"

Writing extracted values back to the FHIR store as coded Observations (the document → indicator → FHIR pipeline) is in progress — the transcoder and UCUM normalization ship; terminology mapping and write-back are planned (src/fhir/README.md phases 3–4). (verify)

Next steps

File Processing (concept)

The pipeline at a glance

Data Flow

Where uploads fit

Data Mapping

UCUM normalization

Testing

Run the transcoder tests

​Overview

​Supported formats & build gates

​The API

​What each path emits

​Threading (PDF)

​Try it — the document CLI

​Units

​Tests

​Next steps

File Processing (concept)

Data Flow

Data Mapping

Testing

Overview

Supported formats & build gates

The API

What each path emits

Threading (PDF)

Try it — the `document` CLI

Units

Tests

Next steps