Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion docs/src/content/docs/bindings/javascript.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ Ruby readings remain searchable annotations whose `anchor` points back to the
base-text range.

```ts
import { getMdiTextBlocks, sourceSpansForTextRange } from "@illusions-lab/mdi";
import { getMdiTextBlocks, resolveMdiSourceSpan, sourceSpansForTextRange } from "@illusions-lab/mdi";

const result = getMdiTextBlocks("# 題\n\n{東京|とうきょう}");
const paragraph = result.blocks[1];
Expand All @@ -69,13 +69,25 @@ const match = { start: "2:1", end: "2:3" } as const;
console.log(paragraph.text); // 東京
console.log(paragraph.annotations[0].text); // とうきょう
console.log(sourceSpansForTextRange(paragraph, match)); // UTF-8 source spans
console.log(resolveMdiSourceSpan("# 題\n\n{東京|とうきょう}", { startByte: 8, endByte: 14 }));
```

`sourceMap.synthetic` identifies separators added by the projection, such as
table tabs and row newlines; these deliberately produce no source span.
`parseMdiTextPosition`, `formatMdiTextPosition`, and `formatMdiTextRange` are
stateless helpers for the canonical coordinate spelling.

`resolveMdiSourceSpan(source, span)` performs the inverse lookup in Rust. The
span is half-open UTF-8 bytes and must use uint32 values in source order, stay
within the source, and end on code-point boundaries. It returns ordered
`blockText` and zero-based `annotation` matches plus `complete`, `partial`, or
`none` coverage. Ruby base/readings are independent channels. A match is
`exact` only when its full forward coverage equals the input; otherwise it is
`overlap`. Empty spans return no caret-like neighbor. Structural delimiters,
synthetic separators, and unmapped text produce no invented canonical range,
so forward/reverse mapping is not promised to be bijective across annotations,
multi-to-one tokens, partial graphemes, discontinuities, or unmapped text.

## Choose the export level

The one-argument EPUB and DOCX calls are synchronous Rust baseline exports:
Expand Down
9 changes: 9 additions & 0 deletions docs/src/content/docs/bindings/rust.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,15 @@ match render_pdf(source, &PdfOptions::default()) {

`MDI_IR_VERSION` and `MDI_SPEC_VERSION` are `&'static str` constants exported directly — check them if you're persisting a `ParseOutput` and reloading it later, the same way any other binding must. `SourceSpan { start_byte: u32, end_byte: u32 }` is a half-open UTF-8 byte range, exactly as described in [Diagnostics and UTF-8 source spans](/core/diagnostics/) — being in Rust doesn't change the unit; it's still bytes, not `char` indices, because `str` in Rust is itself UTF-8 bytes and indexing by anything else would require an extra pass every binding would have to pay for.

For searchable canonical text, use `get_mdi_text_blocks(source)`. Its inverse,
`resolve_mdi_source_span(source, span)`, validates ordering, bounds, and UTF-8
boundaries, then returns all maximal block-text and annotation grapheme ranges.
Coverage is `Complete`, `Partial`, or `None`; matches are `Exact` only when the
complete forward coverage equals the input. Empty spans return no matches, and
structural, synthetic, or unmapped bytes do not gain invented ranges. Ruby's
separate channels and multi-to-one/discontinuous mappings mean round trips are
not generally bijective.

## Current implementation status

Parsing (`parse_document`/`parse_output`), serialization (`serialize_mdi`), and every renderer (`render_html`, `render_text_format`, `render_epub`, `render_docx`, `render_pdf`) are implemented today, at the "baseline" level described on [Rust Core API status](/core/rust-api/#not-yet-implemented). There is no separate `validate`/`normalize` API distinct from `parse_output`/`serialize_mdi` — see that same page for exactly what's missing.
Expand Down
13 changes: 12 additions & 1 deletion docs/src/content/docs/core/rust-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,24 @@ This page lists only symbols present in [`mdi-core/src/lib.rs`](https://github.c

## Public data types

`ParseOutput`, `ParserCapabilities`, `Diagnostic`, `DiagnosticSeverity`, `SourceSpan`, `Document`, `Frontmatter`, `FrontmatterEntry`, `MdiTextBlocksResult`, `MdiTextBlock`, `MdiTextPosition`, `MdiTextRange`, `MdiTextSourceMap`, `MdiTextSourceRun`, `MdiTextAnnotation`, `PdfOptions`, `EpubCover`, `ResolvedExportProfile` and its nested profile/Chromium print types (current-generation API); `MdiSyntaxDocument`, `MdiBlock`, `PagebreakVariant`, `Inline`, `RubyReading` (the older, `parse_mdi_syntax`-only shape — `Inline`/`RubyReading` are also reused internally to build the current-generation `Document`'s MDI nodes, but their `serde` output is what appears inside `Document.children`, not `MdiSyntaxDocument`).
`ParseOutput`, `ParserCapabilities`, `Diagnostic`, `DiagnosticSeverity`, `SourceSpan`, `Document`, `Frontmatter`, `FrontmatterEntry`, `MdiTextBlocksResult`, `MdiTextBlock`, `MdiTextPosition`, `MdiTextRange`, `MdiTextSourceMap`, `MdiTextSourceRun`, `MdiTextAnnotation`, `MdiSourceSpanTextResolution`, `MdiSourceSpanTextMatch`, `MdiSourceSpanCoverage`, `MdiSourceSpanRelation`, `MdiSourceSpanResolutionError`, `PdfOptions`, `EpubCover`, `ResolvedExportProfile` and its nested profile/Chromium print types (current-generation API); `MdiSyntaxDocument`, `MdiBlock`, `PagebreakVariant`, `Inline`, `RubyReading` (the older, `parse_mdi_syntax`-only shape — `Inline`/`RubyReading` are also reused internally to build the current-generation `Document`'s MDI nodes, but their `serde` output is what appears inside `Document.children`, not `MdiSyntaxDocument`).

Use `get_mdi_text_blocks(source)` or `get_mdi_text_blocks_json(source)` for the
Rust-owned plaintext search projection. It returns source-order blocks with
one-based Unicode-grapheme positions, UTF-8 source-map boundaries, ruby reading
annotations, and the same document/diagnostic envelope as `parse_output`.

Use `resolve_mdi_source_span(source, span)` to map a validated half-open UTF-8
`SourceSpan` back to maximal canonical grapheme ranges. It returns block text
before zero-based annotation channels in deterministic block order. Coverage
is `Complete`, `Partial`, or `None`, and a range is `Exact` only when its full
forward source coverage equals the requested span. Reversed, out-of-bounds, or
non-code-point-boundary inputs return `MdiSourceSpanResolutionError`; an empty
span is valid and has no matches. Pure structural, synthetic, and unmapped
bytes do not create ranges. Ruby's two channels, multi-to-one tokens, partial
graphemes, discontinuous mappings, and unmapped text mean this is not a general
inverse of every forward lookup.

## Not yet implemented

These exist as concepts in `ARCHITECTURE.md`/`SYNTAX.md` but have **no corresponding function in `mdi-core` today** — don't assume they exist because the architecture diagram mentions the concept:
Expand Down
13 changes: 12 additions & 1 deletion docs/src/content/docs/ja/bindings/javascript.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ block、完全な document IR、diagnostics を返します。`3:18` は三つ
annotation として検索でき、`anchor` は base text の range を指します。

```ts
import { getMdiTextBlocks, sourceSpansForTextRange } from "@illusions-lab/mdi";
import { getMdiTextBlocks, resolveMdiSourceSpan, sourceSpansForTextRange } from "@illusions-lab/mdi";

const result = getMdiTextBlocks("# 題\n\n{東京|とうきょう}");
const paragraph = result.blocks[1];
Expand All @@ -67,13 +67,24 @@ const match = { start: "2:1", end: "2:3" } as const;
console.log(paragraph.text); // 東京
console.log(paragraph.annotations[0].text); // とうきょう
console.log(sourceSpansForTextRange(paragraph, match)); // UTF-8 source span
console.log(resolveMdiSourceSpan("# 題\n\n{東京|とうきょう}", { startByte: 8, endByte: 14 }));
```

`sourceMap.synthetic` は table の tab や row newline など projection が追加した
separator を示し、source span は作りません。`parseMdiTextPosition`、
`formatMdiTextPosition`、`formatMdiTextRange` は canonical な座標表記用の
stateless helper です。

`resolveMdiSourceSpan(source, span)` は Rust で逆引きします。入力は half-open
UTF-8 byte の uint32 で、source 内・昇順・code-point boundary でなければ
なりません。結果は block 順、本文優先、zero-based annotation 順の canonical
range と `complete | partial | none` coverage です。forward coverage 全体が入力と
一致する match だけが `exact`、それ以外は `overlap` です。ruby base と reading
は別 channel です。空 span は caret と解釈せず match を返しません。純粋な構造
delimiter、synthetic、unmapped byte に range は作られないため、annotation、
multi-to-one token、partial grapheme、discontinuous mapping を含む round trip は
一般には bijection ではありません。

## baseline と設定付き EPUB/DOCX

一引数の API は synchronous Rust baseline export です。
Expand Down
8 changes: 8 additions & 0 deletions docs/src/content/docs/ja/bindings/rust.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,14 @@ match mdi_core::render_pdf(source, &mdi_core::PdfOptions::default()) {

`MDI_IR_VERSION` と `MDI_SPEC_VERSION` は exported constant です。永続化した `ParseOutput` を読み直すなら version を確認してください。`SourceSpan { start_byte, end_byte }` は UTF-8 byte の半開 range で、`char` index ではありません。

検索用 canonical text は `get_mdi_text_blocks(source)`、逆引きは
`resolve_mdi_source_span(source, span)` を使います。後者は順序、範囲、UTF-8
boundary を検証し、本文と annotation の maximal grapheme range、
`Complete | Partial | None` coverage、`Exact | Overlap` relation を返します。
空 span、純構造 delimiter、synthetic、unmapped source は range を作りません。
Ruby の別 channel や multi-to-one/discontinuous mapping により、round trip は
一般に bijection ではありません。

## 現在の実装状況

parse、`serialize_mdi`、HTML/TXT/EPUB/DOCX/PDF renderer はすべて実装済みです。baseline の正確な範囲は [Rust Core API](/ja/core/rust-api/#not-yet-implemented) を参照してください。
Expand Down
7 changes: 6 additions & 1 deletion docs/src/content/docs/ja/core/rust-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,12 @@ description: 現在 mdi-core/src/lib.rs に実在する public symbol の一覧

## Public data types

current-generation API は `ParseOutput`、`ParserCapabilities`、`Diagnostic`、`SourceSpan`、`Document`、`Frontmatter`、`PdfOptions`、`EpubCover`、`ResolvedExportProfile` とその nested profile/Chromium print types です。旧 shape は `MdiSyntaxDocument`、`MdiBlock`、`PagebreakVariant`、`Inline`、`RubyReading` です。
current-generation API は `ParseOutput`、`ParserCapabilities`、`Diagnostic`、`SourceSpan`、`Document`、`Frontmatter`、`MdiTextBlocksResult`、`MdiSourceSpanTextResolution`、`MdiSourceSpanTextMatch`、`MdiSourceSpanCoverage`、`MdiSourceSpanRelation`、`MdiSourceSpanResolutionError`、`PdfOptions`、`EpubCover`、`ResolvedExportProfile` とその nested profile/Chromium print types です。旧 shape は `MdiSyntaxDocument`、`MdiBlock`、`PagebreakVariant`、`Inline`、`RubyReading` です。

`get_mdi_text_blocks(source)` は grapheme 単位の canonical text projection を返し、
`resolve_mdi_source_span(source, span)` は half-open UTF-8 source span を本文と ruby
annotation の range に逆引きします。coverage、relation、boundary、delimiter、
synthetic/unmapped、round-trip の制約は [Rust binding](/ja/bindings/rust/) を参照してください。

## Not yet implemented

Expand Down
12 changes: 11 additions & 1 deletion docs/src/content/docs/zh-tw/bindings/javascript.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Unicode grapheme。Ruby 讀音會作為可搜尋的獨立 annotation channel,
仍指回正文 base text 的 range。

```ts
import { getMdiTextBlocks, sourceSpansForTextRange } from "@illusions-lab/mdi";
import { getMdiTextBlocks, resolveMdiSourceSpan, sourceSpansForTextRange } from "@illusions-lab/mdi";

const result = getMdiTextBlocks("# 題\n\n{東京|とうきょう}");
const paragraph = result.blocks[1];
Expand All @@ -65,12 +65,22 @@ const match = { start: "2:1", end: "2:3" } as const;
console.log(paragraph.text); // 東京
console.log(paragraph.annotations[0].text); // とうきょう
console.log(sourceSpansForTextRange(paragraph, match)); // UTF-8 source spans
console.log(resolveMdiSourceSpan("# 題\n\n{東京|とうきょう}", { startByte: 8, endByte: 14 }));
```

`sourceMap.synthetic` 指出 projection 額外加入的 separator,例如 table 的 tab 與
row newline;它們不會偽造 source span。`parseMdiTextPosition`、
`formatMdiTextPosition`、`formatMdiTextRange` 是 canonical 座標格式的無狀態 helper。

`resolveMdiSourceSpan(source, span)` 由 Rust 執行反向解析。輸入是 half-open UTF-8
byte uint32,必須有序、位於 source 範圍內且落在 code-point boundaries。結果依
block、正文優先、零基底 annotation index 排序,coverage 為
`complete | partial | none`。只有 match 的完整 forward coverage 恰等於輸入時
才是 `exact`,其餘交集為 `overlap`。Ruby base 與 reading 是獨立 channel;空
span 不視為 caret,也不回傳鄰近 range。純結構 delimiter、synthetic 與 unmapped
byte 不會取得虛構 range,因此 annotation、多對一 token、partial grapheme、
discontinuous mapping 等情況不保證 round trip 是雙射。

## baseline 與可設定 EPUB/DOCX

一個參數的 API 是 synchronous Rust baseline export:
Expand Down
7 changes: 7 additions & 0 deletions docs/src/content/docs/zh-tw/bindings/rust.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,13 @@ match render_pdf(source, &PdfOptions::default()) {

`MDI_IR_VERSION` 與 `MDI_SPEC_VERSION` 為 exported `&'static str` constants;儲存 `ParseOutput` 後再載入時應檢查。`SourceSpan { start_byte: u32, end_byte: u32 }` 是 half-open UTF-8 byte range,詳見[診斷](/zh-tw/core/diagnostics/)。

搜尋用 canonical text 可用 `get_mdi_text_blocks(source)`;反向查詢使用
`resolve_mdi_source_span(source, span)`。它會驗證順序、範圍與 UTF-8 boundaries,
回傳正文及 annotation 的最大 grapheme ranges、`Complete | Partial | None`
coverage 和 `Exact | Overlap` relation。空 span、純結構 delimiter、synthetic 與
unmapped source 都不會產生 range。Ruby 雙 channel 與多對一/不連續 mapping
代表 round trip 通常不是雙射。

## 目前實作狀態

Parsing、`serialize_mdi` 及所有 renderer(`render_html`、`render_text_format`、`render_epub`、`render_docx`、`render_pdf`)皆已實作,限制見 [Rust Core API 尚未實作項目](/zh-tw/core/rust-api/#尚未實作)。沒有獨立 `validate`/`normalize` API,分別由 `parse_output`/`serialize_mdi` 擔任。
Expand Down
7 changes: 6 additions & 1 deletion docs/src/content/docs/zh-tw/core/rust-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,12 @@ description: "`mdi-core/src/lib.rs` 現在實際公開的所有 symbol,包含

## Public data types

`ParseOutput`、`ParserCapabilities`、`Diagnostic`、`DiagnosticSeverity`、`SourceSpan`、`Document`、`Frontmatter`、`FrontmatterEntry`、`PdfOptions`、`EpubCover`、`ResolvedExportProfile` 與其 nested profile/Chromium print types(目前 API);`MdiSyntaxDocument`、`MdiBlock`、`PagebreakVariant`、`Inline`、`RubyReading`(較舊、只供 `parse_mdi_syntax` 的 shape;`Inline`/`RubyReading` 也在內部用來建立目前 `Document` 的 MDI nodes)。
`ParseOutput`、`ParserCapabilities`、`Diagnostic`、`DiagnosticSeverity`、`SourceSpan`、`Document`、`Frontmatter`、`FrontmatterEntry`、`MdiTextBlocksResult`、`MdiSourceSpanTextResolution`、`MdiSourceSpanTextMatch`、`MdiSourceSpanCoverage`、`MdiSourceSpanRelation`、`MdiSourceSpanResolutionError`、`PdfOptions`、`EpubCover`、`ResolvedExportProfile` 與其 nested profile/Chromium print types(目前 API);`MdiSyntaxDocument`、`MdiBlock`、`PagebreakVariant`、`Inline`、`RubyReading`(較舊、只供 `parse_mdi_syntax` 的 shape;`Inline`/`RubyReading` 也在內部用來建立目前 `Document` 的 MDI nodes)。

`get_mdi_text_blocks(source)` 提供 grapheme canonical text projection;
`resolve_mdi_source_span(source, span)` 把 half-open UTF-8 source span 反解為正文與
ruby annotation ranges。Coverage、relation、boundary、delimiter、synthetic/unmapped
及 round-trip 限制請參閱 [Rust 綁定](/zh-tw/bindings/rust/)。

## 尚未實作

Expand Down
5 changes: 5 additions & 0 deletions mdi-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@ Use `get_mdi_text_blocks` when a search or annotation index needs source-order
plaintext blocks with one-based Unicode-grapheme coordinates and exact UTF-8
source maps. The result includes the same document IR and diagnostics, and the
source is parsed only once.
Use `resolve_mdi_source_span` for the inverse lookup: a validated half-open
UTF-8 source span becomes ordered canonical block-text and ruby-annotation
ranges. Its `complete`, `partial`, or `none` coverage reports whether every
requested source byte belongs to at least one mapped grapheme; synthetic and
unmapped projection text never creates a match.
When rendering one parsed document in multiple formats, use the
`*_document` functions, such as `render_html_document`, to avoid parsing it
again.
Expand Down
32 changes: 26 additions & 6 deletions mdi-core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,12 @@ pub use publication_profile::{
prepare_chromium_print_profile_resolved, resolve_export_profile, resolve_export_profile_json,
};
pub use text_projection::{
MDI_TEXT_PROJECTION_VERSION, MdiAnnotationSourceMap, MdiTextAnnotation, MdiTextBlock,
MdiTextBlockKind, MdiTextBlocksResult, MdiTextPosition, MdiTextRange, MdiTextSourceMap,
MdiTextSourceRun, get_mdi_text_blocks, get_mdi_text_blocks_json,
MDI_TEXT_PROJECTION_VERSION, MdiAnnotationSourceMap, MdiSourceSpanCoverage,
MdiSourceSpanRelation, MdiSourceSpanResolutionError, MdiSourceSpanTextMatch,
MdiSourceSpanTextResolution, MdiTextAnnotation, MdiTextBlock, MdiTextBlockKind,
MdiTextBlocksResult, MdiTextPosition, MdiTextRange, MdiTextSourceMap, MdiTextSourceRun,
get_mdi_text_blocks, get_mdi_text_blocks_json, resolve_mdi_source_span,
resolve_mdi_source_span_json,
};

/// MDI syntax version implemented by this crate.
Expand Down Expand Up @@ -3984,12 +3987,12 @@ fn classify_block_macro(source: &str) -> BlockMacroClass {
#[cfg(feature = "wasm")]
mod wasm {
use super::{
BlockMacroClass, EpubCover, PagebreakVariant, RubyReading, TextFormat,
BlockMacroClass, EpubCover, PagebreakVariant, RubyReading, SourceSpan, TextFormat,
apply_pdf_profile_json, classify_block_macro, get_mdi_text_blocks_json,
page_size_catalog_json, parse_json, prepare_chromium_print_profile_json, render_docx,
render_docx_with_profile, render_epub, render_epub_with_profile, render_html, render_text,
render_text_format, resolve_export_profile_json, serialize_mdi, split_ruby, unescape_mdi,
unescape_ruby,
render_text_format, resolve_export_profile_json, resolve_mdi_source_span_json,
serialize_mdi, split_ruby, unescape_mdi, unescape_ruby,
};
use wasm_bindgen::prelude::*;

Expand All @@ -4007,6 +4010,23 @@ mod wasm {
get_mdi_text_blocks_json(source)
}

/// Resolve a half-open UTF-8 source span to canonical text ranges in Rust.
#[wasm_bindgen(js_name = resolveMdiSourceSpanJson)]
pub fn wasm_resolve_mdi_source_span_json(
source: &str,
start_byte: u32,
end_byte: u32,
) -> Result<String, JsValue> {
resolve_mdi_source_span_json(
source,
SourceSpan {
start_byte,
end_byte,
},
)
.map_err(|error| JsValue::from_str(&error.to_string()))
}

/// Render source through the Rust parser and Rust HTML renderer.
#[wasm_bindgen(js_name = renderHtml)]
pub fn wasm_render_html(source: &str) -> String {
Expand Down
Loading
Loading