- Section/subsection headings now appear in BOTH heading_events and
emitted_block_events (emitted_as='heading') for complete event stream
- Backmatter headings (Funding/Acknowledgments/Data Availability/Conflicts)
now generate heading_events → appear as structure tree sections with
their own body_units
- Page-qualified block IDs (p{page}:{block_id}) in tree own/subtree_block_ids
and body_units block_map: eliminates ambiguous block_id collisions when
same block_id appears on multiple pages in real OCR data
- TC-9.2 wording fix, KEYS empty guard, global unit_id uniqueness check
- RenderOutput dataclass with heading_events/emitted_block_events tracking
- Stack-algorithm structure tree with own/subtree block IDs
- Recursive body_units builder with role helper + token cap splitting
- Schema v4: section_path_json, section_level, section_title, part_ordinal
- Phase 4/5 reorder in ocr_rebuild.py and ocr.py
- 12 new test cases for tree nesting, own_block_ids, rendered-order intervals
P0: disambiguate table field keys from block_page_by_id in ocr_render.py
P1: add _is_tail_backmatter_continuation() guard, order backmatter correctly
P2: phase 4b tail nonref guard with tail_backmatter_blocks list
P3: page-continuation-marker false reference gate in ocr_families.py
P4: force reference heading before same-page refs, ref_item bypass usable gate
Two-part fix:
1. ocr_bio.py: clear zone ('') when post_ref_bio_cleanup converts
block to backmatter_body, so render doesn't interleave with refs
(was reference_zone from zone inference)
2. ocr_render.py: skip_section_grouping path (no reference_heading
on page) grouped backmatter_body/backmatter_heading into non_ref,
placing them before the same page's refs. On mixed-column tail
pages this put bios between [170] and [171]. Now backmatter roles
are output after refs within each page.
Fixes 2HEUD5P9 page 26 author bios appearing before [171]-[188].
Three issues fixed:
1. score_structured_insert: _in_visual_container weight 0.3->0.45
(evidence-driven admission is now reliable)
2. score_structured_insert: add sidebar_header_phrase check (+0.2)
for 'available with this article' etc.
3. _container_text set to block text (was True bool)
4. Normalize newlines in text before keyword matching
5. Safeguard renderer for non-string _container_text
6. Bump span_visual_container_version to force backfill re-run
When two figures share the same figure_id (e.g. supplementary
"Figure S.1" and main "Figure 1" both mapped to "figure_001"),
the second occurrence is renamed with an "s" prefix:
figure_001 -> figure_s001, figure_ss001, etc.
This avoids file overwrite in ocr_objects.py and ensures both
body and supplementary figures appear at their correct positions
in the rendered fulltext.
Changes:
- _resolve_figure_id_collisions() in ocr_figures.py
- figure_id passthrough in ocr_figure_reader.py (reader + normalize)
- _reader_figure_embed_target prefers explicit figure_id
- 11 unit tests covering single/triple/multi collisions, mixed buckets,
empty IDs, realistic body+supplementary layout
Moves the block_id fetch and consumed_table_block_keys check before
_SKIPPED_BODY_ROLES so table note removal is a contract-based decision
(ownership) not a role-based accident (footnote role).
Part of table ownership contract refactor.
- Add rebuild raw-block dedup: strip pdf_text_layer_fallback blocks
whose 5-gram overlap with same-page body exceeds 80%
- Propagate _text_source / _ocr_raw_status / _ocr_raw_error_type to
structured blocks for downstream debugging
- Add render-level dedup: skip body_paragraph if _text_source marks
it as backfill with >=80% overlap vs emitted page text
- Exclude noise/page_footer/page_header/frontmatter_noise roles from
table note candidate detection
On two-column tail pages with reference items but no explicit reference
heading (e.g. K7R8PEKW page 16), forced backmatter section grouping
overrides the natural column-sorted reading order.
- _reorder_tail_run: add skip_section_grouping param. When set, emit
blocks in column-sorted order with reference items grouped at end,
bypassing the backmatter/body section grouping logic entirely.
- _order_tail_blocks: detect pages with ref items + no ref heading and
pass skip_section_grouping=True.
- _normalize_backmatter_roles_after_boundary: remove forced conversion
of body_paragraph to backmatter_body inside backmatter region (body
paragraphs attach naturally via _find_owning_heading in Phase 1).
Add _merge_box_content() post-processing to normalize_document_structure:
detects structured_insert blocks whose text starts with Box N or Key
insights, then consumes following body-like blocks as box content.
Consumed blocks get role=noise + render_default=False to suppress
double-rendering. Column-crossing boxes (common in multi-column
layouts) are handled by removing the x-overlap heuristic.
- ocr_render.py: extend consumed frontmatter roles skip from page 1 to pages <= 2,
preventing paper_title appearing twice on pre-proof papers
- ocr_document.py: add image/media block promotion in backmatter zone normalization
so raw image blocks become media_asset instead of backmatter_body; this fixes
build_figure_inventory producing 0 matched_figures. Also check seed_role in
_sanitize_reference_zone_boundary since it runs before role resolution
- scripts/dev/ocr_rebuild_paper.py: single-paper rebuild + block_trace regeneration
entry point (will be wired into CLI later)
- tests: boundary protection and completeness check tests
- DW trace regenerated with correct role distribution
