Commit graph

9 commits

Author SHA1 Message Date
LLLin000
45022c3e16 docs: archive superpowers specs/plans, commit PR9A-C plan files
- Archive 173 design/plan/analysis docs from docs/superpowers/ →
  docs/archive/superpowers/ (all features shipped)
- Commit 5 PR9A-C plan files to docs/plans/ (historical record)
- Rebuild plugin bundle (main.js: 2172→49 lines, minified)
- Fix settings.ts type assertion
2026-07-09 00:39:39 +08:00
LLLin000
0dfbf2a68d feat: PR 8 — Object Vector Retrieval (paperforge_objects collection)
New paperforge_objects collection in ChromaDB for figure/table caption
vector search, completing the memory layer retrieval triad:

  paperforge_fulltext → legacy_chunk
  paperforge_body     → body_unit
  paperforge_objects  → object_unit (NEW)

Changes:
- manifest.py: compute_object_units_hash(), manifest records both hashes
- _chroma.py: _COLLECTION_NAMES includes paperforge_objects
- builder.py: get_object_units_for_embedding() + embed_object_units()
- status.py: object_chunk_count, total_chunks
- state_snapshot.py: write_vector_runtime() extended with new fields
- search.py: 3rd collection, dict-based source mapping, object_kind/label
- __init__.py: exports for new functions
- commands/embed.py: _has_object_units_in_db, body+object structured
  path routing with dual-hash resume, extended counters
- Test fix: test_build_paper_manifest data complete for hash functions

Verification: 20 object vectors embedded, all per-paper counts match,
status total consistent (58266 = 57435 + 811 + 20),
object caption queries return object_unit results via merge_retrieve.
Unit tests: 153 pass, 1 skip.
2026-07-06 19:58:46 +08:00
LLLin000
e551f79a77 fix: PR 7 post-test hardening — object units, node_id, expanded tests
P0 fixes from test report review:
- object unit_id: fallback to f"{obj_type}:p{page}:{block_id}" when
  figure_id/table_id missing (prevents DB overwrite on INSERT OR REPLACE)
- object caption_key: page-qualified f"p{page}:{caption_bid}" to match
  page-qualified subtree_block_ids in find_owning_node
- object block_map: page-qualified keys for consistency with body_units
- structure node_id: changed from sec:{block_id} to sec:p{page}:{block_id}
  for stability (reduces vector ID churn from emitted_order changes)

New tests:
- Section 4b: Object units DB persistence (unit_id unique, non-empty labels,
  DB count == list count after upsert)
- Section 5b: Coverage gate (v2_tree / render_map / body_papers counts)

Layer A: 1669/1669 pass (+60 checks from 4b + 5b)
Unit tests: 153 pass, 1 skip
2026-07-06 19:34:05 +08:00
LLLin000
01461330bb fix: final hardening — policy revision + Layer A all green
- _body_unit_role_kind now excludes only reference_item/reference_heading
  (per policy: everything in fulltext except ref zone)
- backmatter_body unit_kind removed; all units are 'body'
- bounds_map in structure_tree uses page-qualified key (fixes collision
  when same block_id appears on different pages)
- Layer A test plan: emitted_ids uses page-qualified keys + handles
  block_id=0 (falsy value)
- Layer A: 1609 passed, 0 failed on real vault (22 papers)
- Unit tests: 152 passed, 1 pre-existing skip
2026-07-06 19:08:54 +08:00
LLLin000
77a6e0cfec fix: heading→emitted events, backmatter→heading events, page-qualified keys
- Section/subsection headings now appear in BOTH heading_events and
  emitted_block_events (emitted_as='heading') for complete event stream
- Backmatter headings (Funding/Acknowledgments/Data Availability/Conflicts)
  now generate heading_events → appear as structure tree sections with
  their own body_units
- Page-qualified block IDs (p{page}:{block_id}) in tree own/subtree_block_ids
  and body_units block_map: eliminates ambiguous block_id collisions when
  same block_id appears on multiple pages in real OCR data
- TC-9.2 wording fix, KEYS empty guard, global unit_id uniqueness check
2026-07-06 18:19:17 +08:00
LLLin000
897afdaecb fix: token cap recursive split + policy version check + test plan update
- _split_if_oversized: recursive _halve_text until every part <= 1000 tokens
- _incremental_units_only: also checks retrieval_policy_version match
- _rebuild_paper_units: use RETRIEVAL_POLICY_VERSION constant
- Functional test plan: Layer A (artifact) + Layer B (API), all 8 review
  fixes incorporated (persisted artifacts, no jump-assertion, stronger
  object/backmatter/abstract tests, proper embed flow)
2026-07-06 17:05:13 +08:00
LLLin000
1ef0899fec docs: comprehensive functional test plan for memory layer
10 sections covering Render Events, Structure Tree, Body Units,
Object Units, FTS, Content Discovery, Memory Builder, Embed Build,
Retrieve Merge, and Regression. Includes auto-test script skeleton.
2026-07-06 16:59:31 +08:00
LLLin000
288829bb5b feat: PR 1 — nested structure tree + body_units correctness
- RenderOutput dataclass with heading_events/emitted_block_events tracking
- Stack-algorithm structure tree with own/subtree block IDs
- Recursive body_units builder with role helper + token cap splitting
- Schema v4: section_path_json, section_level, section_title, part_ordinal
- Phase 4/5 reorder in ocr_rebuild.py and ocr.py
- 12 new test cases for tree nesting, own_block_ids, rendered-order intervals
2026-07-06 16:42:08 +08:00
LLLin000
7b29ab68ca feat: improve vnext figure matching and split OCR status
This batch implements the validated PR-2 / PR-3 workstreams.

PR-2b: continued captions in vnext
- add figure continuation helpers
- mark continuation legends in candidate indexing
- run ContinuationCaptionPass before PrimarySamePagePass
- emit unique continuation records with trace metadata
- skip continuation legends in ordinary same-page proposals

PR-2a: short-caption geometry
- reuse _TRUNCATED_LEGEND_ONLY_PATTERN
- add narrow short-caption branch at the vnext group/page seam
- require same page, same column/x-overlap, tight gap, and local uniqueness
- avoid global threshold changes or score_figure_match edits

PR-3: completeness normalization and additive status split
- normalize legacy/vnext figure completeness shapes
- keep overall unchanged
- add structure_status, figure_status, table_status,
  dimension_statuses, status_reasons

Verification:
- 443 targeted tests passing
- real-vault rebuilds:
  * SKXTCE6M: unmatched_legends 6->0, overall green
  * SRNJDAA2: unmatched_legends 9->0, figure green
  * 6QNRHRKX: matched 0->4, figure yellow (improved, not solved)
  * 9DM6MCIF: unmatched_legends 3->0, figure green
  * XGT9Z257: unmatched_legends 4->0, figure green
  * V4UTP5X7: figure green while structure remains red
2026-07-06 00:55:40 +08:00