Commit graph

35 commits

Author SHA1 Message Date
Charles Kelsoe
9cfc3ecaae docs: disclose scanned capabilities with facts in README and privacy
The Obsidian preview scan flags three real capabilities: filesystem access
(outside-vault media download), vault file enumeration (dedup + folder
auto-complete), and clipboard (copy debug log, write only). All are legitimate
and necessary or opt-in. Document each plainly in a README Permissions section
and reconcile PRIVACY.md (add device-access detail, fix the enumeration nuance,
drop the BRAT reference, bump the date) rather than removing the functionality.
2026-06-15 19:25:20 -04:00
Charles Kelsoe
7c2cb2d8da docs: restore README badges with reliable providers
Bring back the badge row, fixed properly instead of removed. CI and Release
status now use GitHub native badge.svg endpoints (served by GitHub, no
shields.io token pool). Downloads, stars, and latest release use badgen.net,
which does not share shields.io rate-limited token pool that was rendering
"Unable to select next GitHub token from pool". License and Obsidian version
stay static since their values are constant.
2026-06-15 19:15:25 -04:00
Charles Kelsoe
1f0631c512 docs: use static README badges to avoid shields.io token-pool errors
The GitHub-API-backed shields badges (downloads, stars, latest, license, CI and
release status) intermittently render "unable to select next github token from
the pool" when shields.io is rate-limited, which looks broken on a marketplace
README. Keep only the two static badges (Obsidian version, MIT license) that do
not depend on the GitHub API. Dropping the CI/release status badges also keeps
CI internals off the public README.
2026-06-15 19:12:52 -04:00
Charles Kelsoe
d6fb1b3a37 docs: make Community plugins the primary install, drop BRAT
The plugin goes straight to the Community plugins store, so lead Installation
with the in-app Browse flow and keep manual install as the fallback. Remove the
BRAT section and the not-yet-in-store / beta wording.
2026-06-15 19:06:40 -04:00
Charles Kelsoe
bd7ec5beaf docs: refresh README for 1.0 feature set
Drop the early-development warning now that 1.0 is stable. Document this
cycle's import-window additions: endless scroll, Select all, loaded/selected
counts, and the per-feed Re-detect source button.
2026-06-15 18:39:44 -04:00
Charles Kelsoe
c30a50ceb5 CI: suppress js-yaml dev-only OSV advisory GHSA-h67p-54hq-rp68
A newly disclosed moderate DoS in js-yaml (<= 4.1.1) is flagged by OSV-Scanner
against js-yaml@3.14.2, a deep transitive dev dependency from the test-coverage
toolchain (ts-jest -> istanbul -> load-nyc-config -> js-yaml). It is never in
the shipped main.js and only parses our own trusted coverage config. The patch
is js-yaml 4.2.0, but load-nyc-config needs the removed 3.x safeLoad API, so no
non-breaking override exists. Document the acceptance in osv-scanner.toml and
point the scanner at it, rather than dropping the severity gate.
2026-06-15 18:35:07 -04:00
Charles Kelsoe
ae8dbd073e Release 1.0.0: first stable release
First stable release of the free and public-content importer. Bundles this
cycle's import-window work (endless scroll, select-all, loaded and selected
counts), custom-domain Substack detection via the feed generator marker, and a
per-feed Re-detect button to upgrade a feed saved as generic in place.

Importing full bodies of paid Substack posts is deferred to a later release.
2026-06-15 18:30:22 -04:00
Charles Kelsoe
e83caf2a3d Add endless scroll, select-all, item counts, and custom-domain Substack detection
Import window:
- Endless scroll: an IntersectionObserver auto-loads older Substack archive
  pages as a bottom sentinel scrolls into view. The "Load older" button stays
  as a fallback and end-of-archive indicator.
- Select-all toggle that checks every available (not imported, not dismissed)
  loaded item and flips to "Deselect all" once all are selected.
- Toolbar counts: a running loaded count that reads "All N items" once the
  archive is exhausted, and an "x of y selected" count beside the toggle.

Resolver:
- Detect a custom-domain Substack by its <generator>Substack</generator> feed
  marker, so it classifies as substack (and gains archive backfill) instead of
  falling back to generic. Backward compatible: feeds without the marker stay
  generic.

Settings:
- A "Re-detect" button on each feed editor page re-resolves the feed and
  updates its source type in place, upgrading a feed saved as generic without
  removing and re-adding it. feedId is left untouched so dedup and dismissals
  stay valid.

Adds unit coverage for selectableItemIds, selectAllControlState, and the
generator-marker detection on both the probe and explicit-feed-url paths.
2026-06-15 17:34:02 -04:00
Charles Kelsoe
4dc9ecfc2a README: add early-development/use-at-own-risk notice, requirements, full feature list
State clearly that the plugin is in early development, not production-ready, use
at own risk, keep backups, and not yet in the Community Plugins store (install via
BRAT or manually). Add a Requirements section (Obsidian 1.13.0+, desktop only).
Expand Features to everything shipped through 0.4.0 (media download, archive
backfill, per-feed cleanup + command, feed-tags property, per-feed overrides).
Add Changelog pointer.
2026-06-14 17:51:37 -04:00
Charles Kelsoe
67e7732b01 Release 0.4.0: per-feed deterministic cleanup rules 2026-06-14 17:34:57 -04:00
Charles Kelsoe
aebec603c4 Feature: per-feed deterministic cleanup (link-host + trim rules)
Strip promotional clutter from note bodies by LINK TARGET and structure, never by
body wording (so rules survive rewording). Per-feed link-host list (e.g.
buymeacoffee.com, substack.com/app, /subscribe) removes short CTA/footer blocks;
an optional trim-after-last-horizontal-rule drops a trailing footer region. Long
real paragraphs that merely cite such a link are preserved. Rules apply at import
and via a new 'Clean up imported notes' command that re-cleans existing notes with
frontmatter untouched. cleanup.ts is pure + idempotent. +32 tests (323 total).
2026-06-14 17:34:05 -04:00
Charles Kelsoe
ad7c0e4113 Release 0.3.0: Substack archive backfill 2026-06-14 17:19:56 -04:00
Charles Kelsoe
8f078f8248 Feature: Substack archive backfill (load older than the RSS window)
A 'Load older' control in the import window pages back through a Substack
publication's archive (/api/v1/archive?offset=) beyond the ~20-item RSS window,
fetching each older post's body on demand (/api/v1/posts/<slug>). ListItemsOptions
gains an offset; offset 0 keeps the inline-body RSS path. Generic RSS feeds expose
only their recent window, so the control is Substack-only. Defensive JSON parsing;
selection preserved across paged loads. +20 tests.
2026-06-14 17:18:56 -04:00
Charles Kelsoe
327f2eea94 Release 0.2.0: per-feed media download; feed-tags property; import-window fixes 2026-06-14 17:01:02 -04:00
Charles Kelsoe
a0d5359072 Import window: flip imported badge immediately; trim the summary
Imported items now show the imported badge right after import instead of only
after reopening: the metadataCache index updates asynchronously, so the just-
written notes are marked imported directly from the import result. The result
summary now shows a one-line count plus failures only, instead of listing every
item (which overflowed the modal).
2026-06-14 16:59:45 -04:00
Charles Kelsoe
de018c18e8 Feature: per-feed media (audio/video) download
Download podcast/audio/video enclosures into a vault subfolder (vault.createBinary)
or to an outside-vault folder (Node fs, desktop-only, behind a Platform.isDesktop
guard) for media you do not want syncing. Global defaults plus per-feed overrides
(download on/off, location, subfolder). Notes record a media-file frontmatter line
and point the body link at the local file. Failures are best-effort (logged, never
abort the item). +20 tests.
2026-06-14 16:59:44 -04:00
Charles Kelsoe
91a6f9a206 Tags: write feed tags to a feed-tags property by default
Feed and item tags now go to a plain feed-tags frontmatter property instead of the
global Obsidian tags, so they no longer flood the tag pane, search, and graph. A
new 'Tag destination' setting switches back to Obsidian tags when wanted. A
leading '#' is now stripped from a tag (a #-prefixed tag could previously turn
'tags: [#x]' into a YAML comment and drop every tag, the likely cause of tags not
appearing). Tests added for both destinations and the # case.
2026-06-14 16:44:55 -04:00
Charles Kelsoe
11105ba5d6 Release 0.1.2: preserve import selection on dismiss; strip Substack clutter 2026-06-14 16:30:57 -04:00
Charles Kelsoe
b06dd6911d Converter: strip Substack app-install / subscribe / CTA clutter
Remove email-truncation and 'read in the app' promos, subscribe forms, and CTA
buttons by their Substack data-component-name (InstallSubstackAppToDOM,
SubscribeWidgetToDOM, ButtonCreateButton) plus the matching class families
(install-substack-app, subscription-widget, preamble, digest-post-embed,
fake-input/button, cta-caption). Real images (Image2ToDOM) and prose are kept.
Structural match only, never body text. Test added.
2026-06-14 16:27:53 -04:00
Charles Kelsoe
7654740d6a Fix dismiss/import wiping other checked items
Selection lived only in the checkbox DOM, so toggleDismiss and the post-import
refresh (both re-render the whole list) reset every checkbox. Track selected ids
in a Set restored on each render; dismiss drops only its own item and imported
items leave the selection. (DOM-level selection still needs a jsdom UI harness to
unit-test; logic verified manually for now.)
2026-06-14 16:27:52 -04:00
Charles Kelsoe
f1bb6350ef Release 0.1.1: fix load failure and add-feed Save button 2026-06-14 16:16:47 -04:00
Charles Kelsoe
26b4debd2e Fix add-feed Save button doing nothing
Save mixed ButtonComponent.setDisabled() with raw toggleAttribute() and required
a separate Resolve click, so it could stay disabled/unresponsive. Save is now
always enabled and resolves the feed on demand before saving, with a re-entry
guard. Added a regression test for the resolve-on-save path.
2026-06-14 16:16:47 -04:00
Charles Kelsoe
9f083443da Fix plugin failing to load (settings tab getter/setter conflict)
RssImporterSettingTab declared a getter-only `get plugin()`, but the
PluginSettingTab base constructor assigns `this.plugin = plugin`, which throws
'Cannot set property plugin which has only a getter' -- the plugin was disabled
on load before any UI registered. Use a plain typed field (the shell-path-copy
pattern) instead. onload now wraps its body in a guard that surfaces a Notice on
failure instead of failing silently. Added a regression test that constructs the
settings tab against the faithful base mock (no test did before).
2026-06-14 16:16:47 -04:00
Charles Kelsoe
9e563d8ffe Phase 5: apply adversarial-review fixes
Correctness: resolver classifies /p/ as Substack only on a substack.com host;
redirect cap aligned to 5; extractFeedItemId unescapes in a single pass (round
trips literal backslash escapes). Feature: podcast/media notes now record
media-url frontmatter and append an [Episode audio]/[Media] body link. Safety:
the import command guards synchronous NoteWriter construction (a bad destination
folder now surfaces a Notice, not a silent throw); loadSettings validates
per-feed literal fields from data.json. Diagnosability: empty-body imports and
skipped image downloads are logged. Structure: FeedPickerModal and
DuplicatePromptModal moved to their own files; id non-empty guard at the mapping
boundary; doc + type-dedup cleanups. 244 tests; build + lint green.

Deferred (low/cosmetic): shared header helper, Atom updated-vs-published date
precedence, the sanctioned settings-tab cast.
2026-06-14 15:26:45 -04:00
Charles Kelsoe
eec329d526 Phase 4: end-to-end pipeline test (source to note to dedup index)
Wires real source + converter + note-writer + vault-index against the captured
fixtures: generic feed maps podcast/article correctly, a written note round-trips
through the dedup index so re-import is a no-op, a Substack paid teaser is flagged
truncated with the warning note shape, and conversion is deterministic.
2026-06-14 15:05:12 -04:00
Charles Kelsoe
69fff137d1 Wave E: plugin wiring (main.ts)
onload registers the import/add-feed/debug commands, settings tab, and ribbon;
requestUrl-backed HttpFetcher with SyntaxError-only safeJson; per-session
FetchPacer; source selection by feed type; per-feed NoteWriter + ImportRunner
with optional image download; feed picker, add-feed, and duplicate-policy
prompt modals; runGuarded wraps async command bodies.
2026-06-14 15:02:00 -04:00
Charles Kelsoe
a2d53fc070 Wave D: import services and UI shell
- import-runner: per-item orchestration (fetchBody, convert, optional image
  download, write); per-item failures never abort the batch; abort + progress;
  formatImportNotice. image-downloader: best-effort download + markdown rewrite.
- UI: createStackedRow helper, FolderSuggest (AbstractInputSuggest), AddFeedModal
  (resolve + preview + commit), ImportModal (three-state list, live progress,
  summary), declarative RssImporterSettingTab (Feeds list + Defaults group),
  real styles.css.
- Extend the obsidian test mock with SettingPage + settings-tab methods.
2026-06-14 15:01:59 -04:00
Charles Kelsoe
3b6b74a910 Add settings contract (RssImporterSettings, FeedConfig, defaults) 2026-06-14 14:44:17 -04:00
Charles Kelsoe
9bd52e729f Docs: MVP-accurate README, PRIVACY, SECURITY
README gains Features/Usage and the correct minAppVersion badge (1.13.0).
PRIVACY/SECURITY replace the template's no-network-code default with the real
on-device flow: requests only to user-configured feed hosts, redirect targets,
the Substack public profile API, and referenced image/media URLs; no telemetry,
no maintainer server.
2026-06-14 14:40:39 -04:00
Charles Kelsoe
19e47bd9ad Wave C: feed sources (generic + substack) and shared mapping
- source-common: fetchAndParseFeed, mapRawItemToFeedItem (the shared
  RawFeedItem to FeedItem mapping), buildResolvedFeed, applyLimit, FeedFetchError
- source-generic: GenericRssFeedSource (RSS/Atom/podcast, body from RSS)
- source-substack: SubstackFeedSource (free RSS bodies, paywall detection sets
  audience/isTruncated; section deferred to the JSON API in a later release)
32 source tests; 194 total green; tsc + eslint clean.
2026-06-14 14:40:39 -04:00
Charles Kelsoe
e89a35e122 Wave B: XML parser, HTML-to-Markdown converter, feed resolver
- feed-xml: RSS/Atom/podcast parse via DOMParser, content:encoded/enclosure/
  namespaced-tag extraction, ISO date normalization, FeedParseError
- html-converter: Turndown + gfm + feed rules (figure/caption, widget strip,
  code-fence language, footnotes), deterministic output
- feed-resolver: input classification, Substack @handle profile resolution,
  manual redirect walking (hop cap 5), XML probe
- turndown-plugin-gfm.d.ts: standalone ambient declaration (module-scoped
  augmentation in types.d.ts was inert under the build config)
162 tests green (jsdom for DOM modules, node for resolver); tsc + eslint clean.
2026-06-14 14:32:39 -04:00
Charles Kelsoe
42ed0f59ab Wave A: leaf modules (logger, pacer, paywall, dismiss, index, note-writer)
Six contract-only modules with full unit tests (93 tests):
- debug-logger: ring buffer, no-op when disabled, injected clock, exportable
- fetch-pacer: sequential queue, inter-request delay, 429/Retry-After backoff
- paywall-detector: structural teaser classifier (no localized-text matching)
- dismiss-store: data.json-backed reversible per-feed dismissed set
- vault-index: metadataCache frontmatter scan scoped per destination folder
- note-writer: sanitize, ensureFolder, collision guard, duplicate policy, vault.process
All green: tsc, eslint (obsidianmd), jest.
2026-06-14 14:11:12 -04:00
Charles Kelsoe
4cc3f6fbf7 Add jest test harness: obsidian mock + moduleNameMapper wiring 2026-06-14 14:03:11 -04:00
Charles Kelsoe
bed5740610 Add feed-source contract (FeedItem, FeedSource, HttpFetcher, frontmatter keys) 2026-06-14 13:59:15 -04:00
Charles Kelsoe
35ac67a806 Scaffold from standard plugin template
Bootstrap obsidian-rss-importer from plugin-templates/standard: placeholders
substituted, isDesktopOnly set true, Turndown + jsdom test deps added, esbuild
bumped to 0.28.1 to clear advisory GHSA-gv7w-rqvm-qjhr (matches shell-path-copy).
Build and lint green; main.ts is a wiring-only stub pending feature phases.
2026-06-14 13:57:19 -04:00