diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..57d12624 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,157 @@ +# CLAUDE.md — indiekit-blog + +Personal [Indiekit](https://getindiekit.com/) deployment for [blog.giersig.eu](https://blog.giersig.eu). +Built on the [rmdes/indiekit](https://github.com/rmdes/indiekit) fork ecosystem with a set of +patch scripts that fix issues that cannot be upstreamed. + +## Memory files + +Detailed architecture and lessons learned live in `memory/`: + +- [`memory/project_activitypub.md`](memory/project_activitypub.md) — AP data flows, own-post timeline, compose paths, all AP patches +- [`memory/project_architecture.md`](memory/project_architecture.md) — FreeBSD jails, nginx, MongoDB collections, actor URLs, patch infrastructure +- [`memory/feedback_patches.md`](memory/feedback_patches.md) — Patch authoring patterns, known fragile points, compose-path gotchas + +**Always read the relevant memory file before investigating or modifying anything in this repo.** + +--- + +## Running + +```sh +npm run serve # apply all patches then start Indiekit +npm run postinstall # re-apply patches after npm install (postinstall runs automatically) +``` + +Do **not** restart with `node` directly — patches must run first. + +--- + +## Patch system + +All fixes to `node_modules` live in `scripts/patch-*.mjs`. +Both `postinstall` and `serve` in `package.json` run them in order. + +### Writing a patch + +```js +const MARKER = "// [patch] my-patch-name"; // idempotency check +const OLD_SNIPPET = `exact source text`; +const NEW_SNIPPET = `replacement ${MARKER}`; + +// 1. Skip if MARKER already present +// 2. Warn if OLD_SNIPPET not found (upstream changed) +// 3. Replace + writeFile +``` + +- Match node_modules text **byte-for-byte** (spaces, not tabs; exact line endings). +- Always include both candidate paths: `node_modules/@rmdes/...` and + `node_modules/@indiekit/indiekit/node_modules/@rmdes/...`. +- Template literals in patch strings: escape `` \` `` and `\${}`. +- Append new AP patches **after** `patch-ap-federation-bridge-base-url` in both + `postinstall` and `serve` in `package.json`. +- Some patches are `serve`-only (not `postinstall`) — check both when adding microsub patches. + +--- + +## ActivityPub actor + +| Field | Value | +|---|---| +| Handle | `svemagie` (env `AP_HANDLE`) | +| Full handle | `@svemagie@blog.giersig.eu` | +| Actor URL | `https://blog.giersig.eu/activitypub/users/svemagie` | +| Mastodon migration alias | `@svemagie@troet.cafe` (`AP_ALSO_KNOWN_AS`) | + +AP syndicator `info.uid = publicationUrl` (`https://blog.giersig.eu/`), `info.checked = true`. + +--- + +## Post type discovery + +`getPostType(postTypes, properties)` in `post-type-discovery.js`: + +| Property present | Type | +|---|---| +| `in-reply-to` | `reply` → `/replies/{slug}/` | +| `like-of` | `like` → `/likes/{slug}/` | +| `repost-of` | `repost` → `/reposts/{slug}/` | +| `photo` | `photo` → `/photos/{slug}/` | +| _(none)_ | `note` → `/notes/{slug}/` | + +Only KEY presence matters, not value. If `in-reply-to` is silently missing from the JF2 +object, the post becomes `"note"` with no error — this is a common threading bug root cause. + +--- + +## Reply threading — two compose paths + +| Path | How AP target is pre-checked | +|---|---| +| AP reader `/activitypub/admin/reader/compose` | `target.defaultChecked = target.checked === true` (patched) | +| Microsub reader `/microsub/admin/reader/compose` | `target.checked` from Micropub `q=config` (already `true` in config) | +| Mastodon client API `POST /api/v1/statuses` | `mp-syndicate-to` hardcoded to `publicationUrl` (always AP) | + +The AP reader template uses `target.defaultChecked`, NOT `target.checked` — these are different. +`patch-ap-compose-default-checked` maps the config's `checked: true` through to `defaultChecked`. + +### Mastodon API reply-to-reply (patched) + +`POST /api/v1/statuses` resolves `in_reply_to_id` via `findTimelineItemById(ap_timeline, id)`. +Own posts were previously not inserted into `ap_timeline` until the Eleventy build webhook fired +(30–120 s). `patch-ap-mastodon-reply-threading` inserts a provisional item immediately after +`postContent.create()` using `addTimelineItem()` (`$setOnInsert` — idempotent). + +--- + +## ap_timeline + +Key collection. Documents keyed by `uid`. + +- **Inbound posts** — inserted by `inbox-handlers.js` on Create/Update/Announce activities +- **Own posts** — inserted by `syndicator.js` after delivery (after build) AND now immediately + by `POST /api/v1/statuses` (via patch) +- **Status IDs** — `encodeCursor(published)` = ms-since-epoch string; `findTimelineItemById` + resolves them with a ±1 s MongoDB `$dateFromString` range query to handle TZ-offset strings + +--- + +## Fork dependencies + +Four packages come from GitHub forks (not npm): + +| Package | Fork | +|---|---| +| `@rmdes/indiekit-endpoint-activitypub` | `github:svemagie/indiekit-endpoint-activitypub` | +| `@rmdes/indiekit-endpoint-blogroll` | `github:svemagie/indiekit-endpoint-blogroll#bookmark-import` | +| `@rmdes/indiekit-endpoint-microsub` | `github:svemagie/indiekit-endpoint-microsub#bookmarks-import` | +| `@rmdes/indiekit-endpoint-youtube` | `github:svemagie/indiekit-endpoint-youtube` | + +To pull latest fork commits: `npm install github:svemagie/`. + +--- + +## Environment variables (key ones) + +| Var | Purpose | +|---|---| +| `AP_HANDLE` | ActivityPub handle (default: `svemagie` from `GITHUB_USERNAME`) | +| `PUBLICATION_URL` | Canonical blog URL (`https://blog.giersig.eu`) | +| `INDIEKIT_URL` | Application URL (same as publication URL here) | +| `MONGO_HOST` / `MONGO_URL` | MongoDB connection | +| `REDIS_URL` | Redis for AP message queue and KV store (production) | +| `AP_ALSO_KNOWN_AS` | Migration alias (old Mastodon handle) | +| `GH_CONTENT_TOKEN` | GitHub token for writing posts to the `blog` repo | + +--- + +## Common debugging starting points + +- **Post created as "note" instead of "reply"** → `in-reply-to` missing from JF2. Check: form + hidden field, `submitComposeController`, Mastodon API `findTimelineItemById`, `formEncodedToJf2`. +- **Reply not federated to AP** → `mp-syndicate-to` not set. Check: `target.defaultChecked` / + `target.checked` in compose form, `getSyndicateToProperty` in `jf2.js`. +- **AP object lookup 302 redirect** → nginx not forwarding `Host` / `X-Forwarded-Proto`; see + `patch-ap-federation-bridge-base-url`. +- **Timeline item not found by Mastodon client** → `findTimelineItemById` date mismatch; stored + date is TZ-offset string, lookup decodes cursor to UTC ISO — relies on `$dateFromString` range. diff --git a/memory/MEMORY.md b/memory/MEMORY.md new file mode 100644 index 00000000..88f8b90d --- /dev/null +++ b/memory/MEMORY.md @@ -0,0 +1,5 @@ +# Memory Index + +- [project_activitypub.md](project_activitypub.md) - ActivityPub federation architecture and patch chain dependencies +- [project_architecture.md](project_architecture.md) - Server architecture: FreeBSD jails, nginx, internal fetch URLs +- [feedback_patches.md](feedback_patches.md) - Patch management lessons learned diff --git a/memory/feedback_patches.md b/memory/feedback_patches.md new file mode 100644 index 00000000..8bba0df0 --- /dev/null +++ b/memory/feedback_patches.md @@ -0,0 +1,69 @@ +# Patch Management — Lessons Learned + +## Patch Script Pattern + +```javascript +const MARKER = "// [patch] my-patch-name"; +const OLD_SNIPPET = `exact text to find`; +const NEW_SNIPPET = `replacement text ${MARKER}`; + +// Check MARKER first → skip if already applied +// Check OLD_SNIPPET → warn if not found (upstream may have changed) +// Replace and write only if source changed +``` + +Use `$setOnInsert` for MongoDB upserts in patches that add timeline items — idempotent, +safe to call multiple times (e.g. from both patch and syndicator). + +## Target File Candidates + +Always include both paths: +```javascript +const candidates = [ + "node_modules/@rmdes/indiekit-endpoint-activitypub/...", + "node_modules/@indiekit/indiekit/node_modules/@rmdes/indiekit-endpoint-activitypub/...", +]; +``` + +## Known Fragile Patterns + +- **Exact whitespace matters.** Patch OLD_SNIPPETs must match node_modules source byte-for-byte, + including indentation (spaces not tabs) and exact line endings. +- **Template literals in patches** — escape backticks and `${}` in patch script string literals + using `\`` and `\${}`. +- **`patch-microsub-reader-ap-dispatch`** is in `serve` only (not `postinstall`). Reason unknown + but may relate to timing or the microsub package being rebuilt differently. Check both scripts + when adding new microsub patches. + +## AP Threading — Two Compose Paths + +Replies can come from two different UIs. Each has different syndication logic: + +| Path | Syndication checkbox pre-checked by | +|------|-------------------------------------| +| AP reader (`/activitypub/admin/reader/compose`) | `target.defaultChecked` set in `composeController()` | +| Microsub reader (`/microsub/admin/reader/compose`) | `target.checked` from Micropub `q=config` response | +| Mastodon client API (`POST /api/v1/statuses`) | `mp-syndicate-to` hardcoded to `publicationUrl` (always AP) | + +The AP reader compose form uses `target.defaultChecked` (NOT `target.checked`) in its template. +If `composeController()` doesn't set `defaultChecked` correctly, the AP checkbox is unchecked +even though `target.checked = true` from the Micropub config. + +## ap_timeline Insertion Timing + +Own posts reach `ap_timeline` via TWO paths: +1. **Mastodon API** (`POST /api/v1/statuses`): Now inserts immediately after `postContent.create()` + via `patch-ap-mastodon-reply-threading`. The syndicator's later upsert is a no-op (`$setOnInsert`). +2. **Micropub + syndication webhook**: AP syndicator inserts after Eleventy build completes (30–120 s). + +Always ensure new post creation paths insert to `ap_timeline` immediately if the post may be +replied to again before the build webhook fires. + +## Post Type Discovery Dependency + +`getPostType(postTypes, properties)` uses `Map.has(key)` — only KEY presence matters, not value. +- `"in-reply-to"` present (any value, even empty) → `"reply"` +- If `in-reply-to` is absent → falls through to `"note"` + +If `in-reply-to` is silently lost (null inReplyTo in Mastodon API, unchecked form field, etc.), +the post silently becomes a note with no error. Check this first when debugging "wrong post type" issues. diff --git a/memory/project_activitypub.md b/memory/project_activitypub.md new file mode 100644 index 00000000..2b883602 --- /dev/null +++ b/memory/project_activitypub.md @@ -0,0 +1,117 @@ +# ActivityPub Federation — Architecture & Patch Chain + +## Key Files + +| File | Role | +|------|------| +| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/controllers/compose.js` | AP reader compose form: GET builds the form with `syndicationTargets`, POST submits to Micropub | +| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/mastodon/routes/statuses.js` | Mastodon-compatible API: `POST /api/v1/statuses` creates posts via Micropub pipeline | +| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/syndicator.js` | AP syndicator: federates posts to followers, adds own posts to `ap_timeline` | +| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/jf2-to-as2.js` | Converts JF2 properties to ActivityPub Note/Create activity | +| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/storage/timeline.js` | `addTimelineItem()` — atomic upsert (`$setOnInsert`) to `ap_timeline` | +| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/mastodon/entities/status.js` | Serializes `ap_timeline` documents to Mastodon Status JSON | +| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/mastodon/helpers/pagination.js` | `encodeCursor(date)` / `decodeCursor(id)` — ms-since-epoch as status ID | +| `node_modules/@indiekit/endpoint-micropub/lib/post-type-discovery.js` | `getPostType()` — returns "reply" if `in-reply-to` key present, else "note" | +| `node_modules/@indiekit/endpoint-micropub/lib/post-data.js` | Micropub create pipeline: normalise → getPostType → save | +| `node_modules/@rmdes/indiekit-endpoint-microsub/lib/controllers/reader.js` | Microsub reader compose: auto-selects syndication targets via `detectProtocol()` | +| `node_modules/@rmdes/indiekit-endpoint-activitypub/views/activitypub-compose.njk` | AP reader compose template: uses `target.defaultChecked` for checkbox state | +| `node_modules/@rmdes/indiekit-endpoint-activitypub/views/partials/ap-item-card.njk` | Timeline card: reply button passes `item.url` as `replyTo` query param | + +## Data Flow: Reply via AP Reader UI + +``` +User clicks reply on timeline item + → /activitypub/admin/reader/compose?replyTo= + → composeController() fetches syndicationTargets from Micropub q=config + → sets target.defaultChecked = target.checked === true [patch: ap-compose-default-checked] + → renders activitypub-compose.njk with hidden in-reply-to field + → user submits form + → submitComposeController() POSTs to Micropub with in-reply-to + → Micropub: formEncodedToJf2 → normaliseProperties → getPostType("reply") → save to /replies/{slug}/ + → syndication webhook fires → AP syndicator.syndicate(properties) + → jf2ToAS2Activity creates Note with inReplyTo + → delivered to followers + original author's inbox +``` + +## Data Flow: Reply via Mastodon Client API (Phanpy/Elk) + +``` +Client sends POST /api/v1/statuses { status, in_reply_to_id } + → findTimelineItemById(ap_timeline, in_reply_to_id) + → decodes cursor (ms-since-epoch) → looks up by published date + → inReplyTo = replyItem.uid || replyItem.url + → jf2["in-reply-to"] = inReplyTo (if resolved) + → jf2["mp-syndicate-to"] = [publicationUrl] (always set — AP-only) + → postData.create → postContent.create + → addTimelineItem immediately [patch: ap-mastodon-reply-threading] + → returns minimal status JSON to client + → build webhook → syndicator → AP delivery +``` + +## Own Post in ap_timeline + +The AP syndicator stores outbound posts with: +- `uid = url = properties.url` (blog URL, e.g. `https://blog.giersig.eu/replies/slug/`) +- `published = properties.published` (ISO 8601 with TZ offset, e.g. `"2026-03-21T16:33:50+01:00"`) +- `inReplyTo = properties["in-reply-to"]` (original Mastodon URL or own blog URL) + +The Mastodon API encodes status IDs as `encodeCursor(published)` = ms-since-epoch string. +`findTimelineItemById` uses a ±1 s range query with `$dateFromString` to handle TZ-offset strings. + +## Syndication Target Config + +The AP syndicator's `info` object: +```javascript +{ + checked: true, // from indiekit.config.mjs: checked: true + name: `@${handle}@${hostname}`, // e.g. "@svemagie@blog.giersig.eu" + uid: publicationUrl, // e.g. "https://blog.giersig.eu/" + service: { name: "ActivityPub (Fediverse)", ... } +} +``` + +The Micropub `q=config` endpoint returns this with `checked: true`. +Both compose forms read this value: +- **Microsub reader**: uses `target.checked` directly (always pre-checked if `checked: true` in config) +- **AP reader**: uses `target.defaultChecked` set by `composeController()` — was broken, now fixed + +## Patches Applied (AP threading) + +### `patch-ap-compose-default-checked` +**File:** `lib/controllers/compose.js` +**Problem:** `target.defaultChecked` was hardcoded to `name === "@rick@rmendes.net"` — never matches. +**Fix:** `target.defaultChecked = target.checked === true` +**Effect:** AP syndication checkbox is pre-checked in the AP reader compose form, matching +the `checked: true` config — replies via the AP reader UI are now federated. + +### `patch-ap-mastodon-reply-threading` +**File:** `lib/mastodon/routes/statuses.js` +**Problem:** After `POST /api/v1/statuses`, own post was NOT inserted into `ap_timeline` until +the Eleventy build webhook fired (30–120 s later). Follow-up replies during that window +would fail `findTimelineItemById` → `inReplyTo = null` → no `in-reply-to` in JF2 → +`getPostType` returned `"note"` → reply saved at `/notes/` with no `inReplyTo` in the AP activity. +**Fix:** Eagerly insert a provisional timeline item via `addTimelineItem()` immediately after +`postContent.create()`. Uses `$setOnInsert` (idempotent); syndicator's later upsert is a no-op. +**Effect:** `in_reply_to_id` can be resolved immediately → correct `"reply"` post type → proper +`inReplyTo` in the AP Note → thread displays correctly on Mastodon. + +## Post Type Discovery + +`getPostType(postTypes, properties)` in `post-type-discovery.js`: +- Checks `propertiesMap.has("in-reply-to")` → `"reply"` +- Checks `propertiesMap.has("like-of")` → `"like"` +- Checks `propertiesMap.has("repost-of")` → `"repost"` +- Falls through to `"note"` if none match + +The `"reply"` post type in `indiekit.config.mjs` has no `discovery` field — standard PTD spec applies. + +## detectProtocol() in Microsub Reader + +`detectProtocol(url)` in `reader.js` classifies URLs for syndication auto-selection: +- `"atmosphere"` — bsky.app / bluesky +- `"fediverse"` — mastodon., mstdn., fosstodon., troet., social., misskey., pixelfed., hachyderm., infosec.exchange, chaos.social (extended by `patch-microsub-reader-ap-dispatch`) +- `"web"` — everything else, including own blog URLs + +Own blog URLs return `"web"`, so auto-selection doesn't trigger for reply-to-own-reply in the +microsub reader. This is harmless because `checked: true` in the config already pre-checks the +AP target in the microsub reader's `target.checked` field. diff --git a/memory/project_architecture.md b/memory/project_architecture.md new file mode 100644 index 00000000..76cad018 --- /dev/null +++ b/memory/project_architecture.md @@ -0,0 +1,56 @@ +# Server Architecture + +## Infrastructure + +- **FreeBSD jails** — Indiekit runs in an isolated jail +- **nginx** — reverse proxy; must forward `Host: blog.giersig.eu` and `X-Forwarded-Proto: https` + for Fedify to construct correct canonical URLs (see `patch-ap-federation-bridge-base-url`) +- **MongoDB** — `10.100.0.20:27017`, database `indiekit`, auth source `admin` +- **Redis** — optional; URL via `REDIS_URL` env var; used for AP activity queue + +## Publication URLs + +- `publicationBaseUrl` = `https://blog.giersig.eu` (from `PUBLICATION_URL` env or hardcoded default) +- `applicationBaseUrl` = same (from `INDIEKIT_URL` env) +- GitHub repo: `svemagie/blog`, branch `main` + +## Internal Fetch + +Several patches rewrite outbound HTTP fetches to use internal jail addresses instead of +going through the public internet / nginx: +- `patch-micropub-fetch-internal-url` — Micropub post creation fetches +- `patch-bluesky-syndicator-internal-url` — Bluesky syndicator +- `_toInternalUrl()` helper in microsub/activitypub controllers + +## Collections (MongoDB) + +| Collection | Contents | +|------------|----------| +| `posts` | Micropub post data (path + properties) | +| `ap_timeline` | Incoming + outgoing AP posts; key: `uid` | +| `ap_notifications` | Mentions, replies, likes, boosts received | +| `ap_followers` | Follower actor URLs | +| `ap_following` | Following actor URLs | +| `ap_activities` | Activity log (outbound + inbound) | +| `ap_profile` | Own actor profile (name, icon, url) | +| `ap_interactions` | Likes and boosts performed by own account | + +## ActivityPub Actor + +- Handle: `activityPubHandle` from `AP_HANDLE` env → `GITHUB_USERNAME` (`svemagie`) → hostname prefix +- Full handle: `@svemagie@blog.giersig.eu` +- Actor URL: `https://blog.giersig.eu/activitypub/actor` +- AP objects served at: `https://blog.giersig.eu/activitypub/objects/note/{+id}` + - Own reply posts: `/activitypub/objects/note/replies/{slug}` + +## Patch Infrastructure + +Patches live in `scripts/patch-*.mjs`. Each script: +1. Checks if already applied (MARKER string) +2. Looks for OLD_SNIPPET in node_modules target file +3. Replaces with NEW_SNIPPET if found +4. Reports result to stdout + +Both `postinstall` and `serve` scripts in `package.json` run all patches in order. +Some patches (e.g. `patch-microsub-reader-ap-dispatch`) only appear in `serve`, not `postinstall`. +New AP patches are appended at the end of the AP patch chain (after `patch-ap-federation-bridge-base-url`).