giersig.eu/indiekit-server
1
0
Fork 0
Code Issues Actions Packages Projects Releases Activity

docs: add CLAUDE.md and memory files for AP threading context

Browse Source 
CLAUDE.md covers patch authoring rules, post-type discovery, the two
reply compose paths, ap_timeline insertion timing, fork dependencies,
and common debugging entry points.

memory/ contains three files:
- project_activitypub.md — data flows, syndicator config, all AP patches
- project_architecture.md — FreeBSD jails, MongoDB collections, actor URLs
- feedback_patches.md — patch pattern, known fragile points, threading gotchas

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Sven
2026-03-30 08:30:32 +02:00
parent 97d99976ea
commit 5e50d7aceb
5 changed files with 404 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

157
CLAUDE.md Normal file
View File

@@ -0,0 +1,157 @@
# CLAUDE.md — indiekit-blog
Personal [Indiekit](https://getindiekit.com/) deployment for [blog.giersig.eu](https://blog.giersig.eu).
Built on the [rmdes/indiekit](https://github.com/rmdes/indiekit) fork ecosystem with a set of
patch scripts that fix issues that cannot be upstreamed.
## Memory files
Detailed architecture and lessons learned live in `memory/`:
- [`memory/project_activitypub.md`](memory/project_activitypub.md) — AP data flows, own-post timeline, compose paths, all AP patches
- [`memory/project_architecture.md`](memory/project_architecture.md) — FreeBSD jails, nginx, MongoDB collections, actor URLs, patch infrastructure
- [`memory/feedback_patches.md`](memory/feedback_patches.md) — Patch authoring patterns, known fragile points, compose-path gotchas
**Always read the relevant memory file before investigating or modifying anything in this repo.**
---
## Running
```sh
npm run serve # apply all patches then start Indiekit
npm run postinstall # re-apply patches after npm install (postinstall runs automatically)
```
Do **not** restart with `node` directly — patches must run first.
---
## Patch system
All fixes to `node_modules` live in `scripts/patch-*.mjs`.
Both `postinstall` and `serve` in `package.json` run them in order.
### Writing a patch
```js
const MARKER = "// [patch] my-patch-name"; // idempotency check
const OLD_SNIPPET = `exact source text`;
const NEW_SNIPPET = `replacement ${MARKER}`;
// 1. Skip if MARKER already present
// 2. Warn if OLD_SNIPPET not found (upstream changed)
// 3. Replace + writeFile
```
- Match node_modules text **byte-for-byte** (spaces, not tabs; exact line endings).
- Always include both candidate paths: `node_modules/@rmdes/...` and
`node_modules/@indiekit/indiekit/node_modules/@rmdes/...`.
- Template literals in patch strings: escape `` \` `` and `\${}`.
- Append new AP patches **after** `patch-ap-federation-bridge-base-url` in both
`postinstall` and `serve` in `package.json`.
- Some patches are `serve`-only (not `postinstall`) — check both when adding microsub patches.
---
## ActivityPub actor
| Field | Value |
|---|---|
| Handle | `svemagie` (env `AP_HANDLE`) |
| Full handle | `@svemagie@blog.giersig.eu` |
| Actor URL | `https://blog.giersig.eu/activitypub/users/svemagie` |
| Mastodon migration alias | `@svemagie@troet.cafe` (`AP_ALSO_KNOWN_AS`) |
AP syndicator `info.uid = publicationUrl` (`https://blog.giersig.eu/`), `info.checked = true`.
---
## Post type discovery
`getPostType(postTypes, properties)` in `post-type-discovery.js`:
| Property present | Type |
|---|---|
| `in-reply-to` | `reply` → `/replies/{slug}/` |
| `like-of` | `like` → `/likes/{slug}/` |
| `repost-of` | `repost` → `/reposts/{slug}/` |
| `photo` | `photo` → `/photos/{slug}/` |
| _(none)_ | `note` → `/notes/{slug}/` |
Only KEY presence matters, not value. If `in-reply-to` is silently missing from the JF2
object, the post becomes `"note"` with no error — this is a common threading bug root cause.
---
## Reply threading — two compose paths
| Path | How AP target is pre-checked |
|---|---|
| AP reader `/activitypub/admin/reader/compose` | `target.defaultChecked = target.checked === true` (patched) |
| Microsub reader `/microsub/admin/reader/compose` | `target.checked` from Micropub `q=config` (already `true` in config) |
| Mastodon client API `POST /api/v1/statuses` | `mp-syndicate-to` hardcoded to `publicationUrl` (always AP) |
The AP reader template uses `target.defaultChecked`, NOT `target.checked` — these are different.
`patch-ap-compose-default-checked` maps the config's `checked: true` through to `defaultChecked`.
### Mastodon API reply-to-reply (patched)
`POST /api/v1/statuses` resolves `in_reply_to_id` via `findTimelineItemById(ap_timeline, id)`.
Own posts were previously not inserted into `ap_timeline` until the Eleventy build webhook fired
(30–120 s). `patch-ap-mastodon-reply-threading` inserts a provisional item immediately after
`postContent.create()` using `addTimelineItem()` (`$setOnInsert` — idempotent).
---
## ap_timeline
Key collection. Documents keyed by `uid`.
- **Inbound posts** — inserted by `inbox-handlers.js` on Create/Update/Announce activities
- **Own posts** — inserted by `syndicator.js` after delivery (after build) AND now immediately
by `POST /api/v1/statuses` (via patch)
- **Status IDs** — `encodeCursor(published)` = ms-since-epoch string; `findTimelineItemById`
resolves them with a ±1 s MongoDB `$dateFromString` range query to handle TZ-offset strings
---
## Fork dependencies
Four packages come from GitHub forks (not npm):
| Package | Fork |
|---|---|
| `@rmdes/indiekit-endpoint-activitypub` | `github:svemagie/indiekit-endpoint-activitypub` |
| `@rmdes/indiekit-endpoint-blogroll` | `github:svemagie/indiekit-endpoint-blogroll#bookmark-import` |
| `@rmdes/indiekit-endpoint-microsub` | `github:svemagie/indiekit-endpoint-microsub#bookmarks-import` |
| `@rmdes/indiekit-endpoint-youtube` | `github:svemagie/indiekit-endpoint-youtube` |
To pull latest fork commits: `npm install github:svemagie/<package-name>`.
---
## Environment variables (key ones)
| Var | Purpose |
|---|---|
| `AP_HANDLE` | ActivityPub handle (default: `svemagie` from `GITHUB_USERNAME`) |
| `PUBLICATION_URL` | Canonical blog URL (`https://blog.giersig.eu`) |
| `INDIEKIT_URL` | Application URL (same as publication URL here) |
| `MONGO_HOST` / `MONGO_URL` | MongoDB connection |
| `REDIS_URL` | Redis for AP message queue and KV store (production) |
| `AP_ALSO_KNOWN_AS` | Migration alias (old Mastodon handle) |
| `GH_CONTENT_TOKEN` | GitHub token for writing posts to the `blog` repo |
---
## Common debugging starting points
- **Post created as "note" instead of "reply"** → `in-reply-to` missing from JF2. Check: form
hidden field, `submitComposeController`, Mastodon API `findTimelineItemById`, `formEncodedToJf2`.
- **Reply not federated to AP** → `mp-syndicate-to` not set. Check: `target.defaultChecked` /
`target.checked` in compose form, `getSyndicateToProperty` in `jf2.js`.
- **AP object lookup 302 redirect** → nginx not forwarding `Host` / `X-Forwarded-Proto`; see
`patch-ap-federation-bridge-base-url`.
- **Timeline item not found by Mastodon client** → `findTimelineItemById` date mismatch; stored
date is TZ-offset string, lookup decodes cursor to UTC ISO — relies on `$dateFromString` range.

5
memory/MEMORY.md Normal file
View File

@@ -0,0 +1,5 @@
# Memory Index
- [project_activitypub.md](project_activitypub.md) - ActivityPub federation architecture and patch chain dependencies
- [project_architecture.md](project_architecture.md) - Server architecture: FreeBSD jails, nginx, internal fetch URLs
- [feedback_patches.md](feedback_patches.md) - Patch management lessons learned

69
memory/feedback_patches.md Normal file
View File

@@ -0,0 +1,69 @@
# Patch Management — Lessons Learned
## Patch Script Pattern
```javascript
const MARKER = "// [patch] my-patch-name";
const OLD_SNIPPET = `exact text to find`;
const NEW_SNIPPET = `replacement text ${MARKER}`;
// Check MARKER first → skip if already applied
// Check OLD_SNIPPET → warn if not found (upstream may have changed)
// Replace and write only if source changed
```
Use `$setOnInsert` for MongoDB upserts in patches that add timeline items — idempotent,
safe to call multiple times (e.g. from both patch and syndicator).
## Target File Candidates
Always include both paths:
```javascript
const candidates = [
"node_modules/@rmdes/indiekit-endpoint-activitypub/...",
"node_modules/@indiekit/indiekit/node_modules/@rmdes/indiekit-endpoint-activitypub/...",
];
```
## Known Fragile Patterns
- **Exact whitespace matters.** Patch OLD_SNIPPETs must match node_modules source byte-for-byte,
including indentation (spaces not tabs) and exact line endings.
- **Template literals in patches** — escape backticks and `${}` in patch script string literals
using `\`` and `\${}`.
- **`patch-microsub-reader-ap-dispatch`** is in `serve` only (not `postinstall`). Reason unknown
but may relate to timing or the microsub package being rebuilt differently. Check both scripts
when adding new microsub patches.
## AP Threading — Two Compose Paths
Replies can come from two different UIs. Each has different syndication logic:
| Path | Syndication checkbox pre-checked by |
|------|-------------------------------------|
| AP reader (`/activitypub/admin/reader/compose`) | `target.defaultChecked` set in `composeController()` |
| Microsub reader (`/microsub/admin/reader/compose`) | `target.checked` from Micropub `q=config` response |
| Mastodon client API (`POST /api/v1/statuses`) | `mp-syndicate-to` hardcoded to `publicationUrl` (always AP) |
The AP reader compose form uses `target.defaultChecked` (NOT `target.checked`) in its template.
If `composeController()` doesn't set `defaultChecked` correctly, the AP checkbox is unchecked
even though `target.checked = true` from the Micropub config.
## ap_timeline Insertion Timing
Own posts reach `ap_timeline` via TWO paths:
1. **Mastodon API** (`POST /api/v1/statuses`): Now inserts immediately after `postContent.create()`
via `patch-ap-mastodon-reply-threading`. The syndicator's later upsert is a no-op (`$setOnInsert`).
2. **Micropub + syndication webhook**: AP syndicator inserts after Eleventy build completes (30–120 s).
Always ensure new post creation paths insert to `ap_timeline` immediately if the post may be
replied to again before the build webhook fires.
## Post Type Discovery Dependency
`getPostType(postTypes, properties)` uses `Map.has(key)` — only KEY presence matters, not value.
- `"in-reply-to"` present (any value, even empty) → `"reply"`
- If `in-reply-to` is absent → falls through to `"note"`
If `in-reply-to` is silently lost (null inReplyTo in Mastodon API, unchecked form field, etc.),
the post silently becomes a note with no error. Check this first when debugging "wrong post type" issues.

117
memory/project_activitypub.md Normal file
View File

@@ -0,0 +1,117 @@
# ActivityPub Federation — Architecture & Patch Chain
## Key Files
| File | Role |
|------|------|
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/controllers/compose.js` | AP reader compose form: GET builds the form with `syndicationTargets`, POST submits to Micropub |
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/mastodon/routes/statuses.js` | Mastodon-compatible API: `POST /api/v1/statuses` creates posts via Micropub pipeline |
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/syndicator.js` | AP syndicator: federates posts to followers, adds own posts to `ap_timeline` |
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/jf2-to-as2.js` | Converts JF2 properties to ActivityPub Note/Create activity |
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/storage/timeline.js` | `addTimelineItem()` — atomic upsert (`$setOnInsert`) to `ap_timeline` |
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/mastodon/entities/status.js` | Serializes `ap_timeline` documents to Mastodon Status JSON |
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/mastodon/helpers/pagination.js` | `encodeCursor(date)` / `decodeCursor(id)` — ms-since-epoch as status ID |
| `node_modules/@indiekit/endpoint-micropub/lib/post-type-discovery.js` | `getPostType()` — returns "reply" if `in-reply-to` key present, else "note" |
| `node_modules/@indiekit/endpoint-micropub/lib/post-data.js` | Micropub create pipeline: normalise → getPostType → save |
| `node_modules/@rmdes/indiekit-endpoint-microsub/lib/controllers/reader.js` | Microsub reader compose: auto-selects syndication targets via `detectProtocol()` |
| `node_modules/@rmdes/indiekit-endpoint-activitypub/views/activitypub-compose.njk` | AP reader compose template: uses `target.defaultChecked` for checkbox state |
| `node_modules/@rmdes/indiekit-endpoint-activitypub/views/partials/ap-item-card.njk` | Timeline card: reply button passes `item.url` as `replyTo` query param |
## Data Flow: Reply via AP Reader UI
```
User clicks reply on timeline item
→ /activitypub/admin/reader/compose?replyTo=<url>
→ composeController() fetches syndicationTargets from Micropub q=config
→ sets target.defaultChecked = target.checked === true [patch: ap-compose-default-checked]
→ renders activitypub-compose.njk with hidden in-reply-to field
→ user submits form
→ submitComposeController() POSTs to Micropub with in-reply-to
→ Micropub: formEncodedToJf2 → normaliseProperties → getPostType("reply") → save to /replies/{slug}/
→ syndication webhook fires → AP syndicator.syndicate(properties)
→ jf2ToAS2Activity creates Note with inReplyTo
→ delivered to followers + original author's inbox
```
## Data Flow: Reply via Mastodon Client API (Phanpy/Elk)
```
Client sends POST /api/v1/statuses { status, in_reply_to_id }
→ findTimelineItemById(ap_timeline, in_reply_to_id)
→ decodes cursor (ms-since-epoch) → looks up by published date
→ inReplyTo = replyItem.uid || replyItem.url
→ jf2["in-reply-to"] = inReplyTo (if resolved)
→ jf2["mp-syndicate-to"] = [publicationUrl] (always set — AP-only)
→ postData.create → postContent.create
→ addTimelineItem immediately [patch: ap-mastodon-reply-threading]
→ returns minimal status JSON to client
→ build webhook → syndicator → AP delivery
```
## Own Post in ap_timeline
The AP syndicator stores outbound posts with:
- `uid = url = properties.url` (blog URL, e.g. `https://blog.giersig.eu/replies/slug/`)
- `published = properties.published` (ISO 8601 with TZ offset, e.g. `"2026-03-21T16:33:50+01:00"`)
- `inReplyTo = properties["in-reply-to"]` (original Mastodon URL or own blog URL)
The Mastodon API encodes status IDs as `encodeCursor(published)` = ms-since-epoch string.
`findTimelineItemById` uses a ±1 s range query with `$dateFromString` to handle TZ-offset strings.
## Syndication Target Config
The AP syndicator's `info` object:
```javascript
{
checked: true, // from indiekit.config.mjs: checked: true
name: `@${handle}@${hostname}`, // e.g. "@svemagie@blog.giersig.eu"
uid: publicationUrl, // e.g. "https://blog.giersig.eu/"
service: { name: "ActivityPub (Fediverse)", ... }
}
```
The Micropub `q=config` endpoint returns this with `checked: true`.
Both compose forms read this value:
- **Microsub reader**: uses `target.checked` directly (always pre-checked if `checked: true` in config)
- **AP reader**: uses `target.defaultChecked` set by `composeController()` — was broken, now fixed
## Patches Applied (AP threading)
### `patch-ap-compose-default-checked`
**File:** `lib/controllers/compose.js`
**Problem:** `target.defaultChecked` was hardcoded to `name === "@rick@rmendes.net"` — never matches.
**Fix:** `target.defaultChecked = target.checked === true`
**Effect:** AP syndication checkbox is pre-checked in the AP reader compose form, matching
the `checked: true` config — replies via the AP reader UI are now federated.
### `patch-ap-mastodon-reply-threading`
**File:** `lib/mastodon/routes/statuses.js`
**Problem:** After `POST /api/v1/statuses`, own post was NOT inserted into `ap_timeline` until
the Eleventy build webhook fired (30–120 s later). Follow-up replies during that window
would fail `findTimelineItemById` → `inReplyTo = null` → no `in-reply-to` in JF2 →
`getPostType` returned `"note"` → reply saved at `/notes/` with no `inReplyTo` in the AP activity.
**Fix:** Eagerly insert a provisional timeline item via `addTimelineItem()` immediately after
`postContent.create()`. Uses `$setOnInsert` (idempotent); syndicator's later upsert is a no-op.
**Effect:** `in_reply_to_id` can be resolved immediately → correct `"reply"` post type → proper
`inReplyTo` in the AP Note → thread displays correctly on Mastodon.
## Post Type Discovery
`getPostType(postTypes, properties)` in `post-type-discovery.js`:
- Checks `propertiesMap.has("in-reply-to")` → `"reply"`
- Checks `propertiesMap.has("like-of")` → `"like"`
- Checks `propertiesMap.has("repost-of")` → `"repost"`
- Falls through to `"note"` if none match
The `"reply"` post type in `indiekit.config.mjs` has no `discovery` field — standard PTD spec applies.
## detectProtocol() in Microsub Reader
`detectProtocol(url)` in `reader.js` classifies URLs for syndication auto-selection:
- `"atmosphere"` — bsky.app / bluesky
- `"fediverse"` — mastodon., mstdn., fosstodon., troet., social., misskey., pixelfed., hachyderm., infosec.exchange, chaos.social (extended by `patch-microsub-reader-ap-dispatch`)
- `"web"` — everything else, including own blog URLs
Own blog URLs return `"web"`, so auto-selection doesn't trigger for reply-to-own-reply in the
microsub reader. This is harmless because `checked: true` in the config already pre-checks the
AP target in the microsub reader's `target.checked` field.

56
memory/project_architecture.md Normal file
View File

@@ -0,0 +1,56 @@
# Server Architecture
## Infrastructure
- **FreeBSD jails** — Indiekit runs in an isolated jail
- **nginx** — reverse proxy; must forward `Host: blog.giersig.eu` and `X-Forwarded-Proto: https`
for Fedify to construct correct canonical URLs (see `patch-ap-federation-bridge-base-url`)
- **MongoDB** — `10.100.0.20:27017`, database `indiekit`, auth source `admin`
- **Redis** — optional; URL via `REDIS_URL` env var; used for AP activity queue
## Publication URLs
- `publicationBaseUrl` = `https://blog.giersig.eu` (from `PUBLICATION_URL` env or hardcoded default)
- `applicationBaseUrl` = same (from `INDIEKIT_URL` env)
- GitHub repo: `svemagie/blog`, branch `main`
## Internal Fetch
Several patches rewrite outbound HTTP fetches to use internal jail addresses instead of
going through the public internet / nginx:
- `patch-micropub-fetch-internal-url` — Micropub post creation fetches
- `patch-bluesky-syndicator-internal-url` — Bluesky syndicator
- `_toInternalUrl()` helper in microsub/activitypub controllers
## Collections (MongoDB)
| Collection | Contents |
|------------|----------|
| `posts` | Micropub post data (path + properties) |
| `ap_timeline` | Incoming + outgoing AP posts; key: `uid` |
| `ap_notifications` | Mentions, replies, likes, boosts received |
| `ap_followers` | Follower actor URLs |
| `ap_following` | Following actor URLs |
| `ap_activities` | Activity log (outbound + inbound) |
| `ap_profile` | Own actor profile (name, icon, url) |
| `ap_interactions` | Likes and boosts performed by own account |
## ActivityPub Actor
- Handle: `activityPubHandle` from `AP_HANDLE` env → `GITHUB_USERNAME` (`svemagie`) → hostname prefix
- Full handle: `@svemagie@blog.giersig.eu`
- Actor URL: `https://blog.giersig.eu/activitypub/actor`
- AP objects served at: `https://blog.giersig.eu/activitypub/objects/note/{+id}`
- Own reply posts: `/activitypub/objects/note/replies/{slug}`
## Patch Infrastructure
Patches live in `scripts/patch-*.mjs`. Each script:
1. Checks if already applied (MARKER string)
2. Looks for OLD_SNIPPET in node_modules target file
3. Replaces with NEW_SNIPPET if found
4. Reports result to stdout
Both `postinstall` and `serve` scripts in `package.json` run all patches in order.
Some patches (e.g. `patch-microsub-reader-ap-dispatch`) only appear in `serve`, not `postinstall`.
New AP patches are appended at the end of the AP patch chain (after `patch-ap-federation-bridge-base-url`).