240 lines
14 KiB
Markdown
240 lines
14 KiB
Markdown
# ActivityPub Federation — Architecture & Patch Chain
|
||
|
||
## Key Files
|
||
|
||
| File | Role |
|
||
|------|------|
|
||
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/controllers/compose.js` | AP reader compose form: GET builds the form with `syndicationTargets`, POST submits to Micropub |
|
||
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/mastodon/routes/statuses.js` | Mastodon-compatible API: `POST /api/v1/statuses` creates posts via Micropub pipeline |
|
||
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/syndicator.js` | AP syndicator: federates posts to followers, adds own posts to `ap_timeline` |
|
||
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/jf2-to-as2.js` | Converts JF2 properties to ActivityPub Note/Create activity |
|
||
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/storage/timeline.js` | `addTimelineItem()` — atomic upsert (`$setOnInsert`) to `ap_timeline` |
|
||
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/mastodon/entities/status.js` | Serializes `ap_timeline` documents to Mastodon Status JSON |
|
||
| `node_modules/@rmdes/indiekit-endpoint-activitypub/lib/mastodon/helpers/pagination.js` | `encodeCursor(date)` / `decodeCursor(id)` — ms-since-epoch as status ID |
|
||
| `node_modules/@indiekit/endpoint-micropub/lib/post-type-discovery.js` | `getPostType()` — returns "reply" if `in-reply-to` key present, else "note" |
|
||
| `node_modules/@indiekit/endpoint-micropub/lib/post-data.js` | Micropub create pipeline: normalise → getPostType → save |
|
||
| `node_modules/@rmdes/indiekit-endpoint-microsub/lib/controllers/reader.js` | Microsub reader compose: auto-selects syndication targets via `detectProtocol()` |
|
||
| `node_modules/@rmdes/indiekit-endpoint-activitypub/views/activitypub-compose.njk` | AP reader compose template: uses `target.defaultChecked` for checkbox state |
|
||
| `node_modules/@rmdes/indiekit-endpoint-activitypub/views/partials/ap-item-card.njk` | Timeline card: reply button passes `item.url` as `replyTo` query param |
|
||
|
||
## Data Flow: Reply via AP Reader UI
|
||
|
||
```
|
||
User clicks reply on timeline item
|
||
→ /activitypub/admin/reader/compose?replyTo=<url>
|
||
→ composeController() fetches syndicationTargets from Micropub q=config
|
||
→ sets target.defaultChecked = target.checked === true [patch: ap-compose-default-checked]
|
||
→ renders activitypub-compose.njk with hidden in-reply-to field
|
||
→ user submits form
|
||
→ submitComposeController() POSTs to Micropub with in-reply-to
|
||
→ Micropub: formEncodedToJf2 → normaliseProperties → getPostType("reply") → save to /replies/{slug}/
|
||
→ syndication webhook fires → AP syndicator.syndicate(properties)
|
||
→ jf2ToAS2Activity creates Note with inReplyTo
|
||
→ delivered to followers + original author's inbox
|
||
```
|
||
|
||
## Data Flow: Reply via Mastodon Client API (Phanpy/Elk)
|
||
|
||
```
|
||
Client sends POST /api/v1/statuses { status, in_reply_to_id }
|
||
→ findTimelineItemById(ap_timeline, in_reply_to_id)
|
||
→ decodes cursor (ms-since-epoch) → looks up by published date
|
||
→ inReplyTo = replyItem.uid || replyItem.url
|
||
→ jf2["in-reply-to"] = inReplyTo (if resolved)
|
||
→ jf2["mp-syndicate-to"] = [publicationUrl] (always set — AP-only)
|
||
→ postData.create → postContent.create
|
||
→ addTimelineItem immediately [patch: ap-mastodon-reply-threading]
|
||
→ returns minimal status JSON to client
|
||
→ build webhook → syndicator → AP delivery
|
||
```
|
||
|
||
## Own Post in ap_timeline
|
||
|
||
The AP syndicator stores outbound posts with:
|
||
- `uid = url = properties.url` (blog URL, e.g. `https://blog.giersig.eu/replies/slug/`)
|
||
- `published = properties.published` (ISO 8601 with TZ offset, e.g. `"2026-03-21T16:33:50+01:00"`)
|
||
- `inReplyTo = properties["in-reply-to"]` (original Mastodon URL or own blog URL)
|
||
|
||
The Mastodon API encodes status IDs as `encodeCursor(published)` = ms-since-epoch string.
|
||
`findTimelineItemById` uses a ±1 s range query with `$dateFromString` to handle TZ-offset strings.
|
||
|
||
## Syndication Target Config
|
||
|
||
The AP syndicator's `info` object:
|
||
```javascript
|
||
{
|
||
checked: true, // from indiekit.config.mjs: checked: true
|
||
name: `@${handle}@${hostname}`, // e.g. "@svemagie@blog.giersig.eu"
|
||
uid: publicationUrl, // e.g. "https://blog.giersig.eu/"
|
||
service: { name: "ActivityPub (Fediverse)", ... }
|
||
}
|
||
```
|
||
|
||
The Micropub `q=config` endpoint returns this with `checked: true`.
|
||
Both compose forms read this value:
|
||
- **Microsub reader**: uses `target.checked` directly (always pre-checked if `checked: true` in config)
|
||
- **AP reader**: uses `target.defaultChecked` set by `composeController()` — was broken, now fixed
|
||
|
||
## Patches Applied (AP threading)
|
||
|
||
### `patch-ap-compose-default-checked`
|
||
**File:** `lib/controllers/compose.js`
|
||
**Problem:** `target.defaultChecked` was hardcoded to `name === "@rick@rmendes.net"` — never matches.
|
||
**Fix:** `target.defaultChecked = target.checked === true`
|
||
**Effect:** AP syndication checkbox is pre-checked in the AP reader compose form, matching
|
||
the `checked: true` config — replies via the AP reader UI are now federated.
|
||
|
||
### `patch-ap-mastodon-reply-threading`
|
||
**File:** `lib/mastodon/routes/statuses.js`
|
||
**Problem:** After `POST /api/v1/statuses`, own post was NOT inserted into `ap_timeline` until
|
||
the Eleventy build webhook fired (30–120 s later). Follow-up replies during that window
|
||
would fail `findTimelineItemById` → `inReplyTo = null` → no `in-reply-to` in JF2 →
|
||
`getPostType` returned `"note"` → reply saved at `/notes/` with no `inReplyTo` in the AP activity.
|
||
**Fix:** Eagerly insert a provisional timeline item via `addTimelineItem()` immediately after
|
||
`postContent.create()`. Uses `$setOnInsert` (idempotent); syndicator's later upsert is a no-op.
|
||
**Effect:** `in_reply_to_id` can be resolved immediately → correct `"reply"` post type → proper
|
||
`inReplyTo` in the AP Note → thread displays correctly on Mastodon.
|
||
|
||
## Post Type Discovery
|
||
|
||
`getPostType(postTypes, properties)` in `post-type-discovery.js`:
|
||
- Checks `propertiesMap.has("in-reply-to")` → `"reply"`
|
||
- Checks `propertiesMap.has("like-of")` → `"like"`
|
||
- Checks `propertiesMap.has("repost-of")` → `"repost"`
|
||
- Falls through to `"note"` if none match
|
||
|
||
The `"reply"` post type in `indiekit.config.mjs` has no `discovery` field — standard PTD spec applies.
|
||
|
||
## Inbound AP Activity Pipeline
|
||
|
||
Activities from remote servers follow this path:
|
||
|
||
```
|
||
Remote server → nginx → Express (body buffered in createFedifyMiddleware)
|
||
→ Fedify signature check (uses req._rawBody for digest)
|
||
→ Fedify Redis message queue (if Redis configured)
|
||
→ Fedify queue worker → inbox listener (inbox-listeners.js)
|
||
→ enqueueActivity() → ap_inbox_queue (MongoDB)
|
||
→ startInboxProcessor() (1s poll) → routeToHandler()
|
||
→ handleLike / handleAnnounce / handleCreate
|
||
→ addNotification() → ap_notifications
|
||
```
|
||
|
||
**Critical: `collections._publicationUrl`** is set in `index.js` (`_publicationUrl: this._publicationUrl`)
|
||
AND by `patch-ap-inbox-publication-url` in `federation-setup.js`. Both set `"https://blog.giersig.eu/"`.
|
||
|
||
Notification conditions gate on `pubUrl && objectId.startsWith(pubUrl)`:
|
||
- `handleLike`: only notifies for likes of our own content
|
||
- `handleAnnounce` PATH 1: only notifies for boosts of our content
|
||
- `handleCreate`: only notifies for replies to our posts (`inReplyTo.startsWith(pubUrl)`)
|
||
|
||
**Body buffering** (`createFedifyMiddleware`): `application/activity+json` bodies are buffered
|
||
into `req._rawBody` before `express.json()` (which only handles `application/json`) touches them.
|
||
`fromExpressRequest` passes `req._rawBody` verbatim to the Fedify `Request` object so the
|
||
HTTP Signature Digest check passes.
|
||
|
||
**Fedify inbox log suppression**: `["fedify","federation","inbox"]` was hardcoded to `"fatal"`
|
||
(`patch-ap-inbox-delivery-debug` fixes this to `"error"` so real failures are visible).
|
||
|
||
**Diagnosing inbox delivery issues:**
|
||
- Set `AP_DEBUG=1` → logs `[AP-inbox] POST /activitypub/users/svemagie/inbox ct=... body=...B`
|
||
BEFORE Fedify's signature check. If this doesn't appear, activities aren't reaching our server.
|
||
- With inbox log level now `"error"`: signature failures show as Fedify error logs.
|
||
- Queue processing failures: `[inbox-queue] Failed processing ...` — always logged.
|
||
|
||
### `patch-ap-signature-host-header` *(2026-04-01)*
|
||
**File:** `lib/controllers/federation-bridge.js` → `fromExpressRequest()`
|
||
**Problem:** `patch-ap-federation-bridge-base-url` fixed Fedify URL routing to use the canonical
|
||
`publicationUrl`, but left the `host` header in the copied Headers object untouched. nginx forwards
|
||
an internal host (e.g. `10.100.0.20`) which Fedify reads from `request.headers.get("host")` when
|
||
reconstructing the signed-string for Cavage HTTP Signatures. Signed-string mismatch → every inbox
|
||
POST returns 401 → remote servers exhaust retries and stop delivering.
|
||
**Fix:** After the header-copy loop in `fromExpressRequest()`, override `"host"` with
|
||
`new URL(publicationUrl).host` (`"blog.giersig.eu"`) when `publicationUrl` is provided.
|
||
**Effect:** HTTP Signature verification now succeeds for all inbound AP activities.
|
||
|
||
### `patch-ap-mastodon-status-id` *(2026-04-01)*
|
||
**File:** `lib/mastodon/routes/statuses.js`
|
||
**Problem:** `POST /api/v1/statuses` returned `id: String(Date.now())` — the wall-clock time of the
|
||
response. The `ap_timeline` item uses `published: data.properties.published`, set before the Gitea
|
||
write (which can take 5–15 s). When the client replies to the freshly created post, it sends
|
||
`in_reply_to_id: <Date.now() id>`, which is 5–15 s later than the stored `published` → the ±1 s
|
||
range query misses → `inReplyTo = null` → reply saved as note.
|
||
**Fix:** Use `encodeCursor(data.properties.published)` as the status ID in the creation response
|
||
(falls back to `String(Date.now())` if published is missing). Response ID now matches what
|
||
`findTimelineItemById` will resolve.
|
||
|
||
### `patch-ap-interactions-send-guard` *(2026-04-01)*
|
||
**File:** `lib/mastodon/helpers/interactions.js`
|
||
**Problem:** `likePost` and `boostPost` call `ctx.sendActivity(...)` without try/catch. Any Fedify
|
||
or Redis error propagates → 500 response → the `ap_interactions` DB write never runs → interaction
|
||
not recorded locally.
|
||
**Fix:** Wrap both `sendActivity` calls in try/catch so delivery failures are non-fatal. Interaction
|
||
still recorded in `ap_interactions`; client sees correct UI state.
|
||
|
||
### `patch-ap-syndicate-dedup` *(2026-04-01)*
|
||
**File:** `lib/syndicator.js` → `syndicate()`
|
||
**Problem:** The CI webhook calls `/syndicate?source_url=X&force=true` after every Eleventy build.
|
||
When `syndicateToTargets()` saves the syndication URL it commits to Gitea → triggers another build
|
||
→ second CI call also hits the syndicate endpoint → duplicate `Create(Note)` activity sent.
|
||
Root cause: the AP syndicator UID (`publicationUrl`) shares the same origin as the syndication URL
|
||
(`properties.url`), so `force` mode re-selects it.
|
||
**Fix:** At the start of `syndicate()`, query `ap_activities` for an existing outbound
|
||
Create/Announce/Update for `properties.url`. If found, return the existing URL without re-federating.
|
||
|
||
### `patch-ap-mastodon-delete-fix` *(2026-04-01)*
|
||
**File:** `lib/mastodon/routes/statuses.js` (delete route) + `index.js`
|
||
**Bug 1 (ReferenceError):** Delete route used `objectId` (undefined) instead of `item._id` from
|
||
`findTimelineItemById` → every delete threw ReferenceError → 500 → timeline entry never removed.
|
||
**Bug 2 (no AP broadcast):** Route called `postContent.delete()` directly, bypassing the Indiekit
|
||
syndicator framework → no `Delete(Note)` activity sent to followers → post persists on Mastodon.
|
||
**Fix:** (a) Add `broadcastDelete: (url) => pluginRef.broadcastDelete(url)` to
|
||
`mastodonPluginOptions` in `index.js`. (b) Call `req.app.locals.mastodonPluginOptions.broadcastDelete(postUrl)`
|
||
after removing the timeline entry.
|
||
|
||
### `patch-micropub-delete-propagation` + `patch-bluesky-syndicator-delete` *(2026-04-01)*
|
||
**Files:** `node_modules/@indiekit/endpoint-micropub/lib/action.js` + Bluesky syndicator
|
||
**Problem:** Micropub `action=delete` only deleted the post from the content store. AP and Bluesky
|
||
syndications persisted.
|
||
**Fix:** After `postContent.delete()`, iterate `publication.syndicationTargets` and call
|
||
`syndicator.delete(url, syndication)` fire-and-forget for any syndicator exposing `.delete()`.
|
||
Bluesky syndicator extended with `deletePost(bskyUrl)` (via `com.atproto.repo.deleteRecord`) and
|
||
`delete(url, syndication)` that resolves the bsky.app URL from the preserved `_deletedProperties`.
|
||
|
||
### `patch-ap-inbox-publication-url` *(via 63bc41ebb, 2026-04-01)*
|
||
**File:** `lib/controllers/federation-setup.js`
|
||
**Problem:** `collections._publicationUrl` was never set in `federation-setup.js`, so every
|
||
`pubUrl && objectId.startsWith(pubUrl)` guard in `handleCreate`/`handleAnnounce` always evaluated
|
||
to `undefined` → no reply notifications, no boost notifications for own content, replies from
|
||
non-followers not stored in `ap_timeline`.
|
||
**Fix:** Set `collections._publicationUrl = publicationUrl` before `registerInboxListeners()`.
|
||
Also added else-if branch in `handleCreate` to store replies to own posts in `ap_timeline` even
|
||
when sender is not in `ap_following`.
|
||
|
||
### `patch-ap-status-reply-id` *(2026-04-01)*
|
||
**Files:** `lib/mastodon/entities/status.js` + `lib/mastodon/routes/statuses.js`
|
||
**Problem:** `in_reply_to_id` in the status serializer was a tautological `item.inReplyTo ? null : null`
|
||
(unfilled TODO) — always `null`. Mastodon clients (Phanpy/Elk) use this field to display reply
|
||
threading; without it, own replies appear as standalone posts.
|
||
**Fix (two parts):**
|
||
(A) `status.js`: return `item.inReplyToId || null` instead of the tautological null.
|
||
(B) `statuses.js` POST handler: when pre-inserting own posts into `ap_timeline` (reply-threading
|
||
patch), also store `inReplyToId: inReplyToId || null` — the raw `in_reply_to_id` cursor from
|
||
the client is already a valid `encodeCursor` value.
|
||
**Note:** Inbound AP replies from remote servers still have `inReplyToId = null` (separate patch
|
||
needed). Own replies via the Mastodon client API are fully fixed.
|
||
**Effect:** Own replies are threaded correctly in Phanpy/Elk.
|
||
|
||
---
|
||
|
||
## detectProtocol() in Microsub Reader
|
||
|
||
`detectProtocol(url)` in `reader.js` classifies URLs for syndication auto-selection:
|
||
- `"atmosphere"` — bsky.app / bluesky
|
||
- `"fediverse"` — mastodon., mstdn., fosstodon., troet., social., misskey., pixelfed., hachyderm., infosec.exchange, chaos.social (extended by `patch-microsub-reader-ap-dispatch`)
|
||
- `"web"` — everything else, including own blog URLs
|
||
|
||
Own blog URLs return `"web"`, so auto-selection doesn't trigger for reply-to-own-reply in the
|
||
microsub reader. This is harmless because `checked: true` in the config already pre-checks the
|
||
AP target in the microsub reader's `target.checked` field.
|