diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..e4bf928 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,299 @@ +# CLAUDE.md - Blogroll Endpoint + +## Package Overview + +`@rmdes/indiekit-endpoint-blogroll` is an Indiekit plugin that provides a comprehensive blogroll management system. It aggregates blog feeds from multiple sources (OPML files/URLs, Microsub subscriptions), fetches and caches recent items, and exposes both an admin UI and public JSON API. + +**Key Capabilities:** +- Aggregates blogs from OPML (URL or file), JSON feeds, or manual entry +- Integrates with Microsub plugin to mirror subscriptions +- Background feed fetching with configurable intervals +- Admin UI for managing sources, blogs, and viewing recent items +- Public read-only JSON API for frontend integration +- OPML export functionality + +**npm Package:** `@rmdes/indiekit-endpoint-blogroll` +**Version:** 1.0.17 +**Mount Path:** `/blogrollapi` (default, configurable) + +## Architecture + +### Data Flow + +``` +Sources (OPML/Microsub) → Blogs → Items + ↓ ↓ ↓ + blogrollSources blogrollBlogs blogrollItems + microsub_items (reference) +``` + +1. **Sources** define where blogs come from (OPML URL, OPML file, Microsub channels) +2. **Blogs** are individual feed subscriptions with metadata +3. **Items** are recent posts/articles from blogs (cached for 7 days by default) + +**Special Case: Microsub Integration** +- Microsub-sourced blogs store REFERENCES (`microsubFeedId`) not copies +- Items are queried from `microsub_items` collection directly (no duplication) +- Blogroll API transparently joins data from both sources + +### MongoDB Schema + +**blogrollSources** +```javascript +{ + _id: ObjectId, + type: "opml_url" | "opml_file" | "manual" | "json_feed" | "microsub", + name: String, // Display name + url: String | null, // For opml_url, json_feed + opmlContent: String | null, // For opml_file + // Microsub-specific + channelFilter: String | null, // Specific channel UID or null for all + categoryPrefix: String, // Prefix for blog categories + enabled: Boolean, + syncInterval: Number, // Minutes between syncs + lastSyncAt: String | null, // ISO 8601 + lastSyncError: String | null, + createdAt: String, // ISO 8601 + updatedAt: String // ISO 8601 +} +``` + +**blogrollBlogs** +```javascript +{ + _id: ObjectId, + sourceId: ObjectId | null, // Reference to blogrollSources + title: String, + description: String | null, + feedUrl: String, // Unique identifier + siteUrl: String | null, + feedType: "rss" | "atom" | "jsonfeed", + category: String, // For grouping/filtering + tags: String[], + photo: String | null, // Blog icon/avatar + author: Object | null, // { name: String } + status: "active" | "error" | "deleted", + lastFetchAt: String | null, // ISO 8601 + lastError: String | null, + itemCount: Number, + pinned: Boolean, + hidden: Boolean, + notes: String | null, + // Microsub-specific (when source === "microsub") + source: "microsub" | null, + microsubFeedId: String | null, // Reference to microsub_feeds._id + microsubChannelId: String | null, + microsubChannelName: String | null, + skipItemFetch: Boolean, // True for Microsub blogs + createdAt: String, // ISO 8601 + updatedAt: String // ISO 8601 +} +``` + +**blogrollItems** +```javascript +{ + _id: ObjectId, + blogId: ObjectId, // Reference to blogrollBlogs + uid: String, // Unique hash from feedUrl + itemId + url: String, + title: String, + content: { html: String, text: String }, + summary: String, + published: String, // ISO 8601 + updated: String | null, // ISO 8601 + author: Object | null, // { name: String } + photo: String[] | null, // Image URLs + categories: String[], + fetchedAt: String // ISO 8601 +} +``` + +**blogrollMeta** +```javascript +{ + key: "syncStats", + lastFullSync: String, // ISO 8601 + duration: Number, // Milliseconds + sources: { total: Number, success: Number, failed: Number }, + blogs: { total: Number, success: Number, failed: Number }, + items: { added: Number, deleted: Number } +} +``` + +## Key Files + +### Entry Point +- **index.js** - Plugin class, route registration, initialization + +### Controllers (Protected Routes) +- **lib/controllers/dashboard.js** - Main dashboard, sync triggers +- **lib/controllers/sources.js** - CRUD for sources (OPML/Microsub) +- **lib/controllers/blogs.js** - CRUD for blogs, manual refresh +- **lib/controllers/api.js** - Both protected and public API endpoints + +### Storage (MongoDB Operations) +- **lib/storage/sources.js** - Source CRUD, sync status +- **lib/storage/blogs.js** - Blog CRUD, upsert for sync, status updates +- **lib/storage/items.js** - Item CRUD, transparent Microsub integration + +### Sync Engine +- **lib/sync/scheduler.js** - Background sync, interval management +- **lib/sync/opml.js** - OPML parsing, fetch from URL, export +- **lib/sync/microsub.js** - Microsub channel/feed sync, webhook handler +- **lib/sync/feed.js** - RSS/Atom/JSON Feed parsing, item fetching + +### Utilities +- **lib/utils/feed-discovery.js** - Auto-discover feeds from website URLs + +## Configuration + +### Plugin Options +```javascript +new BlogrollEndpoint({ + mountPath: "/blogrollapi", // Admin UI and API base path + syncInterval: 3600000, // 1 hour (in milliseconds) + maxItemsPerBlog: 50, // Items to fetch per blog + maxItemAge: 7, // Days - older items deleted (encourages discovery) + fetchTimeout: 15000 // 15 seconds per feed fetch +}) +``` + +### Environment/Deployment +- Requires MongoDB (uses Indiekit's database connection) +- Background sync starts 15 seconds after server startup +- Periodic sync runs at `syncInterval` (default 1 hour) + +## Routes + +### Protected Routes (Admin UI) +``` +GET /blogrollapi/ Dashboard (stats, recent activity) +POST /blogrollapi/sync Manual sync trigger +POST /blogrollapi/clear-resync Clear all items and resync + +GET /blogrollapi/sources List sources +GET /blogrollapi/sources/new New source form +POST /blogrollapi/sources Create source +GET /blogrollapi/sources/:id Edit source form +POST /blogrollapi/sources/:id Update source +POST /blogrollapi/sources/:id/delete Delete source +POST /blogrollapi/sources/:id/sync Sync single source + +GET /blogrollapi/blogs List blogs +GET /blogrollapi/blogs/new New blog form +POST /blogrollapi/blogs Create blog +GET /blogrollapi/blogs/:id Edit blog form +POST /blogrollapi/blogs/:id Update blog +POST /blogrollapi/blogs/:id/delete Delete blog (soft delete) +POST /blogrollapi/blogs/:id/refresh Refresh single blog + +GET /blogrollapi/api/discover Feed discovery (protected) +POST /blogrollapi/api/microsub-webhook Microsub webhook handler +GET /blogrollapi/api/microsub-status Microsub integration status +``` + +### Public Routes (Read-Only API) +``` +GET /blogrollapi/api/blogs List blogs (JSON) +GET /blogrollapi/api/blogs/:id Get blog with recent items (JSON) +GET /blogrollapi/api/items List items across all blogs (JSON) +GET /blogrollapi/api/categories List categories with counts (JSON) +GET /blogrollapi/api/status Sync status (JSON) +GET /blogrollapi/api/opml Export all blogs as OPML +GET /blogrollapi/api/opml/:category Export category as OPML +``` + +### API Query Parameters +- **GET /api/blogs**: `?category=Tech&limit=100&offset=0` +- **GET /api/items**: `?blog=&category=Tech&limit=50&offset=0` + +## Inter-Plugin Relationships + +### Microsub Integration +- **Detection:** Checks `application.collections.get("microsub_channels")` for availability +- **Sync:** Reads `microsub_channels` and `microsub_feeds` to create blogroll references +- **Items:** Queries `microsub_items` directly (no duplication) +- **Webhook:** Receives notifications when feeds are subscribed/unsubscribed +- **Orphan Cleanup:** Soft-deletes blogs whose Microsub feed no longer exists + +### Homepage Plugin +- Provides homepage sections: None (this plugin doesn't register homepage sections) +- Can be used BY homepage plugin through public API endpoints + +### Data Dependencies +- **Requires:** MongoDB connection via Indiekit +- **Creates Collections:** `blogrollSources`, `blogrollBlogs`, `blogrollItems`, `blogrollMeta` +- **Reads Collections:** `microsub_channels`, `microsub_feeds`, `microsub_items` (when Microsub plugin is installed) + +## Known Gotchas + +### Date Handling +- **Store dates as ISO strings** (`new Date().toISOString()`), NOT Date objects +- The Nunjucks `| date` filter crashes on Date objects +- Controllers convert Date objects to ISO strings before passing to templates +- See CLAUDE.md root: "CRITICAL: Indiekit Date Handling Convention" + +### Microsub Reference Architecture +- Microsub blogs have `source: "microsub"` and `skipItemFetch: true` +- Items are NOT copied to `blogrollItems` - queried from `microsub_items` directly +- The `getItems()` and `getItemsForBlog()` functions transparently join both sources +- DO NOT run feed fetch on Microsub blogs - Microsub handles that + +### Soft Deletion +- Blogs are soft-deleted (`status: "deleted"`, `hidden: true`) not removed +- This prevents OPML/Microsub sync from recreating manually deleted blogs +- `upsertBlog()` skips blogs with `status: "deleted"` + +### Item Retention +- Items older than `maxItemAge` (default 7 days) are auto-deleted on each sync +- This is intentional to encourage discovery of fresh content +- Adjust `maxItemAge` for longer retention + +### Flash Messages +- Uses session-based flash messages for user feedback +- `consumeFlashMessage(request)` extracts and clears messages +- Returns `{ success, error }` for Indiekit's native `notificationBanner` + +## Dependencies + +```json +{ + "@indiekit/error": "^1.0.0-beta.25", + "@indiekit/frontend": "^1.0.0-beta.25", + "express": "^5.0.0", + "feedparser": "^2.2.10", // RSS/Atom parsing + "sanitize-html": "^2.13.0", // XSS prevention for feed content + "xml2js": "^0.6.2" // OPML parsing +} +``` + +## Testing Notes + +- **No test suite configured** (manual testing only) +- Test against real feeds: RSS, Atom, JSON Feed +- Test OPML import (nested categories) +- Test Microsub integration (requires `@rmdes/indiekit-endpoint-microsub`) +- Test soft delete behavior (re-sync should not recreate deleted blogs) + +## Common Tasks + +### Add a New Source Type +1. Add type to `createSource()` in `lib/storage/sources.js` +2. Implement sync function in `lib/sync/` (e.g., `syncJsonFeedSource()`) +3. Add handler in `runFullSync()` in `lib/sync/scheduler.js` +4. Update source form UI + +### Change Item Retention Period +- Modify `maxItemAge` plugin option (default 7 days) +- Items older than this are deleted on each sync + +### Debug Sync Issues +- Check `blogrollMeta.syncStats` document for last sync results +- Check `blogs.lastError` and `sources.lastSyncError` for failures +- Tail logs for `[Blogroll]` prefix messages + +### Integrate with Frontend +- Use public API endpoints (`/blogrollapi/api/blogs`, `/blogrollapi/api/items`) +- OPML export available at `/blogrollapi/api/opml` +- All public endpoints return JSON (except OPML which returns XML) diff --git a/README.md b/README.md new file mode 100644 index 0000000..2bce0da --- /dev/null +++ b/README.md @@ -0,0 +1,241 @@ +# Blogroll Endpoint for Indiekit + +An Indiekit plugin that provides a comprehensive blogroll management system with feed aggregation, admin UI, and public API. + +## Features + +- **Multiple Source Types:** Import blogs from OPML files/URLs, Microsub subscriptions, or add manually +- **Background Feed Fetching:** Automatically syncs blogs and caches recent items +- **Microsub Integration:** Mirror your Microsub subscriptions as a blogroll (zero duplication) +- **Admin UI:** Manage sources, blogs, and view recent activity +- **Public JSON API:** Read-only endpoints for frontend integration +- **OPML Export:** Export your blogroll as OPML (all or by category) +- **Feed Discovery:** Auto-discover feeds from website URLs +- **Item Retention:** Automatic cleanup of old items (encourages fresh content discovery) + +## Installation + +```bash +npm install @rmdes/indiekit-endpoint-blogroll +``` + +## Configuration + +Add to your `indiekit.config.js`: + +```javascript +import BlogrollEndpoint from "@rmdes/indiekit-endpoint-blogroll"; + +export default { + plugins: [ + new BlogrollEndpoint({ + mountPath: "/blogrollapi", // Admin UI and API base path + syncInterval: 3600000, // 1 hour (in milliseconds) + maxItemsPerBlog: 50, // Items to fetch per blog + maxItemAge: 7, // Days - older items auto-deleted + fetchTimeout: 15000 // 15 seconds per feed fetch + }) + ] +}; +``` + +## Requirements + +- **Indiekit:** `>=1.0.0-beta.25` +- **MongoDB:** Required for data storage +- **Optional:** `@rmdes/indiekit-endpoint-microsub` for Microsub integration + +## Usage + +### Admin UI + +Navigate to `/blogrollapi` in your Indiekit instance to access: + +- **Dashboard:** View sync status, blog counts, recent activity +- **Sources:** Manage OPML and Microsub sources +- **Blogs:** Add/edit/delete individual blogs, refresh feeds +- **Manual Sync:** Trigger immediate sync or clear and resync + +### Source Types + +1. **OPML URL:** Point to a public OPML file (e.g., your feed reader's export) +2. **OPML File:** Paste OPML XML directly into the form +3. **Microsub:** Import subscriptions from your Microsub channels +4. **Manual:** Add individual blog feeds one at a time + +### Public API + +All API endpoints return JSON (except OPML export which returns XML). + +**List Blogs** +``` +GET /blogrollapi/api/blogs?category=Tech&limit=100&offset=0 +``` + +**Get Blog with Recent Items** +``` +GET /blogrollapi/api/blogs/:id +``` + +**List Items Across All Blogs** +``` +GET /blogrollapi/api/items?blog=&category=Tech&limit=50&offset=0 +``` + +**List Categories** +``` +GET /blogrollapi/api/categories +``` + +**Sync Status** +``` +GET /blogrollapi/api/status +``` + +**Export OPML** +``` +GET /blogrollapi/api/opml (all blogs) +GET /blogrollapi/api/opml/:category (specific category) +``` + +### Example Response + +**GET /blogrollapi/api/blogs** +```json +{ + "items": [ + { + "id": "507f1f77bcf86cd799439011", + "title": "Example Blog", + "description": "A great blog about tech", + "feedUrl": "https://example.com/feed", + "siteUrl": "https://example.com", + "feedType": "rss", + "category": "Tech", + "tags": ["programming", "web"], + "photo": "https://example.com/icon.png", + "status": "active", + "itemCount": 25, + "pinned": false, + "lastFetchAt": "2026-02-13T10:30:00.000Z" + } + ], + "total": 42, + "hasMore": true +} +``` + +**GET /blogrollapi/api/items** +```json +{ + "items": [ + { + "id": "507f1f77bcf86cd799439011", + "url": "https://example.com/post/hello", + "title": "Hello World", + "summary": "My first blog post...", + "published": "2026-02-13T10:00:00.000Z", + "isFuture": false, + "author": { "name": "Jane Doe" }, + "photo": ["https://example.com/image.jpg"], + "categories": ["announcement"], + "blog": { + "id": "507f1f77bcf86cd799439011", + "title": "Example Blog", + "siteUrl": "https://example.com", + "category": "Tech", + "photo": "https://example.com/icon.png" + } + } + ], + "hasMore": false +} +``` + +## Microsub Integration + +If you have `@rmdes/indiekit-endpoint-microsub` installed, the blogroll can mirror your subscriptions: + +1. Create a Microsub source in the admin UI +2. Select specific channels or sync all channels +3. Add a category prefix (optional) to distinguish Microsub blogs +4. Blogs and items are referenced, not duplicated + +**Benefits:** +- Zero data duplication - items are served directly from Microsub +- Automatic orphan cleanup when feeds are unsubscribed +- Webhook support for real-time updates + +## Background Sync + +The plugin automatically syncs in the background: + +1. **Initial Sync:** Runs 15 seconds after server startup +2. **Periodic Sync:** Runs every `syncInterval` milliseconds (default 1 hour) +3. **What it Does:** + - Syncs enabled sources (OPML/Microsub) + - Fetches new items from active blogs + - Deletes items older than `maxItemAge` days + - Updates sync statistics + +**Manual Sync:** +- Trigger from the dashboard +- Use `POST /blogrollapi/sync` (protected endpoint) +- Use `POST /blogrollapi/clear-resync` to clear and resync all + +## Feed Discovery + +The plugin includes auto-discovery for finding feeds from website URLs: + +```javascript +// In the admin UI, when adding a blog, paste a website URL +// The plugin will: +// 1. Check tags in HTML +// 2. Try common feed paths (/feed, /rss, /atom.xml, etc.) +// 3. Suggest discovered feeds +``` + +## Item Retention + +By default, items older than 7 days are automatically deleted during sync. This encourages discovery of fresh content rather than archiving everything. + +**To Change Retention:** +```javascript +new BlogrollEndpoint({ + maxItemAge: 30 // Keep items for 30 days instead +}) +``` + +## Blog Status + +- **active:** Blog is working, fetching items normally +- **error:** Last fetch failed (see `lastError` for details) +- **deleted:** Soft-deleted, won't be recreated by sync + +## Navigation + +The plugin adds itself to Indiekit's navigation: + +- **Menu Item:** "Blogroll" (requires database) +- **Shortcut:** Bookmark icon in admin dashboard + +## Security + +- **Protected Routes:** Admin UI and management endpoints require authentication +- **Public Routes:** Read-only API endpoints are publicly accessible +- **XSS Prevention:** Feed content is sanitized with `sanitize-html` +- **Feed Discovery:** Protected to prevent abuse (requires authentication) + +## Supported Feed Formats + +- RSS 2.0 +- Atom 1.0 +- JSON Feed 1.0 + +## Contributing + +Report issues at: https://github.com/rmdes/indiekit-endpoint-blogroll/issues + +## License + +MIT