mirror of
https://github.com/svemagie/indiekit-endpoint-blogroll.git
synced 2026-04-02 07:24:57 +02:00
docs: add CLAUDE.md and README.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Ricardo
299
CLAUDE.md
Normal file
299
CLAUDE.md
Normal file
@@ -0,0 +1,299 @@
|
||||
# CLAUDE.md - Blogroll Endpoint
|
||||
|
||||
## Package Overview
|
||||
|
||||
`@rmdes/indiekit-endpoint-blogroll` is an Indiekit plugin that provides a comprehensive blogroll management system. It aggregates blog feeds from multiple sources (OPML files/URLs, Microsub subscriptions), fetches and caches recent items, and exposes both an admin UI and public JSON API.
|
||||
|
||||
**Key Capabilities:**
|
||||
- Aggregates blogs from OPML (URL or file), JSON feeds, or manual entry
|
||||
- Integrates with Microsub plugin to mirror subscriptions
|
||||
- Background feed fetching with configurable intervals
|
||||
- Admin UI for managing sources, blogs, and viewing recent items
|
||||
- Public read-only JSON API for frontend integration
|
||||
- OPML export functionality
|
||||
|
||||
**npm Package:** `@rmdes/indiekit-endpoint-blogroll`
|
||||
**Version:** 1.0.17
|
||||
**Mount Path:** `/blogrollapi` (default, configurable)
|
||||
|
||||
## Architecture
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
Sources (OPML/Microsub) → Blogs → Items
|
||||
↓ ↓ ↓
|
||||
blogrollSources blogrollBlogs blogrollItems
|
||||
microsub_items (reference)
|
||||
```
|
||||
|
||||
1. **Sources** define where blogs come from (OPML URL, OPML file, Microsub channels)
|
||||
2. **Blogs** are individual feed subscriptions with metadata
|
||||
3. **Items** are recent posts/articles from blogs (cached for 7 days by default)
|
||||
|
||||
**Special Case: Microsub Integration**
|
||||
- Microsub-sourced blogs store REFERENCES (`microsubFeedId`) not copies
|
||||
- Items are queried from `microsub_items` collection directly (no duplication)
|
||||
- Blogroll API transparently joins data from both sources
|
||||
|
||||
### MongoDB Schema
|
||||
|
||||
**blogrollSources**
|
||||
```javascript
|
||||
{
|
||||
_id: ObjectId,
|
||||
type: "opml_url" | "opml_file" | "manual" | "json_feed" | "microsub",
|
||||
name: String, // Display name
|
||||
url: String | null, // For opml_url, json_feed
|
||||
opmlContent: String | null, // For opml_file
|
||||
// Microsub-specific
|
||||
channelFilter: String | null, // Specific channel UID or null for all
|
||||
categoryPrefix: String, // Prefix for blog categories
|
||||
enabled: Boolean,
|
||||
syncInterval: Number, // Minutes between syncs
|
||||
lastSyncAt: String | null, // ISO 8601
|
||||
lastSyncError: String | null,
|
||||
createdAt: String, // ISO 8601
|
||||
updatedAt: String // ISO 8601
|
||||
}
|
||||
```
|
||||
|
||||
**blogrollBlogs**
|
||||
```javascript
|
||||
{
|
||||
_id: ObjectId,
|
||||
sourceId: ObjectId | null, // Reference to blogrollSources
|
||||
title: String,
|
||||
description: String | null,
|
||||
feedUrl: String, // Unique identifier
|
||||
siteUrl: String | null,
|
||||
feedType: "rss" | "atom" | "jsonfeed",
|
||||
category: String, // For grouping/filtering
|
||||
tags: String[],
|
||||
photo: String | null, // Blog icon/avatar
|
||||
author: Object | null, // { name: String }
|
||||
status: "active" | "error" | "deleted",
|
||||
lastFetchAt: String | null, // ISO 8601
|
||||
lastError: String | null,
|
||||
itemCount: Number,
|
||||
pinned: Boolean,
|
||||
hidden: Boolean,
|
||||
notes: String | null,
|
||||
// Microsub-specific (when source === "microsub")
|
||||
source: "microsub" | null,
|
||||
microsubFeedId: String | null, // Reference to microsub_feeds._id
|
||||
microsubChannelId: String | null,
|
||||
microsubChannelName: String | null,
|
||||
skipItemFetch: Boolean, // True for Microsub blogs
|
||||
createdAt: String, // ISO 8601
|
||||
updatedAt: String // ISO 8601
|
||||
}
|
||||
```
|
||||
|
||||
**blogrollItems**
|
||||
```javascript
|
||||
{
|
||||
_id: ObjectId,
|
||||
blogId: ObjectId, // Reference to blogrollBlogs
|
||||
uid: String, // Unique hash from feedUrl + itemId
|
||||
url: String,
|
||||
title: String,
|
||||
content: { html: String, text: String },
|
||||
summary: String,
|
||||
published: String, // ISO 8601
|
||||
updated: String | null, // ISO 8601
|
||||
author: Object | null, // { name: String }
|
||||
photo: String[] | null, // Image URLs
|
||||
categories: String[],
|
||||
fetchedAt: String // ISO 8601
|
||||
}
|
||||
```
|
||||
|
||||
**blogrollMeta**
|
||||
```javascript
|
||||
{
|
||||
key: "syncStats",
|
||||
lastFullSync: String, // ISO 8601
|
||||
duration: Number, // Milliseconds
|
||||
sources: { total: Number, success: Number, failed: Number },
|
||||
blogs: { total: Number, success: Number, failed: Number },
|
||||
items: { added: Number, deleted: Number }
|
||||
}
|
||||
```
|
||||
|
||||
## Key Files
|
||||
|
||||
### Entry Point
|
||||
- **index.js** - Plugin class, route registration, initialization
|
||||
|
||||
### Controllers (Protected Routes)
|
||||
- **lib/controllers/dashboard.js** - Main dashboard, sync triggers
|
||||
- **lib/controllers/sources.js** - CRUD for sources (OPML/Microsub)
|
||||
- **lib/controllers/blogs.js** - CRUD for blogs, manual refresh
|
||||
- **lib/controllers/api.js** - Both protected and public API endpoints
|
||||
|
||||
### Storage (MongoDB Operations)
|
||||
- **lib/storage/sources.js** - Source CRUD, sync status
|
||||
- **lib/storage/blogs.js** - Blog CRUD, upsert for sync, status updates
|
||||
- **lib/storage/items.js** - Item CRUD, transparent Microsub integration
|
||||
|
||||
### Sync Engine
|
||||
- **lib/sync/scheduler.js** - Background sync, interval management
|
||||
- **lib/sync/opml.js** - OPML parsing, fetch from URL, export
|
||||
- **lib/sync/microsub.js** - Microsub channel/feed sync, webhook handler
|
||||
- **lib/sync/feed.js** - RSS/Atom/JSON Feed parsing, item fetching
|
||||
|
||||
### Utilities
|
||||
- **lib/utils/feed-discovery.js** - Auto-discover feeds from website URLs
|
||||
|
||||
## Configuration
|
||||
|
||||
### Plugin Options
|
||||
```javascript
|
||||
new BlogrollEndpoint({
|
||||
mountPath: "/blogrollapi", // Admin UI and API base path
|
||||
syncInterval: 3600000, // 1 hour (in milliseconds)
|
||||
maxItemsPerBlog: 50, // Items to fetch per blog
|
||||
maxItemAge: 7, // Days - older items deleted (encourages discovery)
|
||||
fetchTimeout: 15000 // 15 seconds per feed fetch
|
||||
})
|
||||
```
|
||||
|
||||
### Environment/Deployment
|
||||
- Requires MongoDB (uses Indiekit's database connection)
|
||||
- Background sync starts 15 seconds after server startup
|
||||
- Periodic sync runs at `syncInterval` (default 1 hour)
|
||||
|
||||
## Routes
|
||||
|
||||
### Protected Routes (Admin UI)
|
||||
```
|
||||
GET /blogrollapi/ Dashboard (stats, recent activity)
|
||||
POST /blogrollapi/sync Manual sync trigger
|
||||
POST /blogrollapi/clear-resync Clear all items and resync
|
||||
|
||||
GET /blogrollapi/sources List sources
|
||||
GET /blogrollapi/sources/new New source form
|
||||
POST /blogrollapi/sources Create source
|
||||
GET /blogrollapi/sources/:id Edit source form
|
||||
POST /blogrollapi/sources/:id Update source
|
||||
POST /blogrollapi/sources/:id/delete Delete source
|
||||
POST /blogrollapi/sources/:id/sync Sync single source
|
||||
|
||||
GET /blogrollapi/blogs List blogs
|
||||
GET /blogrollapi/blogs/new New blog form
|
||||
POST /blogrollapi/blogs Create blog
|
||||
GET /blogrollapi/blogs/:id Edit blog form
|
||||
POST /blogrollapi/blogs/:id Update blog
|
||||
POST /blogrollapi/blogs/:id/delete Delete blog (soft delete)
|
||||
POST /blogrollapi/blogs/:id/refresh Refresh single blog
|
||||
|
||||
GET /blogrollapi/api/discover Feed discovery (protected)
|
||||
POST /blogrollapi/api/microsub-webhook Microsub webhook handler
|
||||
GET /blogrollapi/api/microsub-status Microsub integration status
|
||||
```
|
||||
|
||||
### Public Routes (Read-Only API)
|
||||
```
|
||||
GET /blogrollapi/api/blogs List blogs (JSON)
|
||||
GET /blogrollapi/api/blogs/:id Get blog with recent items (JSON)
|
||||
GET /blogrollapi/api/items List items across all blogs (JSON)
|
||||
GET /blogrollapi/api/categories List categories with counts (JSON)
|
||||
GET /blogrollapi/api/status Sync status (JSON)
|
||||
GET /blogrollapi/api/opml Export all blogs as OPML
|
||||
GET /blogrollapi/api/opml/:category Export category as OPML
|
||||
```
|
||||
|
||||
### API Query Parameters
|
||||
- **GET /api/blogs**: `?category=Tech&limit=100&offset=0`
|
||||
- **GET /api/items**: `?blog=<id>&category=Tech&limit=50&offset=0`
|
||||
|
||||
## Inter-Plugin Relationships
|
||||
|
||||
### Microsub Integration
|
||||
- **Detection:** Checks `application.collections.get("microsub_channels")` for availability
|
||||
- **Sync:** Reads `microsub_channels` and `microsub_feeds` to create blogroll references
|
||||
- **Items:** Queries `microsub_items` directly (no duplication)
|
||||
- **Webhook:** Receives notifications when feeds are subscribed/unsubscribed
|
||||
- **Orphan Cleanup:** Soft-deletes blogs whose Microsub feed no longer exists
|
||||
|
||||
### Homepage Plugin
|
||||
- Provides homepage sections: None (this plugin doesn't register homepage sections)
|
||||
- Can be used BY homepage plugin through public API endpoints
|
||||
|
||||
### Data Dependencies
|
||||
- **Requires:** MongoDB connection via Indiekit
|
||||
- **Creates Collections:** `blogrollSources`, `blogrollBlogs`, `blogrollItems`, `blogrollMeta`
|
||||
- **Reads Collections:** `microsub_channels`, `microsub_feeds`, `microsub_items` (when Microsub plugin is installed)
|
||||
|
||||
## Known Gotchas
|
||||
|
||||
### Date Handling
|
||||
- **Store dates as ISO strings** (`new Date().toISOString()`), NOT Date objects
|
||||
- The Nunjucks `| date` filter crashes on Date objects
|
||||
- Controllers convert Date objects to ISO strings before passing to templates
|
||||
- See CLAUDE.md root: "CRITICAL: Indiekit Date Handling Convention"
|
||||
|
||||
### Microsub Reference Architecture
|
||||
- Microsub blogs have `source: "microsub"` and `skipItemFetch: true`
|
||||
- Items are NOT copied to `blogrollItems` - queried from `microsub_items` directly
|
||||
- The `getItems()` and `getItemsForBlog()` functions transparently join both sources
|
||||
- DO NOT run feed fetch on Microsub blogs - Microsub handles that
|
||||
|
||||
### Soft Deletion
|
||||
- Blogs are soft-deleted (`status: "deleted"`, `hidden: true`) not removed
|
||||
- This prevents OPML/Microsub sync from recreating manually deleted blogs
|
||||
- `upsertBlog()` skips blogs with `status: "deleted"`
|
||||
|
||||
### Item Retention
|
||||
- Items older than `maxItemAge` (default 7 days) are auto-deleted on each sync
|
||||
- This is intentional to encourage discovery of fresh content
|
||||
- Adjust `maxItemAge` for longer retention
|
||||
|
||||
### Flash Messages
|
||||
- Uses session-based flash messages for user feedback
|
||||
- `consumeFlashMessage(request)` extracts and clears messages
|
||||
- Returns `{ success, error }` for Indiekit's native `notificationBanner`
|
||||
|
||||
## Dependencies
|
||||
|
||||
```json
|
||||
{
|
||||
"@indiekit/error": "^1.0.0-beta.25",
|
||||
"@indiekit/frontend": "^1.0.0-beta.25",
|
||||
"express": "^5.0.0",
|
||||
"feedparser": "^2.2.10", // RSS/Atom parsing
|
||||
"sanitize-html": "^2.13.0", // XSS prevention for feed content
|
||||
"xml2js": "^0.6.2" // OPML parsing
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Notes
|
||||
|
||||
- **No test suite configured** (manual testing only)
|
||||
- Test against real feeds: RSS, Atom, JSON Feed
|
||||
- Test OPML import (nested categories)
|
||||
- Test Microsub integration (requires `@rmdes/indiekit-endpoint-microsub`)
|
||||
- Test soft delete behavior (re-sync should not recreate deleted blogs)
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Add a New Source Type
|
||||
1. Add type to `createSource()` in `lib/storage/sources.js`
|
||||
2. Implement sync function in `lib/sync/` (e.g., `syncJsonFeedSource()`)
|
||||
3. Add handler in `runFullSync()` in `lib/sync/scheduler.js`
|
||||
4. Update source form UI
|
||||
|
||||
### Change Item Retention Period
|
||||
- Modify `maxItemAge` plugin option (default 7 days)
|
||||
- Items older than this are deleted on each sync
|
||||
|
||||
### Debug Sync Issues
|
||||
- Check `blogrollMeta.syncStats` document for last sync results
|
||||
- Check `blogs.lastError` and `sources.lastSyncError` for failures
|
||||
- Tail logs for `[Blogroll]` prefix messages
|
||||
|
||||
### Integrate with Frontend
|
||||
- Use public API endpoints (`/blogrollapi/api/blogs`, `/blogrollapi/api/items`)
|
||||
- OPML export available at `/blogrollapi/api/opml`
|
||||
- All public endpoints return JSON (except OPML which returns XML)
|
||||
241
README.md
Normal file
241
README.md
Normal file
@@ -0,0 +1,241 @@
|
||||
# Blogroll Endpoint for Indiekit
|
||||
|
||||
An Indiekit plugin that provides a comprehensive blogroll management system with feed aggregation, admin UI, and public API.
|
||||
|
||||
## Features
|
||||
|
||||
- **Multiple Source Types:** Import blogs from OPML files/URLs, Microsub subscriptions, or add manually
|
||||
- **Background Feed Fetching:** Automatically syncs blogs and caches recent items
|
||||
- **Microsub Integration:** Mirror your Microsub subscriptions as a blogroll (zero duplication)
|
||||
- **Admin UI:** Manage sources, blogs, and view recent activity
|
||||
- **Public JSON API:** Read-only endpoints for frontend integration
|
||||
- **OPML Export:** Export your blogroll as OPML (all or by category)
|
||||
- **Feed Discovery:** Auto-discover feeds from website URLs
|
||||
- **Item Retention:** Automatic cleanup of old items (encourages fresh content discovery)
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install @rmdes/indiekit-endpoint-blogroll
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Add to your `indiekit.config.js`:
|
||||
|
||||
```javascript
|
||||
import BlogrollEndpoint from "@rmdes/indiekit-endpoint-blogroll";
|
||||
|
||||
export default {
|
||||
plugins: [
|
||||
new BlogrollEndpoint({
|
||||
mountPath: "/blogrollapi", // Admin UI and API base path
|
||||
syncInterval: 3600000, // 1 hour (in milliseconds)
|
||||
maxItemsPerBlog: 50, // Items to fetch per blog
|
||||
maxItemAge: 7, // Days - older items auto-deleted
|
||||
fetchTimeout: 15000 // 15 seconds per feed fetch
|
||||
})
|
||||
]
|
||||
};
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Indiekit:** `>=1.0.0-beta.25`
|
||||
- **MongoDB:** Required for data storage
|
||||
- **Optional:** `@rmdes/indiekit-endpoint-microsub` for Microsub integration
|
||||
|
||||
## Usage
|
||||
|
||||
### Admin UI
|
||||
|
||||
Navigate to `/blogrollapi` in your Indiekit instance to access:
|
||||
|
||||
- **Dashboard:** View sync status, blog counts, recent activity
|
||||
- **Sources:** Manage OPML and Microsub sources
|
||||
- **Blogs:** Add/edit/delete individual blogs, refresh feeds
|
||||
- **Manual Sync:** Trigger immediate sync or clear and resync
|
||||
|
||||
### Source Types
|
||||
|
||||
1. **OPML URL:** Point to a public OPML file (e.g., your feed reader's export)
|
||||
2. **OPML File:** Paste OPML XML directly into the form
|
||||
3. **Microsub:** Import subscriptions from your Microsub channels
|
||||
4. **Manual:** Add individual blog feeds one at a time
|
||||
|
||||
### Public API
|
||||
|
||||
All API endpoints return JSON (except OPML export which returns XML).
|
||||
|
||||
**List Blogs**
|
||||
```
|
||||
GET /blogrollapi/api/blogs?category=Tech&limit=100&offset=0
|
||||
```
|
||||
|
||||
**Get Blog with Recent Items**
|
||||
```
|
||||
GET /blogrollapi/api/blogs/:id
|
||||
```
|
||||
|
||||
**List Items Across All Blogs**
|
||||
```
|
||||
GET /blogrollapi/api/items?blog=<id>&category=Tech&limit=50&offset=0
|
||||
```
|
||||
|
||||
**List Categories**
|
||||
```
|
||||
GET /blogrollapi/api/categories
|
||||
```
|
||||
|
||||
**Sync Status**
|
||||
```
|
||||
GET /blogrollapi/api/status
|
||||
```
|
||||
|
||||
**Export OPML**
|
||||
```
|
||||
GET /blogrollapi/api/opml (all blogs)
|
||||
GET /blogrollapi/api/opml/:category (specific category)
|
||||
```
|
||||
|
||||
### Example Response
|
||||
|
||||
**GET /blogrollapi/api/blogs**
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"id": "507f1f77bcf86cd799439011",
|
||||
"title": "Example Blog",
|
||||
"description": "A great blog about tech",
|
||||
"feedUrl": "https://example.com/feed",
|
||||
"siteUrl": "https://example.com",
|
||||
"feedType": "rss",
|
||||
"category": "Tech",
|
||||
"tags": ["programming", "web"],
|
||||
"photo": "https://example.com/icon.png",
|
||||
"status": "active",
|
||||
"itemCount": 25,
|
||||
"pinned": false,
|
||||
"lastFetchAt": "2026-02-13T10:30:00.000Z"
|
||||
}
|
||||
],
|
||||
"total": 42,
|
||||
"hasMore": true
|
||||
}
|
||||
```
|
||||
|
||||
**GET /blogrollapi/api/items**
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"id": "507f1f77bcf86cd799439011",
|
||||
"url": "https://example.com/post/hello",
|
||||
"title": "Hello World",
|
||||
"summary": "My first blog post...",
|
||||
"published": "2026-02-13T10:00:00.000Z",
|
||||
"isFuture": false,
|
||||
"author": { "name": "Jane Doe" },
|
||||
"photo": ["https://example.com/image.jpg"],
|
||||
"categories": ["announcement"],
|
||||
"blog": {
|
||||
"id": "507f1f77bcf86cd799439011",
|
||||
"title": "Example Blog",
|
||||
"siteUrl": "https://example.com",
|
||||
"category": "Tech",
|
||||
"photo": "https://example.com/icon.png"
|
||||
}
|
||||
}
|
||||
],
|
||||
"hasMore": false
|
||||
}
|
||||
```
|
||||
|
||||
## Microsub Integration
|
||||
|
||||
If you have `@rmdes/indiekit-endpoint-microsub` installed, the blogroll can mirror your subscriptions:
|
||||
|
||||
1. Create a Microsub source in the admin UI
|
||||
2. Select specific channels or sync all channels
|
||||
3. Add a category prefix (optional) to distinguish Microsub blogs
|
||||
4. Blogs and items are referenced, not duplicated
|
||||
|
||||
**Benefits:**
|
||||
- Zero data duplication - items are served directly from Microsub
|
||||
- Automatic orphan cleanup when feeds are unsubscribed
|
||||
- Webhook support for real-time updates
|
||||
|
||||
## Background Sync
|
||||
|
||||
The plugin automatically syncs in the background:
|
||||
|
||||
1. **Initial Sync:** Runs 15 seconds after server startup
|
||||
2. **Periodic Sync:** Runs every `syncInterval` milliseconds (default 1 hour)
|
||||
3. **What it Does:**
|
||||
- Syncs enabled sources (OPML/Microsub)
|
||||
- Fetches new items from active blogs
|
||||
- Deletes items older than `maxItemAge` days
|
||||
- Updates sync statistics
|
||||
|
||||
**Manual Sync:**
|
||||
- Trigger from the dashboard
|
||||
- Use `POST /blogrollapi/sync` (protected endpoint)
|
||||
- Use `POST /blogrollapi/clear-resync` to clear and resync all
|
||||
|
||||
## Feed Discovery
|
||||
|
||||
The plugin includes auto-discovery for finding feeds from website URLs:
|
||||
|
||||
```javascript
|
||||
// In the admin UI, when adding a blog, paste a website URL
|
||||
// The plugin will:
|
||||
// 1. Check <link rel="alternate"> tags in HTML
|
||||
// 2. Try common feed paths (/feed, /rss, /atom.xml, etc.)
|
||||
// 3. Suggest discovered feeds
|
||||
```
|
||||
|
||||
## Item Retention
|
||||
|
||||
By default, items older than 7 days are automatically deleted during sync. This encourages discovery of fresh content rather than archiving everything.
|
||||
|
||||
**To Change Retention:**
|
||||
```javascript
|
||||
new BlogrollEndpoint({
|
||||
maxItemAge: 30 // Keep items for 30 days instead
|
||||
})
|
||||
```
|
||||
|
||||
## Blog Status
|
||||
|
||||
- **active:** Blog is working, fetching items normally
|
||||
- **error:** Last fetch failed (see `lastError` for details)
|
||||
- **deleted:** Soft-deleted, won't be recreated by sync
|
||||
|
||||
## Navigation
|
||||
|
||||
The plugin adds itself to Indiekit's navigation:
|
||||
|
||||
- **Menu Item:** "Blogroll" (requires database)
|
||||
- **Shortcut:** Bookmark icon in admin dashboard
|
||||
|
||||
## Security
|
||||
|
||||
- **Protected Routes:** Admin UI and management endpoints require authentication
|
||||
- **Public Routes:** Read-only API endpoints are publicly accessible
|
||||
- **XSS Prevention:** Feed content is sanitized with `sanitize-html`
|
||||
- **Feed Discovery:** Protected to prevent abuse (requires authentication)
|
||||
|
||||
## Supported Feed Formats
|
||||
|
||||
- RSS 2.0
|
||||
- Atom 1.0
|
||||
- JSON Feed 1.0
|
||||
|
||||
## Contributing
|
||||
|
||||
Report issues at: https://github.com/rmdes/indiekit-endpoint-blogroll/issues
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Reference in New Issue
Block a user