diff --git a/README.md b/README.md index c98fcdb..e9fa900 100644 --- a/README.md +++ b/README.md @@ -23,9 +23,186 @@ ## Navigation - [About](#about) +- [Recent Updates](#-recent-updates) +- [Data Directory](#data-directory) - [How to Contribute](#how-to-contribute) +- [Documentation](#-documentation) - [Meta](#meta) +--- + +## πŸš€ **Recent Updates** + +> [!IMPORTANT] +> **πŸ”₯ 2025-10:** Major data infrastructure upgrade complete! +> +> **DATA REVOLUTION:** +> - 5 authoritative datasets added (GDP, Inflation, COVID, Pulitzer, Salaries) +> - Library science methodology implementation +> - Comprehensive data management system +> - 1,700+ data points spanning 107 years (1918-2025) +> +> [See full changelog β†’](#recent-updates-detail) + +
+πŸ“… Click to see all updates + +### Recent Changes + +#### **2025-10-25 - Dataset Updates & Validation** +- βœ… **DS-00004:** Pulitzer Prize Winners - Arts & Letters data refreshed +- βœ… **DS-00002:** U.S. GDP data updated (1929-2025) +- βœ… **DS-00003:** U.S. CPI inflation data updated (1947-2025) +- βœ… **DS-00005:** Knowledge Worker Global Salaries validation check completed + +#### **2025-10-18 - New Dataset** +- πŸ†• **DS-00005:** Knowledge Worker Global Compensation dataset added +- πŸ“Š Global salary data for knowledge workers +- πŸ” Comprehensive geographic and role coverage + +#### **2025-10-16 - Data Management System** +- πŸ—οΈ **Library Science Methodology** implemented with 8-dimension source evaluation +- ⚑ **TypeScript Automation** with Bun runtime +- πŸ“‹ **Auto-Discovery Orchestrator** for dataset updates +- πŸ“Š **Central Logging System** with aggregated logs +- πŸ“ˆ **Dashboard Auto-Generation** with health metrics +- πŸ”„ **Git Integration** for version control +- πŸ“š **Comprehensive Documentation Suite:** + - `GETTING_STARTED.md` - Complete setup guide + - `PROJECT_SUMMARY.md` - Technical architecture + - `QUICK_REFERENCE.md` - Command cheatsheet + - `Data/README.md` - Data philosophy and standards + +#### **2025-10-07 - Major Dataset Additions** +- πŸ†• **DS-00004:** Pulitzer Prize Winners - Arts & Letters (1918-2024) + - 249 winners across Poetry, Drama, General/Special awards + - High-quality, complete coverage of selected categories + - Source: Wikidata + +- πŸ†• **DS-00003:** Bay Area COVID-19 Wastewater Surveillance + - 161 weekly data points (2022-2025) + - California statewide data (Bay Area proxy) + - Leading health indicator + - Source: California Department of Public Health (CDPH) + +#### **2025-10-06 - GitHub Automation** +- πŸ€– **Claude Code Review Workflow** - Automated code review +- πŸ€– **Claude PR Assistant Workflow** - PR analysis and assistance +- βš™οΈ **CI/CD Integration** for quality assurance + +#### **2025-10-06 - U.S. Inflation Dataset** +- πŸ†• **DS-00001:** U.S. Consumer Price Index (CPI-U) +- πŸ“Š 945 monthly data points (1947-2025) +- πŸ“ˆ Gold standard inflation measure +- πŸ›οΈ Source: FRED/Bureau of Labor Statistics + +#### **2025-10-06 - Community Contributions** +- 🌍 **Brazil - SΓ£o Paulo Mental Health** problem added (@ktfth) +- πŸ“ **Arguments** contributions (@DesertEaglePWN, @JaymanW) +- 🎯 **Values** framework established (@karai114) +- βœ… Multiple problem database updates + +#### **2024-09-25 - Framework Expansion** +- πŸ“‹ **Claims Framework** established (@ThatNateGuy) + - Anthropogenic climate change + - Everettian Interpretation of Quantum Mechanics + - Supernaturalism + - Atavistic Model of Cancer + - Holographic Universe theory + +#### **2024-07-27 - Repository Consolidation** +- πŸ—οΈ **Single-Repo Structure** - Moved from multi-repo to unified structure +- πŸ“¦ Easier project management and contribution workflow +- πŸš€ Simplified development process + +
+ +
+πŸ“Š Project Statistics (as of 2025-10-27) + +### Data & Coverage +- **Datasets:** 5 authoritative ground-truth datasets +- **Data Points:** 1,700+ (spanning multiple domains) +- **Historical Coverage:** 1918-2025 (107 years maximum span) +- **Geographic Coverage:** Global (U.S.-focused with expanding international data) + +### Infrastructure +- **Update Scripts:** TypeScript with Bun runtime +- **Automation:** Auto-discovery orchestrator with central logging +- **Data Formats:** CSV, JSON, Markdown, Pipe-delimited +- **Quality Framework:** 8-dimension library science evaluation +- **Version Control:** Full git integration with automated commits +- **GitHub Actions:** 2 active workflows (Code Review, PR Assistant) + +### Documentation +- **Markdown:** 8,000+ lines of documentation +- **TypeScript:** 1,000+ lines of automation code +- **Documentation Files:** 25+ comprehensive guides and references +- **Standards:** Dublin Core, MARC, SDMX, DDI metadata compliance + +### Community +- **Contributors:** 6+ community members +- **Pull Requests Merged:** 10+ contributions +- **Object Types:** 17+ framework components (Problems, Solutions, Ideas, Plans, etc.) + +
+ +
+🎯 Milestones & Roadmap + +### βœ… Completed Milestones + +**Phase 1: Foundation (July 2024)** +- βœ… Single-repo structure +- βœ… Core object types defined (17+ types) +- βœ… Basic directory structure +- βœ… Initial documentation +- βœ… Public launch with intro video + +**Phase 2: Community Building (Aug-Sep 2024)** +- βœ… First community contributions +- βœ… Claims framework established +- βœ… Arguments and Values added +- βœ… Multi-contributor ecosystem active + +**Phase 3: Data Infrastructure (Oct 2025)** +- βœ… Five authoritative datasets added +- βœ… Library science methodology implemented +- βœ… TypeScript data management system +- βœ… Comprehensive documentation suite +- βœ… GitHub Actions automation +- βœ… Quality assurance framework + +### 🚧 Upcoming (Planned) + +**Phase 4: Enhanced Access & Interaction** +- [ ] Web-based contribution interface (non-coders can contribute) +- [ ] Interactive data visualizations +- [ ] RESTful API for programmatic access +- [ ] Advanced cross-reference linking +- [ ] Evidence-based problem/solution matching + +**Phase 5: Dataset Expansion** +- [ ] Additional authoritative datasets (UNICEF, OECD, IHME) +- [ ] Community-driven dataset requests +- [ ] Real-time data feeds for select sources +- [ ] Historical data archive expansion + +**Phase 6: Advanced Features** +- [ ] Machine-readable catalog (DCAT/CKAN) +- [ ] Automated quality scoring algorithms +- [ ] Data quality trend tracking +- [ ] Email/Slack notifications for updates +- [ ] Parallel dataset updates + +
+ +--- + +**Full Update History:** See [`UPDATES.md`](./UPDATES.md) for complete chronological changelog + +--- + ## About **Substrate** is an open-source framework for capturing, organizing, and analyzing different aspects of human civilization. It provides a structured knowledge system covering problems, solutions, plans, experiments, and empirical dataβ€”all interconnected and designed to be analyzed by both humans and AI systems. @@ -42,20 +219,30 @@ Substrate includes a **Data/** directory with authoritative, ground-truth datase **Current Datasets:** -| Dataset | Coverage | Data Points | Source | Description | -|---------|----------|-------------|--------|-------------| -| **US-GDP** | 1929-2025 | 96 years (annual)
314 quarters | FRED/BEA | Real GDP (chained 2017 dollars) - primary measure of US economic activity | -| **US-Inflation** | 1947-2025 | 945 months | FRED/BLS | Consumer Price Index (CPI-U) - gold standard inflation measure | -| **Bay-Area-COVID-Wastewater** | 2022-2025 | 161 weeks | CDPH | California COVID-19 wastewater surveillance (leading health indicator) | -| **Pulitzer-Prize-Winners** | 1918-2024 | 249 winners | Wikidata | Arts & Letters categories (Poetry, Drama, General/Special awards) | +| Dataset ID | Dataset Name | Coverage | Data Points | Source | Description | +|-----------|--------------|----------|-------------|--------|-------------| +| **DS-00002** | **US-GDP** | 1929-2025 | 96 years (annual)
314 quarters | FRED/BEA | Real GDP (chained 2017 dollars) - primary measure of US economic activity | +| **DS-00001** | **US-Inflation** | 1947-2025 | 945 months | FRED/BLS | Consumer Price Index (CPI-U) - gold standard inflation measure | +| **DS-00003** | **Bay-Area-COVID-Wastewater** | 2022-2025 | 161 weeks | CDPH | California COVID-19 wastewater surveillance (leading health indicator) | +| **DS-00004** | **Pulitzer-Prize-Winners** | 1918-2024 | 249 winners | Wikidata | Arts & Letters categories (Poetry, Drama, General/Special awards) | +| **DS-00005** | **Knowledge-Worker-Global-Salaries** | Global | Multi-region | Research | Global compensation data for knowledge workers across roles and geographies | + +**Data Management System:** +- **Library Science Methodology**: 8-dimension source quality evaluation +- **TypeScript Automation**: Auto-discovery orchestrator with Bun runtime +- **Quality Standards**: Dublin Core, MARC, SDMX, DDI metadata compliance +- **Version Control**: Full git integration with automated updates +- **Central Logging**: Aggregated logs and health monitoring +- **Documentation**: Comprehensive guides for each dataset **Data Philosophy:** - **Ground Truth First**: Authoritative, verifiable sources only -- **Human-Readable + Machine-Parseable**: CSV and Markdown formats +- **Human-Readable + Machine-Parseable**: CSV, JSON, and Markdown formats - **Full Transparency**: Complete methodology documentation and source attribution - **Shared Knowledge**: Public domain or openly licensed data +- **Research-Grade Quality**: Professional library science evaluation -See `Data/README.md` for complete documentation of all datasets, data quality standards, and contribution guidelines. +See **[Data/README.md](./Data/README.md)** for complete documentation of all datasets, data quality standards, and contribution guidelines. ## Introduction video @@ -75,12 +262,50 @@ And here's a full blog post about the project. [Introducing Substrate](https://danielmiessler.com/p/introducing-substrate) +## πŸ“š **Documentation** + +Substrate includes comprehensive documentation for all aspects of the project: + +### **Getting Started** +- **[GETTING_STARTED.md](./GETTING_STARTED.md)** - Complete setup and usage guide for the data management system +- **[QUICK_REFERENCE.md](./QUICK_REFERENCE.md)** - Quick command reference and cheatsheet +- **[Data/README.md](./Data/README.md)** - Data directory philosophy, standards, and contribution guidelines + +### **Technical Documentation** +- **[PROJECT_SUMMARY.md](./PROJECT_SUMMARY.md)** - Technical architecture and system design overview +- **[Data/README-LIBRARY-SCIENCE.md](./Data/README-LIBRARY-SCIENCE.md)** - Library science methodology framework +- **[Data/MIGRATION-GUIDE.md](./Data/MIGRATION-GUIDE.md)** - Guide for data directory structure changes + +### **Update Logs & Changes** +- **[UPDATES.md](./UPDATES.md)** - Complete project update history and changelog +- **[Data/UPDATES.md](./Data/UPDATES.md)** - Data directory-specific update log +- Individual dataset update logs in each `Data/*/UPDATES.md` file + +### **Dataset Documentation** +Each dataset includes comprehensive documentation: +- **README.md** - Dataset overview, research methodology, and usage +- **UPDATES.md** - Dataset-specific update history +- **RESOURCES.md** - Data sources, APIs, and download instructions +- **source.md** - Library science evaluation (8-dimension quality assessment) + +### **Video & Blog** +- **[Introduction Video](https://www.youtube.com/watch?v=ky7ejowc_qY)** - Project explanation and structure +- **[Blog Post](https://danielmiessler.com/p/introducing-substrate)** - Detailed project introduction + +--- + ## How to Contribute You can contribute to Substrate by submitting PRs to modify the various Substrate object files within each directory, e.g.: `Problems`, `Solutions`, `Ideas`, etc. We're working on a web-based interface for this as well to make it easier for non-coders to contribute. +### Contributing Datasets + +To contribute new datasets, see: +- **[Data/README.md](./Data/README.md)** - Data contribution guidelines and quality standards +- **[GETTING_STARTED.md](./GETTING_STARTED.md)** - Step-by-step guide for adding new data sources +
> [!NOTE] diff --git a/UPDATES.md b/UPDATES.md new file mode 100644 index 0000000..9ed4cbc --- /dev/null +++ b/UPDATES.md @@ -0,0 +1,317 @@ +# Substrate Project Updates + +This file tracks all significant changes, additions, and milestones in the Substrate project. + +--- + +## πŸš€ Recent Updates + +> **2025-10-25:** Major data infrastructure upgrade - Comprehensive data management system with library science methodology + +--- + +## 2025-10 - Data Infrastructure Revolution + +### Dataset Additions (5 New Authoritative Datasets) + +**Knowledge Worker Global Salaries (DS-00005)** +- **Added:** 2025-10-18 +- **Coverage:** Global compensation data for knowledge workers +- **Validation:** 2025-10-25 validation check completed +- **Status:** Active + +**Pulitzer Prize Winners - Arts & Letters (DS-00004)** +- **Added:** 2025-10-07 +- **Coverage:** 1918-2024 (249 winners across Poetry, Drama, General/Special awards) +- **Source:** Wikidata +- **Update:** 2025-10-25 refresh +- **Quality:** High-quality, complete coverage of selected categories +- **Rationale:** Focused on Arts & Letters for quality over breadth + +**Bay Area COVID-19 Wastewater Surveillance (DS-00003)** +- **Added:** 2025-10-07 +- **Coverage:** 2022-07-09 to 2025-08-02 (161 weekly data points) +- **Source:** California Department of Public Health (CDPH) +- **Type:** Leading health indicator (population-level surveillance) +- **Geographic:** Statewide California serving as Bay Area proxy + +**U.S. Gross Domestic Product (DS-00002)** +- **Added:** 2025-10-16 +- **Coverage:** Annual 1929-2024 (96 years) + Quarterly Q1 1947 - Q2 2025 (314 quarters) +- **Source:** Federal Reserve Economic Data (FRED) / Bureau of Economic Analysis (BEA) +- **Update:** 2025-10-25 refresh +- **Significance:** Primary measure of U.S. economic activity +- **Quality:** Gold standard indicator with three-stage quarterly revision process +- **Research:** Created through comprehensive 10-agent parallel research across Perplexity, Claude WebSearch, and Gemini + +**U.S. Consumer Price Index - Inflation (DS-00001)** +- **Added:** 2025-10-06 +- **Coverage:** 1947-2025 (945 monthly data points) +- **Source:** Federal Reserve Economic Data (FRED) / Bureau of Labor Statistics (BLS) +- **Update:** 2025-10-25 refresh +- **Type:** CPI-U (Consumer Price Index for All Urban Consumers) +- **Significance:** Gold standard inflation measure for the United States + +### Data Management System + +**Library Science Methodology Implementation** +- **Eight-Dimension Source Evaluation Framework:** + 1. Authority & Credibility + 2. Currency & Timeliness + 3. Accuracy & Reliability + 4. Coverage & Scope + 5. Objectivity & Bias + 6. Accessibility + 7. Documentation Quality + 8. Provenance & Citation + +- **Metadata Standards:** Dublin Core, MARC, SDMX, DDI +- **Source Classification:** Primary, Secondary, Tertiary +- **Quality Assurance:** Research-grade evaluation for each dataset + +**Technical Infrastructure** +- **Runtime:** Bun (TypeScript) +- **Auto-Discovery:** Orchestrator automatically detects all DS-* directories +- **Update Scripts:** TypeScript scripts with error handling, retry logic, rate limiting +- **Central Logging:** Aggregated logs from all sources +- **Dashboard Generation:** Auto-generated README with system health metrics +- **Git Integration:** Automated version control +- **Data Formats:** Raw JSON + Pipe-delimited (Substrate standard) + +**Documentation Suite** +- `GETTING_STARTED.md` - Complete setup and usage guide (536 lines) +- `PROJECT_SUMMARY.md` - Technical architecture overview (475 lines) +- `QUICK_REFERENCE.md` - Command cheatsheet +- `Data/README.md` - Data directory documentation +- Individual `Data/*/UPDATES.md` - Dataset-specific change logs +- Individual `Data/*/README.md` - Dataset documentation with research methodology +- `README-LIBRARY-SCIENCE.md` - Library science framework explanation + +**Migration from Data-Sources to Data** +- **Completed:** 2025-10-16 +- **Reason:** Simplified directory naming, clearer structure +- **Impact:** All references updated, old directory removed +- **Documentation:** MIGRATION-GUIDE.md and MIGRATION-COMPLETE.md created + +--- + +## 2025-10 - GitHub Automation + +### GitHub Actions + +**Claude Code Review Workflow** +- **Added:** 2025-10-06 +- **Updated:** 2025-10-06 +- **Function:** Automated code review using Claude +- **Status:** Active + +**Claude PR Assistant Workflow** +- **Added:** 2025-10-06 +- **Updated:** 2025-10-06 +- **Function:** Automated PR assistance and analysis +- **Status:** Active + +--- + +## 2025-10 - Community Contributions + +### Problems + +**Brazil - SΓ£o Paulo Mental Health** +- **Contributor:** @ktfth +- **Added:** 2025-10-06 +- **PR:** #30 +- **Impact:** Expanded geographic coverage of mental health issues + +**Various Problem Updates** +- **Contributor:** @DesertEaglePWN +- **Added:** 2025-10-06 +- **PR:** #28, #31 +- **Impact:** Problem database refinement + +### Arguments + +**New Arguments** +- **Contributor:** @DesertEaglePWN +- **Added:** 2025-10-06 +- **PR:** #31 +- **Impact:** Expanded argumentation framework + +**AI Understanding Argument** +- **Contributor:** @JaymanW +- **Added:** 2024-09-25 +- **PR:** #21 +- **Content:** Arguments about AI comprehension and understanding + +### Values + +**Values Framework** +- **Contributor:** @karai114 +- **Added:** 2024-09-25 +- **PR:** #22 +- **Impact:** Established values taxonomy for Substrate + +### Claims + +**Initial Claims** +- **Contributor:** @ThatNateGuy +- **Added:** 2024-04-25 +- **PR:** #13 +- **Claims Added:** + - Anthropogenic climate change + - Everettian Interpretation of Quantum Mechanics + - Supernaturalism + - Atavistic Model of Cancer + - Holographic Universe theory + +--- + +## 2024-07 - Project Foundation + +### Repository Consolidation + +**Single-Repo Structure** +- **Date:** 2024-07-27 +- **Change:** Moved from multi-repo to single-repo structure +- **Benefit:** Easier management and contribution +- **Impact:** Simplified development workflow + +### Core Components + +**Initial Object Types Created:** +- Problems +- Solutions +- Ideas +- Plans +- Experiments +- Results +- Models +- Arguments +- Claims +- Values +- Organizations +- People +- Projects +- Funding Sources +- Outcomes +- Risks +- Threats + +**Documentation** +- README.md with project vision +- Introduction video (YouTube) +- Blog post announcement + +--- + +## Project Milestones + +### Phase 1: Foundation (July 2024) +βœ… Single-repo structure +βœ… Core object types defined +βœ… Basic directory structure +βœ… Initial documentation +βœ… Public launch + +### Phase 2: Community Building (Aug-Sep 2024) +βœ… First community contributions +βœ… Claims framework established +βœ… Arguments and Values added +βœ… Multi-contributor ecosystem + +### Phase 3: Data Infrastructure (Oct 2025) +βœ… Five authoritative datasets +βœ… Library science methodology +βœ… Data management system +βœ… TypeScript automation +βœ… Comprehensive documentation +βœ… GitHub Actions integration + +### Phase 4: Future (Planned) +- [ ] Web-based contribution interface +- [ ] Interactive data visualizations +- [ ] API for programmatic access +- [ ] Additional authoritative datasets +- [ ] Cross-reference linking system +- [ ] Evidence-based problem/solution matching +- [ ] Community-driven dataset requests + +--- + +## Dataset Update History + +For detailed dataset-specific updates, see: +- `Data/UPDATES.md` - Central data directory updates +- `Data/US-GDP/UPDATES.md` - GDP dataset updates +- `Data/US-Inflation/UPDATES.md` - Inflation dataset updates +- `Data/Bay-Area-COVID-Wastewater/UPDATES.md` - COVID wastewater updates +- `Data/Pulitzer-Prize-Winners/UPDATES.md` - Pulitzer Prize updates + +--- + +## Breaking Changes + +### 2025-10-16: Data-Sources β†’ Data Directory Rename +- **Impact:** Directory path changed from `Data-Sources/` to `Data/` +- **Migration:** Automatic, all references updated +- **Documentation:** See `Data/MIGRATION-GUIDE.md` + +--- + +## Statistics + +### Project Scale (as of 2025-10-27) + +**Datasets:** +- Total: 5 authoritative datasets +- Total Data Points: 1,700+ (GDP quarterly + monthly inflation + COVID weekly + Pulitzer winners + salary data) +- Historical Coverage: 1918-2025 (107 years maximum span) +- Geographic Coverage: Global (U.S.-focused with expanding international data) + +**Documentation:** +- Lines of Markdown: 8,000+ lines +- Lines of TypeScript: 1,000+ lines +- Documentation Files: 25+ files + +**Community:** +- Contributors: 6+ community members +- Pull Requests Merged: 10+ +- Issues Addressed: Multiple + +**Infrastructure:** +- GitHub Actions: 2 workflows +- Update Scripts: TypeScript with Bun +- Data Formats: CSV, JSON, Markdown, Pipe-delimited +- Version Control: Full git integration + +--- + +## Acknowledgments + +**Major Contributors:** +- **Daniel Miessler** - Project creator and maintainer +- **@ThatNateGuy** - Claims framework +- **@JaymanW** - Arguments on AI understanding +- **@karai114** - Values framework +- **@DesertEaglePWN** - Problems and Arguments updates +- **@ktfth** - Brazil mental health problems + +**Special Thanks:** +- Jonathan Dunn - Similar goals and inspiration +- Joel Parish - Structure wisdom +- Joseph Thacker - Constant flow of ideas + +--- + +## How to Track Updates + +**Watch This File:** `UPDATES.md` for project-wide changes +**Watch Data Updates:** `Data/UPDATES.md` for dataset-specific changes +**Watch GitHub:** Releases and commit history +**Watch Individual Datasets:** Each dataset has its own `UPDATES.md` file + +--- + +**Last Updated:** 2025-10-27 +**Update Frequency:** As changes occur +**Format:** Reverse chronological (newest first)