From 92edf8ee44e1396ce2d849b48c7335177d40e0e9 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Mon, 24 Nov 2025 22:18:46 -0800 Subject: [PATCH] docs: add GETTING_STARTED.md and QUICK_REFERENCE.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes #38 - Created GETTING_STARTED.md with complete setup walkthrough, API keys guide, and contribution guidelines - Created QUICK_REFERENCE.md with command cheatsheet for common tasks πŸ€– Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- GETTING_STARTED.md | 271 +++++++++++++++++++++++++++++++++++++++++++++ QUICK_REFERENCE.md | 200 +++++++++++++++++++++++++++++++++ 2 files changed, 471 insertions(+) create mode 100644 GETTING_STARTED.md create mode 100644 QUICK_REFERENCE.md diff --git a/GETTING_STARTED.md b/GETTING_STARTED.md new file mode 100644 index 0000000..bb723da --- /dev/null +++ b/GETTING_STARTED.md @@ -0,0 +1,271 @@ +# Getting Started with Substrate + +This guide walks you through setting up and using Substrate, the infrastructure for human knowledge and progress. + +--- + +## Prerequisites + +### Required +- **Git** - To clone the repository +- A text editor or IDE + +### Optional (for automation) +- **[Bun](https://bun.sh)** - JavaScript/TypeScript runtime for running update scripts + ```bash + curl -fsSL https://bun.sh/install | bash + ``` + +--- + +## Installation + +### 1. Clone the Repository + +```bash +git clone https://github.com/danielmiessler/Substrate.git +cd Substrate +``` + +### 2. Explore the Structure + +```bash +# Core datasets (GDP, inflation, COVID wastewater, etc.) +ls Data/ + +# Wellbeing data sources (FRED, CDC, Census, BLS, EPA) +ls Data-Sources/ + +# Knowledge components +ls Problems/ +ls Solutions/ +ls Arguments/ +ls Claims/ +``` + +### 3. Install Dependencies (Optional) + +If you want to run the TypeScript automation scripts: + +```bash +bun install +``` + +--- + +## Viewing the Data + +All data is stored in human-readable formats: + +- **CSV files** - Raw data you can open in Excel, Google Sheets, or any spreadsheet app +- **Markdown files** - Documentation and metadata + +No special tools required - just browse the repository! + +### Example: View US GDP Data + +```bash +# See the data files +ls Data/US-GDP/ + +# View the CSV in terminal +cat Data/US-GDP/us-gdp-annual.csv | head -20 + +# Or open in your favorite spreadsheet app +open Data/US-GDP/us-gdp-annual.csv +``` + +--- + +## Running Updates + +> **Note:** This is optional. The data is already included in the repository. + +### Update a Single Dataset + +```bash +cd Data/US-GDP +bun run update.ts +``` + +### Update a Wellbeing Data Source + +Most wellbeing sources require API keys: + +```bash +# Set your API key +export FRED_API_KEY="your_key_here" + +# Run the update +cd Data-Sources/DS-00004β€”FRED_Economic_Wellbeing +bun run update.ts +``` + +### Update All Datasets + +```bash +bun run scripts/update-all.ts +``` + +--- + +## API Keys + +Most data sources are free but require registration: + +| Data Source | Get Key | Rate Limit | +|-------------|---------|------------| +| **FRED Economic** | [fred.stlouisfed.org/docs/api](https://fred.stlouisfed.org/docs/api/api_key.html) | 120 req/min | +| **Census ACS** | [api.census.gov/data/key_signup](https://api.census.gov/data/key_signup.html) | 500 req/day | +| **EPA Air Quality** | Email: aqs.support@epa.gov | 10 req/min | +| **BLS JOLTS** | [bls.gov/developers/home](https://www.bls.gov/developers/home.htm) | 500 req/day | +| **CDC WONDER** | No key required | Fair use | + +### Setting Up Environment Variables + +Create a `.env` file in the repository root: + +```bash +FRED_API_KEY=your_fred_key +CENSUS_API_KEY=your_census_key +EPA_EMAIL=your_email@example.com +EPA_KEY=your_epa_key +BLS_API_KEY=your_bls_key +``` + +Or export them in your shell: + +```bash +export FRED_API_KEY="your_fred_key" +export CENSUS_API_KEY="your_census_key" +``` + +--- + +## Contributing + +We welcome contributions! Here's how to add to Substrate: + +### What You Can Contribute + +| Type | Directory | Description | +|------|-----------|-------------| +| **Problems** | `Problems/` | Documented challenges with evidence | +| **Solutions** | `Solutions/` | Proven approaches with results | +| **Arguments** | `Arguments/` | Reasoning chains with quality scores | +| **Claims** | `Claims/` | Assertions linked to evidence | +| **Data** | `Data/` or `Data-Sources/` | Authoritative datasets | +| **People** | `People/` | Researchers and practitioners | +| **Organizations** | `Organizations/` | Groups working on problems | +| **Projects** | `Projects/` | Active initiatives | +| **Plans** | `Plans/` | Actionable strategies | +| **Ideas** | `Ideas/` | Frameworks and concepts | +| **Values** | `Values/` | Guiding principles | + +### How to Submit + +1. **Fork the repository** on GitHub + +2. **Create a branch** for your contribution: + ```bash + git checkout -b add-my-contribution + ``` + +3. **Add your content** following the format in each directory's README + +4. **Commit your changes**: + ```bash + git add . + git commit -m "Add: description of your contribution" + ``` + +5. **Push and create a Pull Request**: + ```bash + git push origin add-my-contribution + ``` + +### Contribution Guidelines + +- **Follow existing formats** - Check the README in each directory for templates +- **Include evidence** - Link claims to data sources where possible +- **Use clear IDs** - Follow the naming conventions (e.g., `PR-00001` for problems) +- **Document your sources** - Include provenance and methodology +- **Be specific** - Vague contributions are harder to use + +### Quality Standards + +We don't gatekeep ideas, but we do maintain quality: + +- **Problems** should be specific and evidence-backed +- **Solutions** should include outcomes or expected results +- **Arguments** should have clear reasoning chains +- **Data** should include methodology and limitations + +--- + +## Directory Structure + +``` +Substrate/ +β”œβ”€β”€ Data/ # Core datasets +β”‚ β”œβ”€β”€ US-GDP/ # US GDP data (1929-present) +β”‚ β”œβ”€β”€ US-Inflation/ # US inflation data +β”‚ β”œβ”€β”€ Bay-Area-COVID/ # COVID wastewater monitoring +β”‚ β”œβ”€β”€ Pulitzer-Prize/ # Pulitzer Prize winners +β”‚ └── Knowledge-Worker-Salaries/ +β”‚ +β”œβ”€β”€ Data-Sources/ # Wellbeing data sources +β”‚ β”œβ”€β”€ DS-00001β€”WHO_Global_Health_Observatory/ +β”‚ β”œβ”€β”€ DS-00002β€”UN_SDG_Indicators/ +β”‚ β”œβ”€β”€ DS-00003β€”World_Bank_Open_Data/ +β”‚ β”œβ”€β”€ DS-00004β€”FRED_Economic_Wellbeing/ +β”‚ β”œβ”€β”€ DS-00005β€”CDC_WONDER_Mortality/ +β”‚ β”œβ”€β”€ DS-00006β€”Census_ACS_Social_Wellbeing/ +β”‚ β”œβ”€β”€ DS-00007β€”BLS_JOLTS_Labor_Market/ +β”‚ └── DS-00008β€”EPA_Air_Quality_System/ +β”‚ +β”œβ”€β”€ Problems/ # Documented challenges +β”œβ”€β”€ Solutions/ # Proven approaches +β”œβ”€β”€ Arguments/ # Reasoning chains +β”œβ”€β”€ Claims/ # Evidence-linked assertions +β”œβ”€β”€ Plans/ # Actionable strategies +β”œβ”€β”€ Ideas/ # Frameworks and concepts +β”œβ”€β”€ People/ # Researchers and practitioners +β”œβ”€β”€ Organizations/ # Groups working on issues +β”œβ”€β”€ Projects/ # Active initiatives +β”œβ”€β”€ Values/ # Guiding principles +β”‚ +β”œβ”€β”€ scripts/ # Automation scripts +β”œβ”€β”€ README.md # Main documentation +β”œβ”€β”€ GETTING_STARTED.md # This file +β”œβ”€β”€ QUICK_REFERENCE.md # Command cheatsheet +└── UPDATES.md # Changelog +``` + +--- + +## Next Steps + +1. **Explore the data** - Browse `Data/` and `Data-Sources/` to see what's available +2. **Read the documentation** - Each dataset has its own README with methodology +3. **Try the automation** - Run an update script to see how data is refreshed +4. **Contribute** - Add a problem, solution, or data source you care about + +--- + +## Getting Help + +- **Issues** - [github.com/danielmiessler/Substrate/issues](https://github.com/danielmiessler/Substrate/issues) +- **Discussions** - Start a discussion in the repository +- **Twitter** - [@danielmiessler](https://twitter.com/danielmiessler) + +--- + +## Related Projects + +- **[TELOS](https://github.com/danielmiessler/Telos)** - Goals and strategy framework +- **[Fabric](https://github.com/danielmiessler/fabric)** - AI augmentation framework + +--- + +**[← Back to README](./README.md)** | **[Quick Reference β†’](./QUICK_REFERENCE.md)** diff --git a/QUICK_REFERENCE.md b/QUICK_REFERENCE.md new file mode 100644 index 0000000..2d44fcb --- /dev/null +++ b/QUICK_REFERENCE.md @@ -0,0 +1,200 @@ +# Substrate Quick Reference + +Command cheatsheet for working with Substrate. + +--- + +## Setup + +```bash +# Clone repository +git clone https://github.com/danielmiessler/Substrate.git +cd Substrate + +# Install Bun (if needed) +curl -fsSL https://bun.sh/install | bash + +# Install dependencies +bun install +``` + +--- + +## Browse Data + +```bash +# List core datasets +ls Data/ + +# List wellbeing data sources +ls Data-Sources/ + +# View a specific dataset +cat Data/US-GDP/us-gdp-annual.csv | head -20 + +# Open in spreadsheet app (macOS) +open Data/US-GDP/us-gdp-annual.csv +``` + +--- + +## Update Data + +### Single Dataset + +```bash +cd Data/US-GDP +bun run update.ts +``` + +### Wellbeing Sources (Require API Keys) + +```bash +# Set API key first +export FRED_API_KEY="your_key" + +# Then run update +cd Data-Sources/DS-00004β€”FRED_Economic_Wellbeing +bun run update.ts +``` + +### All Datasets + +```bash +bun run scripts/update-all.ts +``` + +--- + +## API Keys + +| Source | Get Key | Env Variable | +|--------|---------|--------------| +| FRED | [fred.stlouisfed.org/docs/api](https://fred.stlouisfed.org/docs/api/api_key.html) | `FRED_API_KEY` | +| Census | [api.census.gov/data/key_signup](https://api.census.gov/data/key_signup.html) | `CENSUS_API_KEY` | +| EPA | Email: aqs.support@epa.gov | `EPA_KEY` | +| BLS | [bls.gov/developers/home](https://www.bls.gov/developers/home.htm) | `BLS_API_KEY` | +| CDC WONDER | No key needed | β€” | + +```bash +# Set all keys at once +export FRED_API_KEY="xxx" +export CENSUS_API_KEY="xxx" +export EPA_KEY="xxx" +export BLS_API_KEY="xxx" +``` + +--- + +## Contributing + +### Fork & Clone + +```bash +# Fork on GitHub, then: +git clone https://github.com/YOUR_USERNAME/Substrate.git +cd Substrate +git remote add upstream https://github.com/danielmiessler/Substrate.git +``` + +### Create Branch + +```bash +git checkout -b add-my-contribution +``` + +### Commit & Push + +```bash +git add . +git commit -m "Add: description" +git push origin add-my-contribution +``` + +### Stay Updated + +```bash +git fetch upstream +git merge upstream/main +``` + +--- + +## Directory Quick Reference + +| Directory | Contains | ID Format | +|-----------|----------|-----------| +| `Data/` | Core datasets | Folder names | +| `Data-Sources/` | Wellbeing sources | `DS-00001` | +| `Problems/` | Documented challenges | `PR-00001` | +| `Solutions/` | Proven approaches | `SO-00001` | +| `Arguments/` | Reasoning chains | `AR-00001` | +| `Claims/` | Evidence-linked assertions | `CL-00001` | +| `Plans/` | Actionable strategies | `PL-00001` | +| `Ideas/` | Frameworks | `ID-00001` | +| `People/` | Researchers | `PE-00001` | +| `Organizations/` | Groups | `OR-00001` | +| `Projects/` | Initiatives | `PJ-00001` | +| `Values/` | Principles | `VA-00001` | + +--- + +## Data Sources Quick Reference + +| ID | Source | Key Indicators | +|----|--------|----------------| +| DS-00001 | WHO Global Health | Health indicators (194 countries) | +| DS-00002 | UN SDG | Sustainable Development Goals | +| DS-00003 | World Bank | Development metrics | +| DS-00004 | FRED Economic | Debt, unemployment, inequality | +| DS-00005 | CDC WONDER | Overdoses, suicides, mortality | +| DS-00006 | Census ACS | Social isolation, commute, digital divide | +| DS-00007 | BLS JOLTS | Quit rate, job openings, layoffs | +| DS-00008 | EPA Air Quality | PM2.5, ozone, air quality | + +--- + +## Common Tasks + +### Find a Dataset + +```bash +# Search by name +ls Data/ | grep -i gdp +ls Data-Sources/ | grep -i fred + +# Search in README files +grep -r "unemployment" Data-Sources/*/README.md +``` + +### Check Data Freshness + +```bash +# View last update time +ls -la Data/US-GDP/*.csv +cat Data/US-GDP/update-log.md | tail -20 +``` + +### Validate Data + +```bash +# Count rows +wc -l Data/US-GDP/us-gdp-annual.csv + +# Check for missing values +grep -c ",," Data/US-GDP/us-gdp-annual.csv +``` + +--- + +## Links + +- **README**: [./README.md](./README.md) +- **Getting Started**: [./GETTING_STARTED.md](./GETTING_STARTED.md) +- **Updates**: [./UPDATES.md](./UPDATES.md) +- **Data Philosophy**: [./Data/README.md](./Data/README.md) +- **Library Science**: [./Data/README-LIBRARY-SCIENCE.md](./Data/README-LIBRARY-SCIENCE.md) + +--- + +**[← Back to README](./README.md)** | **[Getting Started β†’](./GETTING_STARTED.md)**