svemagie/Substrate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add GETTING_STARTED.md and QUICK_REFERENCE.md

Browse Source 
Fixes #38

- Created GETTING_STARTED.md with complete setup walkthrough, API keys guide, and contribution guidelines
- Created QUICK_REFERENCE.md with command cheatsheet for common tasks

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Daniel Miessler
2025-11-24 22:18:46 -08:00
parent 73f46e0efa
commit 92edf8ee44
2 changed files with 471 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

271
GETTING_STARTED.md Normal file
View File

@@ -0,0 +1,271 @@
# Getting Started with Substrate
This guide walks you through setting up and using Substrate, the infrastructure for human knowledge and progress.
---
## Prerequisites
### Required
- **Git** - To clone the repository
- A text editor or IDE
### Optional (for automation)
- **[Bun](https://bun.sh)** - JavaScript/TypeScript runtime for running update scripts
```bash
curl -fsSL https://bun.sh/install | bash
```
---
## Installation
### 1. Clone the Repository
```bash
git clone https://github.com/danielmiessler/Substrate.git
cd Substrate
```
### 2. Explore the Structure
```bash
# Core datasets (GDP, inflation, COVID wastewater, etc.)
ls Data/
# Wellbeing data sources (FRED, CDC, Census, BLS, EPA)
ls Data-Sources/
# Knowledge components
ls Problems/
ls Solutions/
ls Arguments/
ls Claims/
```
### 3. Install Dependencies (Optional)
If you want to run the TypeScript automation scripts:
```bash
bun install
```
---
## Viewing the Data
All data is stored in human-readable formats:
- **CSV files** - Raw data you can open in Excel, Google Sheets, or any spreadsheet app
- **Markdown files** - Documentation and metadata
No special tools required - just browse the repository!
### Example: View US GDP Data
```bash
# See the data files
ls Data/US-GDP/
# View the CSV in terminal
cat Data/US-GDP/us-gdp-annual.csv | head -20
# Or open in your favorite spreadsheet app
open Data/US-GDP/us-gdp-annual.csv
```
---
## Running Updates
> **Note:** This is optional. The data is already included in the repository.
### Update a Single Dataset
```bash
cd Data/US-GDP
bun run update.ts
```
### Update a Wellbeing Data Source
Most wellbeing sources require API keys:
```bash
# Set your API key
export FRED_API_KEY="your_key_here"
# Run the update
cd Data-Sources/DS-00004—FRED_Economic_Wellbeing
bun run update.ts
```
### Update All Datasets
```bash
bun run scripts/update-all.ts
```
---
## API Keys
Most data sources are free but require registration:
| Data Source | Get Key | Rate Limit |
|-------------|---------|------------|
| **FRED Economic** | [fred.stlouisfed.org/docs/api](https://fred.stlouisfed.org/docs/api/api_key.html) | 120 req/min |
| **Census ACS** | [api.census.gov/data/key_signup](https://api.census.gov/data/key_signup.html) | 500 req/day |
| **EPA Air Quality** | Email: aqs.support@epa.gov | 10 req/min |
| **BLS JOLTS** | [bls.gov/developers/home](https://www.bls.gov/developers/home.htm) | 500 req/day |
| **CDC WONDER** | No key required | Fair use |
### Setting Up Environment Variables
Create a `.env` file in the repository root:
```bash
FRED_API_KEY=your_fred_key
CENSUS_API_KEY=your_census_key
EPA_EMAIL=your_email@example.com
EPA_KEY=your_epa_key
BLS_API_KEY=your_bls_key
```
Or export them in your shell:
```bash
export FRED_API_KEY="your_fred_key"
export CENSUS_API_KEY="your_census_key"
```
---
## Contributing
We welcome contributions! Here's how to add to Substrate:
### What You Can Contribute
| Type | Directory | Description |
|------|-----------|-------------|
| **Problems** | `Problems/` | Documented challenges with evidence |
| **Solutions** | `Solutions/` | Proven approaches with results |
| **Arguments** | `Arguments/` | Reasoning chains with quality scores |
| **Claims** | `Claims/` | Assertions linked to evidence |
| **Data** | `Data/` or `Data-Sources/` | Authoritative datasets |
| **People** | `People/` | Researchers and practitioners |
| **Organizations** | `Organizations/` | Groups working on problems |
| **Projects** | `Projects/` | Active initiatives |
| **Plans** | `Plans/` | Actionable strategies |
| **Ideas** | `Ideas/` | Frameworks and concepts |
| **Values** | `Values/` | Guiding principles |
### How to Submit
1. **Fork the repository** on GitHub
2. **Create a branch** for your contribution:
```bash
git checkout -b add-my-contribution
```
3. **Add your content** following the format in each directory's README
4. **Commit your changes**:
```bash
git add .
git commit -m "Add: description of your contribution"
```
5. **Push and create a Pull Request**:
```bash
git push origin add-my-contribution
```
### Contribution Guidelines
- **Follow existing formats** - Check the README in each directory for templates
- **Include evidence** - Link claims to data sources where possible
- **Use clear IDs** - Follow the naming conventions (e.g., `PR-00001` for problems)
- **Document your sources** - Include provenance and methodology
- **Be specific** - Vague contributions are harder to use
### Quality Standards
We don't gatekeep ideas, but we do maintain quality:
- **Problems** should be specific and evidence-backed
- **Solutions** should include outcomes or expected results
- **Arguments** should have clear reasoning chains
- **Data** should include methodology and limitations
---
## Directory Structure
```
Substrate/
├── Data/ # Core datasets
│ ├── US-GDP/ # US GDP data (1929-present)
│ ├── US-Inflation/ # US inflation data
│ ├── Bay-Area-COVID/ # COVID wastewater monitoring
│ ├── Pulitzer-Prize/ # Pulitzer Prize winners
│ └── Knowledge-Worker-Salaries/
│
├── Data-Sources/ # Wellbeing data sources
│ ├── DS-00001—WHO_Global_Health_Observatory/
│ ├── DS-00002—UN_SDG_Indicators/
│ ├── DS-00003—World_Bank_Open_Data/
│ ├── DS-00004—FRED_Economic_Wellbeing/
│ ├── DS-00005—CDC_WONDER_Mortality/
│ ├── DS-00006—Census_ACS_Social_Wellbeing/
│ ├── DS-00007—BLS_JOLTS_Labor_Market/
│ └── DS-00008—EPA_Air_Quality_System/
│
├── Problems/ # Documented challenges
├── Solutions/ # Proven approaches
├── Arguments/ # Reasoning chains
├── Claims/ # Evidence-linked assertions
├── Plans/ # Actionable strategies
├── Ideas/ # Frameworks and concepts
├── People/ # Researchers and practitioners
├── Organizations/ # Groups working on issues
├── Projects/ # Active initiatives
├── Values/ # Guiding principles
│
├── scripts/ # Automation scripts
├── README.md # Main documentation
├── GETTING_STARTED.md # This file
├── QUICK_REFERENCE.md # Command cheatsheet
└── UPDATES.md # Changelog
```
---
## Next Steps
1. **Explore the data** - Browse `Data/` and `Data-Sources/` to see what's available
2. **Read the documentation** - Each dataset has its own README with methodology
3. **Try the automation** - Run an update script to see how data is refreshed
4. **Contribute** - Add a problem, solution, or data source you care about
---
## Getting Help
- **Issues** - [github.com/danielmiessler/Substrate/issues](https://github.com/danielmiessler/Substrate/issues)
- **Discussions** - Start a discussion in the repository
- **Twitter** - [@danielmiessler](https://twitter.com/danielmiessler)
---
## Related Projects
- **[TELOS](https://github.com/danielmiessler/Telos)** - Goals and strategy framework
- **[Fabric](https://github.com/danielmiessler/fabric)** - AI augmentation framework
---
**[← Back to README](./README.md)** | **[Quick Reference →](./QUICK_REFERENCE.md)**

200
QUICK_REFERENCE.md Normal file
View File

@@ -0,0 +1,200 @@
# Substrate Quick Reference
Command cheatsheet for working with Substrate.
---
## Setup
```bash
# Clone repository
git clone https://github.com/danielmiessler/Substrate.git
cd Substrate
# Install Bun (if needed)
curl -fsSL https://bun.sh/install | bash
# Install dependencies
bun install
```
---
## Browse Data
```bash
# List core datasets
ls Data/
# List wellbeing data sources
ls Data-Sources/
# View a specific dataset
cat Data/US-GDP/us-gdp-annual.csv | head -20
# Open in spreadsheet app (macOS)
open Data/US-GDP/us-gdp-annual.csv
```
---
## Update Data
### Single Dataset
```bash
cd Data/US-GDP
bun run update.ts
```
### Wellbeing Sources (Require API Keys)
```bash
# Set API key first
export FRED_API_KEY="your_key"
# Then run update
cd Data-Sources/DS-00004—FRED_Economic_Wellbeing
bun run update.ts
```
### All Datasets
```bash
bun run scripts/update-all.ts
```
---
## API Keys
| Source | Get Key | Env Variable |
|--------|---------|--------------|
| FRED | [fred.stlouisfed.org/docs/api](https://fred.stlouisfed.org/docs/api/api_key.html) | `FRED_API_KEY` |
| Census | [api.census.gov/data/key_signup](https://api.census.gov/data/key_signup.html) | `CENSUS_API_KEY` |
| EPA | Email: aqs.support@epa.gov | `EPA_KEY` |
| BLS | [bls.gov/developers/home](https://www.bls.gov/developers/home.htm) | `BLS_API_KEY` |
| CDC WONDER | No key needed | — |
```bash
# Set all keys at once
export FRED_API_KEY="xxx"
export CENSUS_API_KEY="xxx"
export EPA_KEY="xxx"
export BLS_API_KEY="xxx"
```
---
## Contributing
### Fork & Clone
```bash
# Fork on GitHub, then:
git clone https://github.com/YOUR_USERNAME/Substrate.git
cd Substrate
git remote add upstream https://github.com/danielmiessler/Substrate.git
```
### Create Branch
```bash
git checkout -b add-my-contribution
```
### Commit & Push
```bash
git add .
git commit -m "Add: description"
git push origin add-my-contribution
```
### Stay Updated
```bash
git fetch upstream
git merge upstream/main
```
---
## Directory Quick Reference
| Directory | Contains | ID Format |
|-----------|----------|-----------|
| `Data/` | Core datasets | Folder names |
| `Data-Sources/` | Wellbeing sources | `DS-00001` |
| `Problems/` | Documented challenges | `PR-00001` |
| `Solutions/` | Proven approaches | `SO-00001` |
| `Arguments/` | Reasoning chains | `AR-00001` |
| `Claims/` | Evidence-linked assertions | `CL-00001` |
| `Plans/` | Actionable strategies | `PL-00001` |
| `Ideas/` | Frameworks | `ID-00001` |
| `People/` | Researchers | `PE-00001` |
| `Organizations/` | Groups | `OR-00001` |
| `Projects/` | Initiatives | `PJ-00001` |
| `Values/` | Principles | `VA-00001` |
---
## Data Sources Quick Reference
| ID | Source | Key Indicators |
|----|--------|----------------|
| DS-00001 | WHO Global Health | Health indicators (194 countries) |
| DS-00002 | UN SDG | Sustainable Development Goals |
| DS-00003 | World Bank | Development metrics |
| DS-00004 | FRED Economic | Debt, unemployment, inequality |
| DS-00005 | CDC WONDER | Overdoses, suicides, mortality |
| DS-00006 | Census ACS | Social isolation, commute, digital divide |
| DS-00007 | BLS JOLTS | Quit rate, job openings, layoffs |
| DS-00008 | EPA Air Quality | PM2.5, ozone, air quality |
---
## Common Tasks
### Find a Dataset
```bash
# Search by name
ls Data/ | grep -i gdp
ls Data-Sources/ | grep -i fred
# Search in README files
grep -r "unemployment" Data-Sources/*/README.md
```
### Check Data Freshness
```bash
# View last update time
ls -la Data/US-GDP/*.csv
cat Data/US-GDP/update-log.md | tail -20
```
### Validate Data
```bash
# Count rows
wc -l Data/US-GDP/us-gdp-annual.csv
# Check for missing values
grep -c ",," Data/US-GDP/us-gdp-annual.csv
```
---
## Links
- **README**: [./README.md](./README.md)
- **Getting Started**: [./GETTING_STARTED.md](./GETTING_STARTED.md)
- **Updates**: [./UPDATES.md](./UPDATES.md)
- **Data Philosophy**: [./Data/README.md](./Data/README.md)
- **Library Science**: [./Data/README-LIBRARY-SCIENCE.md](./Data/README-LIBRARY-SCIENCE.md)
---
**[← Back to README](./README.md)** | **[Getting Started →](./GETTING_STARTED.md)**