From 1e71e808d364f52c38bceaf32dab23df07444683 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Mon, 27 Oct 2025 02:26:56 +0100 Subject: [PATCH] Add transparency/OSINT use cases and system diagrams MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TONE ADJUSTMENTS: - Soften 'information crisis' language to neutral 'disconnected knowledge' - Remove politically-charged framing - Focus on practical transparency applications NEW SECTION: Transparency & Accountability πŸ” OSINT & Investigation: - Cross-reference public records, corporate filings, government data - Link people, organizations, financial flows across datasets - Surface connections invisible in isolated databases - Track influence networks and funding relationships πŸ“Š Criminal Investigation Support: - Connect individuals to organizations, transactions, locations (public data) - Build evidence chains linking claims to verifiable records - Correlate timeline data across authoritative sources - Enable pattern detection (fraud, corruption, illicit networks) 🌐 Public Accountability: - Verify claims against documented evidence - Track campaign promises vs actual policy outcomes - Link political donations to voting records and positions - Monitor government spending vs stated goals NEW SECTION: System Diagrams - Potential Linkages πŸ“Š Entity Relationship Diagram: - Shows Problem β†’ Solutions β†’ Experiments β†’ Metrics flow - Links Claims β†’ Arguments β†’ Data β†’ Evidence - Connects People β†’ Organizations β†’ Projects β†’ Plans πŸ” OSINT Investigation Flow: - Public tip β†’ Person of Interest β†’ Organizations β†’ Transactions - Corporate filings β†’ Funding sources β†’ Politicians β†’ Voting records - Multi-source data β†’ Evidence chain β†’ Pattern detection βš–οΈ Argument Quality Chain: - Demonstrates scoring: Authoritative sources (7/10) vs Blog posts (2/10) - Shows verification through FBI UCR, DOJ Stats, peer review - Contrasts strong vs weak arguments 🌐 Full System Integration: - TELOS Framework ↔ Substrate Core ↔ Data Layer ↔ Entities - Complete system showing all major linkages Future Capabilities: - Real-time link discovery, pattern detection, graph visualization - API access, ML-suggested connections --- README.md | 231 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 229 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index d4df85d..caa5ba4 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ Substrate is built on a simple but powerful idea: **meaningful progress requires connecting ideas to evidence, and claims to data**. -We face an information crisisβ€”not from lack of data, but from fragmented knowledge. Claims float without evidence. Arguments lack supporting data. Problems are discussed without reference to proven solutions. Data exists in silos, disconnected from the debates it could inform. +Knowledge exists everywhere, but it's often disconnected. Claims are made without clear evidence. Arguments reference data that's hard to verify. Solutions are proposed without examining what's worked before. Public records sit in separate databases, difficult to cross-reference. Substrate addresses this by creating an **interconnected knowledge system** where every component links to related components: @@ -47,7 +47,29 @@ This dual accessibility enables a new kind of collaboration: humans contribute k By connecting claims to arguments, arguments to data, and data to authoritative sources, Substrate creates a **foundation for mutual understanding**. When we disagree, we can trace our reasoning back through evidence to see exactly where our perspectives diverge. When we agree, we can build on verified knowledge rather than assumptions. -This isn't just an academic exerciseβ€”it's infrastructure for human progress. Whether you're a researcher seeking ground-truth data, a policymaker evaluating solutions, or a citizen trying to understand complex issues, Substrate provides a trusted foundation built on transparency and evidence. +### Transparency & Accountability + +Substrate's interconnected structure enables powerful transparency use cases: + +**πŸ” OSINT & Investigation** +- Cross-reference public records, corporate filings, and government data +- Link people, organizations, and financial flows across datasets +- Surface connections that would be invisible in isolated databases +- Track influence networks and funding relationships + +**πŸ“Š Criminal Investigation Support** +- Connect individuals to organizations, transactions, and locations using public data +- Build evidence chains linking claims to verifiable records +- Correlate timeline data across multiple authoritative sources +- Enable pattern detection across datasets (fraud, corruption, illicit networks) + +**🌐 Public Accountability** +- Verify claims made by public figures against documented evidence +- Track campaign promises against actual policy outcomes and metrics +- Link political donations to voting records and policy positions +- Monitor government spending against stated goals and results + +This isn't just an academic exerciseβ€”it's infrastructure for transparency and progress. Whether you're a researcher seeking ground-truth data, an investigator cross-referencing public records, a journalist fact-checking claims, or a citizen trying to understand complex issues, Substrate provides a trusted foundation built on verifiable evidence and open data. ### How It Works: Community Contributions @@ -97,6 +119,211 @@ Substrate integrates with the **[TELOS framework](https://github.com/danielmiess Together, they create a complete system: TELOS provides the *intention* (goals, strategies), Substrate provides the *evidence* (data, arguments, proven solutions). +### System Diagrams: Potential Linkages + +
+πŸ“Š Entity Relationship Diagram + +```mermaid +graph TD + P[Problem: Toxic Water] --> S1[Solution: Filtration System] + P --> S2[Solution: Infrastructure Replacement] + P --> D1[Data: WHO Water Quality] + P --> D2[Data: CDC Health Records] + + S1 --> E1[Experiment: Flint MI Pilot] + S1 --> M1[Metrics: PPM Lead Reduction] + + C[Claim: Filtration Effective] --> A1[Argument: Cost-Effective] + C --> A2[Argument: Fast Deployment] + + A1 --> D3[Data: Implementation Costs] + A1 --> D1 + A2 --> E1 + + O1[Org: EPA] --> D1 + O2[Org: Local Water Authority] --> E1 + + PE1[Person: City Manager] --> O2 + PE1 --> PL1[Plan: 5-Year Remediation] + + PL1 --> S1 + PL1 --> M1 + + style P fill:#ff6b6b + style S1 fill:#4ecdc4 + style C fill:#ffe66d + style D1 fill:#95e1d3 + style A1 fill:#f38181 +``` + +
+ +
+πŸ” OSINT Investigation Flow + +```mermaid +graph LR + START[Public Tip: Suspicious Activity] --> P1[Person of Interest] + + P1 --> O1[Organization A] + P1 --> O2[Organization B] + P1 --> T1[Transaction Records] + + O1 --> D1[Corporate Filings] + O1 --> F1[Funding Source: PAC] + O2 --> D2[Tax Records] + + T1 --> D3[Bank Records Public] + T1 --> D4[Property Transfers] + + F1 --> P2[Person B: Politician] + D4 --> P3[Person C: Associate] + + P2 --> V1[Voting Record] + P2 --> C1[Campaign Promises] + + V1 --> POL[Policy Outcome] + C1 --> POL + F1 --> POL + + D1 --> E1[Evidence Chain] + D2 --> E1 + D3 --> E1 + D4 --> E1 + V1 --> E1 + + E1 --> R[Report: Pattern Detected] + + style START fill:#ff6b6b + style E1 fill:#4ecdc4 + style R fill:#95e1d3 +``` + +
+ +
+βš–οΈ Argument Quality Chain + +```mermaid +graph TD + A[Argument: Policy X Reduces Crime] --> Q{Quality Check} + + Q --> D1[Data Source 1: FBI UCR] + Q --> D2[Data Source 2: DOJ Stats] + Q --> D3[Data Source 3: Peer Review] + + D1 --> V1{Verification} + D2 --> V1 + D3 --> V1 + + V1 -->|Authoritative| SCORE1[Score: +3] + V1 -->|Peer Reviewed| SCORE2[Score: +2] + V1 -->|Multi-Source| SCORE3[Score: +2] + + SCORE1 --> FINAL[Final Rating: 7/10 - Highly Credible] + SCORE2 --> FINAL + SCORE3 --> FINAL + + A2[Argument: Policy Y Ineffective] --> Q2{Quality Check} + Q2 --> D4[Data: Blog Post] + Q2 --> D5[Data: Opinion Article] + + D4 --> V2{Verification} + D5 --> V2 + + V2 -->|Anecdotal| SCORE4[Score: -2] + V2 -->|No Peer Review| SCORE5[Score: -1] + + SCORE4 --> FINAL2[Final Rating: 2/10 - Weak] + SCORE5 --> FINAL2 + + style FINAL fill:#4ecdc4 + style FINAL2 fill:#ff6b6b +``` + +
+ +
+🌐 Full System Integration + +```mermaid +graph TB + subgraph "TELOS Framework" + TG[Goals] + TS[Strategies] + TC[Challenges] + TM[Metrics] + end + + subgraph "Substrate Core" + P[Problems] + SO[Solutions] + AR[Arguments] + CL[Claims] + PL[Plans] + end + + subgraph "Data Layer" + D1[US GDP Data] + D2[Inflation Data] + D3[Health Data] + D4[Public Records] + D5[Corporate Data] + end + + subgraph "Entities" + OR[Organizations] + PE[People] + PR[Projects] + FU[Funding] + end + + TG -.-> P + TS -.-> SO + TC -.-> D3 + TM -.-> PL + + P --> SO + P --> D3 + SO --> AR + AR --> CL + CL --> D1 + CL --> D2 + + SO --> PL + PL --> TM + + OR --> PR + OR --> FU + PE --> OR + PE --> PL + + D4 --> PE + D4 --> OR + D5 --> OR + D5 --> FU + + AR --> D1 + AR --> D2 + AR --> D3 + AR --> D4 + + style TG fill:#ffe66d + style P fill:#ff6b6b + style D1 fill:#95e1d3 + style OR fill:#f38181 +``` + +
+ +**Future Capabilities:** +- Real-time link discovery across new data sources +- Automated pattern detection for investigative workflows +- Interactive graph visualization for exploring connections +- API access for programmatic queries across linked entities +- Machine learning to suggest relevant connections based on context + ### History Substrate was launched in **July 2024** with a vision to create shared infrastructure for human knowledge. The project began with 17 core object types (Problems, Solutions, Ideas, Plans, Values, Models, Arguments, Claims, etc.) and has grown through community contributions and a major data infrastructure initiative in October 2025.