svemagie/Substrate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Simplify README: remove sensationalist language, focus on data and sources

Browse Source
This commit is contained in:
Daniel Miessler
2025-11-10 00:12:08 -08:00
parent 43758bc2bb
commit 6321f5bbbc
1 changed files with 190 additions and 322 deletions
Download Patch File Download Diff File Expand all files Collapse all files

512
research/data-utilization-global-analysis-november-2025/README.md
View File

@@ -1,154 +1,52 @@
# Global Data Generation and Utilization Analysis # Global Data Generation and Utilization Rates
**Research Study** **Research Study**
**Date:** November 9-10, 2025 **Date:** November 9-10, 2025
**Researcher:** Daniel Miessler (with Kai AI research infrastructure) **Researcher:** Daniel Miessler
**Classification:** Global Data Economics and Utilization Assessment **Research Design:** Multi-agent parallel investigation (9 specialized agents)
**Research Design:** Multi-Agent Parallel Investigation (9 specialized agents)
---
## Abstract
This study presents a comprehensive analysis of global data generation rates and utilization patterns across all major data categories. Through a multi-agent research methodology employing nine parallel specialized research agents across four distinct AI platforms (Claude, Perplexity, Gemini, WebSearch), we investigated data generation volumes, examination rates by humans and AI systems, and the massive gap between what we create and what we consume.
**Key Finding:** Of the 149 zettabytes of data generated globally in 2024, only **12-15% is ever examined** by humans or AI systems. The remaining **85-88% (approximately 129 zettabytes) is never looked at by anyone**—representing a massive utilization crisis, not a data scarcity problem.
**Critical Discovery:** The "AI is running out of training data" narrative measures exhausted historical archives while ignoring the massive waterfall of fresh data generated daily that AI cannot access due to context window limitations, encryption, legal barriers, and architectural constraints.
--- ---
## Research Question ## Research Question
**Primary Research Question:**
What percentage of data generated globally is actually viewed, analyzed, or acted upon by humans or AI systems? What percentage of data generated globally is actually viewed, analyzed, or acted upon by humans or AI systems?
**Sub-Questions:**
1. How much data is generated daily/annually across all major categories?
2. What percentage of enterprise data is examined (documents, communications, code, logs)?
3. What percentage of video content is watched (streaming, surveillance, user-generated)?
4. What percentage of IoT sensor data is analyzed?
5. What percentage is examined by humans vs AI vs both vs neither?
6. How does the "running out of data" narrative relate to daily generation rates?
**Motivating Context:**
When AI researchers claim we're "running out of training data," are they measuring the right thing? If businesses generate trillions of words daily but ignore 75-90%, doesn't that suggest an **examination crisis** rather than a **data scarcity crisis**?
--- ---
## Research Methodology ## Methodology
### Research Design: Multi-Agent Parallel Investigation ### Research Design
**Methodological Framework:** Multi-agent parallel research utilizing 9 specialized AI research agents distributed across 4 platforms (Perplexity AI, Claude, Gemini, WebSearch) to gather and cross-validate data from industry reports, platform statistics, and academic studies.
Parallel mixed-methods research utilizing nine specialized AI research agents distributed across four distinct AI platforms to ensure multi-perspective coverage and reduce platform-specific bias.
**Research Duration:** 6 hours across 2 sessions (November 9-10, 2025) **Research Duration:** 6 hours across 2 sessions
**Source Coverage:** 150+ authoritative sources (2024-2025 data)
**Agent Distribution:** ### Agent Assignments
- **Perplexity AI:** 3 agents - Real-time web research, industry reports, market statistics
- **Claude (Anthropic):** 3 agents - Deep technical analysis, cross-referencing, synthesis
- **Gemini (Google):** 3 agents - Ecosystem analysis, trend identification, validation
- **WebSearch:** Fallback when Gemini API encountered 404 errors
**Total Source Coverage:** 150+ authoritative publications, reports, and studies from 2024-2025 **Phase 1: Enterprise Data (6 agents)**
1. Enterprise dark data statistics (Veritas, IDC, Gartner)
### Research Agents & Assignment 2. Communication engagement rates (email, Slack, Teams)
3. Document access patterns (Google Docs, Word, Confluence)
**Phase 1: Enterprise Data Utilization (6 agents)** 4. Code review coverage (GitHub, GitLab)
5. Security log analysis rates (SIEM, observability tools)
**Agent 1: perplexity-researcher** 6. AI automation penetration in enterprises
*Topic:* Enterprise dark data statistics and utilization rates
*Focus:* What percentage of enterprise data is collected but never analyzed; Veritas, IDC, Gartner studies; storage vs usage patterns; industry breakdowns
**Agent 2: perplexity-researcher**
*Topic:* Communication engagement rates (email, Slack, Teams, meetings)
*Focus:* Email open/response rates; Slack/Teams engagement; meeting notes access; time decay curves; utilization funnel (sent → read → acted upon)
**Agent 3: claude-researcher**
*Topic:* Document creation vs access patterns
*Focus:* Google Docs, Word, Confluence access statistics; percentage never opened after creation; collaboration rates; knowledge base utilization
**Agent 4: claude-researcher**
*Topic:* Code review coverage rates
*Focus:* GitHub/GitLab PR statistics; human review vs automated analysis; rubber stamp rates; security scanning coverage; the "dark zone" of unreviewed code
**Agent 5: gemini-researcher**
*Topic:* Security log and monitoring analysis rates
*Focus:* SIEM coverage; alert investigation rates; unmonitored assets; observability data utilization; SOC analyst capacity constraints
**Agent 6: gemini-researcher**
*Topic:* AI automation penetration in enterprises
*Focus:* What percentage of data is processed by AI systems; RPA coverage; ML analytics adoption; automation vs human analysis ratios
**Phase 2: Global Data Breakdown (3 agents)** **Phase 2: Global Data Breakdown (3 agents)**
7. Global data generation by type (149 ZB total)
8. Video content utilization (streaming, surveillance, user-generated)
9. IoT sensor data utilization (21.1B devices)
**Agent 7: perplexity-researcher** ### Quality Assurance
*Topic:* Global data generation breakdown by type
*Focus:* Composition of 149 ZB annually; video vs IoT vs enterprise vs machine logs; internet traffic analysis; human vs machine-generated ratios
**Agent 8: perplexity-researcher**
*Topic:* Video content utilization (streaming, surveillance, UGC)
*Focus:* YouTube view distribution; surveillance footage review rates; Twitch zero-viewer streams; the "82% of traffic is video" myth clarification
**Agent 9: claude-researcher**
*Topic:* IoT sensor data utilization
*Focus:* 21.1B devices generating 79.4 ZB/year; edge vs cloud processing; industrial vs consumer utilization; the 99% loss rate before decisions
### Analytical Framework
**Multi-Source Validation:**
- Minimum 3 sources per major claim - Minimum 3 sources per major claim
- Cross-platform verification (multiple AI research agents) - Cross-platform verification
- Official reports prioritized over secondary sources - Confidence levels assigned to all findings
- Recent data emphasized (2024-2025)
**Confidence Level Assignment:**
- High (90%+): Multiple authoritative sources, official statistics, large samples
- Medium (70-90%): Calculated from authoritative sources, logical extrapolation
- Low (50-70%): Limited data, vendor claims, future projections
**Bias Mitigation:**
- Multi-platform AI agent distribution
- Contradictory evidence documented - Contradictory evidence documented
- Assumptions explicitly stated
- Limitations acknowledged
--- ---
## Research Outputs ## Primary Finding
### Primary Deliverables
1. **SOURCES.md** - Comprehensive documentation of 150+ sources across all 9 research reports
2. **README.md** (this file) - Research overview, methodology, key findings
3. **findings/SYNTHESIS.md** - Unified synthesis across all research categories
4. **findings/REAL_UTILIZATION.md** - Corrected global utilization calculation
5. **findings/THE_ARGUMENT.md** - Blog post-ready argument framework
### Supporting Research Reports
**Enterprise Utilization (6 reports, 134KB):**
- `findings/dark-data-statistics.md` (25KB)
- `findings/communication-engagement.md` (23KB)
- `findings/document-access-patterns.md` (16KB)
- `findings/code-review-coverage.md` (18KB)
- `findings/security-log-analysis.md` (23KB)
- `findings/ai-automation-penetration.md` (29KB)
**Global Breakdown (3 reports, 47KB):**
- `findings/data-types-breakdown.md` (12KB)
- `findings/video-utilization.md` (17KB)
- `findings/iot-utilization.md` (19KB)
**Blog-Ready Materials:**
- `data-utilization-table.md` - Markdown table with sources for blog posts
---
## Key Findings Summary
### Primary Finding: The 85-88% Utilization Crisis
**Of 149 zettabytes generated globally in 2024:** **Of 149 zettabytes generated globally in 2024:**
- **12-15% examined** by humans or AI (~20 ZB) - **12-15% examined** by humans or AI (~20 ZB)
@@ -158,243 +56,213 @@ Parallel mixed-methods research utilizing nine specialized AI research agents di
- Humans only: 8-10% - Humans only: 8-10%
- AI only: 5-10% - AI only: 5-10%
- Both human and AI: ~3% - Both human and AI: ~3%
- **Neither: 85-88%** - Neither: 85-88%
### Secondary Finding: Worst Offenders
**Data types with lowest utilization:**
1. **Surveillance Video** (22% of all data)
- 1+ billion cameras worldwide
- 5.5 million TB/day of footage
- **95-99% never watched**
- Source: Grand View Research, security industry studies
2. **IoT Sensor Data** (23% of all data)
- 21.1 billion devices
- 79.4 zettabytes/year
- **95%+ never analyzed**
- 99% lost before reaching operational decisions
- Source: IoT Analytics, McKinsey Digital
3. **Machine Logs & Telemetry** (14% of all data)
- **80-90% never examined**
- >90% of observability data never read
- Stored for compliance, not analysis
- Source: Coralogix, SANS SOC Survey
4. **Enterprise Data** (9% of all data)
- Documents: 41-80% never accessed after creation
- Communications: 85-91% never meaningfully consumed
- **70-75% overall never examined**
- Source: NetApp, Veritas, Forrester
5. **User-Generated Video** (~15% of all data)
- YouTube: 91% of videos <1,000 views
- Twitch: 80-90% of streams have zero viewers
- **60-80% never watched**
- Source: YouTube statistics, TwitchTracker
### Tertiary Finding: The Context Window Bottleneck
**AI cannot access most daily data even if utilization improved:**
- Global generation: 4-5 trillion words/day
- Claude's 200K token context: **1 in 5 trillion** of daily data
- We generate GPT-3's entire training corpus **every 11 seconds**
- Context windows are O(n²) - fundamental architectural limitation
**Five Walls Blocking AI Access:**
1. Legal (GDPR, 137 countries with data protection)
2. Technical (E2E encryption, processing speed limits)
3. Corporate (firewalls, DLP, explicit policies)
4. Platform (API rate limiting)
5. Consent (users/companies refusing access)
**Result:** 81-90% of generated data is permanently inaccessible to AI training
### Quaternary Finding: The "82% is Video" Myth
**Critical Clarification:**
- "82% of internet traffic is video" (Cisco VNI)
- This measures DATA TRANSMITTED (watched content only)
- NOT data GENERATED (includes unwatched content)
**Actual video utilization:**
- Streaming video: 60-70% watched (created for consumption)
- Surveillance: 1-5% watched (99% stored but ignored)
- User-generated: 10-30% meaningful viewership
- **Overall video: 20-30% utilization**
### Quinary Finding: The Utilization Funnel
**From generation to action:**
```
100% Generated (149 ZB)
↓
12-15% Examined (humans or AI)
↓
5-10% Analyzed (deeper examination)
↓
1-5% Informs Decisions
↓
0.02-0.10% Drives Action
```
**The devastating reality:**
- 85-88% never examined
- 99.9% never drives business decisions
--- ---
## Strategic Implications ## Data Generation and Utilization by Category
### For the "Running Out of Data" Debate | Data Type | Annual Volume | % of Total | Utilization Rate | Source Confidence |
|-----------|---------------|------------|------------------|-------------------|
| **Streaming Video** | 45 ZB | 30% | 60-70% watched | High (90%+) |
| **Surveillance Video** | 33 ZB | 22% | 1-5% watched | High (90%+) |
| **IoT Sensor Data** | 34 ZB | 23% | <5% analyzed | High (90%+) |
| **Machine Logs** | 21 ZB | 14% | 10-20% examined | Medium (70-90%) |
| **Enterprise Data** | 13 ZB | 9% | 25-30% examined | High (90%+) |
| **Social Media** | 3 ZB | 2% | 30-40% viewed | Medium (70-90%) |
**The Paradigm Error:** **Total Global Generation:** 149 ZB/year (IDC Data Age 2025)
- Epoch AI projections measure exhausted historical archives (Common Crawl, books, papers)
- This is measuring the RESERVOIR, not the WATERFALL
- Daily generation (350M TB/day) is 1,320x larger than entire Common Crawl (250 PB total)
**The Real Constraint:**
- NOT data scarcity (we generate GPT-3's corpus every 11 seconds)
- NOT quality (85% ignored includes high-value insights)
- ACTUAL: Context window scaling + five-wall access barriers
**The Winning Quote:**
> "AI researchers panic about 'running out of training data' while humanity generates 149 zettabytes annually—and ignores 88% of it. We have a billion surveillance cameras recording footage nobody watches, 21 billion IoT sensors taking readings nobody analyzes, and trillions of machine logs nobody reads. The problem isn't data scarcity. We're drowning in ignored data."
### For Enterprise Strategy
**The Opportunity:**
- 129 zettabytes/year ignored = massive untapped value
- 5-10% of ignored data has high value (6-13 ZB/year)
- Organizations that solve selective attention win trillions
**The Challenge:**
- Volume overwhelm (fundamentally impossible to examine everything)
- Signal-to-noise (50% false positives, 33% ROT data)
- Economic reality ($2.16B/year wasted on unused observability data)
**The Path Forward:**
- NOT examining more data (impossible)
- INSTEAD: Selective attention at scale (unsolved AI problem)
- Knowing which 0.01% to process in real-time vs which 99.99% to ignore
--- ---
## Research Confidence Levels ## Detailed Findings by Category
### High Confidence Findings (90%+ certainty) ### Surveillance Video (22% of global data)
- Surveillance: 95-99% never watched (1+ billion cameras, multiple sources) **Generation:**
- IoT: 90% dark data (IoT Analytics, McKinsey confirmation) - 1+ billion cameras worldwide
- Enterprise dark data: 68-85% (Veritas, IDC, Gartner consensus) - 5.5 million TB/day of footage
- YouTube: 91% <1,000 views (official YouTube data) - 33 zettabytes/year
- Security alerts: 44% uninvestigated (SANS 2024 SOC Survey)
- Documents: 41-80% never accessed (NetApp 2024)
- Global generation: 149 ZB (IDC Data Age 2025)
### Medium Confidence Findings (70-90% certainty) **Utilization:**
- 95-99% never watched
- Footage stored for compliance/legal requirements
- Only reviewed if incident reported
- Communication: 9-15% net utilization (platform stats) **Sources:** Grand View Research 2024, security industry studies
- Code review: 10-15% thorough (GitHub Octoverse + Codacy) **Confidence:** High (90%+)
- Machine logs: 80-90% ignored (observability studies)
- AI automation: 15-25% processing (category extrapolation)
- Global utilization: 12-15% (weighted bottom-up calculation)
### Lower Confidence Findings (50-70% certainty) ---
- Exact data type percentages (source taxonomy varies) ### IoT Sensor Data (23% of global data)
**Generation:**
- 21.1 billion connected devices
- 79.4 zettabytes/year (projected 2025)
- Industrial sensors, smart homes, wearables, vehicles
**Utilization:**
- 90% becomes "dark data" (collected but never analyzed)
- 30-50% filtered at edge before storage
- <5% of stored data analyzed
- 99% lost before reaching operational decisions (industrial settings)
**Sources:** IoT Analytics 2024, McKinsey Digital
**Confidence:** High (90%+)
---
### Machine Logs & Telemetry (14% of global data)
**Generation:**
- Server logs, application logs, network telemetry
- Cloud infrastructure monitoring
- 21 zettabytes/year
**Utilization:**
- 80-90% never examined
- >90% of observability data never read
- Stored for compliance and debugging, not active analysis
- 44% of security alerts uninvestigated
**Sources:** Coralogix 2024, SANS SOC Survey 2024
**Confidence:** Medium-High (80%+)
---
### Enterprise Data (9% of global data)
**Generation:**
- Documents, communications, code, databases
- 13 zettabytes/year
**Utilization:**
- **Documents:** 41-80% never accessed after creation
- **Communications:** 85-91% never meaningfully consumed
- **Code:** 70-85% never reviewed after initial commit
- **Security logs:** 44% of alerts uninvestigated
- **Overall:** 70-75% never examined
**Sources:** NetApp 2024, Veritas Global Databerg Report, GitHub Octoverse, SANS
**Confidence:** High (90%+)
---
### Streaming Video (30% of global data)
**Generation:**
- Netflix, YouTube, TikTok, streaming services
- 45 zettabytes/year
**Utilization:**
- 60-70% watched (content created for consumption)
- Long-tail distribution: small fraction of content drives majority of views
- Catalog utilization varies by platform (proprietary data)
**Note:** "82% of internet traffic is video" (Cisco VNI) measures DATA TRANSMITTED (watched content), not DATA GENERATED (includes unwatched content)
**Sources:** Cisco VNI 2024, streaming service analytics
**Confidence:** Medium (70-80%)
---
### User-Generated Video (subset of video, ~15% of global data)
**Generation:**
- YouTube, Twitch, social media video
- ~22 zettabytes/year
**Utilization:**
- YouTube: 91% of videos receive <1,000 views
- Twitch: 80-90% of streams have zero concurrent viewers
- 60-80% never achieves meaningful viewership
**Sources:** YouTube statistics 2024, TwitchTracker
**Confidence:** High (90%+)
---
### Social Media (non-video) (2% of global data)
**Generation:**
- Text posts, images (non-video)
- 3 zettabytes/year
**Utilization:**
- 30-40% viewed with meaningful engagement
- Power law distribution: small fraction gets most attention
- Brief visibility window, rapid decay
**Sources:** Social media engagement studies 2024
**Confidence:** Medium (70-80%)
---
## Confidence Levels
### High Confidence (90%+)
- Surveillance: 95-99% never watched
- IoT: 90% dark data
- Enterprise dark data: 68-85%
- YouTube view distribution: 91% <1,000 views
- Security alerts: 44% uninvestigated
- Documents: 41-80% never accessed
- Global generation: 149 ZB
### Medium Confidence (70-90%)
- Communication utilization: 9-15%
- Code review thoroughness: 10-15%
- Machine logs: 80-90% ignored
- AI automation: 15-25% processing
- Global utilization rate: 12-15%
### Lower Confidence (50-70%)
- Exact category percentages (taxonomy varies by source)
- Streaming catalog utilization (proprietary data) - Streaming catalog utilization (proprietary data)
- Future projections (inherently speculative) - Future projections
--- ---
## Limitations and Future Research ## Study Limitations
### Study Limitations 1. **Temporal:** Data reflects 2024-2025 landscape
2. **Definitions:** "Examined" vs "analyzed" vs "acted upon" varies by source
1. **Rapidly Evolving Landscape:** Technology changes quickly (2024-2025 data) 3. **Data availability:** Proprietary systems don't publish statistics
2. **Definition Inconsistencies:** "Examined" vs "analyzed" vs "acted upon" thresholds vary 4. **Category overlap:** Some double-counting potential (video ⊆ surveillance ⊆ IoT)
3. **Data Availability:** Limited public data on proprietary systems (streaming catalogs, Fortune 500) 5. **Source reliability:** Some data from vendors (not independently audited)
4. **Category Overlap:** Video ⊆ Surveillance ⊆ IoT creates potential double-counting
5. **Vendor Claims:** Some statistics from vendors (not independently audited)
### Recommended Future Research
**Longitudinal Studies:**
1. Re-evaluate in 12 months to track utilization trends
2. Monitor AI automation maturation (2025-2027)
3. Track edge computing impact on IoT utilization
**Deep Dives:**
1. Industry-specific utilization (healthcare vs finance vs manufacturing)
2. ROI case studies for dark data monetization
3. Selective attention algorithms and success patterns
**Gap Filling:**
1. Streaming service catalog utilization (currently proprietary)
2. Fortune 500 production AI deployments (confidential)
3. Network traffic inspection coverage (limited public data)
--- ---
## Conclusion ## Documentation
This multi-agent research investigation reveals that humanity generates 149 zettabytes of data annually but examines only 12-15%—leaving 85-88% (129 zettabytes) permanently ignored. The "AI is running out of data" narrative measures exhausted historical archives while missing the massive waterfall of fresh data we create daily but cannot access due to context window limitations, encryption, legal barriers, and architectural constraints. **Complete source documentation:** See SOURCES.md (150+ sources with full citations)
**Critical Insight:** The problem is not data scarcity. We generate GPT-3's entire training corpus every 11 seconds. The problem is threefold: **Detailed methodology:** See METHODOLOGY.md (research design, agent assignments, validation protocols)
1. **Human overwhelm:** 85-88% ignored (surveillance, IoT, logs we'll never examine) **Individual research reports:** See findings/ directory (12 detailed reports)
2. **AI access barriers:** Context windows + five walls block highest-value data
3. **Attention-to-action gap:** Even examined data rarely drives decisions (99.9%)
**Meta-Finding:** The next decade of AI won't be defined by how much data we generate (already infinite) but by how effectively we solve selective attention—knowing which 0.01% of daily generation to process in real-time and which 99.99% to ignore. That's a $10+ trillion unsolved problem. **Blog-ready table:** See data-utilization-table.md
--- ---
## Citation ## Citation
**For Academic Use:** **Academic:**
``` ```
Miessler, D. (2024). Global Data Generation and Utilization Analysis Miessler, D. (2025). Global Data Generation and Utilization Rates.
[Technical Report]. Multi-Agent Research Investigation. Retrieved from Multi-Agent Research Investigation. Retrieved from
Substrate/research/data-utilization-global-analysis-november-2024/ https://github.com/danielmiessler/Substrate/tree/main/research/data-utilization-global-analysis-november-2025
``` ```
**For Blog/Article Use:** **General:**
``` ```
Research conducted via multi-agent AI framework, November 2025. Research conducted via multi-agent AI framework, November 2025.
150+ authoritative sources including Veritas, IDC, NetApp, SANS, GitHub, 150+ sources including IDC, Veritas, NetApp, SANS, GitHub, IoT Analytics, McKinsey.
IoT Analytics, McKinsey, and others. Complete documentation available.
``` ```
--- ---
## Appendices
- **Appendix A:** Comprehensive sources documentation (SOURCES.md)
- **Appendix B:** Unified synthesis (findings/SYNTHESIS.md)
- **Appendix C:** Corrected global calculation (findings/REAL_UTILIZATION.md)
- **Appendix D:** Blog post argument framework (findings/THE_ARGUMENT.md)
- **Appendix E:** Enterprise utilization reports (findings/*)
- **Appendix F:** Global breakdown reports (findings/*)
- **Appendix G:** Blog-ready table (data-utilization-table.md)
---
## Document History
- **Version 1.0** (2024-11-10): Initial research completion and documentation
- **Research Duration:** 6 hours (2 sessions: Nov 9-10, 2024)
- **Total Output:** 9 comprehensive reports, 200KB+ documentation
- **Total Sources:** 150+ authoritative publications (2024-2025)
- **Confidence Level:** High (85-90%) on core 12-15% utilization finding
---
**Research Infrastructure:** Kai AI System (Multi-Agent Research Framework) **Research Infrastructure:** Kai AI System (Multi-Agent Research Framework)
**Primary Researcher:** Daniel Miessler **Primary Researcher:** Daniel Miessler
**Research Dates:** November 9-10, 2025 **Research Dates:** November 9-10, 2025
**Document Status:** Final - Comprehensive Research Documentation **Document Status:** Final