Open-source prediction tracking & accuracy scoring framework.
Track predictions from anyone — media, analysts, politicians, CEOs, AI models. Score accuracy over time. Build leaderboards. Hold the world accountable.
Everyone makes predictions. Almost nobody tracks them.
- Elon Musk said FSD would be ready “next year” — five years in a row
- Jim Cramer’s stock picks are famously inverse-correlated with outcomes
- Media outlets make bold forecasts and quietly move on when they’re wrong
Signal Tracker gives you the tools to track all of this systematically.
Created by Crene — the AI-powered prediction intelligence platform that tracks 420+ sources across tech, finance, politics, and geopolitics.
pip install signal-tracker
from signal_tracker import SignalTracker
from datetime import date
# Initialize
tracker = SignalTracker()
# Add sources to track
elon = tracker.add_source("Elon Musk", source_type="person", category="tech")
cramer = tracker.add_source("Jim Cramer", source_type="person", category="finance")
# Add predictions
claim1 = tracker.add_claim(
source=elon,
text="Tesla will achieve full self-driving by end of 2025",
target_date=date(2025, 12, 31),
)
claim2 = tracker.add_claim(
source=cramer,
text="Netflix will hit $800 by Q2 2025",
target_date=date(2025, 6, 30),
)
# Verify when outcomes are known
tracker.verify(claim1, outcome="wrong", reasoning="FSD not achieved by deadline")
tracker.verify(claim2, outcome="correct", reasoning="Netflix reached $820 in May 2025")
# Build leaderboard
board = tracker.leaderboard()
for entry in board.entries:
print(f"{entry.rank}. {entry.source.name}: {entry.score.accuracy_score}%")
# Save state
tracker.save("my_tracker.json")
Track specific, verifiable claims with time bounds, measurable targets, and clear success criteria.
claim = tracker.add_claim(
source=source,
text="Bitcoin will reach $150k by end of 2025",
target_date=date(2025, 12, 31),
category="crypto",
)
Multiple scoring modes — simple, time-windowed, and recency-weighted.
score = tracker.score(source)
print(f"{source.name}: {score.accuracy_score}% ({score.correct_claims}/{score.total_claims})")
Automatic ranking with risers, fallers, and notable results.
board = tracker.leaderboard(min_claims=3)
board.top_accurate # Best predictors
board.worst_accurate # Worst predictors
board.biggest_risers # Improving fast
board.biggest_fallers # Getting worse
board.notable_wrongs # High-profile misses
Extract predictions from text — rule-based (fast) or LLM-powered (accurate).
# Rule-based (no API needed)
claims = tracker.extract_claims(
text="Elon Musk said Tesla will achieve full self-driving by 2025.",
source=elon,
)
# LLM-powered (bring your own LLM function)
def my_llm(prompt: str) -> str:
return response.text
tracker = SignalTracker(llm_fn=my_llm)
claims = tracker.extract_claims(text, source=elon, use_llm=True)
Consensus-based verification like Crene’s 4-LLM system.
tracker.verify_with_consensus(claim, [
{"outcome": "correct", "verifier": "ai:claude", "confidence": 0.9},
{"outcome": "correct", "verifier": "ai:gpt-4", "confidence": 0.85},
{"outcome": "wrong", "verifier": "ai:gemini", "confidence": 0.6},
])
Automatically rate how verifiable a prediction is.
from signal_tracker import QualityScorer
scorer = QualityScorer()
score = scorer.score(claim) # 0-100
JSON file or SQLite for larger datasets.
tracker.save("tracker.json")
tracker = SignalTracker.load("tracker.json")
from signal_tracker.storage import SQLiteBackend
backend = SQLiteBackend("tracker.db")
signal-tracker/
├── tracker.py # SignalTracker — main interface
├── models.py # Source, Claim, Verification, ScoreSnapshot
├── scoring.py # AccuracyScorer, QualityScorer
├── extractors.py # ClaimExtractor (rules + LLM)
├── leaderboard.py # Leaderboard engine
└── storage.py # SQLiteBackend
Design principles: Zero required dependencies (stdlib only), bring your own LLM, pluggable storage, plain dataclasses everywhere.
| Use Case | Who It’s For |
|---|---|
| Track media accuracy | Journalists, researchers |
| Score analyst predictions | Finance professionals |
| Monitor political promises | Civic organizations |
| Track AI model forecasts | ML engineers |
| Build prediction markets | Developers |
| Personal prediction journal | Anyone |
Crene is an AI-powered prediction intelligence platform that tracks 420+ sources across tech, finance, politics, and geopolitics. We use a 4-LLM consensus system (Claude, GPT-4, Gemini, Grok) to verify claims and score credibility.
Signal Tracker is the open-source framework extracted from Crene’s production system. The framework is free — the data is the moat.
We welcome contributions! See CONTRIBUTING.md.
git clone https://github.com/Creneinc/signal-tracker.git
cd signal-tracker
pip install -e ".[all]"
pytest
MIT License. See LICENSE.