scienthoon/jev-ood-calibration

Skip(New discovery — enrichment pending)

Independent calibration test of TypeSafe's Jev on a task it cannot have seen: 900 rule-generated support tickets (choice / score / boolean) plus 3 public benchmarks via Vercel AI Gateway. Raw responses, ECE with noise floor, temperature refit, per-type sign of miscalibration. Reproducible for ~$0.06.

Why this repo matters

Evaluates Jev's "epistemically honest probabilities" on unseen tasks.
Measures calibration and overconfidence on out-of-distribution data.
Provides raw responses and detailed calibration metrics for transparency.

Key Metrics

Stars: 7
Forks: 0
Open Issues: 0
Stars (7d delta): 0
Stars (30d delta): 0

Scores & Metadata

Trend Velocity Score: 0.00
Opportunity Score: 25.00
Confidence Score: 10.00
Language: Python
License: MIT
First Seen: 2 weeks ago

Links & Sources

Related Repositories

sst/opencode

203.6kTypeScript

Shubhamsaboo/awesome-llm-apps

100+ AI Agents, Agent Skills and RAG Apps - Free and Open Source.

138.8kPython

Comfy-Org/ComfyUI

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.

131.2kPython

openai/codex

Lightweight coding agent that runs in your terminal

122.2kRust