· ai
Why 90% Accuracy in Text-to-SQL is 100% Useless
The eternal promise of self-service analytics The post Why 90% Accuracy in Text-to-SQL is 100% Useless appeared first on Towards Data Science....
-
- · ai
How to Build an AI Agent Evaluation Framework That Scales
The Scaling Problem So, you've built a great AI agent. You've tested it with a few dozen examples, and it works perfectly. Now, you're ready to deploy it to pr...
- · ai
When Ai Learns to Admit Its Mistakes Trust Becomes a Real Responsibility
Introduction OpenAI’s latest research direction marks a significant evolution in how advanced AI systems are trained and evaluated, raising fundamental questio...
- · ai
Running Evals on a Bloated RAG Pipeline
Comparing metrics across datasets and models The post Running Evals on a Bloated RAG Pipeline appeared first on Towards Data Science....
- · ai
Measuring AI Ability to Complete Long Tasks
Article URL: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ Comments URL: https://news.ycombinator.com/item?id=46342166 Points: 1...
- · ai
Measuring AI Ability to Complete Long Tasks: Opus 4.5 has 50% horizon of 4h49M
Article URL: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ Comments URL: https://news.ycombinator.com/item?id=46342166 Points: 3...
- · ai
AI agents fail 63% of the time on complex tasks. Patronus AI says its new 'living' training worlds can fix that.
Patronus AI, the artificial intelligence evaluation startup backed by $20 million from investors including Lightspeed Venture Partners and Datadog, unveiled a n...
- · ai
Auto-grading decade-old Hacker News discussions with hindsight
!hnherohttps://bear-images.sfo2.cdn.digitaloceanspaces.com/karpathy/hnhero.webp Yesterday I stumbled on this HN thread — Show HN: Gemini Pro 3 hallucinates the...
- · ai
How to use System prompts as Ground Truth for Evaluation
The Problem: Lack of Clear Ground Truth Most teams struggle to evaluate their AI agents because they don’t have a well‑defined ground truth. Typical workflow:...
- · ai
Gemini 3 Pro scores 69% trust in blinded testing up from 16% for Gemini 2.5: The case for evaluating AI on real-world trust, not academic benchmarks
Just a few short weeks ago, Google debuted its Gemini 3 model, claiming it scored a leadership position in multiple AI benchmarks. But the challenge with vendor...
- · ai
I Drop a Test, 5 Out of 6 SOTA LLMs Drop Their Pants Off
The Hypothesis I've been researching what makes an entity 'deeply' intelligent—not just smart or capable, but understanding reality in a way that transcends pa...