Measurement on Mike Olsenhttps://molsen.ca/tags/measurement/Recent content in Measurement on Mike OlsenHugo -- gohugo.ioen-caTue, 31 Dec 2024 00:00:00 +0000What AI Benchmarks Miss About Real Professional Workhttps://molsen.ca/writing/what-benchmarks-arent-measuring/Tue, 31 Dec 2024 00:00:00 +0000https://molsen.ca/writing/what-benchmarks-arent-measuring/We do not know how much valuable professional work AI can competently perform. Existing benchmarks do not provide that measure. Tame Problems OpenAI’s GDPval benchmark represents substantial investment in evaluation. It recruits credentialed lawyers, accountants, engineers, and physicians to compare AI outputs with human work in blind evaluations of realistic tasks. It is carefully designed and transparently documented, improving on simpler metrics while measuring the wrong things for this purpose.