How we tested it
Where the accuracy figures on the home page come from, step by step, and what they do not show.
Agrees with an expert 99% as often as experts agree with each other.
Of the chords an expert wrote, our analysis agreed with 69%. Of the chords one expert wrote, the other expert agreed with 70%. The 99% is the first figure divided by the second (69.39% against 69.74%). Here "agreed" means the same chord, in the same key, with the same note in the bass, for more than half of the chord's length.
1. Pieces chosen by a rule, before any results
We used only pieces that two different groups of experts had analysed, chord by chord, and that were in none of the sets we had used before, either to build the tool or to test it. Both analyses also had to fit our copy of the score: at least 85% of the notes had to belong to the chords they name, which rules out analyses that do not line up with the score.
From each of four styles we then took 9 pieces, evenly spaced through the alphabetical list of the candidates. The rule and the list of 36 pieces were written down on 6 November 2026, before any of our analyses of them were made.
2. The expert analyses
All of them are published research collections, gathered in one place with their scores by When in Rome:
- DCML: analyses by the Digital and Cognitive Musicology Lab at EPFL in Lausanne, of Mozart and Beethoven sonatas and Chopin mazurkas, among others.
- BPS-FH: Tsung-Ping Chen and Li Su's analyses of Beethoven's piano sonata first movements (2018).
- BCMH: the Bach Chorales Melody-Harmony collection.
- TAOM: a collection of analyses being prepared by Dmitri Tymoczko, written by him and other music theorists. His mazurka analyses are credited to "Dmitri Tymoczko and his computer", so they were made with computer help.
| Style | Pieces | Bars scored | Experts compared |
|---|---|---|---|
| Bach chorales | 9 | 127 | TAOM and BCMH |
| Mozart sonata movements | 9 | 405 | DCML and TAOM |
| Beethoven sonata movements | 9 | 409 | two or three of BPS-FH, DCML and TAOM |
| Chopin mazurkas | 9 | 386 | DCML and TAOM |
The two chorale analyses may not be fully independent of each other, which would make the experts' agreement on the chorales look higher than it is.
3. Our analysis, made as the service makes it
Claude Opus, at medium effort, wrote every analysis from the same written method and the same table of the score's notes that our service gives the chat model. It was run from a test script rather than inside the Claude or ChatGPT app. Each piece was run once, and that run is the one scored. In 15 of the 61 passages the first reply contained a mark in a form the drawing step rejects; the model was asked once more, and its second reply was scored.
Long pieces were not analysed whole. From each sonata movement and mazurka we took up to two stretches of about 24 bars, spread evenly through the piece by a fixed rule; each chorale was taken whole. That gave 61 passages, 1,327 bars in all. The tests start from digital scores, so mistakes in reading a PDF are not part of these figures.
4. Scored chord by chord, at several levels
Each chord counts once, however long it lasts. A chord the expert wrote counts as agreed when our analysis says the same thing for more than half of its length. We report six levels:
| Level | What has to match | Ours against an expert | Expert against expert |
|---|---|---|---|
| Key | The key the music is in at that point | 90% | 88% |
| Root | The note the chord is built on, whichever way the chord is turned | 88% | 91% |
| Chord | The same notes: V and V7 count as different | 81% | 81% |
| Chord and bass | The same notes and the same lowest note, so the same inversion | 75% | 75% |
| Function | The same bass note and a chord that does the same job: V and V7 count as the same, and so do a cadential 6/4 and the V it leads to | 79% | 81% |
| Full | Key, chord and bass note all the same. This is the headline level. | 69% | 70% |
Share of the expert's chords agreed with, all 36 pieces.
Counted the other way round, an expert agreed with 72% of our chords at the full level. We also measured by the share of the music's time rather than by chord: at the full level that is 74% for us and 70% between experts. We lead with the chord-by-chord figure because it is the less favourable of the two for us.
5. Experts against each other, on the same bars
Experts disagree, so the fair yardstick is not 100% but how often one expert agrees with another. Each expert's analysis was scored against the other's on exactly the bars we scored, both ways round, the same way as ours. Ours was scored against each expert in turn and the results averaged.
That averaging flatters us slightly. Where the two experts disagree and we side with one of them, we get half credit, while the experts get none for that chord.
6. How sure we can be
A result on 36 pieces depends partly on which pieces they were. To see how much, we drew 2,000 new sets of 36 from the same pieces at random, with repeats allowed, and recomputed the difference each time (a bootstrap). At the full level, by chord, ours minus the experts' is +1.2 percentage points, and 95% of the redrawn sets fall between −2.7 and +5.2. So the test cannot tell us apart from a second expert either way.
This difference is worked out piece by piece and weighted by each piece's bars. Pooling all the chords first, as in the 69% and 70% above, gives −0.4 points instead.
The range covers only the choice of pieces, not the model's own variation from run to run. In an earlier comparison of two runs of nearly the same setup, single pieces differed by about 4 points (standard deviation) at the full level, which would move an average over 6 pieces by about 3.
7. Results by style
| Style | Expert against expert | Ours against an expert | Difference, points (95% range) |
|---|---|---|---|
| Bach chorales | 83% | 77% | −6.3 (−11.5 to −1.0) |
| Mozart sonatas | 79% | 73% | −6.3 (−12.1 to −1.2) |
| Beethoven sonatas | 59% | 63% | +4.3 (−2.4 to +12.1) |
| Chopin mazurkas | 58% | 66% | +8.3 (+2.9 to +12.4) |
| All 36 pieces | 70% | 69% | +1.2 (−2.7 to +5.2) |
Full level, by chord. The difference is worked out piece by piece and weighted by each piece's bars, so it can differ from subtracting the two columns (step 6).
We are below the experts on the chorales and Mozart, where experts agree most, and above them on Beethoven and Chopin, where they disagree most. Part of that lead comes from the half credit described in step 5.
8. Cadences
Only the DCML analyses mark cadences. We used the 18 pieces (all 9 Mozart, 6 Beethoven, 3 Chopin) where the DCML expert marks a cadence within a bar of at least half of its own phrase endings; 6 others were left out. There the expert marked 124 cadences. We marked 63% of them within a beat with the same type, and 73% within a beat with any type. Of our own 141 cadence marks, 36% have no expert cadence nearby; we counted these but did not treat them as errors. We have no expert-against-expert figure for cadences, and the expert also writes a few kinds of cadence (evaded, plagal) that our marks cannot express.
What we don't claim
- Only Claude Opus was measured. ChatGPT and other models have not been tested this way and may do less well.
- It is weaker where experts agree most: the Bach chorales and Mozart.
- The comparison flatters us a little, because siding with either expert earns half credit.
- The model may have seen these analyses. They are published, so they may have been in what it learned from.
- Passages, not whole movements, and four styles of music from Bach to Chopin. Other music, such as Debussy's, was not part of this test.
- Agreeing with an expert is not the same as being right. Even two experts disagree on about 30% of chords at the full level.
Every experiment behind these figures, including the ones that failed, is written up in our engineering log; this test is entry 122, and the chord-by-chord figures on this page are in entry 131.