Most individual results reproduced.
The reported average is still unexplained.

An independent check of an existing evaluation.

Before asking how well ECGFounder works in another setting, I wanted to check whether I could reproduce one of the evaluations in its paper. I used the authors' released 12-lead model on PTB-XL, a dataset they had already tested. I did not train a new model.

Most individual scores came back very close to the published table. I also found an unexplained difference in the reported average and showed how changing the signal preparation can move individual scores.

The authors had already tested ECGFounder on several datasets outside the one used to train it. My contribution here is an independent check of one of those evaluations, with the choices and discrepancies documented. That gives me a clearer starting point for testing it in another setting. Whether it helps clinicians or patients is a further question this experiment does not answer.

A ranking score, not “percent accurate”.

The score used here is AUROC. Imagine choosing one recording with a finding and one without it. Does the model give the first recording the higher score? AUROC measures how well it ranks those pairs: 0.5 is chance-level ranking and 1.0 is perfect ranking on the evaluated data.

It is not a percentage of correct diagnoses. It does not tell us where to draw a yes-or-no threshold, how many false alarms we will get, or whether the model’s probabilities are well calibrated.

Figure 1 · Same 30 findings

Most dots sit on the agreement line.

Each dot is a finding. A dot on the diagonal means the paper and lab report the same AUROC.

Open figure at full size ↗Download all 31 scores (CSV) ↓

Of the 30 scores in the paper's table, 29 were within 0.001 of my run. That gives me confidence that most of the individual results reproduce closely. It also points towards the evaluation script as a useful route for reproducing them, although matching scores don't establish the authors' exact setup.

Sinus rhythm was the exception: 0.923 in the paper and 0.823 in my run. I checked the label mapping and the model's other outputs without resolving the difference. A typo is one possible explanation, but I haven't established the cause.

The headline average needs explaining.

The paper reports an average of 0.924. The plain mean of the 30 scores printed in its table is 0.8977. My mean across those same 30 findings is 0.8943.

Those are two different questions: can I reproduce the individual results, and can I explain the reported average? Most individual results reproduce closely. I still do not know which scope or calculation produced the headline number, so I cannot establish its cause from this run.

The 31-finding mean includes one additional evaluable label, “anterolateral leads”, that is absent from the paper’s 30-row table. That is why 0.8909 should not be presented as a directly comparable mean of those 30 rows.

The instructions don’t quite line up.

The instructions for preparing the ECGs don’t quite line up. The paper describes filtering out noise. The README points to a filtering routine with different settings. But the released PTB-XL evaluation script skips those filters and just centres and rescales each recording.

So I ran the model both ways: following the evaluation script, then following the README’s filtering instructions. The average across 31 findings barely moved—just 0.0004. But underneath that average, some individual scores improved and others got worse, with changes of up to 0.044.

That caught my attention because the model and the recordings hadn’t changed. Preparing the signal differently was enough to shift some results, and looking only at the average would have hidden that. Anyone trying to reproduce or build on this work needs to know which preparation steps were used. This experiment doesn’t establish which approach is best in clinical use.

Figure 3 · The four largest absolute changes

A small average can hide larger movements.

Right of zero: a higher AUROC after filtering. Left: a lower AUROC. These are the four largest changes among all 31 findings.

Open figure at full size ↗Download all 31 scores (CSV) ↓

A matching score doesn’t settle the counts.

I also found a mismatch in the counts. For low-voltage QRS, the paper lists 182, while the released label file I used marks 3,327 recordings with that finding. The label-file count comes from the reference labels used to score the model, not from the model's own predictions.

The score still closely matches the paper, but that doesn't explain the count difference. I need to clarify what the paper's count represents and check how the released labels were defined and loaded before treating the two evaluations as identical.

Keep each patient’s recordings together.

The score changes from filtering are measured differences on this dataset. I haven't yet estimated uncertainty around those changes.

The uncertainty ranges I calculated for the averages also have a limitation: I resampled ECG recordings individually, even though some came from the same person. Recordings from the same person can be related, so those ranges may look too precise.

A stronger check would keep each patient's recordings together and compare both preparation methods on the same groups of patients. Until then, I'll report the changes as observations from this dataset, with their wider reliability still open.

PTB-XL includes 21,799 recordings from 18,869 patients. The planned follow-up will resample patients, keep their recordings together, and compare both preparation paths using the same patient draws. It is an exploratory follow-up after seeing these results, not a newly pre-stated confirmation of them.

One model, two sets of recordings.

Next I want to test the single-lead version of ECGFounder on recordings taken during surgery. That is a different model from the 12-lead version I used in Week 2, and it receives just one electrical view of the heart.

To make the comparison more informative, I plan to run that same single-lead model on lead II from both PTB-XL and VitalDB, using consistent preparation and comparing findings that can be defined in both datasets. That will let me compare the same model across two sets of recordings. It will still leave differences in the patients, recording conditions and labels to consider.

VitalDB Arrhythmia gives me selected stretches of ECG recorded during surgery, with rhythms reviewed by anaesthesiologists. Those stretches were screened for possible rhythm abnormalities, so they are not a complete picture of an operation or everyday life.

This test can show how the model performs on those labelled recordings. It cannot, by itself, tell me how many false alarms it would produce during a full day of monitoring, or whether using it would improve someone's care.

VitalDB is the broader collection of surgical recordings. VitalDB Arrhythmia provides the reviewed labels for selected ECG segments. The labels are used to evaluate model outputs; they are not fed into the AI as hints.

The additional single-lead PTB-XL baseline and patient-grouped comparison are planned research tasks. Their time will be allocated at Sunday planning on 1 November. Neither is a new result on this page.

What this correction changes.

Updated 28 October 2026. This revision of the 25 October post corrects the filtering description, separates the 30- and 31-finding averages, removes certainty about the authors’ exact setup and a possible typo, and makes the label-count and statistical limits explicit. It also distinguishes the next single-lead experiment from Week 2’s 12-lead reproduction.

The underlying saved results have not changed. The headline-average discrepancy and the count questions remain open. I contacted the authors on 21 October; no answer is recorded as of this 28 October update.

• Original ECGFounder paper, version 4 · PDF
• PTB-XL v1.0.3 · Wagner et al. · CC BY 4.0
• VitalDB · VitalDB Arrhythmia
• Saved per-finding scores used in these figures

This work reports model performance against a dataset’s reference labels. It does not provide health advice or establish clinical benefit. The lab repository is planned to become public with the Season 1 research output.