Evaluation procedure

Goals

The aims of the evaluation metrics involved in the segmentation contest are twofold:

  • measure the degree of accuracy of the left ventricular endocardium and epicardium as the right ventricular endocardium. This will be done through global and local measures of similarity with the reference contours;

  • measure the degree of accuracy of the derived clinical indices.


For this contest, we provide the usual clinical and geometrical metrics.

Clinical indice metrics

The clinical metrics are the ones that are the most widely used in cardiac clinical practice. Moreover a set of metrics are computed per structure (i.e. LV cavity, RV cavity and myocardium) to allow the computation of ranking per structure. The following metrics are computed:

  • Left ventricular cavity

    • Correlation coefficient computed from the set of End Diastolic Volumes (EDV) measurements

    • Correlation coefficient computed from the set of End Systolic Volumes (ESV) measurements

    • Correlation coefficient computed from the set of Ejection Fraction (EF) measurements

    • Bias computed from the set of EDV measurements

    • Bias computed from the set of ESV measurements

    • Bias computed from the set of EF measurements

    • Limits of agreement (LOA = 1.96 times the standard deviation) computed from the set of EDV measurements

    • Limits of agreement (LOA = 1.96 times the standard deviation) computed from the set of ESV measurements

    • Limits of agreement (LOA = 1.96 times the standard deviation) computed from the set of EF measurements

  • Right ventricular cavity

    • Correlation coefficient computed from the set of End Diastolic Volumes (EDV) measurements

    • Correlation coefficient computed from the set of End Systolic Volumes (ESV) measurements

    • Correlation coefficient computed from the set of Ejection Fraction (EF) measurements

    • Bias computed from the set of EDV measurements

    • Bias computed from the set of ESV measurements

    • Bias computed from the set of EF measurements

    • Limits of agreement (LOA = 1.96 times the standard deviation) computed from the set of EDV measurements

    • Limits of agreement (LOA = 1.96 times the standard deviation) computed from the set of ESV measurements

    • Limits of agreement (LOA = 1.96 times the standard deviation) computed from the set of EF measurements

  • Myocardium

    • Correlation coefficient computed from the set of Myocardial Mass (at End Diastolic time instant) measurements

    • Correlation coefficient computed from the set of End Systolic Volumes (ESV) measurements

    • Bias computed from the set of Myocardial Mass measurements

    • Bias computed from the set of ESV measurements

    • Limits of agreement (LOA = 1.96 times the standard deviation) computed from the set of Myocardial Mass measurements

    • Limits of agreement (LOA = 1.96 times the standard deviation) computed from the set of ESV measurements

Distance error metrics

A set of geometrical metrics are computed per structure (i.e. LV cavity, RV cavity and myocardium) to allow the computation of ranking per structure. The following metrics are computed:

  • Left ventricular cavity

    • The average Dice value for the left ventricle cavity at ED.

    • The average Dice value for the left ventricle cavity at ES.

    • The average Hausdorff distance for the endocardial contour of the left ventricle at ED.

    • The average Hausdorff distance for the endocardial contour of the left ventricle at ES.

  • Right ventricular cavity

    • The average Dice value for the LV cavity at ED.

    • The average Dice value for the LV cavity at ES.

    • The average Hausdorff distance for the endocardial contour of the right ventricle at ED.

    • The average Hausdorff distance for the endocardial contour of the right ventricle at ES.

  • Myocardium

    • The average Dice value for the myocardium region at ED.

    • The average Dice value for the myocardium region at ES.

    • The average Hausdorff distance for the myocardial contours at ED.

    • The average Hausdorff distance for the myocardial contours at ES.

Please refer to this citation for any use of the ACDC database

  • O. Bernard, A. Lalande, C. Zotti, F. Cervenansky, et al.
    "Deep Learning Techniques for Automatic MRI Cardiac Multi-structures Segmentation and
    Diagnosis: Is the Problem Solved ?" in IEEE Transactions on Medical Imaging,
    vol. 37, no. 11, pp. 2514-2525, Nov. 2018

    doi: 10.1109/TMI.2018.2837502