Challenges - MICCAI 2015

About

The OPTIMA Cyst segmentation challenge was hosted at the MICCAI 2015 conference in Munich as a full day Challenge event on the 5th October. 

The presence of retinal cysts are an important indicator of eye disease such as retinal vein occlusion (RVO) and age-related macular degeneration (AMD), thus their detection and segmentation is beneficial to clinical disease analysis, treatment and treatment progress assessment. Pathologies such as cysts can be imaged using spectral domain optical coherence tomography (SD-OCT), which is the most important ancillary test for the diagnosis of sight degrading diseases today. SD-OCT is a non-invasive modality for acquiring high resolution, 3D cross sectional volumetric images of the retina and the sub-retinal layers, in addition to retinal pathology. Up until now, there has not been a publically available dataset featuring clinically segmented cyst pathology or a universal framework for the evaluation of segmentation methods.

With this challenge, we made available a dataset of SD-OCT scans containing a wide variety of retinal cysts with accompanying clinical ground truth annotation. We want to challenge the medical imaging community to develop new and novel cyst segmentation techniques which can use this dataset for training and testing. In addition a universal evaluation framework has been designed to allow all the methods developed to evaluated and compared with one another.

This challenge is currently closed and the results are on this page below.


Challenge Categories

We discern four different categories of retinal cyst segmentation algorithms : automatic vendor independent segmentation methods, automatic vendor dependent methods, semi-automated vendor independent methods and semi-automated vendor dependent methods.

Category 1: Automatic vendor independent segmentation methods
Submitted methods must be fully automated and applicable on image from any SD-OCT scanner vendor. Evaluation of method will use images from all scanner vendors.

Category 2: Automatic vendor dependent segmentation methods
Submitted methods must be fully automated and may be be tailored for images from a specific scanner vendor. Evaluation of method will use images from individual scanner vendors.

Category 3: Semi-automatic vendor independent segmentation methods
Submitted methods may use a single pixel/voxel as a starting point for each cyst and should be applicable to images from any scanner vendor. Evaluation of methods will use images from all scanner vendors.

Category 4: Semi-automatic vendor dependent segmentation methods
Submitted methods may use a single pixel/voxel as a starting point for each cyst but may be tailored for images from a specific scanner vendor. Evaluation of methods will use images from individual scanner vendors.

Dataset composition

The challenge dataset consists of a training set, stage 1 testing set and stage 2 testing set with 15, 8 and 7 scans respectively. The breakdown of the dataset is as follows:
 

Set

Spectralis

Cirrus

Topcon

Nidek

Total

Training

4

4

4

3

15

Testing 1

2

2

2

2

8

Testing 2

2

2

2

1

7


Initially the training and testing 1 sets were made available for download with testing 2 to be an unseen test data set.

 

 

Results

Presented here are rankings of the submitted methods to the cyst segmentation challenge using a point based system. A point value is assigned to each team based on their performance ranking for each evaluation performed, 1st = 5 points, 2nd = 4 points, 3rd = 3 points, 4th = 2 points, 5th = 1 point. 

Overall Ranking
Overall point based rankings accumulated from all evaluations performed.

Team

Points

de Sisternes et al. (Stanford)

739

Venhuizen et al. (RadboudUMC)

608

Oguz et al. (Iowa)

509

Gopinath and Sivaswamy (RIAG)

282


Cirrus Ranking
Cirrus specific point based rankings accumulated from all evaluations performed.
 

Team

Points

de Sisternes et al. (Stanford)

141

Oguz et al. (Iowa)

128

Venhuizen et al. (RadboudUMC)

115

Gopinath and Sivaswamy (RIAG)

64


Nidek Ranking
Nidek specific point based rankings accumulated from all evaluations performed.

Team

Points

de Sisternes et al. (Stanford)

148

Oguz et al. (Iowa)

120

Venhuizen et al. (RadboudUMC)

104

Gopinath and Sivaswamy (RIAG)

76


Spectralis Ranking
Spectralis specific point based rankings accumulated from all evaluations performed.

Team

Points

de Sisternes et al. (Stanford)

146

Venhuizen et al. (RadboudUMC)

130

Esmaeili et al. (Isfahan)

97

Oguz et al. (Iowa)

63

Gopinath and Sivaswamy (RIAG)

44


Topcon Ranking
Topcon specific point based rankings accumulated from all evaluations performed.
 

Team

Points

de Sisternes et al. (Stanford)

152

Venhuizen et al. (RadboudUMC)

129

Oguz et al. (Iowa)

101

Gopinath and Sivaswamy (RIAG)

66



Dice Coefficients

 

Unmasked - Overall

Evaluated against G1, G2, G1 ∩ G2, and all devices:

Team Mean SD
de Sisternes et al. 0.64 0.14
Venhuizen et al. 0.55 0.20
Oguz et al. 0.48 0.22
Esmaeili et al. 0.45 0.23
Haritz et al. 0.14 0.08

Evaluated against G1, G2, G1 ∩ G2, and Cirrus:

Team Mean SD
de Sisternes et al. 0.62 0.18
Venhuizen et al. 0.56 0.27
Oguz et al. 0.47 0.30
Haritz et al. 0.09 0.06

Evaluated against G1, G2, G1 ∩ G2, and Nidek:

Team Mean SD
de Sisternes et al. 0.62 0.15
Oguz et al. 0.57 0.12
Venhuizen et al. 0.42 0.11
Haritz et al. 0.21 0.05

Evaluated against G1, G2, G1 ∩ G2, and Spectralis:

Team Mean SD
de Sisternes et al. 0.65 0.07
Venhuizen et al. 0.57 0.21
Esmaeili et al. 0.45 0.23
Oguz et al. 0.38 0.23
Haritz et al. 0.14 0.09

Evaluated against G1, G2, G1 ∩ G2, and Topcon:

Team Mean SD
de Sisternes et al. 0.67 0.15
Venhuizen et al. 0.63 0.11
Oguz et al. 0.52 0.13
Haritz et al. 0.14 0.06


Unmasked - G1

Evaluated against G1, and all devices:

Team Mean SD
de Sisternes et al. 0.64 0.14
Venhuizen et al. 0.56 0.20
Oguz et al. 0.48 0.25
Esmaeili et al. 0.46 0.25
Haritz et al. 0.14 0.08

Evaluated against G1, and Cirrus:

Team Mean SD
de Sisternes et al. 0.61 0.20
Venhuizen et al. 0.56 0.30
Oguz et al. 0.47 0.33
Haritz et al. 0.09 0.06

Evaluated against G1, and Nidek:

Team Mean SD
de Sisternes et al. 0.63 0.18
Oguz et al. 0.59 0.14
Venhuizen et al. 0.45 0.13
Haritz et al. 0.20 0.06

Evaluated against G1, and Spectralis:

Team Mean SD
de Sisternes et al. 0.64 0.08
Venhuizen et al. 0.57 0.23
Esmaeili et al. 0.46 0.25
Oguz et al. 0.39 0.26
Haritz et al. 0.14 0.11

Evaluated against G1, and Topcon:

Team Mean SD
de Sisternes et al. 0.67 0.16
Venhuizen et al. 0.64 0.12
Oguz et al. 0.52 0.14
Haritz et al. 0.14 0.06


Unmasked - G2

Evaluated against G2, and all devices

Team Mean SD
de Sisternes et al. 0.63 0.14
Venhuizen et al. 0.55 0.20
Oguz et al. 0.48 0.22
Esmaeili et al. 0.45 0.24
Haritz et al. 0.14 0.08

Evaluated against G2, and Cirrus:

Team Mean SD
de Sisternes et al. 0.61 0.14
Venhuizen et al. 0.56 0.21
Oguz et al. 0.47 0.31
Haritz et al. 0.09 0.04

Evaluated against G2,and Nidek:

Team Mean SD
de Sisternes et al. 0.60 0.17
Oguz et al. 0.57 0.14
Venhuizen et al. 0.42 0.12
Haritz et al. 0.21 0.05

Evaluated against G2, and Spectralis:

Team Mean SD
de Sisternes et al. 0.64 0.07
Venhuizen et al. 0.58 0.23
Esmaeili et al. 0.45 0.24
Oguz et al. 0.38 0.25
Haritz et al. 0.14 0.10

Evaluated against G2, and Topcon:

Team Mean SD
de Sisternes et al. 0.64 0.07
Venhuizen et al. 0.58 0.23
Oguz et al. 0.38 0.25
Haritz et al. 0.14 0.10


Unmasked - G1 ∩ G2

Evaluated against G1 ∩ G2, and all devices:

Team Mean SD
de Sisternes et al. 0.65 0.15
Venhuizen et al. 0.54 0.20
Oguz et al. 0.48 0.22
Esmaeili et al. 0.45 0.25
Haritz et al. 0.14 0.08

Evaluated against G1 ∩ G2, and Cirrus:

Team Mean SD
de Sisternes et al. 0.63 0.21
Venhuizen et al. 0.55 0.28
Oguz et al. 0.47 0.34
Haritz et al. 0.09 0.06

Evaluated against G1 ∩ G2, and Nidek:

Team Mean SD
de Sisternes et al. 0.61 0.17
Oguz et al. 0.55 0.13
Venhuizen et al. 0.38 0.10
Haritz et al. 0.21 0.06

Evaluated against G1 ∩ G2, and Spectralis:

Team Mean SD
de Sisternes et al. 0.66 0.08
Venhuizen et al. 0.56 0.24
Esmaeili et al. 0.45 0.25
Oguz et al. 0.37 0.25
Haritz et al. 0.14 0.10

Evaluated against G1 ∩ G2, and Topcon:

Team Mean SD
de Sisternes et al. 0.69 0.16
Venhuizen et al. 0.63 0.12
Oguz et al. 0.55 0.14
Haritz et al. 0.14 0.06

 

Masked - Overall

Evaluated against G1, G2, G1 ∩ G2, and all devices:

Team Mean SD
de Sisternes et al.

0.68

0.14
Venhuizen et al. 0.601 0.18
Oguz et al. 0.596 0.14
Esmaeili et al. 0.55 0.24
Haritz et al. 0.23 0.15

Evaluated against G1, G2, G1 ∩ G2, and Cirrus:

Team Mean SD
Oguz et al. 0.66 0.17
de Sisternes et al. 0.62 0.18
Venhuizen et al. 0.56 0.27
Haritz et al. 0.14 0.08

Evaluated against G1, G2, G1 ∩ G2, and Nidek:

Team Mean SD
de Sisternes et al. 0.71 0.10
Oguz et al. 0.57 0.12
Venhuizen et al. 0.51 0.07
Haritz et al. 0.42 0.09

Evaluated against G1, G2, G1 ∩ G2, and Spectralis:

Team Mean SD
de Sisternes et al. 0.67 0.06
Venhuizen et al. 0.60 0.16
Oguz et al. 0.56 0.13
Esmaeili et al. 0.55 0.24
Haritz et al. 0.25 0.19

Evaluated against G1, G2, G1 ∩ G2, and Topcon:

Team Mean SD
de Sisternes et al. 0.73 0.18
Venhuizen et al. 0.71 0.09
Oguz et al. 0.60 0.14
Haritz et al. 0.21 0.06


Masked - G1

Evaluated against G1, and all devices:

Team Mean SD
de Sisternes et al. 0.68 0.15
Venhuizen et al. 0.61 0.19
Oguz et al. 0.60 0.15
Esmaeili et al. 0.55 0.27
Haritz et al. 0.23 0.15

Evaluated against G1, and Cirrus:

Team Mean SD
Oguz et al. 0.65 0.18
de Sisternes et al. 0.61 0.19
Venhuizen et al. 0.57 0.30
Haritz et al. 0.14 0.09

Evaluated against G1,and Nidek:

Team Mean SD
de Sisternes et al. 0.73 0.12
Oguz et al. 0.59 0.15
Venhuizen et al. 0.55 0.10
Haritz et al. 0.41 0.12

Evaluated against G1, and Spectralis:

Team Mean SD
de Sisternes et al. 0.66 0.06
Venhuizen et al. 0.60 0.18
Oguz et al. 0.56 0.14
Esmaeili et al. 0.55 0.27
Haritz et al. 0.25 0.22

Evaluated against G1, and Topcon:

Team Mean SD
de Sisternes et al. 0.73 0.19
Venhuizen et al. 0.72 0.09
Oguz et al. 0.59 0.16
Haritz et al. 0.21 0.07


Masked - G2

Evaluated against G2, and all devices:

Team Mean SD
de Sisternes et al. 0.67 0.14
Venhuizen et al. 0.60 0.19
Oguz et al. 0.59 0.15
Esmaeili et al. 0.55 0.27
Haritz et al. 0.23 0.15

Evaluated against G2, and Cirrus:

Team Mean SD
Oguz et al. 0.66 0.20
de Sisternes et al. 0.61 0.08
Venhuizen et al. 0.56 0.15
Haritz et al. 0.14 0.09

Evaluated against G2,and Nidek:

Team Mean SD
de Sisternes et al. 0.70 0.12
Oguz et al. 0.57 0.15
Venhuizen et al. 0.52 0.07
Haritz et al. 0.41 0.11

Evaluated against G2, and Spectralis:

Team Mean SD
de Sisternes et al. 0.67 0.06
Venhuizen et al. 0.61 0.18
Esmaeili et al. 0.5500 0.27
Oguz et al. 0.5496 0.14
Haritz et al. 0.25 0.21

Evaluated against G2, and Topcon:

Team Mean SD
de Sisternes et al. 0.67 0.06
Venhuizen et al. 0.61 0.18
Oguz et al. 0.55 0.14
Haritz et al. 0.25 0.21


Masked - G1 ∩ G2

Evaluated against G1 ∩ G2, and all devices:

Team Mean SD
de Sisternes et al. 0.69 0.15
Oguz et al. 0.60 0.14
Venhuizen et al. 0.59 0.19
Esmaeili et al. 0.55 0.28
Haritz et al. 0.23 0.15

Evaluated against G1 ∩ G2, and Cirrus:

Team Mean SD
Oguz et al. 0.66 0.19
de Sisternes et al. 0.63 0.21
Venhuizen et al. 0.56 0.29
Haritz et al. 0.14 0.10

Evaluated against G1 ∩ G2, and Nidek:

Team Mean SD
de Sisternes et al. 0.69 0.11
Oguz et al. 0.55 0.12
Venhuizen et al. 0.47 0.06
Haritz et al. 0.43 0.12

Evaluated against G1 ∩ G2, and Spectralis:

Team Mean SD
de Sisternes et al. 0.68 0.07
Venhuizen et al. 0.58 0.19
Oguz et al. 0.56 0.13
Esmaeili et al. 0.55 0.28
Haritz et al. 0.25 0.21

Evaluated against G1 ∩ G2, and Topcon:

Team Mean SD
de Sisternes et al. 0.75 0.18
Venhuizen et al. 0.71 0.08
Oguz et al. 0.62 0.14
Haritz et al. 0.21 0.07



Proceedings