Challenges

About

The OPTIMA Cyst segmentation challenge was hosted at the MICCAI 2015 conference in Munich as a full day Challenge event on the 5th October. 

The presence of retinal cysts are an important indicator of eye disease such as retinal vein occlusion (RVO) and age-related macular degeneration (AMD), thus their detection and segmentation is beneficial to clinical disease analysis, treatment and treatment progress assessment. Pathologies such as cysts can be imaged using spectral domain optical coherence tomography (SD-OCT), which is the most important ancillary test for the diagnosis of sight degrading diseases today. SD-OCT is a non-invasive modality for acquiring high resolution, 3D cross sectional volumetric images of the retina and the sub-retinal layers, in addition to retinal pathology. Up until now, there has not been a publically available dataset featuring clinically segmented cyst pathology or a universal framework for the evaluation of segmentation methods.

With this challenge, we made available a dataset of SD-OCT scans containing a wide variety of retinal cysts with accompanying clinical ground truth annotation. We want to challenge the medical imaging community to develop new and novel cyst segmentation techniques which can use this dataset for training and testing. In addition a universal evaluation framework has been designed to allow all the methods developed to evaluated and compared with one another.

This challenge is currently closed and the results are on this page below

Challenge Categories

We discern four different categories of retinal cyst segmentation algorithms : automatic vendor independent segmentation methods, automatic vendor dependent methods, semi-automated vendor independent methods and semi-automated vendor dependent methods.

Category 1: Automatic vendor independent segmentation methods
Submitted methods must be fully automated and applicable on image from any SD-OCT scanner vendor. Evaluation of method will use images from all scanner vendors.
Category 2: Automatic vendor dependent segmentation methods
Submitted methods must be fully automated and may be be tailored for images from a specific scanner vendor. Evaluation of method will use images from individual scanner vendors.
Category 3: Semi-automatic vendor independent segmentation methods
Submitted methods may use a single pixel/voxel as a starting point for each cyst and should be applicable to images from any scanner vendor. Evaluation of methods will use images from all scanner vendors.
Category 4: Semi-automatic vendor dependent segmentation methods
Submitted methods may use a single pixel/voxel as a starting point for each cyst but may be tailored for images from a specific scanner vendor. Evaluation of methods will use images from individual scanner vendors.

Dataset composition

The challenge dataset consists of a training set, stage 1 testing set and stage 2 testing set with 15, 8 and 7 scans respectively. The breakdown of the dataset is as follows:
 

Set	Spectralis	Cirrus	Topcon	Nidek	Total
Training	4	4	4	3	15
Testing 1	2	2	2	2	8
Testing 2	2	2	2	1	7

Initially the training and testing 1 sets were made available for download with testing 2 to be an unseen test data set.

Results

Presented here are rankings of the submitted methods to the cyst segmentation challenge using a point based system. A point value is assigned to each team based on their performance ranking for each evaluation performed, 1st = 5 points, 2nd = 4 points, 3rd = 3 points, 4th = 2 points, 5th = 1 point. 

Overall Ranking
Overall point based rankings accumulated from all evaluations performed.

Team	Points
de Sisternes et al. (Stanford)	739
Venhuizen et al. (RadboudUMC)	608
Oguz et al. (Iowa)	509
Gopinath and Sivaswamy (RIAG)	282

Cirrus Ranking
Cirrus specific point based rankings accumulated from all evaluations performed.
 

Team	Points
de Sisternes et al. (Stanford)	141
Oguz et al. (Iowa)	128
Venhuizen et al. (RadboudUMC)	115
Gopinath and Sivaswamy (RIAG)	64

Nidek Ranking
Nidek specific point based rankings accumulated from all evaluations performed.

Team	Points
de Sisternes et al. (Stanford)	148
Oguz et al. (Iowa)	120
Venhuizen et al. (RadboudUMC)	104
Gopinath and Sivaswamy (RIAG)	76

Spectralis Ranking
Spectralis specific point based rankings accumulated from all evaluations performed.

Team	Points
de Sisternes et al. (Stanford)	146
Venhuizen et al. (RadboudUMC)	130
Esmaeili et al. (Isfahan)	97
Oguz et al. (Iowa)	63
Gopinath and Sivaswamy (RIAG)	44

Topcon Ranking
Topcon specific point based rankings accumulated from all evaluations performed.
 

Team	Points
de Sisternes et al. (Stanford)	152
Venhuizen et al. (RadboudUMC)	129
Oguz et al. (Iowa)	101
Gopinath and Sivaswamy (RIAG)	66

Dice Coefficients

Unmasked – Overall

Evaluated against G1, G2, G1 ∩ G2, and all devices:

Team	Mean	SD
de Sisternes et al.	0.64	0.14
Venhuizen et al.	0.55	0.20
Oguz et al.	0.48	0.22
Esmaeili et al.	0.45	0.23
Haritz et al.	0.14	0.08

Evaluated against G1, G2, G1 ∩ G2, and Cirrus:

Team	Mean	SD
de Sisternes et al.	0.62	0.18
Venhuizen et al.	0.56	0.27
Oguz et al.	0.47	0.30
Haritz et al.	0.09	0.06

Evaluated against G1, G2, G1 ∩ G2, and Nidek:

Team	Mean	SD
de Sisternes et al.	0.62	0.15
Oguz et al.	0.57	0.12
Venhuizen et al.	0.42	0.11
Haritz et al.	0.21	0.05

Evaluated against G1, G2, G1 ∩ G2, and Spectralis:

Team	Mean	SD
de Sisternes et al.	0.65	0.07
Venhuizen et al.	0.57	0.21
Esmaeili et al.	0.45	0.23
Oguz et al.	0.38	0.23
Haritz et al.	0.14	0.09

Evaluated against G1, G2, G1 ∩ G2, and Topcon:

Team	Mean	SD
de Sisternes et al.	0.67	0.15
Venhuizen et al.	0.63	0.11
Oguz et al.	0.52	0.13
Haritz et al.	0.14	0.06

Unmasked – G1

Evaluated against G1, and all devices:

Team	Mean	SD
de Sisternes et al.	0.64	0.14
Venhuizen et al.	0.56	0.20
Oguz et al.	0.48	0.25
Esmaeili et al.	0.46	0.25
Haritz et al.	0.14	0.08

Evaluated against G1, and Cirrus:

Team	Mean	SD
de Sisternes et al.	0.61	0.20
Venhuizen et al.	0.56	0.30
Oguz et al.	0.47	0.33
Haritz et al.	0.09	0.06

Evaluated against G1, and Nidek:

Team	Mean	SD
de Sisternes et al.	0.63	0.18
Oguz et al.	0.59	0.14
Venhuizen et al.	0.45	0.13
Haritz et al.	0.20	0.06

Evaluated against G1, and Spectralis:

Team	Mean	SD
de Sisternes et al.	0.64	0.08
Venhuizen et al.	0.57	0.23
Esmaeili et al.	0.46	0.25
Oguz et al.	0.39	0.26
Haritz et al.	0.14	0.11

Evaluated against G1, and Topcon:

Team	Mean	SD
de Sisternes et al.	0.67	0.16
Venhuizen et al.	0.64	0.12
Oguz et al.	0.52	0.14
Haritz et al.	0.14	0.06

Unmasked – G2

Evaluated against G2, and all devices

Team	Mean	SD
de Sisternes et al.	0.63	0.14
Venhuizen et al.	0.55	0.20
Oguz et al.	0.48	0.22
Esmaeili et al.	0.45	0.24
Haritz et al.	0.14	0.08

Evaluated against G2, and Cirrus:

Team	Mean	SD
de Sisternes et al.	0.61	0.14
Venhuizen et al.	0.56	0.21
Oguz et al.	0.47	0.31
Haritz et al.	0.09	0.04

Evaluated against G2,and Nidek:

Team	Mean	SD
de Sisternes et al.	0.60	0.17
Oguz et al.	0.57	0.14
Venhuizen et al.	0.42	0.12
Haritz et al.	0.21	0.05

Evaluated against G2, and Spectralis:

Team	Mean	SD
de Sisternes et al.	0.64	0.07
Venhuizen et al.	0.58	0.23
Esmaeili et al.	0.45	0.24
Oguz et al.	0.38	0.25
Haritz et al.	0.14	0.10

Evaluated against G2, and Topcon:

Team	Mean	SD
de Sisternes et al.	0.64	0.07
Venhuizen et al.	0.58	0.23
Oguz et al.	0.38	0.25
Haritz et al.	0.14	0.10

Unmasked – G1 ∩ G2

Evaluated against G1 ∩ G2, and all devices:

Team	Mean	SD
de Sisternes et al.	0.65	0.15
Venhuizen et al.	0.54	0.20
Oguz et al.	0.48	0.22
Esmaeili et al.	0.45	0.25
Haritz et al.	0.14	0.08

Evaluated against G1 ∩ G2, and Cirrus:

Team	Mean	SD
de Sisternes et al.	0.63	0.21
Venhuizen et al.	0.55	0.28
Oguz et al.	0.47	0.34
Haritz et al.	0.09	0.06

Evaluated against G1 ∩ G2, and Nidek:

Team	Mean	SD
de Sisternes et al.	0.61	0.17
Oguz et al.	0.55	0.13
Venhuizen et al.	0.38	0.10
Haritz et al.	0.21	0.06

Evaluated against G1 ∩ G2, and Spectralis:

Team	Mean	SD
de Sisternes et al.	0.66	0.08
Venhuizen et al.	0.56	0.24
Esmaeili et al.	0.45	0.25
Oguz et al.	0.37	0.25
Haritz et al.	0.14	0.10

Evaluated against G1 ∩ G2, and Topcon:

Team	Mean	SD
de Sisternes et al.	0.69	0.16
Venhuizen et al.	0.63	0.12
Oguz et al.	0.55	0.14
Haritz et al.	0.14	0.06

Masked – Overall

Evaluated against G1, G2, G1 ∩ G2, and all devices:

Team	Mean	SD
de Sisternes et al.	0.68	0.14
Venhuizen et al.	0.601	0.18
Oguz et al.	0.596	0.14
Esmaeili et al.	0.55	0.24
Haritz et al.	0.23	0.15

Evaluated against G1, G2, G1 ∩ G2, and Cirrus:

Team	Mean	SD
Oguz et al.	0.66	0.17
de Sisternes et al.	0.62	0.18
Venhuizen et al.	0.56	0.27
Haritz et al.	0.14	0.08

Evaluated against G1, G2, G1 ∩ G2, and Nidek:

Team	Mean	SD
de Sisternes et al.	0.71	0.10
Oguz et al.	0.57	0.12
Venhuizen et al.	0.51	0.07
Haritz et al.	0.42	0.09

Evaluated against G1, G2, G1 ∩ G2, and Spectralis:

Team	Mean	SD
de Sisternes et al.	0.67	0.06
Venhuizen et al.	0.60	0.16
Oguz et al.	0.56	0.13
Esmaeili et al.	0.55	0.24
Haritz et al.	0.25	0.19

Evaluated against G1, G2, G1 ∩ G2, and Topcon:

Team	Mean	SD
de Sisternes et al.	0.73	0.18
Venhuizen et al.	0.71	0.09
Oguz et al.	0.60	0.14
Haritz et al.	0.21	0.06

Masked – G1

Evaluated against G1, and all devices:

Team	Mean	SD
de Sisternes et al.	0.68	0.15
Venhuizen et al.	0.61	0.19
Oguz et al.	0.60	0.15
Esmaeili et al.	0.55	0.27
Haritz et al.	0.23	0.15

Evaluated against G1, and Cirrus:

Team	Mean	SD
Oguz et al.	0.65	0.18
de Sisternes et al.	0.61	0.19
Venhuizen et al.	0.57	0.30
Haritz et al.	0.14	0.09

Evaluated against G1,and Nidek:

Team	Mean	SD
de Sisternes et al.	0.73	0.12
Oguz et al.	0.59	0.15
Venhuizen et al.	0.55	0.10
Haritz et al.	0.41	0.12

Evaluated against G1, and Spectralis:

Team	Mean	SD
de Sisternes et al.	0.66	0.06
Venhuizen et al.	0.60	0.18
Oguz et al.	0.56	0.14
Esmaeili et al.	0.55	0.27
Haritz et al.	0.25	0.22

Evaluated against G1, and Topcon:

Team	Mean	SD
de Sisternes et al.	0.73	0.19
Venhuizen et al.	0.72	0.09
Oguz et al.	0.59	0.16
Haritz et al.	0.21	0.07

Masked – G2

Evaluated against G2, and all devices:

Team	Mean	SD
de Sisternes et al.	0.67	0.14
Venhuizen et al.	0.60	0.19
Oguz et al.	0.59	0.15
Esmaeili et al.	0.55	0.27
Haritz et al.	0.23	0.15

Evaluated against G2, and Cirrus:

Team	Mean	SD
Oguz et al.	0.66	0.20
de Sisternes et al.	0.61	0.08
Venhuizen et al.	0.56	0.15
Haritz et al.	0.14	0.09

Evaluated against G2,and Nidek:

Team	Mean	SD
de Sisternes et al.	0.70	0.12
Oguz et al.	0.57	0.15
Venhuizen et al.	0.52	0.07
Haritz et al.	0.41	0.11

Evaluated against G2, and Spectralis:

Team	Mean	SD
de Sisternes et al.	0.67	0.06
Venhuizen et al.	0.61	0.18
Esmaeili et al.	0.5500	0.27
Oguz et al.	0.5496	0.14
Haritz et al.	0.25	0.21

Evaluated against G2, and Topcon:

Team	Mean	SD
de Sisternes et al.	0.67	0.06
Venhuizen et al.	0.61	0.18
Oguz et al.	0.55	0.14
Haritz et al.	0.25	0.21

Masked – G1 ∩ G2

Evaluated against G1 ∩ G2, and all devices:

Team	Mean	SD
de Sisternes et al.	0.69	0.15
Oguz et al.	0.60	0.14
Venhuizen et al.	0.59	0.19
Esmaeili et al.	0.55	0.28
Haritz et al.	0.23	0.15

Evaluated against G1 ∩ G2, and Cirrus:

Team	Mean	SD
Oguz et al.	0.66	0.19
de Sisternes et al.	0.63	0.21
Venhuizen et al.	0.56	0.29
Haritz et al.	0.14	0.10

Evaluated against G1 ∩ G2, and Nidek:

Team	Mean	SD
de Sisternes et al.	0.69	0.11
Oguz et al.	0.55	0.12
Venhuizen et al.	0.47	0.06
Haritz et al.	0.43	0.12

Evaluated against G1 ∩ G2, and Spectralis:

Team	Mean	SD
de Sisternes et al.	0.68	0.07
Venhuizen et al.	0.58	0.19
Oguz et al.	0.56	0.13
Esmaeili et al.	0.55	0.28
Haritz et al.	0.25	0.21

Evaluated against G1 ∩ G2, and Topcon:

Team	Mean	SD
de Sisternes et al.	0.75	0.18
Venhuizen et al.	0.71	0.08
Oguz et al.	0.62	0.14
Haritz et al.	0.21	0.07

Proceedings

Mahdad Esmaeili, Alireza Mehri Dehnavi, Hossein Rabbani, Fedra Hajizadeh: “3D Segmentation of Retinal Cysts from SD-OCT Images by the Use of three dimensional curvelet based K-SVD”

Karthik Gopinath and Jayanthi Sivaswamy: “Domain knowledge assisted cyst segmentation in OCT retinal images”

Ipek Oguz, Li Zhang, Michael D. Abramoff, and Milan Sonka: “Graph-Based Retinal Fluid Segmentation from OCT Images”

Luis de Sisternes, Jerry Hong, Theodore Leng, Daniel L. Rubin: “A Machine Learning Approach for Device-Independent Automated Segmentation of Retinal Cysts in Spectral Domain Optical Coherence Tomography Images”

Freerk G. Venhuizen, Mark J.J.P. van Grinsven, Carel B. Hoyng, Thomas Theelen, Bram van Ginneken, and Clara I. Sanchez: “Vendor Independent Cyst Segmentation in Retinal SD-OCT Volumes using a Combination of Multiple Scale Convolutional Neural Networks”