Summary:
This paper provides a framework for the comparison and evaluation of non-linear registration algorithms. Algorithms are tested in synthetic experiments where the input data is warped using either analytically derived transformations or ones generated from a pool of existing registration algorithms. Both displacement field measures and image similarity metrics are used to obtain quantitative results in an open source, open data evaluation framework.

Hypothesis:
This work is motivated by the lack of existing frameworks for assessing algorithm performance on intra-subject non-linear registration problems. The paper is limited to registration problems of this type and the proposed method is only applied to the analysis of a single CT dataset.

Evidence:
The authors use several displacement field metrics and image similarity measures to evaluate the registration results. The authors claim that the diffeomorphic demons algorithm gives the best overall results, which seems well-supported by the displacement field measures (Tables 2 and 3) but is less clear from the intensity difference measures (Tables 4 and 5).

Open Science:
The authors work adheres well to the concept of Open Science since the experiments are run on publicly available data and the source code is provided. The output images are omitted since they are not generally informative (ie. the differences between registered and original images is not visible), which seems reasonable. Given the code, it appears there is enough information to replicate their experiments.

 

Reproducibility:
I didn't have a chance to try to repeat the authors' experiments.

Use of Open Source Software:
The authors provide their code and describe the registration algorithms that can be found either in ITK or on the Insight website. They mention a potential implementation problem in the itkFastSymmetricDemonsFunction and also evaluate the computational speed of the registration algorithms.

Applicability to other problems:
It would be interesting to see this evaluation framework applied to the intra-subject registration of MR or other imaging modalities.

 

Suggestions for future work:
Extending this framework to a wider variety of registration problems, data sets and modalities would be informative.

Requests for additional information from authors:
A more detailed discussion of the results would be a nice addition to the paper, especially regarding the algorithms that performed well on the evaluation measures. It would be informative to know how the authors interpreted the large number of performance results in tables 2-5. Was performance on displacement field measures considered a better test of an algorithm's performance? Also, additional ways of presenting the results, other than table format, could be useful. In section 3.3, the authors remark on the limitations of collapsing data from 256x256x256 voxels into a single number. Addressing this limitation in the discussion section with respect to the results obtained in the paper would be helpful for the reader in evaluating the results as well as the authors' conclusions regarding which algorithms performed best.

Summary:
The authors address the problem of evaluating the performance of nonrigid intra-subject registration algorithms. To compare different registration algorithms they provide two sets of quantitative evaluation measures. The first set of measures are based on the accuracy of recovering synthetically applied transforms. The second set of measures are based on the similarity of synthetically deformed and registered intensity images. The authors report results for various registration algorithms and for differenct synthetic deformations.

Hypothesis:
The evaluation is restricted  to intra-subject registration algorithms which is clearly stated by the authors. Reported results are based on only CT-CT registration which is not mentioned in the abstract or in the introduction.

Evidence:
The set of proposed evaluation measures are convincing in comparing different registration algorithms.

Open Science:
Reported results are based on publicly available datasets. Pointers to the datasets are given in the paper. In addition, source code and scripts are provided.

Reproducibility:
I did not reproduce the results, however, I downloaded the code and tried to compile it. The code compiles succesfully except for CTCurveRegistration.o, for which I got an error message. The readme file in the source folder points out that I get that error message because I dont have FFTWD libraries.

Use of Open Source Software:
The source code makes use of ITK. In addition,  authors make use  of registration algorithms both available in ITK  and in Insight Journal.

Open Source Contributions:
Authors provide the source code of their framework.  The pyhton scripts make use of some configuration files, which should be modified before using the scripts. Those modifications only include providing the path of the input files. The submission could be made better if those modifications were handled automatically by Cmake. In addition, it would be nice if input images were provided with the submission.

Code Quality:
The code is well documented and easy to read for a person familiar with ITK. Some part of the code includes commented lines. Readme files give enough directions to compile and use the code.

Applicability to other problems:
The submission provides a bunch of generic tools which can be used for validating registration algorithms. These tools are very useful for testing and developing a new registration algorithm.

Suggestions for future work:
To compare the performance of different registration algorithms a synthetic image is generated by applying a synthetic deformation. That synthetic image is used as a fixed image in the registration step. It would be another evaluation criteria if the synthetic image was used as a moving image and if it was measured how closely the composition of the recovered and applied deformation fields approximates the identity transform. 

An extension of the framework to different modalities and to multi-modality image registration methods can be very valuable. In addition, utilizing segmentation labels and using DICE measures might be an option for an independent measure of evaluation.

Another extension of the framework might be to focus on the evaluation of different deformation fields. It would be nice to see how different deformation models perform on recovering synthetic transforms is used with the same similarity metric. 

Requests for additional information from authors:
In discussion section, authors mention that levelset and fastblock methods fail in registration. I would like to hear more details about why those methods fail. Maybe a plot about the parameters used can included in the discussion section.

If the authors suspect that there might be implementation problems in fastblock or symmetric demons algorithms, I think the best way is to exclude those from the reported results, cause it makes the evaluation much harder for the readers.

I would also like to see the difference images at least for a synthetic set. Even if the differences are not visually visible, that would make the results more convincing.

Additional Comments:
Overall, this is a nice framework  for the comparison of different registration algorithms. I really liked the evaluation criteria measuring the differences in the deformation fields. Those might serve as an independent measure as opposed to differences in intensity images which might be used by registration metrics as well. The extension of the framework might be a nice contribution to the research community.

Summary:
This paper addresses the evaluation of non-rigid mono-modal intra-subject registration algorithms and proposes an open-source framework. This problem is of high interest for the medical image analysis community and an open-source framework for it was definitely something missing. The current submission (revision 1) is thus a highly valuable contribution but still needs some work to really become a standard tool. A good point though is that the limitations of the current approach are often well presented by the authors.

The main limitations for me are:

• The use of purely synthetic experiments only
• The absence of imaging noise in the current pipeline
• The fact that, in the paper, only one set of parameters was tested for each registration algorithm

It is really nice to see that the authors took advantage of the open-source workshop and included, in their benchmark, two algorithms - including one from my coauthors and I :) - that have been submitted to the very same workshop! I will try to be objective.

Hypothesis:

• In this work the authors only consider mono-modal registration. This is however not clearly stated in the title or in the paper.
• It should also be stated in the introduction that in its current state, the submission is focused on CT data under the influence of breathing.

Evidence:
Apart from the limitations given above which limit the impact of the paper's claim, the authors use a set of different measurements to compare the different algorithms which is nice.

Open Science:
The paper adheres to the concept of Open Science. The source code is provided and the data can be downloaded from publicly available databases.


Reproducibility:
I didn't reproduce the authors experiments yet but only downloaded the code and compiled it. The code compiled out of the box on my linux (with several gcc warnings though).

Use of Open Source Software:
The code use ITK for the registration algorithm and python for the scripting part. It is really nice to see that the authors take advantage of the open-source nature of the workshop: They include two algorithms submitted to this very same workshop in their benchmark!

Open Source Contributions:
At first sight, their seem to be enough information to reproduce the results.

Code Quality:
The code seems relatively well written and documented but may be improved. The different registration algorithms could for example share a common iteration observer. This would make the comparison in Table 1 of the paper more fair. A lot of time is for example spent in the observer of the diffeomorphic demons (for observation purposes only) but the observer used for the demons seems trivial.

Applicability to other problems:
A similar framework for multi-modal registration problem would also be a great contribution.

Suggestions for future work:

• Another approach for the evaluation which is not cited in the paper would consist in registering pairs of real images that have previously been segmented (through a gold standard). The registration results would then show the agreement between the post-registration segmentations. This would be a great addition to the framework which would then report results that are closer to what typical users expect from a registration algorithm. Freely available segmented data can for example be dowloaded from BrainWeb and IBSR.
• I really think the authors should include an additional step in their pipeline to add some random imaging noise.
• In the paper, it would be great to see the algorithms tested with a range of parameters.
  

Requests for additional information from authors:
In the paper, tables 2 to 5: Why did the authors include the "initial" but not the "affine" results in tables 2 and 3 whereas in table 4 and 5, they include the "affine" but not the "initial" results? It would be nice to see both in all the tables.

Additional Comments:

• Page 5, line 4 from the bottom: Did the author mean "Although intra-subject" instead of "Although inter-subject"?
• Page 7, 2nd paragraph:This paragraph could be expanded a little to make things clearer. A note concerning the bias could for example also be added here.
• Page 7, section 3.3, 2nd paragraph: It is not clear to me if the authors also compute the criteria on the image background. If this is the case, it would be nice to add a note saying how this influences the results. For example if a random synthetic deformation field is used, their will be some deformation in the background. This cannot be recovered since there is no information in this background. In this case, the only thing that affects the deformation found by the algorithms is the regularization term. This could bias the results towards one method...
• Page 8: How is the clamping value (100) chosen? This value can indeed not be generic and needs to be tuned for a specific benchmark. How should this value be changed from a user point of view?
• Page 8: It is not clear if a clamping is used in MAD_int. If so why? I thought the MAD already took outliers into account.
• Page 10 and 11, figure 5 and reference to it: I think the caption is a bit mixed up. From the text I understand that 5a was level-set, 5b block-match and 5c symmetric demons.
• Tables 2 to 5: It would be nice for the reader to put the best results or the 2 best ones in bold characters.
• As a side note: To make a somewhat fairer comparison between the different variants of the demons (demons, symdemons and diffdemons), the authors could potentially use the code that my coauthors and I submitted to the insight-journal. It is slightly fairer since we tried to use a very similar implementation for all the variants and tried to handle the border cases in a consistent manner.