Updating of the SSIM metric class to be compliable with the MS SSIM Metric

[PedroFerreiradaCosta]

MONAI/monai/metrics/regression.py

Line 240 in 8925e3e

class SSIMMetric(RegressionMetric):

I propose four changes to this class to optimise it with regards to MONAI Core code:

• Have the option to return the contrast sensitivity metric, presented in the original paper: Wang, Z., Bovik, A.C., Sheikh, H.R. and Simoncelli, E.P., 2004.
  Image quality assessment: from error visibility to structural similarity.
  IEEE transactions on image processing, 13(4), pp.600-612. -> Sec 3.B - Eq 9
  This value is needed to calculate the MS SSIM.
• Return not only a value, but the metric across batches and channels:
  Not calling .mean() below
  

  MONAI/monai/losses/ssim_loss.py

  Line 124 in 8925e3e

  loss: torch.Tensor = 1 - ssim_value.mean()

• Have the metric not depend on the Loss, optimally it could be the other way around as the Loss is itself implementing the metric. This way the loss can call the metric and return the .mean() of the metric.
  

  MONAI/monai/metrics/regression.py

  Line 295 in 8925e3e

  ssim_value: torch.Tensor = 1 - SSIMLoss(self.win_size, self.k1, self.k2, self.spatial_dims)(

• This is secondary, but it would be great to aim to have both an object and a functional implementation of the metric as it happens with other metrics in MONAI Core.

Let me know if this sounds alright and I'm happy to help implementing these changes.

[Warvito]

@danieltudosiu this might be interesting to you too

[danieltudosiu]

I second @PedroFerreiradaCosta suggestions. I would just make the following change:

• Instead of return the metric across batches and channels by defaults, the user should be able to control the behaviour similarly with the DiceLoss implementation's reduction argument

Furthermore, I see a difference between the SSIMLoss and some of the other SSIM implementations.

• The lack of a Multi-Scale variant which by my understanding should be more stable. The MS-SSIM setup should allow access to the scale weights such that MS-SSIM could be tuned on the fly and transformed into an SSIM later in the training stages.
• It deviates from the original implementation by not using a MATLAB _fspecial equivalent similar to Tensorflow or pytorch-msssim

Lastly, this family of losses has been quite widely used in medical images [1], but it was shown that the basic SSIM is not as well correlated with radiologists' visual assessment as some of its improved versions are [2]. So I would argue that we should try and improve the base implementation and offer more principled defaults for the Loss as well as the Metric. The defaults will differ since some SSIM improvements are non-differentiable as far as I am aware.

[1] Mudeng, V., Kim, M. and Choe, S.W., 2022. Prospects of Structural Similarity Index for Medical Image Analysis. Applied Sciences, 12(8), p.3754.

[2] Renieblas, G.P., Nogués, A.T., González, A.M., León, N.G. and Del Castillo, E.G., 2017. Structural similarity index family for image quality assessment in radiological images. Journal of medical imaging, 4(3), p.035501.

[wyli]

cc @mersad95zd @Can-Zhao @finalelement

[danieltudosiu]

For context, by extending the pytorch-ssim library we internally have implemented the following (combinations and multi-scale are excluded for clarity):

• SSIM [1][2]
• Gradient-SSIM [3]
• 3-SSIM [4]
• 4-SSIM [5]
• *-SSIM [6]

Besides our implementation, we are missing the more exotic and complex ones:

• Gaussian Blur SSIM (GB-SSIM) [7]
• Improved SSIM - Sharpness (I-SSIM-S) [8]
• Contrast Sensitivity Function SSIM (CSF-SSIM) [9]
• Complex Wavelet SSIM [10]
• Improved SSIM [11]

Furthermore, since MONAI should have de facto defaults, the SSIM class' defaults should be based on medical imaging specific recommendations from publications such as [12][13] and medical imaging tested ones like [1-6,10].

[1] Wang, Z., Bovik, A.C., Sheikh, H.R. and Simoncelli, E.P., 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4), pp.600-612.

[2] Wang, Z., Simoncelli, E.P. and Bovik, A.C., 2003, November. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 (Vol. 2, pp. 1398-1402). Ieee.

[3] Chen, G.H., Yang, C.L. and Xie, S.L., 2006, October. Gradient-based structural similarity for image quality assessment. In 2006 International Conference on Image Processing (pp. 2929-2932). IEEE.

[4] Li, C. and Bovik, A.C., 2009, January. Three-component weighted structural similarity index. In Image quality and system performance VI (Vol. 7242, p. 72420Q). International Society for Optics and Photonics.

[5] Li, C. and Bovik, A.C., 2010. Content-partitioned structural similarity index for image quality assessment. Signal Processing: Image Communication, 25(7), pp.517-526.

[6] Rouse, D.M. and Hemami, S.S., 2008, February. Analyzing the role of visual structure in the recognition of natural image content with multi-scale SSIM. In Human Vision and Electronic Imaging XIII (Vol. 6806, p. 680615). International Society for Optics and Photonics.

[7] Chebbi, E., Benzarti, F. and Amiri, H., 2014. An improvement of structural similarity index for image quality assessment. Journal of Computer Science, 10(2), p.353.

[8] Lee, D. and Lim, S., 2016. Improved structural similarity metric for the visible quality measurement of images. Journal of Electronic Imaging, 25(6), p.063015.

[9] Yao, J. and Liu, G., 2018. Improved SSIM IQA of contrast distortion based on the contrast sensitivity characteristics of HVS. IET Image Processing, 12(6), pp.872-879.

[10] Sampat, M.P., Wang, Z., Gupta, S., Bovik, A.C. and Markey, M.K., 2009. Complex wavelet structural similarity: A new image similarity index. IEEE transactions on image processing, 18(11), pp.2385-2401.

[11] Aljanabi, M.A., Hussain, Z.M., Shnain, N.A.A. and Lu, S.F., 2019. Design of a hybrid measure for image similarity: a statistical, algebraic, and information-theoretic approach. European Journal of Remote Sensing, 52(sup4), pp.2-15.

[12] Renieblas, G.P., Nogués, A.T., González, A.M., León, N.G. and Del Castillo, E.G., 2017. Structural similarity index family for image quality assessment in radiological images. Journal of medical imaging, 4(3), p.035501.

[13] Kumar, B., Kumar, S.B. and Kumar, C., 2013, December. Development of improved SSIM quality index for compressed medical images. In 2013 IEEE Second International Conference on Image Information Processing (ICIIP-2013) (pp. 251-255). IEEE.

[mersad95zd]

Thanks @PedroFerreiradaCosta @danieltudosiu, these are all very nice suggestions!
@wyli I'm a bit busy these days (sorry for my late reply), but will soon collaborate with @Can-Zhao to add these features.

[danieltudosiu]

@mersad95zd me and @PedroFerreiradaCosta can work with you on implementing them. Internally we have code that is well documented and is ready to be merged into MONAI core nearly. Would you be ok if we join in on working on this?

[wyli]

thanks @danieltudosiu and @PedroFerreiradaCosta, please go ahead and submit PRs, I think @mersad95zd and @Can-Zhao can help review them...

[danieltudosiu]

Another note about the wide applications of SSIM and the requirements for diverse implementations that could be present in MONAI [1]. Furthermore, we should run a window size analysis since the images in medical imaging have different sizes and there were findings indicating that SSIM and its derivatives have different behaviours at different sizes of images [2]. In my opinion, this could easily be explained by the kernel size to image ratio.

[1] Mudeng, V., Kim, M. and Choe, S.W., 2022. Prospects of Structural Similarity Index for Medical Image Analysis. Applied Sciences, 12(8), p.3754.

[2] Renieblas, G.P., Nogués, A.T., González, A.M., León, N.G. and Del Castillo, E.G., 2017. Structural similarity index family for image quality assessment in radiological images. Journal of medical imaging, 4(3), p.035501.