Detection and classification

[emrekeles-arch]

Hello, there are three questions I want to ask you.

1. What is the purpose of the Breastclip folder? Is it a part of the project? If so, at what stage is it used?

2. I saw a .py file in Breastclip that generates reports from the findings, are these reports obtained from the vindr dataset? Are the reports obtained used in detection and classification tasks, if so, through which files?

3. Which method should I follow if I want to perform detection and birads-classification with the image-text data structure?

Thank you for your work

[shantanu-ai]

Hi,
Thanks for taking an interest in our repo.

1. breastclip folder is the directory with all the necessary files to train Mammo-CLIP. Files like the clip model, contrastive loss, data_loader. When we create the project, we initially name it breastclip so it is there as it is. It is part of the project (core part). It is used from the beginning. If you train the Mammo-CLIP from scratch, the train.py file internally calls the breastclip folder for training. If you use our checkpoints for any downstream tasks, both the classification and detection files calls breastclip for setting up the model.
2. The name of the file is ./src/codebase/augment_text.py. No, the reports are not obtained from VinDr dataset. We obtain the reports from our inhouse UPMC dataset. From VinDr, we use the finding-labels to generate templated-texts in the dataloader dynamically during pre-training of Mammo-CLIP. They dont play any role for classification, detection and zero-shot evaluation. They are only used for pre-training. For classification and detection, if use our checkpoints just follow the steps for classification and detection.
3. For detection, go to this place. Then go to either Linear probe vision encoder Mammo-CLIP on target detection task for linear probe or Finetune vision encoder Mammo-CLIP on target detection task for finetuning. Linear probing means, the vision encoder of Mammo-clip is fixed. For finetuning, we finetune the vision encoder as well. Please place the checkpoints in the proper folder. Also, we include all the scripts and their utilities here. Follow the detector files.

We did not perform birads-classification, we classify density, mass and calcification for VinDr and cancer for RSNA. However, you can do BiRADS very easily. Follow classifcation steps here. If you use VinDr dataset, make sure, your csv file contains the BiRADS column. The in the --label give BiRADS and run ./src/codebase/train_classifier.py file either with linear probe or finetune mode. If you need to change dataset class, we use MammoDataset function in this file.

[emrekeles-arch]

Hi, Thanks for taking an interest in our repo.

1. breastclip folder is the directory with all the necessary files to train Mammo-CLIP. Files like the clip model, contrastive loss, data_loader. When we create the project, we initially name it breastclip so it is there as it is. It is part of the project (core part). It is used from the beginning. If you train the Mammo-CLIP from scratch, the train.py file internally calls the breastclip folder for training. If you use our checkpoints for any downstream tasks, both the classification and detection files calls breastclip for setting up the model.
2. The name of the file is ./src/codebase/augment_text.py. No, the reports are not obtained from VinDr dataset. We obtain the reports from our inhouse UPMC dataset. From VinDr, we use the finding-labels to generate templated-texts in the dataloader dynamically during pre-training of Mammo-CLIP. They dont play any role for classification, detection and zero-shot evaluation. They are only used for pre-training. For classification and detection, if use our checkpoints just follow the steps for classification and detection.
3. For detection, go to this place. Then go to either Linear probe vision encoder Mammo-CLIP on target detection task for linear probe or Finetune vision encoder Mammo-CLIP on target detection task for finetuning. Linear probing means, the vision encoder of Mammo-clip is fixed. For finetuning, we finetune the vision encoder as well. Please place the checkpoints in the proper folder. Also, we include all the scripts and their utilities here. Follow the detector files.

We did not perform birads-classification, we classify density, mass and calcification for VinDr and cancer for RSNA. However, you can do BiRADS very easily. Follow classifcation steps [here]((https://github.com/batmanlab/Mammo-CLIP/tree/main?tab=readme-ov-file#evaluation). If you use VinDr dataset, make sure, your csv file contains the BiRADS column. The in the --label give BiRADS and run ./src/codebase/train_classifier.py file either with linear probe or finetune mode. If you need to change dataset class, we use MammoDataset function in this file.

Thank you so much :)

[shantanu-ai]

If you have any questions, do let me know. Happy coding.

[emrekeles-arch]

What is the main dataset you use to train Breastclip?

[shantanu-ai]

As mentioned in the paper, we use two configurations:

1. We use inhouse UPMC dataset which contains images+report. We preprocess it to create the csv. Then finally we run ./src/codebase/augment_text.py to create the final csv to train Mammo-CLIP. The instructions are mentioned here

2. We also pretrain using UPMC (image+text data) and VinDr (image+label). The labels will be converted to templated texts mentioned in the preprocessing steps.

BDW for classification and detection, u dont need to pretrain Mammo-CLIP. Just use our checkpoints to train the classifier or detector using linear probe or finetuning mode.

[emrekeles-arch]

src/codebase/breastclip/prompts/prompts.py

What is the function of the python file located in the above folder ?

[shantanu-ai]

@emrekeles-arch, this file aims to generate report from the finding-labels of VinDr dataset. In Mammo-CLIP u can use any image-label dataset during the pre-training stage. For image-label dataset, using the labels for Mass, Calcification, distortion etc u need to generate the report sentences first to integrate them in the pre-training. Based on the label and laterality, u get the templated text here. prompts.py uses these texts and the labels to generate sentences for you.

[emrekeles-arch]

Did you aim to create a multimodal data structure here?

[shantanu-ai]

@emrekeles-arch i create the texts from labels in this function src/codebase/breastclip/prompts/prompts.py. This function is getting called from .src/codebase/breastclip/data/datasets/imagetext.py (line 200)

elif hasattr(self.df, "CC_FINDING"):
            cc, mlo = view_list
            cc_findings = ast.literal_eval(self.df[f"{cc}_FINDING"][index])
            mlo_findings = ast.literal_eval(self.df[f"{mlo}_FINDING"][index])
            text = generate_report_from_labels(cc_findings, self.prompt_json, deterministic=(self.split != "train"))
            text2 = generate_report_from_labels(mlo_findings, self.prompt_json, deterministic=(self.split != "train"))

If you want to pretrain Mammo-CLIP with an image-label data, this function will be invoked while creating the dataset class.

[shantanu-ai]

Hi @emrekeles-arch, if your doubts have been clarified, please close this issue. If you have more queries, feel free to ask me.

[emrekeles-arch]

Hello again, I want to do a new training using your model's checkpoints. The dataset I have consists only of images, but I want to do a multimodal training. Would it be useful or unnecessary if I created sentences from the information in the csv file using your prompts.py file, matched each sentence with an image and sent it to the training? Since you are an expert, I wanted to get your opinion.

Thank you for your patience and nice replies.

[shantanu-ai]

Hi @emrekeles-arch, if you want to further pretrain with your dataset (after initialized mammo-clip with our checkpoints), you can create sentences and add them during the pretraining. However, i think this training has a potential that mammo-clip may forget its knowledge from pre-training with image+text data (which we have from UPMC) because you only have image data. Your texts are templated not real one. We have not done such experiment where we train with image+text and then further pretrain with image + templated text data.

Just to note how we pretrain using UPMC (image+text) and VinDr (only image)? Here are the steps for every epoch:

1. We first mix both the datasets. Suppose UPMC has 100 samples and VinDr has 50 samples. Now we have 150 samples total.
2. For each minibatch, we randomly sample images. If the images coming from UPMC (image+text), we use the texts as it is. If it is coming from VinDr (image only), we generate texts from the prompts.py file based on the finding labels of vindr. Suppose in a minibatch size of 16, 10 comes from UPMC and 6 comes from VinDr, we generate texts the 6 from VinDr. Once we have texts ready for all 16, we get embeddings and do contrastive as mentioned in the paper.

Hope it clarifies your question.