Thyroid is a gland in the neck consisting of two lobes. Thyroid nodules are estimated to affect as much as 50% of the population. Triaging them for biopsy is done based on assessment of ultrasound (US) imaging by radiologists. However, interpretation of thyroid ultrasound suffers from high inter-reader variability and overdiagnosis.

Data

Our dataset included US images for 1377 thyroid nodules from 1230 patients (Figure 1). All nodules were assigned ground truth labels (benign or cancer) proven by either biopsy or surgery. For testing, we selected 99 cases that were additionally annotated by multiple radiologists. The remaining 1278 cases were used for algorithm development.

Figure 1: Deidentified ultrasound images of a thyroid nodule.

Deep learning algorithm

We developed a deep learning algorithm to provide management recommendations for thyroid nodules observed on ultrasound images . Code repository for methods applied in this project is available at the following link: github.com/mateuszbuda/thyroid-us.

The main steps of the algorithm are:

1. Extraction of region of interest (ROI) based on calliper marks using Faster R-CNN network.
2. Prediction of malignancy using multi-task CNN.
3. Stratification into risk level and biopsy recommendation.

ROI extraction

Extraction of ROI with a nodule from thyroid US was performed by detection of calliper marks with Faster R-CNN network. ROI was defined as a rectangle enclosing all detected calipers, as shown in Figure 2.

Figure 2. Thyroid US with detected calliper marks (red boxes) and ROI enclosing them (blue box).

Multi-task CNN

For prediction of malignancy, we developed a multi-task CNN (Figure 3). The main task for the network was to predict malignancy of thyroid nodules based on US images. Auxiliary tasks, trained jointly with the main task, were prediction of nodule’s visual features that were highly relevant to the malignancy status. Shared weight were updated based on training signal from all tasks. As a result, the network was able to extract generalizable features, even when trained on a small dataset.

Figure 3. Multi-task CNN network for malignancy prediction of thyroid nodules based on US images.

Results

For the test nodules, the proposed deep learning algorithm achieved AUC of 0.87, which was similar to a committee of three expert radiologists (AUC=0.91).

Figure 4. ROC curves comparing deep learning and radiologists on a test set [1].

The proposed deep learning algorithm achieved 87% sensitivity and 52% specificity. Sensitivity and specificity of the deep learning algorithm for thyroid nodule biopsy recommendations was similar to that of expert radiologists.

References

[1] Mateusz Buda, Benjamin Wildman-Tobriner, Jenny K Hoang, David Thayer, Franklin N Tessler, William D Middleton, Maciej A Mazurowski “Management of thyroid nodules seen on US images: deep learning may match performance of radiologists.” Radiology, 2019.

Links

• Radiology paper: https://doi.org/10.1148/radiol.2019181343.
• GitHub repository: github.com/mateuszbuda/thyroid-us.
• Contributed Talk at the ML in PL Conference 2019: https://youtu.be/VMZnv5vWhUM.

Radiogenomics is a challenging task of distinguishing molecular subtypes of a lesion based on radiological imaging data. The genomic type of a tumor is a valuable information and can help in guiding patient’s treatment.

The dataset used for development was obtained from The Cancer Imaging Archive (TCIA) and involved 110 cases of lower-grade glioma patients. Registers brain MR images with manual FLAIR abnormality segmentation masks are published as a Kaggle Dataset lgg-mri-segmentation.

Figure 1. Randomly sampled patches of tumors extracted from FLAIR sequences of brain MRI masked with manual segmentations. Each row corresponds to a separate genomic subtype cluster.

A method that we applied was transfer learning from a different brain MRI dataset containing scans from cases with tumors of a similar type. Obtained results show a notable association between imaging and genomic data. This provides strong evidence for genomic subtypes being exposed in MRI.

Figure 2. Receiver operating characteristic curve for the task of discriminating betwen tumor genomic subtypes of significantly different survivals times.

In Figure 3, we show network attention heatmaps, which indicate parts of the image responsible for prediction. Increased response by the network was for tumor margin regions of high irregularity.

While deep learning cannot yet replace genomic testing, it shows promise in aiding clinical decisions.

Links

• Full paper: Radiology: Artificial Intelligence
• Data: lgg-mri-segmentation

Thyroid nodules are estimated to affect as much as 50% of the population. Triaging them for biopsy is done based on assessment of ultrasound imaging by radiologists. A system that quantifies the features and provides a recommendation decision proposed by the American College of Radiology (ACR) is called Thyroid Imaging Reporting and Data System (TI-RADS) [1]. ACR TI-RADS was extensively tested in multiple studies which showed its superior performance as compared to other similar risk stratification systems. Still, it was built based on evidence in the literature and expert consensus. We, in turn, developed an improved version using a data-driven approach based on genetic optimization methods and data for 1425 thyroid nodules. An interactive website for our system is available at deckard.duhs.duke.edu/~ai-ti-rads/.

Figure 1: ACR TI-RADS framework introduced in [1]. (FNA = fine needle aspiration biopsy)

Data

Our dataset included 1425 thyroid nodules from 1264 patients. All nodules were assigned ground truth labels (benign or cancer) proven by either biopsy or surgery. Each nodule was assessed for the five ACR TI-RADS feature categories (shown in Figure 1) by a single radiologist, expert in thyroid imaging. For testing, we selected 100 cases that were additionally annotated by multiple radiologists. The remaining 1325 cases were used for algorithm development.

Genetic algorithm

Code repository for a genetic optimization algorithm applied in this project is available at the following link: github.com/mateuszbuda/AI-TI-RADS. It was implemented using Python package DEAP for distributed evolutionary algorithms. The main steps of the algorithm are:

1. initialize population with 100 individuals
2. for 50 generations:
    a. select individuals to create offspring population
    b. crossover randomly selected offspring individuals
    c. mutate randomly selected offspring individuals
3. return individual with the best loss function

An individual was represented with a vector of 19 integer variables with values in a range from 0 to 3, each corresponding to one feature in ACR TI-RADS. Points for two features (anechoic echogenicity and extra-thyroidal extensions margin) were adapted from ACR TI-RADS since we did not have any case with these features in our dataset. Two other features (cystic and spongiform composition) were excluded since in ACR TI-RADS these nodules receive 0 points in total without adding further points for other categories.

In the initial population, all individuals were representing the assignment of points as in ACR TI-RADS.

For selection (step 2a) we used a tournament approach of size 4. Among 4 randomly drawn individuals, we selected one with the best fitness function. This procedure was repeated 100 times in each generation.

The crossover (step 2b) was an exchange of the values between two randomly selected variables. The probability of an individual being subject to crossover with another individual was set to 50%.

The mutation (step 2c) assigned a new random value to a variable. The probability of mutation was set to 20% for an individual and 5% for a variable.

Finally, the optimized loss function used for evaluation of individuals was area under the ROC curve (AUC). The progress of evolution is shown in Figure 2.

Figure 2: Mean and standard deviation for AUC of a population across evolution.

Results

The optimization resulted in AI TI-RADS with new point assignments for the features as shown in Figure 3. Interactive calculator for AI TI-RADS is available online: deckard.duhs.duke.edu/~ai-ti-rads/.

Figure 3: AI TI-RADS diagram from the Radiology article [2]. (FNA = fine needle aspiration biopsy)

On the training set, the AUC for ACR TI-RADS (used for initialization) was 0.79, whereas for AI TI-RADS it was 0.82.

The 100 test cases were assessed for the five ACR TI-RADS feature categories by (1) eight radiologists who interpreted thyroid ultrasound in clinical practice and (2) three ACR TI-RADS committee members whose interpretations were combined into expert consensus using majority vote. Based on these interpretations, we compared sensitivity and specificity for biopsy recommendation of ACR TI-RADS and AI TI-RADS on a boxplot shown in Figure 4.

Figure 4: Comparison between ACR TI-RADS and AI TI-RADS in terms of sensitivity and specificity for biopsy recommendation on the 100 test cases. Boxes represent a set of eight radiologists and blue squares represent an expert consensus.

AI TI-RADS resulted in improved specificity for expert consensus from 50.6% to 54.1% (p=0.096) as well as for mean specificity of the eight radiologists from 47.7% to 55.3% (p<0.001). In terms of sensitivity, for expert consensus, it was the same for both TI-RADS (86.7%), whereas mean sensitivity of the eight radiologists slightly improved from 81.7% to 82.5% (p>0.5).

References

[1] Tessler FN, Middleton WD, Grant EG, Hoang JK, Berland LL, Teefey SA, et al. “ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White Paper of the ACR TI-RADS Committee.” Journal of the American College of Radiology, 2017.

[2] Wildman-Tobriner B, Buda M, Hoang JK, Middleton WD, Thayer D, Short RG, Tessler FN, Mazurowski MA “Using Artificial Intelligence to Revise ACR TI-RADS Risk Stratification of Thyroid Nodules: Diagnostic Accuracy and Utility.” Radiology, 2019.

Links

• AI TI-RADS Radiology paper: doi.org/10.1148/radiol.2019182128.
• AI TI-RADS Calculator: deckard.duhs.duke.edu/~ai-ti-rads/.
• AI TI-RADS GitHub repository: github.com/mateuszbuda/AI-TI-RADS.
• Python DEAP package: deap.readthedocs.io.

Introduction

The most common statistical methods for comparing machine learning models and human readers are p-value and confidence interval. Although receiving some criticism recently, p-value and confidence interval give more insight into results than a raw performance measure, if interpreted correctly, and are required by many journals.

This post shows an example python code utilizing bootstrapping for computing confidence intervals and p-values comparing machine learning models and human readers.

I will not discuss what p-value does or does not mean, what is the right threshold for statistical significance, or how to properly interpret it. Here are some resources that dive deeper into this topic:

• Wasserstein, R.L. and Lazar, N.A., 2016. The ASA’s statement on p-values: context, process, and purpose. The American Statistician, 70(2), pp.129-133.
• Baker, M., 2016. Statisticians issue warning over misuse of p-values. Nature News, 531(7593), p.151.
• Altman, N. and Krzywinski, M., 2016. Points of significance: p-values and the search for significance.
• Benjamin, D.J., Berger, J.O., Johannesson, M., Nosek, B.A., Wagenmakers, E.J., Berk, R., Bollen, K.A., Brembs, B., Brown, L., Camerer, C. and Cesarini, D., 2018. Redefine statistical significance. Nature Human Behaviour, 2(1), p.6.
• McShane, B.B., Gal, D., Gelman, A., Robert, C. and Tackett, J.L., 2019. Abandon statistical significance. The American Statistician, 73(sup1), pp.235-245.

Simple statistical toolset for machine learning

I published a GitHub repository ml-stat-util containing a set of simple functions written in Python for computing p-values and confidence intervals using bootstrapping. I will show how to use it in different common use cases.

A jupyter notebook with all use cases described below is available on GitHub.

Use case #1

Compute AUC with 95% confidence interval for a single model.

from sklearn.metrics import roc_auc_score
import stat_util

score, ci_lower, ci_upper, scores = stat_util.score_ci(
    y_true, y_pred, score_fun=roc_auc_score
)

To get an idea of what happened, we can plot a histogram of bootstrapped scores.

import matplotlib.pyplot as plt

bins = plt.hist(scores)
plt.plot([score, score], [0, np.max(bins[0])], color="tomato")
plt.plot([ci_lower, ci_lower], [0, np.max(bins[0])], color="lime")
plt.plot([ci_upper, ci_upper], [0, np.max(bins[0])], color="lime")

Use case #2

Compare two models by computing p-value for a difference in their performance measured with AUC.

from sklearn.metrics import roc_auc_score
import matplotlib.pyplot as plt
import stat_util

p, z = stat_util.pvalue(y_true, y_pred1, y_pred2, score_fun=roc_auc_score)
bins = plt.hist(z)
plt.plot([0, 0], [0, np.max(bins[0])], color="black")

Use case #3

Compute mean AUC with 95% confidence interval for a set of readers/models.

import numpy as np
from sklearn.metrics import roc_auc_score
import matplotlib.pyplot as plt
import stat_util

mean_score, ci_lower, ci_upper, scores = stat_util.score_stat_ci(
    y_true, y_pred_readers, score_fun=roc_auc_score, stat_fun=np.mean
)
bins = plt.hist(scores)
plt.plot([mean_score, mean_score], [0, np.max(bins[0])], color="tomato")
plt.plot([ci_lower, ci_lower], [0, np.max(bins[0])], color="lime")
plt.plot([ci_upper, ci_upper], [0, np.max(bins[0])], color="lime")

Use case #4

Compare a single model to a set of readers by computing p-value for a difference in their performance measured with AUC.

import numpy as np
from sklearn.metrics import roc_auc_score
import matplotlib.pyplot as plt
import stat_util

p, z = stat_util.pvalue_stat(
    y_true, y_pred, y_pred_readers, score_fun=roc_auc_score, stat_fun=np.mean
)
bins = plt.hist(z)
plt.plot([0, 0], [0, np.max(bins[0])], color="black")

Links

• Code: mateuszbuda/ml-stat-util
• Notebook: mateuszbuda/ml-stat-util/examples.ipynb

Updated on Nov 13, 2020

Overleaf (previously ShareLaTeX) is a convenient tool to collaboratively write papers in LaTeX. Once we have the content of a manuscript ready, it’s time for submission. Here is a step by step guide to preparing a single zip file for uploading on arXiv.

Assumptions:

1. The main file is named ms.tex.
2. Used packages:
   • graphicx
   • hyperref
   • natbib
   • subfiles
3. My setup on Overleaf (ShareLaTeX):
   • Main document class: \documentclass[11pt]{article}
   • Bibliography style: \bibliographystyle{unsrtnat}
   • Path to graphics: \graphicspath{ {images/} }
   • References from references.bib file added using natbib: \bibliography{references}
   • The first line of each subfile is: \documentclass[ms.tex]{subfiles}

    \documentclass[ms.tex]{subfiles}
       
    \begin{document}
    ...
    \end{document}
   

Steps:

1. Download source files and unzip them.
   • Menu
   • Download Source
2. Download the bbl file and rename it to ms.bbl:
   • Logs and output files
   • Other logs & files
   • bbl file

   

   

3. In the main file (ms.tex), below the documentclass declaration line add \pdfoutput=1.

    \documentclass[11pt]{article}
    \pdfoutput=1
    ...
   

4. Remove \documentclass[ms.tex]{subfiles} statement in all subfiles.
5. Compress (zip) all files, but not their parent folder, and upload the zip to arXiv.

Note: It’s possible that this guide does not handle some (edge) cases.

Introduction

Thresholding is a simple method that can improve the accuracy of a classifier in the case when it was trained on an imbalanced dataset. It relies on Bayes’ Theorem and the fact that neural networks estimate posterior distribution (Richard & Lippmann, 1991). In practice, it means that given a datapoint $x$, the output for a neuron representing class $i$ corresponds to

where $p(i)$ is a prior probability for class $i$.

In a standard case, equal priors are assumed for all classes. However, it is not always the case. E.g. in medical datasets some diseases are known to have a prevalence of less than 1%.

A class prior can be estimated based on the number of examples in a training set unless we have a good reason to think that our training set is not reflective of the true class distribution. Otherwise, for class $i$ we have:

where $|i|$ denotes the number of unique examples in class $i$.

Finally, the adjusted network output for class $i$ can be obtained by dividing the original output by the corresponding prior.

Simple example

Let us assume that we have already trained a classifier on a training set with the number of examples per class as following:

Then, we tested our model on a test case $x$ and it returned the vector of class probabilities:

In this case, the predicted class would be:

Now, to apply thresholding method, we have to compute the vector of priors. In this case, it will be:

And adjusted predictions are obtained by dividing original predictions element-wise by the vector of priors:

The predicted class has now changed to:

Python implementation

For impatient readers, the full notebook is available as a GitHub gist.

First, let us build a simple dataset based on two Gaussian distributions.

# 20 positive and 180 negative training examples
X1_train = np.random.normal((1.0, 0.0), 1.0, (20, 2))
X0_train = np.random.normal((-1.0, 0.0), 1.0, (180, 2))

X_train = np.vstack((X0_train, X1_train))
y_train = np.array(180 * [0] + 20 * [1])

X1_test = np.random.normal((1.0, 0.0), 1.0, (500, 2))
X0_test = np.random.normal((-1.0, 0.0), 1.0, (500, 2))

X_test = np.vstack((X0_test, X1_test))
y_test = np.array(500 * [0] + 500 * [1])

The training set looks like this:

We will use it to train a simple neural network with 2 hidden units:

nn = MLPClassifier(solver='lbfgs', hidden_layer_sizes=(2), activation='logistic', random_state=42)
nn.fit(X_train, y_train)

The decision boundary looks like reasonably discriminating positive (blue) and negative (red) examples:

The accuracy evaluated on the test set is:

"Accuracy = {}%".format(100 * nn.score(X_test, y_test))

Accuracy = 75.1%

We will attempt to improve it using thresholding method. To do that, we need to implement functions to estimate class priors and generate adjusted predictions.

def priors(y):
    return np.unique(y, return_counts=True)[1] / float(len(y))

def predict_thresholded(nn, X, p):
    y_pred = nn.predict_proba(X)
    y_pred_th = y_pred / p
    return np.argmax(y_pred_th, axis=1)

Now, we can use them and evaluate the performance again:

p = priors(y_train)
y_pred_test = predict_thresholded(nn, X_test, p)
"Accuracy = {}%".format(100 * accuracy_score(y_test, y_pred_test))

Accuracy = 84.2%

We were able to improve the accuracy by over 9%. However, the thresholding method only scales the outputs by multiplying them by a constant number. It must be noted that the discriminative power of a classifier (e.g. as measured using ROC) does not change in this case.

Links

• Code: GitHub gist
• Class imbalance: arXiv:1710.05381
• Extended report: diva2:1165840

Colaboratory is a tool from Google that lets you run a Python Notebook in the cloud with GPU support. There are some limitations on available memory and time constraints for running a continuous session yet it should be enough to train a decent scale machine learning models. Here is an overview of the setup process together with a sample notebook that shows how to use Colaboratory to start with deep learning on Labeled Faces in the Wild dataset using Keras.

Set-up

The first prerequisite is that you have a Google account and a browser other than Internet Explorer. You probably do, so open your Google Drive. Here, we need to connect the Colaboratory app as shown below.

Then, search for the phrase colaboratory and connect it with your account.

At this point, you should be able to create new Colaboratory Notebook.

To enable GPU support, you need to edit Notebook Settings.

Choose the GPU option from the Hardware accelerator dropdown list and save the setting.

Now, you are ready to do some deep learning.

Deep learning starter notebook

You can start by following a deep learning starter notebook that can be accessed here: DeepLearningStarter.ipynb. It assumes some basic Python programming skills. This notebook is a step by step guide to solve an image classification task. It uses the Labeled Faces in the Wild, “database of face photographs designed for studying the problem of unconstrained face recognition”. It contains images of famous people, e.g. Bill Clinton:

The model employed in this task is an all convolutional neural network implemented in Keras. You should be able to achieve results similar (or even better, after tuning some training hyper-parameters) to the ones shown below.

Figure 1. An overview of deep learning methods. Comparison between training from scratch, transfer learning and deep features approaches.

In the transfer learning, the network is first trained using a different dataset, e.g. ImageNet. Then, the network is “fine-tuned” through additional training with data specific to the problem to be addressed. The idea behind this approach is that solving different visual tasks shares a certain level of processing such as recognition of edges or simple shapes. Another way to address the issue of limited training data is the deep features approach which uses convolutional neural networks which have been trained on a different dataset to extract features from the images. This is done by extracting activations of layers prior to the network’s final layer. Those layers typically have hundreds or thousands of outputs. Then, these outputs are used as inputs to “traditional” classifiers such as linear discriminant analysis, support vector machines, decision trees, etc. This is similar to transfer learning (and is sometimes considered a part of transfer learning) with the difference being that the last layers of a CNN are replaced by a traditional classifier and other layers are not additionally trained.

Links

• Journal paper: Journal of Magnetic Resonance Imaging
  
• arXiv paper: arXiv:1802.08717
  

References

[1] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pages 3320-3328, 2014.

[2] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806–813, 2014.

It is not often taught at universities, even on graduate-level programs, how to write a peer review. I googled around and based on different sources (links provided below) I collected a set of questions to fill each paragraph of a review.

Before writing a review, here is a guide How to Read a Paper by S. Keshav from the University of Waterloo that I follow when reading a paper (it is useful for a literature study as well). Long story short, there are three passes of increasing level of detail. After the first pass, which takes up to 15 minutes, you should be able to decide whether you are capable of serving as a reviewer for a given manuscript.

Here are the questions that I answer to fill each paragraph:

1. Brief summary
• A brief summary of the article and its contribution.
• State what you think the contributions are.
2. Structure
• Is the article well-written and easy to understand?
• Could it be made clearer?
• Is the article well-organized?
• Does the article contain all of the components you would expect (Introduction, Background, Related Work, Methods, Results, Conclusions, Discussion)?
• Are the sections well-developed?
• General comments regarding layout and format, title, abstract, figures and tables.
3. Novelty
• What’s new about the work?
• Does the author do a good job of synthesizing the literature?
• Is there some related work that the authors have missed?
• Does the related work invalidate the contribution, or (more likely) simply change its context or emphasis?
4. Criticism of methodology
• Is the methodology clearly explained?
• Are there gaps or unaddressed issues?
• Are there any apparent technical flaws?
5. Criticism of results
• Does the theory connect to the data?
• Are you convinced by the author’s results? Why or why not?
• Does the author answer the questions he/she sets out to answer?
6. Strong points
• Was there anything you thought was really cool about the paper?
7. Conclusion
• Give a brief recommendation for the paper and your reasons for it.

References

• 10 tips from an editor on undertaking academic peer review for journals
  
• How to conduct a review
  
• How to Write a Peer Review for an Academic Journal: Six Steps from Start to Finish by Tanya Golash-Boza
  

Together with the dataset, we shared a GPU enabled Python code that can be run on Kaggle: kaggle.com/mateuszbuda/brain-segmentation-pytorch. The script implements a data loading, preprocessing, as well as training and inference of a U-Net segmentation model. Trained weights and results can be downloaded form Kaggle.

We applied U-Net architecture for the task of FLAIR abnormality segmentation in brain MRI (Figure 1). Source code for network architecture, training, and inference implemented in PyTorch together with model weights are available on GitHub repository: github.com/mateuszbuda/brain-segmentation-pytorch.

Figure 1. The U-Net architecture used for segmentation.

To evaluate the quality of segmentation, we used Dice similarity coefficient (DSC) with 22-fold cross-validation. The achieved performance was 83.60% mean DSC and 87.33% median DSC. In comparison, DSC of two expert human readers for this kind of tumor is 84% with a standard deviation of 2%. This puts our method on a par with radiologists.

Figure 2. Qualitative results of segmentation. Blue outline corresponds to ground truth and red to automatic segmentation output. Images show FLAIR modality after preprocessing and skull stripping.

Figure 3. Distribution of Dice similarity coefficient among cases. Red line corresponds to mean DSC and green to median DSC.

Then, produced segmentations were used to extract 2D and 3D tumor shape features that are predictive of its genomic subtypes. More detailed description of methods and results is available in our paper Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm published in Computers in Biology and Medicine.

Links

• Full paper: Computers in Biology and Medicine
• Code: github.com/mateuszbuda/brain-segmentation-pytorch
• Data: lgg-mri-segmentation