For quite a long time, researchers and the public were thrilled by the wonders of machine learning. However, over the period of time, the community realized that the machine learning models aren’t a magic wand and they are as best as the data provided to them during the training and development stage. As the world started making decisions based on AI, there were soon conflicts between human and machine intelligence and therefore the need for explainability of black-box models become apparent. Prof. Cynthia Rudin, however, holds the view that for high stake decisions, it is better to use the interpretable model instead of explaining black-box models. To hear her elaborate views on this interesting topic, she was invited as a speaker for the fifth Latent View Colloquium series organized on 28th October 2021. Prof. Rudin is currently a Professor of Computer Science, Electrical & Computer Engineering, Statistical Science, and Biostatistics & Bioinformatics at Duke University and has been bestowed with the prestigious AAAI Squirrel AI award for the year 2022.

Prof. Rudin commenced her talk by giving examples of how the application of black-box based machine learning models have led to bad decisions in the judiciary, medical diagnosis, loan decisions etc.  She next went on to describe the major difference between interpretable and explainable machine learning by explaining that interpretable machine learning obeys a domain-specific set of constraints that makes its computation easier to understand whereas in explainable machine learning we use a black box initially and then explain it afterwards. She then reiterated her belief that we can do away with black-box models in high stakes settings and use the interpretable model instead. She next gave an example of the COMPAS score model which is a propriety model being used in the US justice system but no one knows how it makes the calculations except the company who made it. This black box based model kept a person in jail due to miscalculated COMPAS score which is unfortunate and appalling. Prof. Rudin and her research team took the Florida data that contained demographic and criminal history information of each person and was used for making COMPAS (Correctional Offender management profiling of Alternative sanctions) score. They then compared the accuracy of the COMPAS model with CORELS (Certifiably Optimal RulE ListS) model which was made in their lab and found that both models have similar accuracy.

Prof. Rudin next talked about two types of data: Tabular and Raw. While tabular data included counts and categorical data where features are interpretable whereas in the case of raw data features are individually uninterpretable such as pixels/vowels etc. She explained that these two different types of data require different types of explanations and different techniques as for raw data, only neural networks are currently working whereas for tabular data with minor pre-processing all models give similar performance.

She then went on to give an example of a simple model created by her team for high stake settings to prevent brain damage in critically ill patients when the data was of tabular nature. Ideally, a patient needs to be monitored for seizures using EEG equipment but the major issue here was the high cost of EEG equipment and manpower required to operate it. To solve this issue, Prof. Rudin’s group, with help of doctors and neurologists, developed an interpretable machine learning model named 2HELPS2B. Using this model, if a patient is found to be a high risk he/she is put on EEG for 72 hours and preventive treatment for seizures starts. The model has been validated on an independent multicenter cohort by neurologists and has resulted in a 63.6% reduction in duration of EEG monitoring per patient saving $1135 per patient monitored. The model is just as accurate as black-box model and doctors can decide themselves whether to trust it or not.  Also, the doctors can calibrate the score with information not in the database. Also, this model can be explained to a patient’s relative to give them a reason why a patient was taken off EEG.

Next, Prof. Rudin showed how the interpretable machine learning models can also be applied when data is raw such as in computer vision. She showed how in the case of saliency map explaining a deep neural network doesn’t work as the model shows a lot of area in a photograph as evidence for an explanation that is incorrect. She then explained how her team used two methods: Case-based reasoning and K-nearest parts of prototypes to solve this issue. She then explained that this works as the network adds a prototype layer to any black box and forces the network to do case-based reasoning. Here, the prototypes are learned during training. For this, a layer is added before the last fully connected layer in a standard black box convolutional neural network model which forces the network to do comparisons between the image that comes in and its prototypes. These models specify the small parts as a reason “why something looks like that”.

To check the accuracy of the model,Prof. Rudin’s team used the CUB-200 dataset which contains 200 classes of birds and they found that on this dataset the original black box accuracy ranges between 74.6% (VGG16 model) to 82.3% (Res34 model) but when the team added the layer of interpretability to this model the accuracy is similar or better than any other black-box model. Therefore, even for computer vision, one can still have an interpretable model of the same accuracy as a black box. The team also aims to use this model for mammography which is the hardest task in all of the radiography. Often, radiologists miss one-fifth of breast cancer and half of the women getting an annual mammogram over 10 years will have a false positive and up to ¾ of biopsies come back as benign. An uninterpretable deep learning approach will take mammogram will show probability of malignancy, predict it as benign but not give a reason for it. In the saliency approach, however, it will show the probability of malignancy low, predict it as benign and give the reason as a lesion in the mammogram. Prof. Rudin said that the Interpretable AI algorithm for breast lesions (IAIA-BL) developed by her team takes the mammogram image and breaks it down and it forces the network to look at the parts of the image and make comparisons to prototypical cases and so it tells about the probability of malignancy prediction and reason for it in a much more specific way.

Prof. Rudin also showed how her team used interpretable AI a hard benchmark dataset of the Home equity line of credit dataset. The team participated in an explainable machine learning challenge where the interpretable model developed by the team showed an accuracy of 73.8% and AUC of 0.806 which was similar to the best black-box AUC model.

Prof. Rudin ended her talk by summing up why interpretable models should be used instead of explaining black-box models. The talk was followed by an interesting question-answer session.

The video is available on our YouTube channel: Link.

Keywords

Interpretable Models, Artificial Intelligence