REA : Learn the essentials of machine learning using FastAI (Part 2/2)

Part 2 - Interpreting and improving our model

Machine learning, computer vision, and AI in general is not an easy feat to master, and for some, just getting some experience is difficult enough. Many blog posts I've seen try to teach machine learning from the ground up, but teaching all the underlying concepts can be overwhelming and confusing.

To overcome this, I recently put together a fast-paced and interactive workshop at one of REA Group's Python Dojos, where our Python developers come together to share ideas and learn from one another.During the workshop, I went through training a machine learning modelin just six stepsusing theFastAImachine learning framework.

In Part 1 of this blog series, we produced a baseline computer vision model to detect between impressionism, cubism, and pop artwork art movements. We managed to achieve an accuracy of about 69% in just 127 lines of Python code. The first three steps so far involved the following:

1. We downloaded some training images from Creative Commons;
2. We loaded those images into memory; and
3. We trained the model using transfer learning on the Resnet34 model.

In the second part of this blog, we will cover off the remaining three steps involved in the workshop. In Step 4, we'll going over into how to better interpret what our model has learnt. In Step 5, we'll run some predictions on images our model has never seen before by downloading a few more images online and running it through the model to see what it predicts. In Step 6, we will work towards making improvements on our first-pass, baseline mode, thereby making our model even better than before.

Buckle yourself in, and remember, the tutorial is fullyavailable on GitHuband you can and the Dojo session given at REA isavailable to view on YouTube.

Step 4: Understanding what our model learnt

So far, we've produced a basic image classifier that is about 69% accurate in how well it can distinguish the three forms of art movements we trained on. Now to the interesting part: we can use FastAI to peek open the hood of our machine learning model and interpret what the learner has correctly and incorrectly learnt.

To do this, we can use FastAI'sClassification Interpretationclass, which houses several tools to help us interpret our model's predictions.

Lines 1-2: Creating a Classification Interpreter

Firstly, we need to create an interpreter from a given learner using the from_learnerstatic method.

Lines 2-3: Plotting a confusion matrix

Once we have our interpreter, we can use it to plot a confusion matrix. This confusion matrix shows us the disparity between of correctand incorrectpredictions (of the labels) made by our model for each image in the validation dataset. When executed, a confusion matrix like the one below shows:

Here, we can see that our model:

• is not very good at distinguishing between cubism and the other two art movements, where the model predicted 23 cubism art images as impressionism, and the other 24 as pop art;
• has learnt certain features about impressionism and pop art, and is able to distinguish between them with only a few mistakes (i.e., 60 and 61 images of impressionism and pop art were, respectively, predicted correctly).

Lines 6-8: Viewing images with incorrect predictions

Diving deeper into the incorrect predictions, we can ask our interpreter to show us which images had the highest loss. (The higher the loss, the worse the performance of the model.) Executing the plot_top_lossesmethod on our interpreter shows us:

• images with the highest loss (worst performance), in descending order,
• the predicted label for each image,
• the actual label for each image,

• and the probability of the actual class.

Here's what shows when executed:

It's now clear that we have somebad data! For example we've incorrectly included four images of photographs (image 1, 2, 4, and 8) and some of the other images have been incorrectly classified, or do not fit their respective art movements. The model will become confused, trying to learn certain 'bad' features that it shouldn't be learning (e.g., what a photograph looks like instead of what impressionism art looks like in image 1).

As another example, when I ran the FastAI workshop at the Python Dojo, many people who attended and used the dog breeds example noticed that labels downloaded for Chihuahuawere getting images of theMexican state of the same name, and notthe dog breed!

And here comes an important lesson:a model is only as good as the data you train it.So, bad training data will inevitably lead to a bad model. In step 6, we'll use FastAI to clean up some of this bad data, but for now, let's explore how our model can be used to run predictions on images it has not yet seen before.

Step 5: Predicting on new images

Let's run a prediction using our model on the following three random images:

Lines 1-15: Download random images from the internet with wget

We'll store URLs to the three images above in a list named images, importing the tempfile library to store the downloaded the images in a tempfile. Then, we iterate through each image and can use the wgetcommand (via a shell mixin) to download the image into a tempfile (e.g., somewhere inside the /tmpdirectory).

Lines 16-21: Running a prediction on the learner

Once the image is downloaded into the tempfile at image_path, we can use thepredictmethod on our learner to make a prediction on the image.

The predict method returns a tuple with three key pieces of information:

These pieces of information refer to thevocabulary(i.e., the three labels we used to train the model) within the dataloader produced in Step 2 (see Part 1 of the blog post).

These three pieces of information are:

1. The label the learner ismostconfident in;
2. The index within the vocabulary of that label corresponds to (as aTensorBase-typed integer value);
3. The predictions of all three labels in the vocabulary (a TensorBaselist of floats).

In this case, we can just keep the third element of the tuple and iterate through the data loader's vocabulary (using the vocabproperty on line 20) to see howconfidentthe learner is for each label per image:

1. For the first image of the pug, we can see that the model is most confident that this is pop art; it has learnt the distinguishing colour features that makes pop art different from the other art moves.
2. For the second image of the dog in the suit, there are some mixed confidences; while it thinks the image is also pop art, the confidence of impressionism is also higher than before.
3. Lastly, the image of the sad dog again has a mix of confidences; it's still somewhat confident that this is pop art, but cubism is a close second.

Our model, therefore, appears to be biased against the pop art label.It's making many assumptions that most images are pop art.

While we have a functioning model, it doesn't seem to be that good. In the following step, we'll try looking at cleaning up potentially some bad data that was used to train the model, making our model even better!

Step 6: Improving our model!

A common theme in training machine learning models is to ensure you have good quality data to train the models on. When we interpreted our model in Step 4, we saw that there were some potentially bad training examples, such as images of photographs or other types of art movements we're not interested in.

Bad data fed into our model leads to a model that learns badly. But, it's also a problem we can fix. In this step, we'll learn how to use FastAI to clean up some of this bad data in an easy manner.

FastAI has a handy data-cleansing widget (ImageClassifierCleaner) that can be used to re-classify bad data into their correct labels, or to prune away bad training examples altogether.

Creating and executing a cleaner

When we run the above code, a FastAI widget (an interactive piece of executable code) is imported and executed. These widgets wrapipywidgetsto make previewing, pruning, and marking bad data a lot easier using HTML forms.

To instantiate the widget, we supply the learner we trained up on line 5 and the maximum number of images we wish to clean at a time. When we execute line 6, the resulting code cell constructs a HTML form that:

• shows images in both the validation and training dataset of each label;
• allows us to flag images for deletion; and,
• allows us to reclassify those images with new labels.

Below shows an example using the widget where we clean up the validation dataset for cubism art.

For each image, we can click the dropdown underneath each image and mark the image as:

• to keep the image labelled as the same;
• to remove the image entirely from the dataset; or,
• select a new label to re-classify the image.

Doing this for all images in our dataset is a bit tedious, but it's a good way to get a feel for how the widget works. To speed through the process, click the drop down on the first image, then hit TAB to cycle to the next image or SHIFT+TAB to previous image, and the ↑ and ↓ keys to move between the dropdown options.

Important:Once you have marked a batch of 100 images with the cleaner, move onto the next step below to delete/move the bad data. Then repeat the process for the other training/validation datasets for each label.

If you don't do this, the changes you have marked will be lost!

Deleting and moving bad data

The image cleaner only keeps track of which images we have marked for deletion and which images we have reclassified.It doesn't delete or move the images for you.

In the second code cell in line 6, we use the delete()method on the cleaner to get all cubism training images we marked for deletion, access the ith image to delete under the cleaner's fnsproperty, and run rmto delete the path to that image flagged for deletion:

This removesevery image flagged as in the above step:

In the same code cell at line 12, we use the change()method on the cleaner to get all the cubism training images we wanted to reclassify with a new label (the new_labelvariable), access the path to the ith image using fnsagain, set a new path, and run mvto move the image to the new path:

This changes the file path of every image flagged with a new label to the new directory:

Before going ahead, make sure you repeat this process for the other datasets and labels.

Re-importing cleaned data

Now that we have cleaned up our datasets, the next steps are to:

1. re-import the datainto our DataLoader (i.e., re-run Step 2);
2. re-train our modelgiven the cleaned dataset (i.e., re-run Step 3); and,
3. re-interpret our learnerto see if cleaning the dataset worked (i.e., re-run Step 4).

For Step 2, re-importing the data will make sure that all data that has been loaded into memory no longer holds images of bad data, or has been reassigned to their correct labels.

When we re-run Step 2, we see that the numbers have reduced:

Let's break these new numbers down:

• impressionismhas been reduced from 132 training images to 100, and to 68 validation images to 58 (total of158);

• cubism arthas been reduced from 129 training images to 66, and to 71 validation images to 39 (total of105); and
• pop arthas been reduced from 129 training images to 87, and to 71 validation images to 38 (total of125).

This may introducebiasin our model, we have an uneven dataset, since we have fewer examples of cubism and pop artwork than we do of impressionism artwork. But, when we re-train our model per Step 3, we can see that the model has increased from an overall accuracy of 69% to about85%!

And, if we re-interpret our results, by re-running Step 4, we can see that our confusion matrix has become better:

By cleaning up instances of bad examples, and recategorising them if needed, we have improved the true-positives (actual = predicted) in our validation (i.e., greater numbers along the diagonal top-to-bottom-right). However, we have also introducedbiasin our model; the confusion matrix shows much higher true-positive numbers for impressionism than the other labels. We would need to ensure we have abalanceddataset, so that the model is trained on equal examples for each label.

We can also see that the maximum loss has reduced from7.54(from the image in Step 4) to now4.24, which is a much better result than the loss we had before cleaning:

So, our modelisgetting better; we just need to make continued adjustments and refinements and then re-train as necessary.

Conclusion

In this two-part tutorial, we covered training a model on a dataset of images that we downloaded from the internet, and then interpreted the results of our trained model. We also saw how to clean up the dataset of bad images and re-trained the model to see how well data cleansing improved accuracy and performance.

This primer was by no means exhaustive and extensiveand skips so many important concepts to machine learning. As mentioned in Part One, if you find this stuff interesting and want to learn more, I urge you to check out thefree coursein more detail, or have a look at the FastAI authors' book,Deep Learning for Coders with FastAI and PyTorch: AI Applications without a PhD, which is also available for free onGitHub.

Until next time, happy model training!

PS:I am in no way affiliated with FastAI, its authors, or Google Colab. I just like teaching people cool things