What is Machine Learning?

Machine learning is the science (and art) of programming computers so they can learn from data.

More general definition:

[Machine learning is the] field of study that gives computers the ability to learn without being explicitly programmed.

– Arthur Samuel, 1959

The examples that the system uses to learn are called the training set. Each training example is called a training instance (or sample). The part of a machine learning that learns and makes predictions is called a model. Neural networks and random forests are example of models.

Traditional vs Machine Learning Approach

In traditional programs, a developer designs logic or algorithms to solve a problem (Figure 1). The program applies this logic to input and computes the output.

But in machine learning, a model is built from the data, and that model is the logic (Figure 2). ML programs have two distict phases: Training and Inference.

Why use Machine Learning?

Machine learning is great for:

• Problems for which existing solutions require a lot of fine-tuining or long lists of rules
• Complex problems for which using a traditional approach yields no good solution
• Fluctuating evironments (a machine learning system can easily be retrained on new data, always keeping it up to date)
• Getting insights about complex problems and large amounts of data

Digging into large amounts of data to discover hidden patterns is called data mining, and machine learning excels at it.

Few Examples of Applications

Let’s look at some concrete examples of machine learning tasks, along with the techniques that can tackle them. This will give you a sense of the incredible breadth and complexity of the tasks that machine learning can easily tackle.

• Analzing images of products on a production
  This is image classificaiton, typically performed using CNNs or sometime transformers
• Detecting tumors in brain scans
  This is semantic image segmentation, using CNNs or transformers
• Automatically flagging offensive comments on discussion fourms
  This is natural language processing (NLP), and more specifically text classification, which can be tackled using RNNs and CNNs
• Making your app react to voice commands
  This is speech recognition, which requires processing audio samples: since they are long and complex sequences, they are typically processed using RNNs, CNNs, or transformers
• Detecting credit card fraud
  This is anomly detection, which can be tackled using isolation forests, Gaussian mixture models, or autoencoders
• Recommending a product that a client may be interested in, based on past purchases
  This is a recommender system
• Building an intelligent bot for a game
  This is often tackled using reinforcement learning, which is a branch of machine learning that trains agent to pick the actions that will maximize the rewards over time, within a given environment

Types of Machine Learning Systems

There are so many differents types of machine learning systems that it is useful to classify them in broad categories, based on the following criteria:

• How they are supervised during training (training supervision)
• Whether or not they can learn incrementally on the fly (online vs batch learning)
• Whether they can work by simply comparing new data points to known data points, or instead by detecting a predictive model (instance-based vs model-based learning)

These criteria are not exclusive, you can combine them in any way you like. For example, a state-of-the-art spam filter can be an online, model-based, supervised learning system.

Let’s look at each one of these categoies a bit more closely.

Training Supervision

ML systems can be classifed according to the amount and type of supervision they get during training. There are many categories, but we will discuss the main ones:

Supervised learning

The training set you feed to the algorithm includes the desired solutions, called labels. A typical supervised learning task is classification.

Unsupervised learning

The training data is unlabeled and the system tries to learn without a teacher. Visualization alogrithms, anomaly detection, and dimensionality reduction are few examples of unsupervised learning.

Semi-supervised learning

Since labeling data is usually time-consuming and costly, you will often have plenty of unlabeled instances, and few labeled instances. Some algorithms can deal with data that’s partially labeled. Most semi-supervised learning algorithms are combinations of unsupervised and supervised algorithms.

Some photo-hosting services, such as Google Photos, are good examples of this. Once you upload all your family photos, it automatically recognizes people in multiple photos, and group them, so you can add one label per person and it is able to name everyone in every photo.

Self-supervised learning

Another approach to machine learning involves actually generating a fully labeled dataset from a fully unlabeled one. Again, once the whole dataset is labeled, any supervised learning can be used. This appraoch is called self-supervised learning.

Transferring knowledge from one task to another is called transfer learning.

Reinforcement learning

The learning system, called an agent in this context, can observe the environment, select and perform actions, get rewards or penalties in return. It must then learn by itself what is the best strategy, called a policy, to get the most reward over time. A policy defines what action the agent should choose when it is in a given situation.

Batch vs Online Learning

Batch learning

In batch learning, the system is incapable of learning incrementally: it must be trained using all the available data. This will generally take a lot of time and computing resources, so it is typically done offline.

First the system is trained, and then it is launced into production and runs without learning anymore; it just applies what it has learned.

Online learning

In online learning, you train the system incrementally by feeding it data instances sequentially, either individually or in small groups called mini-batches. Each learning step is fast and cheap, so the system can learn about new data on the fly, as it arrives.

A big challenge with online learning is that if bad data is fed to the system, the system’s performance will decline, possibly quickly (depending on the data quality and learning rate).

Instance-Based vs Model-Based Learning

Instance-based learning

The system learns the examples by heart, then generalizes to new cases by using a similarity measure to compare them to the learned examples.

Model-based learning

Another way to generalize from a set of examples is to build a model of these examples and then use that model to make predictions.

Model selection consists in choosing the type of model and fully specifying its architecture. Training a model means running an algorithm to find the model parameters, that will make it best fit the training data, and hopefully make good predictions on new data.

Summary

Machine learning is about making machines get better at some task by learning from data, instead of having to explicity code rules. There are so many different types of ML systems: supervised or not, batch or online, instance-based or model-based.

In this project, we will be looking at the dataset journey(from gathering to preparation), and then will jump into the python code for creating the model, the FlaskAPI backend, and Streamlit python code.

Dataset

The dataset used is hosted on Kaggle, and the data was captured from Freepik using python script. Once the data was captured, Roboflow was used to organize and annotate. Roboflow can be used for more advanced features like augmentation, pre-processing, getting more examples, and exporting the data in different formats.

I exported the data in folder format which was able to give me the data in the below format:

The dataset consists of around 1300 images. 718 images for training, 208 images for validation, and 106 images for testing.

Python Logic for Model Creation

The full code is accessible on GitHub through this link. But we will go over the important points in this article.

Import Library

import os
import math
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import tensorflow_hub as hub
from tensorflow.keras import layers

print(tf.__version__)

Loading Data

We have loaded the data using ImageDataGenerator with multiple augmentation layers as our dataset is quite small, with only 718 images for training.

train_datagen = tf.keras.preprocessing.image.ImageDataGenerator(
    rescale=1./255,
    rotation_range=40,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    fill_mode='nearest',
    horizontal_flip=True,
)

train_generator = train_datagen.flow_from_directory(
    TRAIN_DIR,
    target_size=(IMG_SIZE, IMG_SIZE),
    batch_size=BATCH_SIZE,
    class_mode='sparse'
)

Found 718 images belonging to 2 classes.

val_datagen = tf.keras.preprocessing.image.ImageDataGenerator(
    rescale=1./255
)

val_generator = val_datagen.flow_from_directory(
    VAL_DIR,
    target_size=(IMG_SIZE, IMG_SIZE),
    batch_size=BATCH_SIZE,
    class_mode='sparse'
)

Found 208 images belonging to 2 classes.

test_datagen = tf.keras.preprocessing.image.ImageDataGenerator(
    rescale=1./255
)

test_generator = test_datagen.flow_from_directory(
    TEST_DIR,
    target_size=(IMG_SIZE, IMG_SIZE),
    batch_size=BATCH_SIZE,
    class_mode='sparse'
)

Found 106 images belonging to 2 classes.

Data Visualization

show_examples(train_generator)

IMAGE SHAPE:  (480, 480, 3)

Now let’s jump over the model architecture and see how we smartly used the pre-trained model as a feature extractor.

Model Architecture

FEATURE_EXTRACTOR_URL = 'https://tfhub.dev/google/imagenet/efficientnet_v2_imagenet21k_ft1k_m/feature_vector/2'

model = tf.keras.Sequential([
    hub.KerasLayer(FEATURE_EXTRACTOR_URL, trainable=False, input_shape=(IMG_SIZE, IMG_SIZE, 3)),
    layers.Dense(2, activation='softmax')
])

model.summary()

model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 keras_layer (KerasLayer)    (None, 1280)              53150388  
                                                                 
 dense (Dense)               (None, 2)                 2562      
                                                                 
=================================================================
Total params: 53,152,950
Trainable params: 2,562
Non-trainable params: 53,150,388
_________________________________________________________________

As we have fewer training images with drastic differences b/w images such as color, size, people, placement & amount of target items, it is a quite time-consuming and complex task to train the model from scratch. That’s why it’s a best practice to check if a pre-trained model can be utilized.

After multiple iterations of custom model architecture, we have used the ‘EfficientNet V2’ model from TensorFlow Hub which was trained on imagenet-21k (Full ImageNet, Fall 2011 release) and fine-tuned on ImageNet1K as a feature extractor and a single dense layer which is used as output layer having 2 nodes defining 2 classes in the dataset.

Model Training

SAVED_MODEL_PATH = './models/pizza_vs_icecream_model.h5'

model_checkpoint = tf.keras.callbacks.ModelCheckpoint(
    SAVED_MODEL_PATH,
    monitor='val_loss',
    save_best_only=True
)

early_stopping = tf.keras.callbacks.EarlyStopping(
    monitor='val_loss',
    patience=5
)

history = model.fit(
    train_generator,
    steps_per_epoch=len(train_generator),
    epochs=50,
    validation_data=val_generator,
    validation_steps=len(val_generator),
    callbacks=[model_checkpoint, early_stopping],
    verbose=2
)

Epoch 1/50
23/23 - 52s - loss: 0.3017 - accuracy: 0.8900 - val_loss: 0.0705 - val_accuracy: 0.9952 - 52s/epoch - 2s/step
Epoch 2/50
23/23 - 35s - loss: 0.0480 - accuracy: 0.9903 - val_loss: 0.0363 - val_accuracy: 0.9952 - 35s/epoch - 2s/step
Epoch 3/50
23/23 - 35s - loss: 0.0348 - accuracy: 0.9889 - val_loss: 0.0221 - val_accuracy: 1.0000 - 35s/epoch - 2s/step
.
.
.
Epoch 38/50
23/23 - 34s - loss: 0.0015 - accuracy: 1.0000 - val_loss: 0.0031 - val_accuracy: 1.0000 - 34s/epoch - 1s/step
Epoch 39/50
23/23 - 34s - loss: 0.0064 - accuracy: 0.9972 - val_loss: 0.0024 - val_accuracy: 1.0000 - 34s/epoch - 1s/step
Epoch 40/50
23/23 - 34s - loss: 0.0050 - accuracy: 0.9972 - val_loss: 0.0027 - val_accuracy: 1.0000 - 34s/epoch - 1s/step

And, definitely in our use case, the pre-trained model has worked wonderfully. Let’s finally evaluate the model.

Model Evaluation

saved_model = tf.keras.models.load_model(
    SAVED_MODEL_PATH,
    custom_objects={'KerasLayer': hub.KerasLayer}
)

saved_model.evaluate(test_generator)

4/4 [==============================] - 7s 1s/step - loss: 0.0043 - accuracy: 1.0000
Out[15]:
[0.004329057410359383, 1.0]

As the model is working flawlessly even on the test data which the model has never seen, it’s time to take this model to production. Upload the saved model to Google Cloud, which will be used in the next section for building the backend services.

Flask API Backend Code

Python code for Flask API is straightforward and the detailed code can be accessible at main.py. We load the saved(Google Cloud) TensorFlow model once the Flask run initially and we get the POST request at the root level along with the image and first thing will be, pre-processing the received image.

The pre-processing steps include:

1. Decoding the base64 image
2. Reading the image using tf.io.decode_image(image, channels=3)
3. Resizing the image to match the image size used to train the model i.e., 480
4. Converting the image to array
5. And finally, normalize the image array by dividing it with 255.0

After pre-processing, we ran the input image array through the model using model.predict(input) and get the output as probability of classes. Using this the high-probability class is identified and returned.

Front-End Streamlit App

To make the model easier to use, let’s build a Streamlit App, which provides astonishing UI with little python code. The code is accessible at app.py.

The purpose of this python code is to provide the upload facility to users, so they can upload an image that will be encoded and sent to the Flask API for prediction. Once the result is returned from the API request, using Streamlit functions, we can show the result in very beautiful formats along with the uploaded images.

Final Connections b/w Streamlit and Flask API

The whole code was uploaded to GitHub and the Flask API code was uploaded to the Google Cloud Run functionality and the Streamlit app code was connected to the Streamlit app.

Results

Result 1

Result 2

Closing Note

This was a fun project for me as the binary classification task was simple, but if we look at the whole project in a nutshell from data collection to taking the model to production with beautiful UI, it was a challenging task too.

A great beginner-friendly project.