Hello everyone, in this post, we will see how we create an image data set in Numpy format. In the previous tutorial, we created an image dataset in CSV format. However, when we use the CSV format to create an image dataset, it takes a long time and has a larger file size than the NumPy format. Therefore we will use Numpy to create our image dataset.

We will use cat and dog images to create this dataset. While creating this dataset, we will use 0 for cat images and 1 for dog images. And since these pictures are of different sizes, we need to make all the pictures the same size. There are many different image sizes used in deep learning applications. These are 28×28, 32×32, 64×64, 227×227 etc. But you don’t have to use these image sizes. In this tutorial, we will use the 64×64 size while creating the dataset. Another important point is whether the pictures are gray or colored. Gray pictures have a single color channel, and color pictures have 3 different color channels. Color pictures are generally preferred in deep learning applications. Therefore, we will use the color image format while creating our data. Let’s start building our dataset by loading the necessary libraries without further ado.

import numpy as np
import pandas as pd
import cv2
import os
from tqdm import tqdm
from glob import glob
  • numpy : is used for matrix operations.
  • pandas : is used for dataset operations.
  • cv2 : is used for image manipulation operations.
  • tqdm : is used to show the percentage of transactions.
  • glob : is used to get file directories of the images.
  • os : is used to list files or folders on the directory.

Before starting to write the code we have to download cat and dog images. To download images use https://www.microsoft.com/en-us/download/details.aspx?id=54765 link. There are 12500 cat images and 12500 dog images we can use.

First, we need to create a python class named CreateDataset for this operation. We use the init method to create predefined variables. Also, we can access these predefined variables from other methods in the class. We define the image height and width, dataframe, and, a dictionary to store label names.

class CreateDataset():
    def __init__(self):
        self.height = 64
        self.width = 64
        self.PetImage_dict = {"Cat":0, "Dog":1}
        self.dataset = list()
        self.isGray = True
            
    def create_dataset(self):
        for name in os.listdir("PetImages/"): # get image folders from directory
            print("PetImage-Name : ", name) # print folder name
            # get all image names with path from in folder
            for image_name in tqdm(glob('PetImages/{}/*.jpg'.format(name))):
                image_name = image_name.replace("\\","/")
                try:                                  # check is there any error
                    if self.isGray:             # load image with opencv
                        image = cv2.imread(image_name)
                    else:                       # load image and convert to grayscale with opencv
                        image = cv2.imread(image_name, cv2.IMREAD_GRAYSCALE)
                    
                    # resize the image dimention with opencv
                    image = cv2.resize(image, (self.height,self.width))
                    
                    # append images and labels to python list
                    self.dataset.append([image,self.PetImage_dict[name]])
                    
                except:
                    pass
        np.random.shuffle(self.dataset) # shuffle the dataset
            
    def save_npy_dataset(self):   # save the dataset to the working directory
        np.save("DogCatDataset.npy", self.dataset, allow_pickle=True)
    def get_data(self):
        return self.dataset

We created the CreateDataset class and we will create an object from this class named dataset like the following code. After creating the dataset object, we create image dataset consists of cat and dog images by calling the create_dataset() method. To save the image dataset which we create to the working directory we will use the save_npy_dataset() method.

dataset = CreateDataset()
dataset.create_dataset()
dataset.save_npy_dataset()

Let’s examine how the image dataset we created looks like by restoring it. We can use the NumPy load method to restore the dataset.

data = np.load("DogCatDataset.npy", allow_pickle=True)
print("data shape : ", data.shape)
data shape :  (24946, 2)

When we restore the dataset and print its shape we will see it has 24946 arrays and each array has two different arrays. The first column of the dataset represents images and the second column represents image labels. Let’s see these images and labels. The following code shows the first image and its label on the screen.

label = data[0][1]
image = data[0][0]
cv2.imshow("image", image)
cv2.waitkey(0)
cv2.destroyAllWindows()