How can Datatset.map be used in Tensorflow to create a dataset of image, label pairs?

TensorFlow's Dataset.map() method applies a transformation function to each element in a dataset. For image classification tasks, we can use it to create (image, label) pairs from file paths by processing each path to load the image and extract its label.

Read More: What is TensorFlow and how Keras work with TensorFlow to create Neural Networks?

Dataset Setup

We'll use a flowers dataset containing thousands of flower images organized in 5 subdirectories (one per class). First, let's create a complete example showing how to use Dataset.map() ?

import tensorflow as tf
import pathlib

# Download and prepare the flowers dataset
dataset_url = "https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz"
data_dir = tf.keras.utils.get_file('flower_photos', origin=dataset_url, untar=True)
data_dir = pathlib.Path(data_dir)

# Create dataset from directory structure
image_count = len(list(data_dir.glob('*/*.jpg')))
print(f"Total images: {image_count}")

# Get class names from directory structure
CLASS_NAMES = [item.name for item in data_dir.glob('*') if item.is_dir()]
print(f"Classes: {CLASS_NAMES}")

Creating the Process Function

The key is defining a function that processes each file path to create (image, label) pairs ?

def process_path(file_path):
    # Extract label from file path
    label = get_label(file_path)
    
    # Load and decode image
    image = tf.io.read_file(file_path)
    image = decode_img(image)
    
    return image, label

def get_label(file_path):
    # Convert path to list of path components
    parts = tf.strings.split(file_path, '/')
    # Get parent directory name (class name)
    class_name = parts[-2]
    # Convert class name to integer label
    return tf.argmax(class_name == CLASS_NAMES)

def decode_img(image):
    # Convert compressed string to 3D uint8 tensor
    image = tf.image.decode_jpeg(image, channels=3)
    # Resize image to fixed input size
    image = tf.image.resize(image, [180, 180])
    # Convert to float values in [0, 1] range
    return tf.cast(image, tf.float32) / 255.0

Applying Dataset.map()

Now we can use Dataset.map() to transform file paths into (image, label) pairs ?

# Create dataset of file paths
list_ds = tf.data.Dataset.list_files(str(data_dir/'*/*'), shuffle=False)
list_ds = list_ds.shuffle(image_count, reshuffle_each_iteration=False)

# Split into train and validation
val_size = int(image_count * 0.2)
train_ds = list_ds.skip(val_size)
val_ds = list_ds.take(val_size)

# Set up for performance
AUTOTUNE = tf.data.AUTOTUNE

# Apply the map transformation
print("Applying Dataset.map() to create (image, label) pairs...")
train_ds = train_ds.map(process_path, num_parallel_calls=AUTOTUNE)
val_ds = val_ds.map(process_path, num_parallel_calls=AUTOTUNE)

# Inspect the results
for image, label in train_ds.take(1):
    print("Image shape:", image.numpy().shape)
    print("Label:", label.numpy())
    print("Class name:", CLASS_NAMES[label.numpy()])

Key Parameters

How It Works

The Dataset.map() method transforms each file path through these steps:

1. Label Extraction: Extract class name from file path and convert to integer
2. Image Loading: Read image file as raw bytes
3. Image Decoding: Decode JPEG to tensor and resize to fixed dimensions
4. Normalization: Convert pixel values to [0,1] range

Performance Optimization

For better performance, you can chain additional optimizations ?

def configure_for_performance(ds):
    ds = ds.cache()  # Cache dataset in memory
    ds = ds.shuffle(buffer_size=1000)  # Shuffle data
    ds = ds.batch(32)  # Create batches
    ds = ds.prefetch(buffer_size=AUTOTUNE)  # Prefetch next batch
    return ds

train_ds = configure_for_performance(train_ds)
val_ds = configure_for_performance(val_ds)

Conclusion

Dataset.map() efficiently transforms file paths into (image, label) pairs by applying a processing function to each element. Use num_parallel_calls=AUTOTUNE for optimal performance when loading and processing multiple images simultaneously.