How to use Boto3 to stop a crawler in AWS Glue Data Catalog

In this article, we will see how a user can stop a crawler present in an AWS Glue Data Catalog using the Boto3 library in Python.

Problem Statement

Use the boto3 library in Python to stop a running crawler in AWS Glue Data Catalog.

Approach to Solve This Problem

• Step 1: Import boto3 and botocore exceptions to handle exceptions.

• Step 2: Define a function that takes crawler_name as a parameter.

• Step 3: Create an AWS session using boto3. Make sure region_name is mentioned in the default profile. If not, explicitly pass the region_name while creating the session.

• Step 4: Create an AWS client for glue service.

• Step 5: Use the stop_crawler function and pass the crawler_name as the Name parameter.

• Step 6: Handle exceptions - CrawlerNotRunningException if the crawler is not running, and generic exceptions for other errors.

Example Code

The following code demonstrates how to stop a crawler using Boto3 ?

import boto3
from botocore.exceptions import ClientError

def stop_a_crawler(crawler_name):
    session = boto3.session.Session()
    glue_client = session.client('glue')
    try:
        response = glue_client.stop_crawler(Name=crawler_name)
        return response
    except ClientError as e:
        raise Exception("boto3 client error in stop_a_crawler: " + e.__str__())
    except Exception as e:
        raise Exception("Unexpected error in stop_a_crawler: " + e.__str__())

# Example usage
print(stop_a_crawler("Data Dimension"))

Output

When the crawler is successfully stopped, the function returns a response metadata ?

{'ResponseMetadata': {'RequestId': '73e50130-*****************8e', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Sun, 28 Mar 2021 07:26:55 GMT', 'content-type': 'application/x-amz-json-1.1', 'content-length': '2', 'connection': 'keep-alive', 'x-amzn-requestid': '73e50130-***************8e'}, 'RetryAttempts': 0}}

Key Points

• The stop_crawler() method only works if the crawler is currently running.

• If the crawler is not running, it will raise a CrawlerNotRunningException.

• The function returns response metadata upon successful execution.

• Proper exception handling ensures graceful error management.

Conclusion

Using Boto3's stop_crawler() method provides a programmatic way to stop AWS Glue crawlers. Always handle exceptions appropriately to manage cases where the crawler is not running or other errors occur.