How to get the list of all crawlers present in an AWS account using Boto3

In this article, we will see how to get the list of all crawlers present in an AWS account using the boto3 library in Python.

What are AWS Glue Crawlers?

AWS Glue crawlers are programs that connect to data stores, determine data schemas, and populate the AWS Glue Data Catalog with table definitions. The list_crawlers() method helps retrieve all crawler names from your AWS Glue service.

Prerequisites

Before running the code, ensure you have:

• AWS credentials configured (via AWS CLI, IAM role, or environment variables)

• boto3 library installed: pip install boto3

• Appropriate permissions to access AWS Glue service

Approach/Algorithm

• Step 1: Import boto3 and botocore.exceptions for error handling

• Step 2: Create an AWS session using boto3.session.Session()

• Step 3: Create an AWS client for the glue service

• Step 4: Use the list_crawlers() method to fetch all crawlers

• Step 5: Handle exceptions appropriately for robust error management

Example

The following code demonstrates how to fetch the list of all crawlers in your AWS account ?

import boto3
from botocore.exceptions import ClientError

def list_of_crawlers():
    session = boto3.session.Session()
    glue_client = session.client('glue')
    try:
        crawler_details = glue_client.list_crawlers()
        return crawler_details
    except ClientError as e:
        raise Exception("boto3 client error in list_of_crawlers: " + str(e))
    except Exception as e:
        raise Exception("Unexpected error in list_of_crawlers: " + str(e))

# Execute the function
result = list_of_crawlers()
print("Crawler Names:", result['CrawlerNames'])
print("Total Crawlers:", len(result['CrawlerNames']))

Output

The output will display all crawler names and metadata ?

Crawler Names: ['crawler_for_s3_file_job', 'crawler_for_employee_data', 'crawler_for_security_data']
Total Crawlers: 3

Understanding the Response

The list_crawlers() method returns a dictionary containing:

• CrawlerNames: A list of all crawler names in your account

• ResponseMetadata: AWS API response details including request ID and HTTP status

Enhanced Version with Filtering

You can also filter crawlers or get detailed information ?

import boto3
from botocore.exceptions import ClientError

def get_crawler_details():
    session = boto3.session.Session()
    glue_client = session.client('glue')
    
    try:
        # Get list of crawler names
        response = glue_client.list_crawlers()
        crawler_names = response['CrawlerNames']
        
        print(f"Found {len(crawler_names)} crawlers:")
        for name in crawler_names:
            print(f"- {name}")
            
        return crawler_names
        
    except ClientError as e:
        print(f"AWS Client Error: {e}")
    except Exception as e:
        print(f"Unexpected Error: {e}")

# Execute the enhanced function
get_crawler_details()

Conclusion

Using boto3's list_crawlers() method provides an efficient way to retrieve all AWS Glue crawlers in your account. This is essential for crawler management and monitoring in data pipeline automation.