How to get the details of a workflow using Boto3

In this article, we will see how to retrieve the resource metadata of an AWS Glue workflow using Boto3. AWS Glue workflows organize and coordinate multiple jobs, crawlers, and triggers into a single unit.

What is AWS Glue Workflow?

An AWS Glue workflow is a collection of related jobs, crawlers, and triggers that work together to complete an ETL process. You can visualize and track the progress of the entire workflow through the AWS Glue console.

Approach to Get Workflow Details

• Step 1: Import boto3 and botocore exceptions to handle errors.

• Step 2: The workflow_name is the required parameter to fetch workflow metadata.

• Step 3: Create an AWS session using boto3. Ensure region_name is configured in your AWS profile.

• Step 4: Create an AWS client for glue service.

• Step 5: Call get_workflow() method with the workflow name.

• Step 6: Handle exceptions appropriately.

Example

The following code demonstrates how to fetch workflow details from AWS Glue −

import boto3
from botocore.exceptions import ClientError

def get_workflow_metadata(workflow_name):
    session = boto3.session.Session()
    glue_client = session.client('glue')
    
    try:
        response = glue_client.get_workflow(Name=workflow_name)
        return response
    except ClientError as e:
        raise Exception("boto3 client error in get_workflow_metadata: " + str(e))
    except Exception as e:
        raise Exception("Unexpected error in get_workflow_metadata: " + str(e))

# Get workflow details
workflow_details = get_workflow_metadata('my-glue-workflow')
print(workflow_details)

Output Structure

The response contains comprehensive workflow information including name, creation date, status, and statistics −

{
    'Workflow': {
        'Name': 'my-glue-workflow',
        'DefaultRunProperties': {},
        'CreatedOn': datetime.datetime(2020, 5, 27, 3, 10, 57, 967000, tzinfo=tzlocal()),
        'LastModifiedOn': datetime.datetime(2020, 5, 27, 3, 10, 57, 967000, tzinfo=tzlocal()),
        'StartedOn': datetime.datetime(2021, 2, 3, 16, 14, 48, 795000, tzinfo=tzlocal()),
        'CompletedOn': datetime.datetime(2021, 2, 3, 16, 28, 6, 207000, tzinfo=tzlocal()),
        'Status': 'COMPLETED',
        'Statistics': {
            'TotalActions': 3,
            'TimeoutActions': 0,
            'FailedActions': 0,
            'StoppedActions': 0,
            'SucceededActions': 3,
            'RunningActions': 0
        }
    },
    'ResponseMetadata': {
        'RequestId': 'b328d064-24ab-48c4-b058-852387a3d474',
        'HTTPStatusCode': 200
    }
}

Key Response Fields

• Name: The workflow name

• Status: Current workflow status (RUNNING, COMPLETED, FAILED, etc.)

• Statistics: Summary of actions including total, succeeded, and failed counts

• CreatedOn/LastModifiedOn: Timestamp information

Conclusion

Use the get_workflow() method from Boto3's Glue client to retrieve comprehensive workflow metadata. This information helps monitor workflow execution and troubleshoot issues in your ETL processes.