Using AWS Step Functions to Manage a Long-Running Process

Get Started
1.25 hours
  • 7 Learning Objectives

About this Hands-on Lab

Lambda functions are a great way to create serverless architectures within AWS. But managing and orchestrating them can be difficult when we use many functions within a pipeline. Managing long-running asynchronous processes is also a problem. Lambda can trigger processes to start, but we should avoid having them wait for long-running processes (more than a few minutes) to conclude. AWS Step Functions is a solution to both these problems. In this hands-on lab, we will use AWS Step Functions to manage Lambda functions and monitor a long-running process — Amazon Transcribe — to trigger a subsequent action when a transcription job is complete.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Create an IAM Role for Step Functions
  1. Navigate to IAM.
  2. Select Roles from the menu on the left.
  3. Select Create role.
  4. Select Step Functions as the trusted entity.
  5. Select Next: Permissions.
  6. Confirm the automated policy selection AWSLambdaRole by selecting Next: Tags.
  7. Follow the remaining prompts and name the role "lab-role-step-functions". 1. Select Create role.
Create a Simple Step Functions State Machine
  1. Navigate to Step Functions.
  2. Select State machines from the left-hand menu.
  3. Select Create state machine.
  4. Give the step function the name "lab-step-functions".
  5. Accept the generic definition for now.
  6. Select Next.
  7. Select Choose an existing IAM role, and select the role we created in the previous task.
  8. Select Create state machine.
Create an IAM Role and Lambda Function

If a Lambda function needs to call other AWS services, we need to create an IAM Role that it can assume at execution time.

Create an IAM Role for our Lambda function to be able to connect with S3, Amazon Transcribe, and CloudWatch Logs:

  1. Navigate to IAM > Roles.

  2. Select Create role.

  3. Select Lambda as the trusted entity.

  4. Select Next: Permissions.

  5. Add the following managed policies:

    • AWSStepFunctionsFullAccess
    • CloudWatchLogsFullAccess
  6. Follow the remaining prompts, and name the role "lab-role-lambda-step-trigger".

Now we can create our Lambda function:

  1. Navigate to Lambda.

  2. Click Create function.

  3. Make sure the Author from scratch option at the top is selected, and then use the following settings:

    • Function name: lab-lambda-step-trigger
    • Runtime: Python 3.6
  4. Expand Choose or create an execution role, select Use an existing role, and select the role created above.

  5. Click Create function.

  6. Edit the function and add the code supplied with this lab.

  7. Scroll down under the function code, and create an environment variable:

    • Called: "STATEMACHINEARN"
    • Value: In a new browser tab, navigate to Step Functions and copy the ARN of the state machine created in a previous step. Paste that value into the environment variable value.
Update S3 to Trigger a Different Lambda Function
  1. Navigate to S3.
  2. Navigate into the input-… bucket.
  3. Navigate into the Properties of the bucket.
  4. Scroll to the Events card.
  5. Select the existing event notification, and delete it.
  6. Click Add notification, and set the following properties:
    • Name: Add a meaningful name for the event.
    • Events: All object create events
    • Send to: Lambda Function
    • Lambda: lab-lambda-step-trigger
    • Click Save.
Update the State Machine to Trigger Our Existing Lambda Function, and Update the Lambda to Work with Step Functions

First, update the state machine definition:

  1. Navigate to Step Functions.

  2. Click Edit.

  3. Using the Generate code snippet menu, select AWS Lambda: Invoke a function and select the full ARN for the lab-lambda-transcribe function.

  4. Use the Copy to clipboard button to copy the definition snippet to your computer’s clipboard.

  5. Carefully paste this definition snippet into the main definition over the whole of the HelloWorld state. To see what this should look like, check the video in this lab.

  6. Fix the validation errors:

    • Change the name of the pasted state to transcribe.
    • Update the StartAt value to transcribe.
    • Change the Next key and value to "End": true.
  7. Click Save > Save anyway.

Now, update the Lambda function:

  1. Navigate to Lambda, and select the lab-lambda-transcribe function.

  2. Replace the three lines that handle the incoming "event" (starting record =, s3bucket =, and s3object =) with the following two lines:

    s3bucket = event['Input']['Bucket']
    s3object = event['Input']['Key']
  3. Select Save.

  4. You can now test the simple pipeline by placing an audio file with speech into the input S3 bucket. Look for the execution details from the Step Functions console page.

Add a *Wait* State and *Choice* State to the Step Function
  1. Copy the entire contents of the file called lab-state-wait.json.

  2. Navigate to Step Functions.

  3. Carefully place your cursor after the curly brace after "End": true. This location is represented here by the X:

            ...
            },
            "End": true
        }X
    ...
  4. Paste your clipboard into this location.

  5. Fix the validation error:

    • Change the first "End": true in the transcribe state to "Next": "transcribe-wait".
  6. Save the state machine.

Create a Lambda Function to Check the Status of the Long-Running Task, and Add It to Our Step Function
  1. Navigate to Lambda.

  2. Click Create a function.

  3. Make sure the Author from scratch option at the top is selected, and then use the following settings:

    • Function name: lab-lambda-status-checker
    • Runtime: Python 3.6
  4. Expand Choose or create an execution role, and select the lab-role-lambda-us-east-1 role.

  5. Click Create function.

  6. From the resources in this lab, locate the file lab-lambda-status-checker.py file and copy its contents into the clipboard.

  7. Paste this code into our newly created Lambda function.

  8. Save the new Lambda function.

Now, we add this function to our state machine definition:

  1. Navigate to Step Functions.

  2. Edit our state machine.

  3. Select Generate code snippet and select AWS Lambda. From the Select function list, select the full ARN for the lab-lambda-status-checker function.

  4. Select Copy to clipboard.

  5. Within the definition, locate the Pass state called transcribe-status. This is a placeholder for some code that will call out new Lambda function. Carefully select the entire transcribe-status step, and paste over the snippet you have in your clipboard.

  6. Fix the validation issues:

    • Rename the pasted step to transcribe-status.
    • Change the value of the Next attribute to transcribe-complete.
  7. Save the state machine. You should now be ready to test. You can use the sample audio files in the resources for this lab, or use your own. For full details and to see it in action, see the last video in this lab.

Additional Resources

Scenario: Our company records its meetings and wants to build an automated pipeline to process meeting audio files. They use Amazon Transcribe, and want to trigger an action after the transcription is complete. But Amazon Transcribe is asynchronous, so we need to find a way to monitor the transcription job, so we can trigger a future action.

The videos will walk through each of the steps, including navigating into the AWS console.

Make sure you are in the us-east-1 region when in the AWS console.

All the resources for this lab, including sample audio files to transcribe, can be found on GitHub.

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Get Started
Who’s going to be learning?

How many seats do you need?

  • $499 USD per seat per year
  • Billed Annually
  • Renews in 12 months

Ready to accelerate learning?

For over 25 licenses, a member of our sales team will walk you through a custom tailored solution for your business.


$2,495.00

Checkout
Sign In
Welcome Back!
Thanks for reaching out!

You’ll hear from us shortly. In the meantime, why not check out what our customers have to say about ACG?