AWS Step Functions

AWS Lambda is for running code. It executes a single function in response to events, for example an S3 upload or an API call. You would use it to perform one specific task such as resizing an image, transforming data or handling a web request. It is stateless and generally short lived (up to 15 minutes). A lambda scales automatically.

AWS Step Functions are used to coordinate multiple tasks into a defined workflow. It manages flow between multiple tasks. It tracks state, handles retries, branching, parallelism and timeouts. Step Functions are used when there is a need to coordinate multiple steps reliably like a loan approval process, data pipeline, or serverless backend with multiple stages.

Amazon SQS (Simple Queue Service) and AWS Step Functions serve different purposes, even though both can help build decoupled and scalable applications.

• Amazon SQS is a message queuing service. It’s used to send, store and receive messages between distributed systems. It enables asynchronous communication and helps decouple producers and consumers, ensuring that messages are delivered even if the receiving service is temporarily unavailable.

• AWS Step Functions is a workflow orchestration service . It coordinates the sequence of steps in a business process, managing execution flow, retries, timeouts and error handling between tasks.

SQS helps move messages between services, while Step Functions helps control how and when those services run as part of a larger workflow.

The key components of AWS Step Functions are the building blocks of a state machine, which defines and manages a workflow:

• State Machine: The overall definition of the workflow, including its structure, logic and behavior.

• States: Each step in the workflow, such as a task, decision, wait, or parallel execution.

• Transitions: Define how the workflow moves from one state to another.

• Input and Output: Data passed into each state, transformed or processed and passed to the next state.

• Execution: A live instance of the state machine, tracking current progress, inputs and results.

• Error Handling: Built-in retry policies and catch blocks to gracefully handle failures.

• Logging and Monitoring: Integrated with CloudWatch to provide visual execution history, logs and performance metrics.

AWS Step Functions integrate with other AWS services in two ways:

• AWS Service Integrations (AWS SDK Integrations), can directly call over 200 AWS service APIs such as DynamoDB, S3, ECS, SageMaker and Glue without using Lambda (custom code). AWS SDK Integrations is cost-effective and low-latency, as it avoids invoking intermediary compute (like Lambda).

• AWS Lambda functions are used in cases such as data transformation is required, conditional logic beyond the capabilities of Step Functions or calling third party APIs.

You could write a Lambda function or some glue code, but AWS Step Functions include debugging, error handling and retries. There is less custom code required and Step Functions are scalable and maintainable.

Step Functions manages process flow whereas a Lambda focuses on its core job. Therefore, this leads to a cleaner, more modular architecture.

In summary, if you just have one task, then use a Lambda. However, if your workflow has multiple steps, conditional logic, or failure handling needs, then AWS Step Functions is best suited, saving time, cost and complexity.

Not necessarily. AWS Step Functions can be cost-efficient at scale when used with the right configuration and workflow type.

Here’s why:

• No infrastructure to manage, you don’t need EC2 instances, containers, or custom orchestrators.

• AWS SDK integrations reduce costs by calling AWS services directly without using Lambda.

• Built-in retries and error handling eliminate the need to build and maintain custom logic.

• Two workflow types let you choose based on your needs:

  • Standard Workflows: Best for long-running, auditable processes with visual execution history.

  • Express Workflows: Optimised for high-volume, low-latency scenarios — much cheaper for real-time or short-lived workflows.

By selecting the appropriate workflow type and minimising unnecessary compute steps, Step Functions can scale efficiently without becoming cost-prohibitive.

Express workflow types are designed for speed. Typically sub-second execution time per state, with the ability to process hundreds of thousands of executions per second. With near instant start time, the workflows are suitable for real-time or event driven apps.

Low latency is achieved as express workflows do not have overheads such as logging and are optimised for speed with no audit trails.

AWS Step Functions work by coordinating a series of steps, called states, into a state machine (workflow), that executes tasks in a defined sequence, with logic for branching, retries, waiting and error handling.

Workflows are a series of event-driven steps. Each step in a workflow is called a state. For example, a task state represents a unit of work that another AWS service performs, such as calling another AWS service or API. Instances of running workflows performing tasks are called executions in Step Functions.

Here is a simplified explanation of how Step Functions work:

1. 1.

   Define a state machine

   This defines the sequence of steps (states), what each state does (e.g., call a Lambda, wait, make a decision) and how to handle success, failure and branching logic.

2. 2.

   Start an execution

   This can be done via the AWS SDK/API, AWS CLI, EventBridge (for event-driven triggers) or a manual start in the AWS Console.

   Note: Each execution is isolated, so multiple can run in parallel.

3. 3.

   Step Functions executes the states

   This involves data input / output passing between steps. Retries, timeouts and error catching. Decisions (via Choice states) and Waiting (via Wait states).

4. 4.

   Monitor and debug the workflow

   AWS provides a visual execution graph for Standard workflows, step by step input / output inspection, logs via CloudWatch and optional metrics and alerts. This makes it easy to troubleshoot and optimise workflows.

5. 5.

   Workflow ends (success or failure)

   An execution ends when it either reaches an End state successfully, or it hits a Fail state or an unhandled error. This can then trigger follow up actions, send notifications or store the results.