Trek word wall1
Developer Experience

CloudFormation Nested Stacks Primer

The limitless potential of CloudFormation is yours to wield following the nested stacks practices.
Jared Short Trek10
Jared Short | Jul 09 2019

Lately, I've seen some discussion around CloudFormation and some of its limitations as well as organizational practices.

CloudFormation nested stacks offer elegant solutions to a lot of these challenges. However, don't just take my word for it.

Understanding all the tricks and features available to you, however, is an adventure in spelunking the proverbial Mines of Moria that is AWS documentation.

How nested stacks work

The first thing to realize is that nested stacks are treated just like any other resource in a CloudFormation template. There is nothing unique or uncomfortable about this situation. A stack is simply a resource to be created and managed like any other resource you would be managing.

# example template resources for a template stack
Resources:
  MyNestedStack:
    Type: AWS::CloudFormation::Stack # required
      Properties: 
        NotificationARNs: # not required
          - String
        Parameters: # conditionally required if the nested stack requires the parameters
          Key : Value
        Tags: # not required
          - Tag
        TemplateURL: String # required
        TimeoutInMinutes: Integer # not required

Typically when I start a project, I create a "root" stack. This root stack exists solely to define the other stacks that exist within my infrastructure.

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  Stage:
    Type: String

Resources:
  DataStores:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: shared/datastores/template.yaml
      Parameters:
        Stage: !Ref Stage
  InternalJobs:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: internal/jobs/template.yaml
      Parameters:
        Stage: !Ref Stage
  PublicAPI:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: public/api/template.yaml
      Parameters:
        Stage: !Ref Stage

This top level template.yaml is supported by a directory tree of the following.

.
├── internal
│   └── jobs
│       └── template.yaml
├── public
│   └── api
│       └── template.yaml
├── shared
│   └── datastores
│       └── template.yaml
└── template.yaml

Finally, for things to go smoothly, we need to embrace the AWS CLI to do all the heavy lifting for us when it comes to managing a template and its lifecycle. aws cloudformation package and aws cloudformation deploy are going to be our core commands for this post.

I have found that embracing the native AWS way of doing things may take some upfront effort, but my ROI is realized within hours of working on the project. You don't want to roll your own cloudformation packaging. If that's the only thing you take away from this post, let it be that.

You'll note that TemplateURL is a file path above. aws cloudformation package manages the process walking a tree of nested stacks and uploading all necessary assets to S3 and rewriting the designated locations in an output template.

Deployment & Management

After a quick aws cloudformation package --template-file template.yaml --output-template packaged.yaml --s3-bucket {your-deployment-s3-bucket} on the root template, you'll get output to packaged.yaml that reflects a new "packaged" template with all necessary assets uploaded to your deployment s3 bucket.

If you find yourself working with CloudFormation a lot, especially if you are already using VS Code, check out this post by Matthew Hodgkins. It introduces some great tools, like linting on your templates. Getting little red squiggles under your mistyped resource names is the difference between finishing a project and debating a career change.

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  Stage:
    Type: String
Resources:
  DataStores:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/serverless-deployment-454679818906-us-east-2/71243994a4224bb9eb149de44022813a.template
      Parameters:
        Stage:
          Ref: Stage
  InternalJobs:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/serverless-deployment-454679818906-us-east-2/71243994a4224bb9eb149de44022813a.template
      Parameters:
        Stage:
          Ref: Stage
  PublicAPI:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/serverless-deployment-454679818906-us-east-2/71243994a4224bb9eb149de44022813a.template
      Parameters:
        Stage:
          Ref: Stage

CloudFormation Nested stacks advantages

These few tricks would let us pretty easily manage multiple stacks and split up resources logically. We don't have to worry about the 200 resource limit, we can make it easier to work with our templates, but what about dependencies between resources?

How do we let AWS Lambda functions in one stack know about, and be delegated permissions to a DynamoDB table or S3 bucket?

Let's talk about CloudFormation Outputs and References. Let's say we have a DynamoDB table that we want to use in our Lambda function. Both in our API and internal compute jobs.

We create the table and tell CloudFormation to make the dynamic table name and ARN available as outputs.

It is almost always best practice to allow CloudFormation to name your resources. in the long run, it makes it easier to manage and maintain your stack, as well as have many stacks in a single account/region without collisions.

# shared/datastores/template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  Stage:
    Type: String

Resources:
  HelloTable:
    Type: AWS::DynamoDB::Table
    Properties: 
      AttributeDefinitions: 
        - AttributeName: id
          AttributeType: S
      BillingMode: PAY_PER_REQUEST
      KeySchema: 
        - AttributeName: id
          KeyType: HASH

Outputs:
  HelloTable:
    Value: !Ref HelloTable
  HelloTableArn:
    Value: !GetAtt HelloTable.Arn

Next, we create an AWS Lambda function using the serverless transform that has access to the DynamoDB table and also passes an environment variable with the table name.

You may note we don't have code in here; rather it points to a file relative to the location of the template. This code gets packaged up and put on S3 for us as part of the single cloudformation package command on the root stack.

If you are curious about what other magic lies in the cloudformation package command, check out the docs!

# internal/jobs/template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'

Parameters:
  Stage:
    Type: String
  HelloTable:
    Type: String
    MinLength: 1
  HelloTableArn:
    Type: String
    MinLength: 1

Resources:
  HelloLambda:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handler.py
      Runtime: python3.7
      CodeUri: src/
      Policies:
        - AWSLambdaExecute
        - Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - dynamodb:BatchGetItem
                - dynamodb:GetItem
                - dynamodb:Query
                - dynamodb:Scan
                - dynamodb:BatchWriteItem
                - dynamodb:PutItem
                - dynamodb:UpdateItem
              Resource: !Ref HelloTableArn
      Environment:
        Variables:
          TABLE_NAME: !Ref HelloTable
      Events:
        PIIScan:
          Type: Schedule
          Properties:
            Schedule: rate(1 day)

The tree, including AWS Lambda functions and their source code, may look something like the following.

.
├── internal
│   └── jobs
│       ├── src
│       │   └── handler.py
│       └── template.yaml
├── packaged.yaml
├── public
│   └── api
│       ├── src
│       │   └── handler.py
│       └── template.yaml
├── shared
│   └── datastores
│       └── template.yaml
└── template.yaml

Wiring up these parameters uses something we are quite familiar with, the GetAtt function. On our root level stack, we add in a couple of new Parameters on our InternalJobs stack.

# Top level stack metadata...
Resources:
  DataStores:
    # stack info / resource config
  InternalJobs:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: internal/jobs/template.yaml
      Parameters:
        Stage: !Ref Stage
        # ------------- ADDED LINES -------------
        HelloTable: !GetAtt DataStores.Outputs.HelloTable
        HelloTableArn: !GetAtt DataStores.Outputs.HelloTableArn
        # ------------- ADDED LINES -------------
  # stack continues...

Now, if our DynamoDB table changes in name or ARN for any reason, our jobs stack references dynamically updates based on those incoming parameters. Development iterations are faster, and correctness is more naturally achieved.

The CloudFormation import / export functionality should only be relied on in the case you are exposing values for other services entirely to depend on. Realize that once an export has been "imported" by another stack, it cannot ever change its value.

A common reason for exports is service discoverability. Instead, take a look at using either SSM Parameters or AWS Cloud Map.

Reusability of Templates

Let's say you have a canonical way of using s3 buckets within your organization. Its configuration sets up all sorts of lifecycle policies, policies, encryption, etc. Your organization creates and provides this template org.s3.template.yaml.

Your team wants 3 buckets in your project. You grab the organizations template, and in your root template, you can use this same template multiple times using parameters for different behavior with CloudFormation conditions in the s3 template allowing a simplified control interface of the template.

# Top level stack metadata...
Resources:
  ProcessStore:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: org.s3.template.yaml
  UploadsStore:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: org.s3.template.yaml
      Parameters:
        AllowPublicUploads: true
  AssetStore:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: org.s3.template.yaml
      Parameters:
        IncludePublicCDN: true

Notes on Deployment and Management

Once your stack is packaged we need to move to deployment. The deployment should reference your output packaged.yaml template.

aws cloudformation deploy --region us-east-2 --template-file packaged.yaml --stack-name blog-post --parameter-overrides Stage=demo

When deploying things that using template transforms or things that create IAM roles/policies you likely need to including --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM to your command acknowledging you are aware of the activities taking place.

Security best practices dictate that you use --role-arn to delegate an IAM Role to the CloudFormation service to run your stack as, rather than running as your credentials issuing the deployment. Learn more about using the service role in the AWS docs.

The rough edges

The CloudFormation changesets feature becomes pretty useless. You can't use it to understand what is going to change in the child stacks. While it is sub-optimal, if you rely on changesets for knowing what is going to happen, I would recommend instead transitioning to cleaner git merges and reviewing source control as an oracle of truth (everything is code defined? right?) rather than the changesets. You still want to make sure you have a solid promotional process (dev -> prod) in place, and make judicious use of stack policies to prevent your data from disappearing.

Drift Detection is still possible, but you need to enumerate over every child stack requesting a drift detection and gathering them to understand what has changed.

Getting into and recovering from bad states can be a little harder to understand. AWS docs, once again, have details worth examining.

If you are working with nested stacks, or anything remotely related to serverless, check out our little community project over at https://serverless.help. We'd love to help guide you in the right direction!

Author
Jared Short Trek10
Jared Short

Similar Blog