Ansible & CloudFormation for Complex Workflows

You got YAML in my YML. You got YML in my YAML. Ansible and CloudFormation provide different strokes for AWS automation

Wed, 07 Aug 2019

CloudFormation StackSets Primer

If you’re unfamiliar with CloudFormation StackSets, I’d highly recommend that you check out the higher level concepts in the documentation. CloudFormation StackSets offer the capability to deploy CloudFormation stacks into multiple regions or multiple accounts. This is super helpful when you want to launch templates across accounts or regions in a standardized manner, or when you want to deploy an app to multiple regions for high-availability or disaster-recovery reasons.

There are many more reasons to use stacksets, but unfortunately, the only way to use them with Cloudformation is through the console or the API. There’s no AWS::CloudFormation::StackSet CloudFormation resource or any native code-defined way to declare stacksets, like other AWS resources. You certainly don’t want to have to write custom stackset orchestration scripts to manage stacksets, and you don’t want to do it manually (because you follow best practices, and all). So this is where Ansible comes in to save the day!

Ansible Primer

Maybe you’ve never used Ansible. Why would you want to learn Ansible over something like Terraform or another alternative? Well, without derailing this blog post into the pro’s and con’s of various deployment mechanisms, Ansible is simple, extensible, and very easy to pick up.

Ultimately, Ansible is a tool that helps you automate tasks and manage the configuration of infrastructure. It doesn’t store state between executions, you just describe exactly what you want and Ansible handles the rest.

How do you install Ansible?

pipx install ansible

Note that we recommend using pipx to reduce installing packages into your global/system python environments.

Now that you’ve got it installed, how do you use Ansible?

You define an Ansible “playbook” in a YAML file, which consists of one-to-many “plays.” Each play contains a list of tasks, which are things you do (execute a command, install a package, deploy AWS infrastructure, etc). To do anything in Ansible, you need to use an Ansible module. You can get a list of modules here. You can even write your own.

Here’s an example playbook to get the AWS Canonical ID:

---
- hosts: localhost # Use 'localhost' since we're not executing this configuration against a remote server, instead we're just executing the playbook
  tasks: # A list of Ansible tasks
    - name: My Debug Task # Each task has a `name`
      debug: # Each task has a module, like `debug` or `command`
        msg: Ansible FTW
    - name: Get canonical user ID
      command: aws s3api list-buckets --query Owner.ID --output text
      register: s3_list_buckets_command # Register the output of the `command` module to this variable name
    - name: Log Canonical ID
      debug:
        msg: '{{ s3_list_buckets_command.stdout }}' # This is how you can reference those variables later on in your playbooks

The defined YAML is an array, where each element of the array is a “play,” making the whole YAML file a “playbook.” To execute the playbook, you just need to run:

$ ansible-playbook my-playbook.yml
PLAY [localhost] ****************************************************************************************************************

TASK [My Debug Task] ************************************************************************************************************
ok: [localhost] => {
    "msg": "Ansible FTW"
}

TASK [Get canonical user ID] ****************************************************************************************************
changed: [localhost]

TASK [Log Canonical ID] *********************************************************************************************************
ok: [localhost] => {
    "msg": "1234567812345678123456781234567812345678123456781234567812345678"
}

PLAY RECAP **********************************************************************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

As you can see we get our two debug logs, one with our static message and the other with our AWS Canonical ID. We also get a few more details on the execution of the playbook.

Defining StackSets in Code

So far everything we’ve done applies to every playbook, but now let’s discuss applying this to CloudFormation StackSets. We’re going to need to use the cloudformation_stack_set module. That module enables you to define a list of accounts and regions to deploy to, the role ARN for CloudFormation to assume when executing the change set, the stackset failure tolerance, and other stackset configuration properties.

This Ansible playbook defines a CloudFormation StackSet to be deployed to 3 accounts, each in one region. When executed, it’ll pull in the ./template.yaml file and deploy it across all three accounts and in the single region configured.

---
- hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Deploy CloudTrail To All Accounts
      cloudformation_stack_set:
        name: cloudtrail
        description: CloudTrail Setup
        state: present
        template_body: '{{ lookup("file", "./template.yaml") }}'
        accounts:
          - '123123123123'
          - '231231231231'
          - '312312312312'
        regions:
          - us-east-1

Notes

  • We’re using the lookup() function to pull in the template at runtime, invoking the file plugin. You can get a list of plugins here. Some notable ones include env for looking up environment variables and aws_ssm and aws_secret for pulling values from AWS SSM and Secrets Manager respectively.

Cross Region Stacks

Now imagine you want to set up Cross-Region replication in code. You need a cloudformation stack in one region, and another one in another region, and you need to connect one stack’s output to another stack’s parameter (i.e. the replication bucket name). Nested stacks, CloudFormation imports and exports, and SSM parameters are all regional. To handle referencing cross-region values you would have to write a script to deploy one stack in region A, query the output, and provide that as a parameter for the stack deployed in region B. However, with the help of ansible, we can define this as configuration in a few lines of yaml and save ourselves a lot of time working out the kinks of cross-region cloudformation stacks.

We have our template for the replication bucket:

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  ReplicationBucket:
    Type: AWS::S3::Bucket
    Properties:
      VersioningConfiguration:
        Status: Enabled
Outputs:
  ReplicationBucketName:
    Description: The name of the replication bucket
    Value: !Ref ReplicationBucket

And another template for the actual bucket:

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  ReplicationBucket:
    Type: String
    Description: The name of the replication bucket
Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      VersioningConfiguration:
        Status: Enabled
      ReplicationConfiguration:
        Role: !GetAtt BucketReplicationRole.Arn
        Rules:
          - Status: Enabled
            Prefix: ''
            Destination:
              Account: !Ref AWS::AccountId
              Bucket: !Sub arn:aws:s3:::${ReplicationBucket}
  BucketReplicationRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: s3.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: BucketReplication
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: s3:*
                Resource: '*'

And so our ansible will look like this:

- hosts: localhost
  gather_facts: false
  tasks:
    - name: Bucket replication stack
      cloudformation:
        region: us-east-2 # us-east-2 is where the main bucket will be replicated
        stack_name: bucket-replication
        state: present
        template_body: '{{ lookup("file", "./replication.yaml") }}'
      register: replication_stack
    - name: Bucket stack
      cloudformation:
        region: us-east-1 # us-east-1 is where our main bucket will be located
        stack_name: bucket
        state: present
        template_body: '{{ lookup("file", "./template.yaml") }}'
        template_parameters:
          ReplicationBucket: "{{ replication_stack.stack_outputs.ReplicationBucketName }}"

Because Ansible isn’t tied to a specific AWS region, it can handle cross-region relationships between regional resources.

This is what makes Ansible so powerful in the AWS CloudFormation and infrastructure orchestration world. You can establish resource relationships however you want: cross-region, cross-account, or even CloudFormation to instance. Ansible gives you the tools to automate your infrastructure a step above CloudFormation. If you’re interested in learning more about Infrastructure as Code, checkout some of our other posts on CloudFormation right here on the CloudProse blog. Follow us @Trek10Inc for more tips on Infrastructure as Code, Serverless, and AWS.

Loading...
Michael Barney

Michael Barney

DevOps Engineer

Michael started his career out in the serverless world, joining the Trek10 team right out of college - a true Serverless Native.More Posts by Michael