CloudFormation YAML is here, and it's awesome

Tue, 20 Sep 2016

You may have seen the announcement recently about what we at Trek10 consider to be the biggest update to CloudFormation since CloudFormation itself. AWS CloudFormation Update – YAML, Cross-Stack References, Simplified Substitution{:target=“_blank”}.

YAML is more readable, easier to work with, easier to skim, and makes my head hurt a lot less… but beyond the glaringly obvious, what has me so pumped?

Inline Comments

“Comment your code” is the mantra we all love to pretend saves us from documentation, but with JSON you had to resort to either a build step to rip out inline comments, or just pretend the problem doesn’t exist. But those days are over, no more thousand line JSON blobs with no inline hints to help you sort out the madness. YAML supports inline comments, and it is beautiful.

Before: (Tell me: What does that security group do? What version of linux does this start?)

{ "Parameters" : { "KeyName" : { "Description" : "The EC2 Key Pair to allow SSH access to the instance", "Type" : "AWS::EC2::KeyPair::KeyName" } }, "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "SecurityGroups" : [ "MyExistingSecurityGroup" ], "KeyName" : { "Ref" : "KeyName"}, "ImageId" : "ami-6869aa05" } } }}

After: (Same questions… answered inline!)

<pre><code>Parameters: KeyName: Description: The EC2 Key Pair to allow SSH access to the instance Type: AWS::EC2::KeyPair::KeyNameResources: Ec2Instance: Type: AWS::EC2::Instance Properties: SecurityGroups: - MyExistingSecurityGroup # SSH access for CloudOps team VPN KeyName: Ref: KeyName ImageId: ami-6869aa05 # Amazon Linux 2016.03.3</code></pre>

Cross Stack References

If you have used CloudFormation for any length of time, no doubt you have asked yourself “Why do I have to put in these parameters for stuff created by this other stack? Why can’t it just know?” We asked ourselves that. Daily.

I am proud to tell you that we started doing Cross Stack referencing before it was cool with AWS Lambda and custom backed resources. However, it was a bloated, ugly, hack of a solution. I have never been happier to take a solution out back and put it out of its misery.

AWS now provides an elegant Cross Stack solution that, if you follow practical naming conventions, is pretty darn powerful.

Before:

<pre><code>{ "Parameters":{ "KeyName" : { "Description" : "The EC2 Key Pair to allow SSH access to the instance", "Type" : "AWS::EC2::KeyPair::KeyName", "Default": "cloudops-key" }, "InstanceType":{ "Type": "String", "Default": "t2.medium" }, "Ec2SecurityGroup":{ "Type": "AWS::EC2::SecurityGroup::Id", "Default": "sg-1234567" }, "ImageId":{ "Type": "String", "Default": "ami-6869aa05" } }, "Resources":{ "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "ImageId" : { "Ref": "ImageId"}, "KeyName" : { "Ref" : "KeyName" }, "InstanceType" : { "Ref" : "InstanceType" }, "SecurityGroups" : [{ "Ref" : "Ec2SecurityGroup" }] } } }}</code></pre>

After:

<pre><code>Resources: Ec2Instance: Type: AWS::EC2::Instance Properties: SecurityGroups: - !ImportValue CloudOpsSecurityGroup InstanceType: !ImportValue DefaultsInstanceType KeyName: !ImportValue CloudOpsSSHKey ImageId: !ImportValue DefaultsImageId</code></pre>

Note: Yes, you could use sub-templates to get somewhat this functionality previously… but if you’ve ever used them in any kind of rapidly moving environment with lots of sub-stack dependencies, you know what I mean when I say “I don’t hate myself quite that much”.

Better “Scripts” and String Substitution

User data and cloud-init files are super handy for configuration of ec2 instances from CloudFormation templates without having to roll your own AMIs through a build pipeline. That said, the syntax in JSON for putting together those scripts was a large pile of 💩, and you can forget about any kind of useful syntax highlighting.

Before: (Don’t even try to read this… just appreciate the hideousness)

<pre><code>"ContainerInstances": { "Type": "AWS::AutoScaling::LaunchConfiguration", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "commands" : { "01_add_instance_to_cluster" : { "command" : { "Fn::Join": [ "", [ "#!/bin/bash\n", "echo ECS_CLUSTER=", { "Ref": "ECSCluster" }, " >> /etc/ecs/ecs.config" ] ] } } }, "files" : { "/etc/cfn/cfn-hup.conf" : { "content" : { "Fn::Join" : ["", [ "[main]\n", "stack=", { "Ref" : "AWS::StackId" }, "\n", "region=", { "Ref" : "AWS::Region" }, "\n" ]]}, "mode" : "000400", "owner" : "root", "group" : "root" }, "/etc/cfn/hooks.d/cfn-auto-reloader.conf" : { "content": { "Fn::Join" : ["", [ "[cfn-auto-reloader-hook]\n", "triggers=post.update\n", "path=Resources.ContainerInstances.Metadata.AWS::CloudFormation::Init\n", "action=/opt/aws/bin/cfn-init -v ", " --stack ", { "Ref" : "AWS::StackName" }, " --resource ContainerInstances ", " --region ", { "Ref" : "AWS::Region" }, "\n", "runas=root\n" ]]} } }, "services" : { "sysvinit" : { "cfn-hup" : { "enabled" : "true", "ensureRunning" : "true", "files" : ["/etc/cfn/cfn-hup.conf", "/etc/cfn/hooks.d/cfn-auto-reloader.conf"] } } } } } }}</code></pre>

After:

ContainerInstances: Type: AWS::AutoScaling::LaunchConfiguration Metadata: AWS::CloudFormation::Init: config: commands: 01_add_instance_to_cluster: command: !Sub | #!/bin/bash echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config files: "/etc/cfn/cfn-hup.conf": content: !Sub | [main] stack=${AWS::StackId} region=${AWS::Region} mode: '000400' owner: root group: root "/etc/cfn/hooks.d/cfn-auto-reloader.conf": content: !Sub | [cfn-auto-reloader-hook] triggers=post.update path=Resources.ContainerInstances.Metadata.AWS::CloudFormation::Init action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource ContainerInstances --region ${AWS::Region} runas=root services: sysvinit: cfn-hup: enabled: 'true' ensureRunning: 'true' files: - "/etc/cfn/cfn-hup.conf" - "/etc/cfn/hooks.d/cfn-auto-reloader.conf"

Night and day difference!

What do I still wish it could do?

Having YAML is spectacular, but we’d also like to leverage some of the more powerful features of the syntax. For example, YAML aliases{:target=“_blank”} so we could leverage the powerful hash merging functionality!

All that said, pretty cool and much needed update!