Hi, Would appreciate if you can share some insights on how you manage Infrastructure as code in AWS. This includes the cloud formation scripts, lambda etc. Currently this is in an S3 bucket. How do you bring this into the SDLC life cycle?
In my organization, we use Atlassian suite of products heavily (Jira, Bamboo, Stash) along with Nexus for SDLC. And we intend to continue with that in AWS. as far as I can see – there is no native way to integrate infrastructure as a code into our build process, keen to understand how others manage this.
If you plan migrate to the cloud you would have to provide your environment anyway. I suggest you start by scripting it, I settled on the following solution :
1 repository for Terraform script initialising the network (VPC, subnets…)
1 repository for Terraform script initialising common stacks (such as monitoring)
1 repository per application
Once you know you can provide your environment with scripts, it is just a matter of maturity of your organisation to integrate IaC. You can provide an environment by using Terraform CLI, and S3 bucket as a backend and do that any time you need a new environment. This way you can keep the deployment method you currently use. Or you could go further, and integrate this in your CI/CD pipeline, and provide a whole new environment for each deployment (i.e. execute all terraform scripts for a delivery).
If your build process do not handle the infrastructure part yet, you won’t be able to find a way to natively integrate IaC. A good start is to create IaC scripts that suits your needs. I guess the "how" you would integrate it depends on which SDLC model your organisation rely on.
In addition to Bruno’s answer, there are 4 paths you can follow for IaaC…
For very simple task write scripts that use the AWS (or an SDK like Boto3 for python). This works up to a point but you’ll quickly start having to write a lot of code to keep track of what you already have running. For example, if you need to have 3 EC2 instances, you’ll need to code your script to check how many you have running already and then launch or terminate instances as needed. For this reason, I wouldn’t recommend scripts for anything other than very simple tasks, quick prototypes or very unusual edge cases.
AWS Cloudformation is AWS’s built-in IaaC service. It allows you to describe the resources you need using YAML documents and it looks after the details of working out what to launch or terminate. I, personally, find the YAML quite hard to work with and I prefer Terraform (see below) however, you will run into a lot of Cloudformation that AWS and others have written so it pays to be familiar with it. One place you would prefer Cloudformation is if you want to make it easy for people to launch your stack on AWS at the click of a button because you can rely on anyone with an AWS account having Cloudformation by default. You can see lots of examples of this on the AWS Quick Starts page
Terraform is the most popular cloud agnostic IaaC solution. Instead of YAML, you define your infrastructure in Terraform’s own language called HCL which provides a leaky abstraction over the AWS CLI (or you can use JSON – useful when you want to generate configurations). Terraform looks after the details of working out what to launch, change and/or terminate just like Cloudformation. One of the major benefits of Terraform is that it works with all major cloud providers so you can define multi-cloud configurations using the same language and tooling
There is a new breed of approach in which you write you IaaC in a normal programming language like Python or Node and submit it to a runtime that does all the hard work of figuring out what needs to chance etc. Examples include AWS CDK and Pulumi but I’ll admit I don’t know all that much about these. I did experiment with Pulumi for a while and found that I didn’t really find much of a practical advantage over Terraform. I suspect this is something that could change as these tools mature and/or I spend a bit more time getting used to their ideoms and best use cases.
Hope this helps