Storage: Block, Object, File & Network

Right type

Block Storage (Attached storage)

Instance Store

EC2 instances have attached devices

EBS Tiers

SSD

• General Purpose
  • Unless you know your IO Operations are throttling or stopping your performance you should probably be using this. GP2 volumes scale operations with size. Many times its more cost-effective to over-provision on volume for a GP2 device than to use Provisioned IOPS if you just need some more performance.
• Provisioned IOPS
  • If you really need some high performing disks then you probably need to use this. Take some time to figure out what you actually need and put some tooling in place to help you re-evaluate or monitor progress. In other words, make sure you are using what you are paying for.
• In some edge cases you may want to evaluate if you are horizontally scaled when the problem could be solved with pIOPS. Rare but this happens.

HDD

• Throughput Optimized
  • If your servers don’t have to hit their disk during normal operation (aside from boot) then consider this as a cost-effective alternative to SSD storage. These are analogous to spinning disk drives, they won’t be performant enough for disk-based workloads. Common use-cases: OS, Logs, scripts/tools.

If you elect to use these disks keep in mind these two implications:

Scaling activities may be slower if these are used for Boot Volumes

High IO Wait can impact performance across the board - potentially triggering scaling activities or tricking you into provisioning more compute than you need!

Cold HDD

File/Network Storage

• EFS
• FSx

Augmenting Storage > AWS Storage Gateway

• Tape Gateway
• File Gateway
• Volume Gateway
• High Availability

Recommendations:

For File/Volume Gateway: Look at this service as a temporary solution to help you while you are migrating or to solve specific pain points with difficulty to move workloads. EBS backed services will be quite expensive, it may make sense in these cases to use something like AWS Cloud Endure to mirror your servers entirely instead of just the storage volume. This would bring the added value of giving you a failover point if your services can be configured to work in AWS.

For Tape Gateway: We don’t have guidance here but we are interested in feedback or guest contributors!

Migration Storage

Cloud Data Migration Services

Looking for contributors for this section. I don’t have much experience with FSx but would like to include some guidance!

Object Storage

S3

Surely, you know that S3 is the cornerstone to any AWS implementation but did you know it is recognized as The Greatest Cloud Service of all Time? S3 is a powerful tool that we recommend learning how to extract the most from. For its traditional purposes familiarize yourself with lifecycle management, versioning, and the types available such as Glacier or Infrequent Mention I mean Access.

Try to use more S3 if:

You are storing log files on individual hosts but aren’t ready to invest in aggregation in something like ElasticSearch or log streaming with Kinesis or EventBridge (CloudWatch Events).

• You have any needs of delivering files, storing files, sharing files that currently rely on block type storage. Examples:
• You currently hosting and serving static assets from fileservers with EBS volumes.

You keep and rotate log files on your servers.

Alternative use cases for S3:

So S3 is great at storing… whatever you want, but you can also use S3 as a low-cost option for:

• A simple queue
• An event source
• A simple webserver (think static websites and assets)
• Version control
• Url signing (more control over retrieval)

Often times if you can use S3 to get the job done it may just be your most cost-effective way to do so. Put this high on your list!

Lifecycle

S3 policies provide a set of tools to manage your data’s lifecycle. Life cycle rules provide two important capabilities:

Transition actions: Use these to change the storage class of your objects.

Expiration actions: Use these to schedule automatic deletion of your items

We recommend using these capabilities to optimize your storage spend by using the appropriate classes and storing only what you must. Be careful however when using these rules that you are aware of minimum storage durations on storage classes evaluate if it is worth it to pay the minimum storage duration pro-rated cost if you delete or transition objects before maturation. You can find these costs and charges here: https://aws.amazon.com/s3/pricing/

Details on how to use lifecycle management are found here: https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html