Storage Solutions in AWS

In general a storage type can be classified into 3 types:

  1. Block storage.
  2. File Storage.
  3. Object Storage.

1.Block Storages:

EBS (Elastic Block Storage):

  • Its a network drive (i.e., not a physical drive) and hence there is a latency.
  • You can detach it from one EC2 and attach it to another.
  • Its locked to an AZ. To move an EBS volume to another AZ you have to snapshot it.
  • A instance can have multiple EBS attached.
  • Its suitable to host databases and OS.
  • Whenever you create an EC2 instance an default Block volume of 8 GB is attached to the EC2 instance, that is called as the Root volume.
  • By default, the Root EBS volume is deleted when an EC2 instance is terminated.
  • By default, any other attached EBS volume is not deleted when an EC2 instance is terminated.
Solid Disk Drives (SDD) volume :

Hard Disk Drives (HDD) volume :
EBS- Multi Attach :
  • Multi attach EBS means the same EBS volume can be attached to multiple EC2 instances in the same AZ.
  • Its only available for io1 and io2 family of EBS.

EC2 Instance Store:

  • Its a physical disk that is attached to an EC2 instance. Its different than the EBS which is basically a Network disk which are good but has little network lagging.
  • This kind of storage being used for high throughput like cache, buffer, scratch data, etc.
  • This volume type is ephemeral (temporary), it gets deleted once you shutdown the EC2 instance.
  • All AMIs (Amazon Machine Image) does not support this type of storage. FYI, the t2.micro free tier type does not have this storage type.

2.File storages:

  • eg. – a shared file system (where other users can access files).
  • Options in AWS:
    • EFS (Elastic File System) (for Linux only)
    • FSx (File Storage for X) (for 3rd party high-performance file systems)

EFS (Elastic File System):

  • It is a managed NFS (Network File System) that can be mounted on many EC2.
  • Can be used by multiple EC2 instances from multiple AZs (unlikely in EBS).
  • Also it can be mounted to on-prem using VPN and Direct Connect (DX).
  • Only for Linux based AMIs.
  • Its expensive (3 times of gp2 EBS), pay per use (you don’t have to provision capacity in advance).
  • Use cases: content management, wordpress, data sharing, web serving.
  • POSIX file system (~Linux) that has a standard file API.
  • It scales automatically, no prior capacity reservation required and its pay-per-use.

FSx :

  • FSx helps to launch 3rd party high-performance file systems on AWS as a fully managed service.
  • For eg.:
    1. FSx for Windows File Server (Supports SMB protocol)
    2. FSx for Lustre (Lustre means Linux cluster) (Supports POSIX compliant protocol)
    3. FSx for NetApp ONTAP (Supports SMB, NFS, iSCSI protocols)
    4. FSx for OpenZFS (Supports NFS protocol)

3.Object storages:

Object storage examples are S3. For a detailed review of S3 and its types please read here.

Storage Extras in AWS

Lets see some extra services related to storage in AWS.

AWS Storage Gateway:

  • AWS Storage Gateway helps in maintaining hybrid storage – some part of your data on cloud and some part on-premises. (Compare it with AWS DataSync)
  • Use cases:
    • -Regulatory compliance.
    • -Disaster recovery.
  • Types of Storage Gateway:
    • S3 File Gateway
    • FSx File Gateway
    • Volume Gateway
    • Tape Gateway

All these gateway maintain local cache on-premises for frequently accessed data.

S3 File Gateway:

  • For native access to S3 files on cloud.
  • Uses NFS/SMB protocol (Network File System/Server Message Block).

FSx File Gateway:

  • For native access to AWS FSx (Windows File Server)
  • Uses SMB protocol.

Volume Gateway:

  • Block storage using iSCSI protocol backed by S3.
  • Backed by EBS snapshots.

Tape Gateway:

  • If your company have data present in physical tapes, then you can take help of Tape Gateway to backup of those onto cloud.
  • It uses iSCSI VTL protocol. VTL (Virtual Tape Library) Is a mix of S3 and Glacier.

Summary pic of Storage Gateway:

AWS DataSync :

  • It help in completely moving the data (with end to end in-flight data encryption) from one place to other. (Compare it with StorageGateway)
  • To completely move large amount of data to and from below places :
    1. On-premises to AWS
    2. AWS to AWS
  • Can synchronize to :
    1. S3 (any storage class – including Glacier)
    2. EFS
    3. FSx
  • Don’t confuse DataSync with AppSync (which uses GraphQL to query data).

Storage Gateway Vs DataSync :

Storage Gateway DataSync
It is a hybrid cloud storage service that gives your on-premises network an virtually unlimited cloud storage by linking it to S3.It is a online data transfer service that simplified and automates the copying of large amounts of data to and from on-premises and cloud over internet or Direct Connect.
Standard storage protocol (SMB and NFS) and iSCSI used.Standard storage protocol (SMB, NFS) used.