Goodbye AWS Lambda Snapshots, Hello AWS Data Lifecyle Manager?

If you have been keeping up with Amazon Web Services (AWS) news, you should know that AWS released a new feature called Data Lifecyle Manager in 2018 to automate your snapshots based on a lifecycle policy.

Previously automated snapshots could do taken through Cloudwatch Events to call the CreateSnapshot API. However this had a major downside that EBS volume tags are not copied over. Therefore, many companies opted to create their Lambda to be called through Cloudwatch Events. This ensured that any volumes created from the snapshots will contain the original tags as long as it is in the Lambda.

Snapshot creation led to another problem, which is cleaning up after them. You will have to create another lambda to be ran periodically to delete the old snapshots, or insert logic in your original lambda to clean up after a certain snapshot creation count has been reached.

Insert Data Lifecycle Manager (DLM)! It is AWS’s automated solution to snapshot management. It is also free to use, but the underlying snapshots does cost storage costs.

Example Template

So what is required? There are only two resources required within Cloudformation which is the IAM service role the Lifecycle Policy uses and the Lifecycle policy for the snapshots itself.

Below is a sample cloudformation for the service role that would need to be created for DLM to use.

DLMSnapshotRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        -
          Effect: "Allow"
          Principal:
            Service:
              - "dlm.amazonaws.com"
          Action:
            - "sts:AssumeRole"
    ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSDataLifecycleManagerServiceRole

After the service role has been created, you can create the actual DLM lifecycle policy. This is a sample lifecycle policy which starts backup at 11am and takes a snapshot every 12 hours.

  BasicLifecyclePolicy:
    Type: "AWS::DLM::LifecyclePolicy"
    Properties:
      Description: "Lifecycle Policy using CloudFormation"
      State: "ENABLED"
      ExecutionRoleArn: !Sub
        - arn:aws:iam::${AWS::AccountId}:role/${DlmRole}
        - { DlmRole: !Ref DLMSnapshotRole }
      PolicyDetails:
        ResourceTypes:
          - "VOLUME"
        TargetTags:
          -
            Key: "Backup"
            Value: "True"

        Schedules:
          -
            Name: "Daily Snapshots"
            CreateRule:
              Interval: 12
              IntervalUnit: "HOURS"
              Times:
                - '11:00'
            RetainRule:
              Count: 5
            CopyTags: true

The Cloudformation YAML above contains some hardcoded values. I have attached a more generalized template here.

Limitations

There are a few limitations for Data Lifecycle Manager that must be considered before using this over Lambda snapshots.

First the interval between snapshots can only be 12 or 24 hours. You cannot get around it by setting multiple start times either. As long as your RPO is 12 hours or more, then this will not be an issue for you.

Another limitation is if anyone of your tags are already part of another lifecycle policy, then you cannot use the same tag for a second policy. This doesn’t really allow for that much customization based on the same key value pair for your snapshots, but you can stagger your snapshots with different values for your tags.

The lifecycle policy also takes about an hour to take effect. If you need to have a snapshot right away, it is recommended to do this manually or trigger your snapshot Lambda.

Conclusion

Data Lifecycle Manager is a very simple and clean way of managing your snapshots. If you can handle the the limitations, this is definitely recommended over Lambda snapshots as it is much easier, cleaner, and more cost efficient. Go try Data Lifecycle Manager today, and let me know your thoughts in the comments!

Further Resources

AWS DLM Article