PROJECT: TESTING BACKUP AND RESTORE
(AWS Backup, SNS, Lambda and EC2)
For many organizations, the data that they possess is one of their most valuable assets for day-to-day operations, continuation and future growth. Such organizations place a great deal of trust in a reliable Information Technology (IT) department to protect such valuable assets through daily/weekly data backups, restorations when required and most likely a regular test plan to ensure data can be protected or recovered efficiently and reliably when needed.
In my backup plan below, I’ll demonstrate how backing up data and testing restores in the cloud is easier when compared to on-premises datacenters by automating the process with an appropriate notification system that will ensure data is backed up frequently, tested to ensure expected recovery and the appropriate people are notified in case of failure.
Part 1: DEPLOY THE INFRASTRUCTURE
Log into your AWS Console (AdministratorAccess permissions) and launch CloudFormation. The CloudFormation service can be found in the Management & Governance tree; or you can use the search feature for discovery.
Let’s follow some steps to create the CloudFormation Stack.
- Click Create Stack, after doing so, select your options as offered:
a. Prepare template.
b. Template source.
c. Amazon S3 URL
(Your screen should reflect the options chosen below).
Notice: The “View in Designer” box is not hiding any text; the S3 URL simply wraps to the next line.
2. Click Next.
Let’s provide some Stack details for Stack name and Parameters:
1. Fields to be filled in:
a. Type your desired Stack name, under Stack name.
b. Select the drop-down menu to select an Availability Zone.
c. I’d recommend leaving the default LatestAmild as is; this retrieves the latest AMI ID for Amazon Linux 2
d. For NotificationEmail, provide an email address to a mailbox that will receive notifications.
2. Click Next, then Next again.
- Scroll to the bottom, select the check box for acknowledgement for the statement above it.
- Click Create stack.
Note: Once Stack creations starts, monitor the issued email inbox. You should receive an email from SNS with the subject “AWS Notification — Subscription Confirmation” Click the link for Confirm subscription of the email to the SNS topic.
3. Go to the Outputs section for your Stack, document the Key and Value for each output; will appear like this;
4. After documenting the Key and Values, you can view the simple application running on your instance by clicking the URL value above.
Part 2: CREATE BACKUP PLAN
Like Part 1; let’s launch the service we’ll be using here, the AWS Backup Console. Also as in Part 1, you can access this service from its tree which is Storage or you can use the Search feature of the AWS Management Console. Click the Create New Plan radio button.
1. Let’s create a Backup Plan; complete these options.
· Select Build a New Plan
· Complete Backup plan name (Your choice)
Let’s continue by setting the backup configuration, complete your configuration as shown below;
2. Once frequency has been established, you will need to specify a backup window — a period of time during which data is being backed up from your data sources. Backups could cause you data sources to be temporarily unavailable, be mindful as to when you run them; business hours vs. scheduled downtime.
If your environment doesn’t have a requirement to store and recover data after extended periods of time, be sure to set the values for Transition to cold storage to Never. Click Create Plan.
Navigate back to the AWS Backup pop-out menu.
3. Click backup vaults, then Create Backup vault.
Note: If you are backing up multiple workloads, it is recommended to use different Backup Vaults.
Complete the backup vault name (Your choice) and select the KMS encryption master key as shown in the following diagram.
Tags are optional, Click Create Backup Vault.
Choose Backup Plans from the side Pop-out menu.
Click the Backup plan you created earlier.
After clicking your backup plan, scroll to the Resource Assignments section, Click Assign resources to display as shown.
Complete the Resource assignment name (Your choice), leave the IAM role as default.
Complete the Assign resources section
· Assign by
· Key
· Value
Note: Tags are case sensitive as well, be mindful of their settings.
Click Assign resources.
PART 3: ENABLE NOTIFICATIONS
We’re setting up notifications to be aware of events within the workload which is easily achieved. AWS Backup leverages AWS SNS to send notifications related to backup activities that are occurring. This will allow visibility into backup job statuses, restore job statuses, or any failures that may have occurred, allowing your Operations teams to respond appropriately.
1. Run your CMD with Administrator privileges to access the resources through the AWS CLI, click here to select the AWS CLI version, the OS platform you’re using as host, and to download/install the tool.
2. You can run the “aws –version” command if you need to verify the version you installed.
3. You can run the “aws configure” command if you need to configure your IAM user to have the privileges to execute an access key and secret key.
a. You can also do this from your graphical user interface in the IAM dashboard.
Edit and run the command below to enable notifications with messages published to the SNS Topic every time a backup or restore is completed to ensure your operation team is aware of any failures.
Note: Use your settings from outputs arn values from your SNSTopicARN Key in your CloudFormation stack.
aws backup put-backup-vault-notifications — region us-east-1 — backup-vault-name BACKUP-LAB-VAULT — backup-vault-events BACKUP_JOB_COMPLETED RESTORE_JOB_COMPLETED — sns-topic-arn <YOUR SNS TOPIC ARN>
PART 4: TEST RESTORE
A backup of a data source is only useful if data can be restored from it, if backups aren’t tested regularly, you might find yourself in a very bad situation where your organization require data sources to be recovered but the backups are faulty and restoring data is not possible. To avoid such situations, backups taken should always be tested to ensure they can be used to recover data.
Let’s test the recovery of a data source:
1. Create and On-Demand Backup from the AWS backup dashboard.
2. Let’s fill in the settings for the following, compare to the diagram that follows;
o Resource type = EC2 type used for the Backup plan
o Instance ID = Is the Physical ID for the resource used
o Backup window = Timeframe to run jobs
o Backup vault = Storage location
o IAM role = Permissions granted to perform job
3. When the job is created, click on Jobs then select Backup jobs
4. Check the Job ID for the details to the Recovery Point ARN, Resource ID and Resource type.
5. You will receive three emails from AWS, each contain details to the Backup job, you can compare them against your configurations in steps 3 through 6.
6. This is my Restore job status and details, I wasn’t lucky enough to catch it as it was running however you can see it was completed successfully.
CONCLUSION:
After all aspects of your Infrastructure has been deployed, Backup Plan(s) has been created and Notifications setup and each has been tested, you can view the logs to what’s happening in your environment to assist you with identifying success and failures more quickly in a fast pace production environment with CloudWatch.
1. Open the AWS CloudWatch console.
2. Click on Log groups under Logs from the left pop-out menu.
3. Under Log group, click the group there to view the streams,
4. Click each log stream to view the output of the Lambda function’s execution to understand the different steps performed by the function to automate this process.
Best Practices Implemented:
1. Identify all data that needs to be backed up and perform backups or reproduce the data from sources.
2. Perform data backup automatically or reproduce the data from sources automatically.
3. Perform periodic recovery of the data to verify backup integrity and processes.
NOTE: Some formatting and code may be unaligned and misrepresented, if you need clarification, please reach out for a .PDF version of this post.
