Monitoring and Metrics Exam Tips for S3

As part of your Sysops Administration duties, you need to ensure you monitor all your core AWS Services.  This includes using CloudWatch and CloudTrail services to monitor your Simple Storage Services (S3).

Automated Tools

• CloudWatch – You can monitor metrics over time and identify trends and configure Amazon SNS to send alerts as and specific thresholds are reached.  You can also configure Autoscaling Policies to perform an action based on the value of the metric
• CloudTrail Log Monitoring – You can share log files between accounts, monitor trails in real time or send them to CloudWatch Logs.

 

CloudWatch Metrics

Two ways into which to use CloudWatch with Amazon S3:

• Daily Storage Metrics for Buckets – monitor bucket storage using CloudWatch. Storage metrics are reported once a day and are provided at no cost
• Request Metrics – Monitor S3 requests to identify and act on operational issues. You can configure metrics at 1-minute intervals.  Key Points to note:
  • Request Metrics are reported for all object operations
  • 1-minute metrics are available at bucket level
  • Use filters for metrics collected using prefix or object tags
• Note: CloudWatch statistics are retained for 15 months

 

Key Metrics for Buckets – Daily Storage Metrics

• BucketSizeBytes – Amount of data in bytes stored in a bucket in Standard, Standard IA, RRS Class
• NumberofObjects – Total number of objects in bucket for all classes except Glacier

 

Key Metrics – Request Metrics

• AllRequests – Total number of HTTP requests. You can use a filter to focus on specific type of objects
• GetRequests – Total number of HTTP Get requests made for objects in a Bucket. Note it does not include list operations.
• PutRequests – Total number of HTTP PUT requests made for objects in bucket
• DeleteRequests – Total number of HTTP DELETE requests
• HeadRequest – Total number of HTTP HEAD requests to a Bucket
• PostRequest – Total number of HTTP POST requests to a Bucket
• ListRequest – Total number of HTTP Requests that list the contents of Bucket
• BytesDownloaded – Number of Bytes downloaded for requests
• BytesUploaded – Number of bytes uploaded that contain a request
• 4xxErrors – Number of HTTP 4xx client errors. Average will show the error rate and sum-statistics will show the count of the type of error
• 5xxErrors – Number of HTTP 5xx errors made to an S3 Bucket
• FirstByteLatency – The per request time to complete request being received by an S3 bucket
• TotalRequestLatency – The elapsed per request time from the first byte to the last byte sent to an s3 bucket

 

Amazon S3 CloudWatch Dimensions

There are three Dimensions used to filter S3 metrics:

• BucketName – Name of the bucket
• Storage Type – StandardStorage, StandardIAStorage, ReducedRedundancyStorage. AllStoragTypes. Note this does not include Glacier Storage Class
• FilterID – filter metrics based on prefix or tags etc.

 

Metric Configuration for S3 Buckets

You can configure CloudWatch to receive 1-minute metrics for Amazon S3.  You must create a metrics configuration for a bucket if you want to get the CloudWatch request metrics for the objects in that bucket.

Key Points to note:

• You can have up to 1000 metrics configuration per bucket.
• Metrics configurations are only necessary for bucket-level and object-level request metrics. The bucket-level daily storage metrics are always turned on and are provided at no additional cost.
• Choose which objects in a bucket are part of which metrics configurations by using filters such as prefix and object tags.
• Each metrics configuration reports all available request metrics.
• Metrics configurations are also reported for operations that list bucket contents, like Get-Bucket (List Objects), GET Bucket Object versions, and List Multipart Uploads, but are not reported as the other operations on buckets that access or modify bucket sub-resources.

 

CloudTrail with Amazon S3

You can use CloudTrail to capture specific API calls made to Amazon S3 from your AWS account and deliver the log files to an Amazon S3 bucket. CloudTrail captures API calls made from the Amazon S3 console or from the Amazon S3 API.

With the logs, you can determine what request was made to Amazon S3, the source IP address from which the request was made, who made the request, when it was made, and so on. You can then track changes made to your AWS resources and to troubleshoot operational issues. CloudTrail makes it easier to ensure compliance with internal policies and regulatory standards.