IAM Policies in Depth

Our recommended way to grant Qloudstat access to your Amazon S3 & CloudFront log files is to create an IAM user with a read-only policy attached. This post is derived from the AWS S3 documentation and CloudFront documentation on IAM policies.

When setting up a new AWS configuration in Qloudstat, you are asked to enter a valid Access Key and Secret Key. This could be your main AWS credentials but this is discouraged. Instead we recommend you to login to the IAM console and create a new user with its dedicated access key.

S3
You can attach the IAM Read Only Policy Template which should suit most needs. A further restricted custom policy with the least grants would be edited like

{
    "Statement":[
        {
             "Effect":"Allow",
             "Action":[
                "s3:GetObject",
                "s3:ListBucket"
             ],
             "Resource":"arn:aws:s3:::logging-target-bucket/*",
             "Condition":{
                "Bool":{
                    "aws:SecureTransport":"true"
                }
             }
        },
        {
            "Effect":"Allow",
            "Action":[
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation",
                "s3:GetBucketLogging"
            ],
            "Resource":"arn:aws:s3:::*",
            "Condition":{
                "Bool":{
                   "aws:SecureTransport":"true"
                }
            }
        }
    ]
}
  • To facilitate handling of your buckets in Qloudstat, we recommend to grant the s3:ListAllMyBuckets to the user.
  • Grant reading the logging status and location of every bucket.
  • Grant listing and fetching files in the target logging bucket named logging-target-bucket. You must repeat this statement for all your logging target buckets or use the wildcard resource name arn:aws:s3:::*
  • All communication must be secured using HTTPS.

You can find additional information in the Qloudstat FAQ.

CloudFront
A policy to fetch log files for CloudFront distributions must allow to read your CloudFront distribution status plus fetching the log files from the S3 logging target bucket.

{
    "Statement": [
        {
          "Action": [
            "s3:Get*",
            "s3:List*"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:s3:::logging-target-bucket/*"
        },
        {
          "Action": [
            "cloudfront:Get*",
            "cloudfront:List*"
          ],
          "Effect": "Allow",
          "Resource": "*"
        }
    ]
}
  • An asterisk (*) is used as the resource when writing a policy to control access to CloudFront distributions. There are no CloudFront resource ARNs (Amazon Resource Names) for you to use in an IAM policy, because IAM cannot control access to specific CloudFront distributions.
  • To facilitate handling of your distributions in Qloudstat, we recommend to grant the cloudfront:ListDistributions to the user. We use a cloudfront:List* wildcard to include both download and streaming (cloudfront:ListStreamingDistributions) API actions.

You can find additional information in the Qloudstat FAQ.

Drill down on dimension values

We aim with Qloudstat to provide much better insights than what static reports of access logs can give. We are thrilled to announced today a major milestone with support for two dimensional queries that allow to drill down on a dimension value and plot these against a second dimension to visualize the combined metrics.

The query interface allows to specify filters on both the primary and drill down dimensions:

To illustrate this we select all URIs with Hits and Completed Download metrics that that had a 404 HTTP status code returned.

Other use cases that come into mind are to plot the combinations of URIs and operating systems in a table and compare the user agents accessing the resource over time in a line chart. Or for CloudFront, you can analyze which edge locations were accessed from which country. Or to get technical, you can plot the HTTP operations against the HTTP response code sent to the client.

Open an account and try it with our own data set!

Tracking CloudFront Cache Results

With the latest update to Amazon CloudFront it cache result types from edge locations are now logged and available for analytics in Qloudstat as of today. The new edge result types reported are as follows:

  • Hit: CloudFront served the object to the viewer from the edge cache.
  • Refresh Hit: CloudFront found the object in the edge cache but it had expired, so CloudFront contacted the origin to verify that the cache has the latest version of the object.
  • Miss: The object wasn’t in the edge cache, so CloudFront requested the object from the origin and served it to the viewer.
  • Limit Exceeded: The request was denied because a CloudFront limit was exceeded.
  • Capacity Exceeded: CloudFront returned a 503 error because the edge location didn’t have enough capacity at the time of the request to serve the object.
  • Error: The request resulted in a client error (4xx) or server error (5xx).

You can drilldown these result types by edge locations and URI and vice versa for more insight.

Note: This will be effective September 12th, 2012 when the new log file format includes these changes.