Analyze historical log files

The regular subscription plans in Qloudstat cover analytics as of the signup date onward. We now offer additional one-time payment subscriptions to fetch log files from prior months.

From your account settings, choose the months backwards from your signup date with unprocessed log files you want to fetch.

A preview will be displayed summarizing the total byte count to be fetched for the selected period with the corresponding price plan. The calculation is based on all unprocessed log files of all your configured endpoints.

IAM Policies in Depth

Our recommended way to grant Qloudstat access to your Amazon S3 & CloudFront log files is to create an IAM user with a read-only policy attached. This post is derived from the AWS S3 documentation and CloudFront documentation on IAM policies.

When setting up a new AWS configuration in Qloudstat, you are asked to enter a valid Access Key and Secret Key. This could be your main AWS credentials but this is discouraged. Instead we recommend you to login to the IAM console and create a new user with its dedicated access key.

S3
You can attach the IAM Read Only Policy Template which should suit most needs. A further restricted custom policy with the least grants would be edited like

{
    "Statement":[
        {
             "Effect":"Allow",
             "Action":[
                "s3:GetObject",
                "s3:ListBucket"
             ],
             "Resource":"arn:aws:s3:::logging-target-bucket/*",
             "Condition":{
                "Bool":{
                    "aws:SecureTransport":"true"
                }
             }
        },
        {
            "Effect":"Allow",
            "Action":[
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation",
                "s3:GetBucketLogging"
            ],
            "Resource":"arn:aws:s3:::*",
            "Condition":{
                "Bool":{
                   "aws:SecureTransport":"true"
                }
            }
        }
    ]
}
  • To facilitate handling of your buckets in Qloudstat, we recommend to grant the s3:ListAllMyBuckets to the user.
  • Grant reading the logging status and location of every bucket.
  • Grant listing and fetching files in the target logging bucket named logging-target-bucket. You must repeat this statement for all your logging target buckets or use the wildcard resource name arn:aws:s3:::*
  • All communication must be secured using HTTPS.

You can find additional information in the Qloudstat FAQ.

CloudFront
A policy to fetch log files for CloudFront distributions must allow to read your CloudFront distribution status plus fetching the log files from the S3 logging target bucket.

{
    "Statement": [
        {
          "Action": [
            "s3:Get*",
            "s3:List*"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:s3:::logging-target-bucket/*"
        },
        {
          "Action": [
            "cloudfront:Get*",
            "cloudfront:List*"
          ],
          "Effect": "Allow",
          "Resource": "*"
        }
    ]
}
  • An asterisk (*) is used as the resource when writing a policy to control access to CloudFront distributions. There are no CloudFront resource ARNs (Amazon Resource Names) for you to use in an IAM policy, because IAM cannot control access to specific CloudFront distributions.
  • To facilitate handling of your distributions in Qloudstat, we recommend to grant the cloudfront:ListDistributions to the user. We use a cloudfront:List* wildcard to include both download and streaming (cloudfront:ListStreamingDistributions) API actions.

You can find additional information in the Qloudstat FAQ.

Drill down on dimension values

We aim with Qloudstat to provide much better insights than what static reports of access logs can give. We are thrilled to announced today a major milestone with support for two dimensional queries that allow to drill down on a dimension value and plot these against a second dimension to visualize the combined metrics.

The query interface allows to specify filters on both the primary and drill down dimensions:

To illustrate this we select all URIs with Hits and Completed Download metrics that that had a 404 HTTP status code returned.

Other use cases that come into mind are to plot the combinations of URIs and operating systems in a table and compare the user agents accessing the resource over time in a line chart. Or for CloudFront, you can analyze which edge locations were accessed from which country. Or to get technical, you can plot the HTTP operations against the HTTP response code sent to the client.

Open an account and try it with our own data set!

Tracking CloudFront Cache Results

With the latest update to Amazon CloudFront it cache result types from edge locations are now logged and available for analytics in Qloudstat as of today. The new edge result types reported are as follows:

  • Hit: CloudFront served the object to the viewer from the edge cache.
  • Refresh Hit: CloudFront found the object in the edge cache but it had expired, so CloudFront contacted the origin to verify that the cache has the latest version of the object.
  • Miss: The object wasn’t in the edge cache, so CloudFront requested the object from the origin and served it to the viewer.
  • Limit Exceeded: The request was denied because a CloudFront limit was exceeded.
  • Capacity Exceeded: CloudFront returned a 503 error because the edge location didn’t have enough capacity at the time of the request to serve the object.
  • Error: The request resulted in a client error (4xx) or server error (5xx).

You can drilldown these result types by edge locations and URI and vice versa for more insight.

Note: This will be effective September 12th, 2012 when the new log file format includes these changes.

 

Multiple Users

As most demanded by enterprise users, you can now add more users to your Qloudstat account to allow these to access analytics and configurations. You can assign either role to  edit configurations or limit a user to view analytics only. We anticipate this makes the use of Qloudstat inside larger organization easier. Only the initial owner of the account is allowed to add users and invitations are sent to confirm the email address and request a password change for new users.

User Management

Updated look & feel for reports

We have updated the dashboard and reports for analytics of the different dimensions and metrics with a simpler user interface.

  • Topmost you find the totals for all metrics for the selected time span including the change in percent to the previous period prominently placed below the endpoint name.

Total of metrics with change

  • The redesigned look has much less navigation overhead. Instead of a navigation menu taking screen real estate, a dropdown selector allows to quickly select the dimension you are interested in such as the URI, country or user agent. Use the tabs to switch between metrics.

Dimension and Metric Selection

  • Further down above the table listing the dimension values you can select the maximum size of the query result set. To search for specific dimension values, enter a glob pattern (Such as  *.jpg for the URI dimension or 20? for the HTTP Status Code) to only include rows with dimension values that match the input. As always, all charts have an export option to download the full result set as a spreadsheet.

Query Options

  • Scroll down for the line chart with the metric values by time (day, week or month) or a column chart showing the distribution by hours over the queried time period. Select dimension values in the table to compare these with the total.

Timeline with Filter

Stay tuned for more updates in the coming weeks. We will always refine the user interface based on customer feedback. And most notably we have drill down support coming to give insights by combining queries with a second dimension.

 

Distribution of metrics per hour

Qloudstat offers a resolution of all metrics by day. To give you an idea how these are distributed over a given day we now additionally plot a graph of accumulated hits per hour to give you a better insight of access characteristics over the day.

Additionally you can now also display the timeline with a resolution of weeks or months if you want a more generic view of time data.

Monitoring with threshold alerts

We are constantly working on the feature set of Qloudstat and today we are introducing new monitoring capabilities that are very useful when you need to track the usage of resources served by Amazon S3 or a CDN.

Qloudstat now allows to configure alerts for a given lower or upper threshold triggered by a dimension and metric. Optionally a filter for the dimension value can be set. Once the threshold for the given period is reached you will get notified by email. The next alert will only be sent again after the period has been elapsed. The accumulation for the configured dimension and metric is reset after each period.

Monitoring threshold with alerts

 

Consider the following usage scenarios where this becomes very useful:

  • Alert when the cost per month reaches a certain limit imposed for a country.
  • Get an alert for 404 HTTP error codes to get notified as soon as possible when you tried to serve a broken link.
  • Get notified when a file is no more downloaded with a lower limit and a filter for the URI dimension.

There is no limit for the number of alerts and they can be configured for daily, weekly or monthly periods.

 

Filtering dimension values

Qloudstat allows you to draw quick conclusions about individual dimension values (such as a single resource in your container) by selecting an entry in the value result list. We now also allow to filter these table results with a custom input supporting the ? and * glob patterns to match dimension values that fit the input pattern.

Glob Pattern for URIs

To give an example consider you want to have the metrics for all files that are stored in a container with a given prefix or suffix only. Navigate to the URI dimension analytics page of your report and enter a glob pattern such as */5902/*.png to only list URIs that contain this pattern in their path. This is quite useful to filter versioned resources or limit the result list to files that are served from a specific location in your bucket or CDN origin.

The dimension values list supports multi-selection which gives you a simple way to bring the selected values in relation. When selecting multiple values a pie chart next to the table shows up with the percentage of each value compared to all selected ones.

Update: The filter with search patterns now also works for enumerated dimension values such as HTTP status code.