Integration with Cyberduck

The latest release of Cyberduck, a cloud storage browser to access S3 or Rackspace accounts, simplifies the setup of access log analytics with Qloudstat with a click of a button. If you already use Cyberduck to configure access logging for CloudFront distributions, S3 buckets or containers in Rackspace Cloudfiles, getting reports in Qloudstat just got a lot easier with the new Read Access for Qloudstat checkbox in the Info panel tab for CDN or S3 buckets.

Rackspace Analytics Setup for Qloudstat in CyberduckS3 Analytics Setup for Qloudstat in Cyberduck.png

For Amazon S3 buckets or CloudFront distributions, this will create a dedicated IAM user with a read only IAM policy for Qloudstat to fetch log files. Unchecking the Read Access for Qloudstat checkbox will remove the IAM user again revoking all access for Qloudstat. When enabled, a clickable link is displayed that redirects to Qloudstat to confirm the new setup.

Confirm Rackspace Setup in Qloudstat

Announcing the Qloudstat API

API

We are happy to announce the beta of the Qloudstat API today. For the impatient, you find all the technical documentation in our API documentation, including samples. The ReSTful API allows you to request data as JSON or CSV depending on the further processing required. You can query your analytics data by values or by time.

  • Queries by value return dimension values (such as the URIs or HTTP status codes) with metrics.
  • With a timeline query, requests that match the query filter (such as 404 in the HTTP status code dimension) are grouped by day.

Authentication is implemented using digest access authentication which is both secure and  is implemented in popular tools such as curl and HTTP client libraries in Python, Ruby or Java or any other popular programming languages. This allows you to quickly wrap up a script solution to integrate the data in Qloudstat with your applications. Multiple access tokens can be created and revoked for your account if you want to grant (temporary) read access to your data to third parties.

The Qloudstat API has the same feature set to allow all queries that you get in the web user interface. This feature parity allows to make a valid API call for every single graph that you see in the web interface.

We think the API is an important step to liberate the data in Qloudstat. While previously data could be downloaded from the web interface as a Excel or CSV, the full blown API allows for much better integration in your existing or new workflows.

To get started, copy the API endpoint URL from your configuration and follow the directions to query by time or values per our API documentation. Please don’t hesitate to let us know how it works for you!

Custom Plan Price Calculator

We have just added a calculator for custom plans based on the estimated log file volume for enterprise customers which need a higher quota than what we offer in the standard plans.

Custom Plan Calculator

To make a educated guess about the expected log file size volume, Qloudstat peeks into the logging targets of your configured endpoints to predict the estimated quota requirement and suggest a plan

Support for Rackspace/Akamai Streaming Logs

Containers in Rackspace Cloudfiles have both a download and streaming CDN URL to distribute content. Log events from the HTTP streaming URL for both US and UK containers are now supported in Qloudstat and give an overview of streaming events triggered by clients (such as connect, play, stop and disconnect) for a given URI with hits and bandwidth metrics. No configuration change is required for your existing endpoints.

Analyze historical log files

The regular subscription plans in Qloudstat cover analytics as of the signup date onward. We now offer additional one-time payment subscriptions to fetch log files from prior months.

From your account settings, choose the months backwards from your signup date with unprocessed log files you want to fetch.

A preview will be displayed summarizing the total byte count to be fetched for the selected period with the corresponding price plan. The calculation is based on all unprocessed log files of all your configured endpoints.

IAM Policies in Depth

Our recommended way to grant Qloudstat access to your Amazon S3 & CloudFront log files is to create an IAM user with a read-only policy attached. This post is derived from the AWS S3 documentation and CloudFront documentation on IAM policies.

When setting up a new AWS configuration in Qloudstat, you are asked to enter a valid Access Key and Secret Key. This could be your main AWS credentials but this is discouraged. Instead we recommend you to login to the IAM console and create a new user with its dedicated access key.

S3
You can attach the IAM Read Only Policy Template which should suit most needs. A further restricted custom policy with the least grants would be edited like

{
    "Statement":[
        {
             "Effect":"Allow",
             "Action":[
                "s3:GetObject",
                "s3:ListBucket"
             ],
             "Resource":"arn:aws:s3:::logging-target-bucket/*",
             "Condition":{
                "Bool":{
                    "aws:SecureTransport":"true"
                }
             }
        },
        {
            "Effect":"Allow",
            "Action":[
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation",
                "s3:GetBucketLogging"
            ],
            "Resource":"arn:aws:s3:::*",
            "Condition":{
                "Bool":{
                   "aws:SecureTransport":"true"
                }
            }
        }
    ]
}
  • To facilitate handling of your buckets in Qloudstat, we recommend to grant the s3:ListAllMyBuckets to the user.
  • Grant reading the logging status and location of every bucket.
  • Grant listing and fetching files in the target logging bucket named logging-target-bucket. You must repeat this statement for all your logging target buckets or use the wildcard resource name arn:aws:s3:::*
  • All communication must be secured using HTTPS.

You can find additional information in the Qloudstat FAQ.

CloudFront
A policy to fetch log files for CloudFront distributions must allow to read your CloudFront distribution status plus fetching the log files from the S3 logging target bucket.

{
    "Statement": [
        {
          "Action": [
            "s3:Get*",
            "s3:List*"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:s3:::logging-target-bucket/*"
        },
        {
          "Action": [
            "cloudfront:Get*",
            "cloudfront:List*"
          ],
          "Effect": "Allow",
          "Resource": "*"
        }
    ]
}
  • An asterisk (*) is used as the resource when writing a policy to control access to CloudFront distributions. There are no CloudFront resource ARNs (Amazon Resource Names) for you to use in an IAM policy, because IAM cannot control access to specific CloudFront distributions.
  • To facilitate handling of your distributions in Qloudstat, we recommend to grant the cloudfront:ListDistributions to the user. We use a cloudfront:List* wildcard to include both download and streaming (cloudfront:ListStreamingDistributions) API actions.

You can find additional information in the Qloudstat FAQ.

Drill down on dimension values

We aim with Qloudstat to provide much better insights than what static reports of access logs can give. We are thrilled to announced today a major milestone with support for two dimensional queries that allow to drill down on a dimension value and plot these against a second dimension to visualize the combined metrics.

The query interface allows to specify filters on both the primary and drill down dimensions:

To illustrate this we select all URIs with Hits and Completed Download metrics that that had a 404 HTTP status code returned.

Other use cases that come into mind are to plot the combinations of URIs and operating systems in a table and compare the user agents accessing the resource over time in a line chart. Or for CloudFront, you can analyze which edge locations were accessed from which country. Or to get technical, you can plot the HTTP operations against the HTTP response code sent to the client.

Open an account and try it with our own data set!

Tracking CloudFront Cache Results

With the latest update to Amazon CloudFront it cache result types from edge locations are now logged and available for analytics in Qloudstat as of today. The new edge result types reported are as follows:

  • Hit: CloudFront served the object to the viewer from the edge cache.
  • Refresh Hit: CloudFront found the object in the edge cache but it had expired, so CloudFront contacted the origin to verify that the cache has the latest version of the object.
  • Miss: The object wasn’t in the edge cache, so CloudFront requested the object from the origin and served it to the viewer.
  • Limit Exceeded: The request was denied because a CloudFront limit was exceeded.
  • Capacity Exceeded: CloudFront returned a 503 error because the edge location didn’t have enough capacity at the time of the request to serve the object.
  • Error: The request resulted in a client error (4xx) or server error (5xx).

You can drilldown these result types by edge locations and URI and vice versa for more insight.

Note: This will be effective September 12th, 2012 when the new log file format includes these changes.

 

Multiple Users

As most demanded by enterprise users, you can now add more users to your Qloudstat account to allow these to access analytics and configurations. You can assign either role to  edit configurations or limit a user to view analytics only. We anticipate this makes the use of Qloudstat inside larger organization easier. Only the initial owner of the account is allowed to add users and invitations are sent to confirm the email address and request a password change for new users.

User Management

Updated look & feel for reports

We have updated the dashboard and reports for analytics of the different dimensions and metrics with a simpler user interface.

  • Topmost you find the totals for all metrics for the selected time span including the change in percent to the previous period prominently placed below the endpoint name.

Total of metrics with change

  • The redesigned look has much less navigation overhead. Instead of a navigation menu taking screen real estate, a dropdown selector allows to quickly select the dimension you are interested in such as the URI, country or user agent. Use the tabs to switch between metrics.

Dimension and Metric Selection

  • Further down above the table listing the dimension values you can select the maximum size of the query result set. To search for specific dimension values, enter a glob pattern (Such as  *.jpg for the URI dimension or 20? for the HTTP Status Code) to only include rows with dimension values that match the input. As always, all charts have an export option to download the full result set as a spreadsheet.

Query Options

  • Scroll down for the line chart with the metric values by time (day, week or month) or a column chart showing the distribution by hours over the queried time period. Select dimension values in the table to compare these with the total.

Timeline with Filter

Stay tuned for more updates in the coming weeks. We will always refine the user interface based on customer feedback. And most notably we have drill down support coming to give insights by combining queries with a second dimension.

 

Distribution of metrics per hour

Qloudstat offers a resolution of all metrics by day. To give you an idea how these are distributed over a given day we now additionally plot a graph of accumulated hits per hour to give you a better insight of access characteristics over the day.

Additionally you can now also display the timeline with a resolution of weeks or months if you want a more generic view of time data.