Qloudstat supports a partial transfer metric for S3 and Rackspace Cloudfiles/Akamai endpoints. We determine partial transfers as follows:

• S3. Log files for S3 include the actual transmitted number of bytes. If it differs from the content length the request is counted as partial.
• Rackspace Cloudfiles. Partial transfer counts are based on a feature by Akamai CDN that marks transfers interrupted by the client with a 000 status code in the log file.

These are also taken into account when counting bandwidth and transfer cost metrics.

The latest release of Cyberduck, a cloud storage browser to access S3 or Rackspace accounts, simplifies the setup of access log analytics with Qloudstat with a click of a button. If you already use Cyberduck to configure access logging for CloudFront distributions, S3 buckets or containers in Rackspace Cloudfiles, getting reports in Qloudstat just got a lot easier with the new Read Access for Qloudstat checkbox in the Info panel tab for CDN or S3 buckets.

We are happy to announce the beta of the Qloudstat API today. For the impatient, you find all the technical documentation in our API documentation, including samples. The ReSTful API allows you to request data as JSON or CSV depending on the further processing required. You can query your analytics data by values or by time.

• Queries by value return dimension values (such as the URIs or HTTP status codes) with metrics.
• With a timeline query, requests that match the query filter (such as 404 in the HTTP status code dimension) are grouped by day.

Authentication is implemented using digest access authentication which is both secure and  is implemented in popular tools such as curl and HTTP client libraries in Python, Ruby or Java or any other popular programming languages. This allows you to quickly wrap up a script solution to integrate the data in Qloudstat with your applications. Multiple access tokens can be created and revoked for your account if you want to grant (temporary) read access to your data to third parties.

The Qloudstat API has the same feature set to allow all queries that you get in the web user interface. This feature parity allows to make a valid API call for every single graph that you see in the web interface.

We think the API is an important step to liberate the data in Qloudstat. While previously data could be downloaded from the web interface as a Excel or CSV, the full blown API allows for much better integration in your existing or new workflows.

To get started, copy the API endpoint URL from your configuration and follow the directions to query by time or values per our API documentation. Please don’t hesitate to let us know how it works for you!

We have just added a calculator for custom plans based on the estimated log file volume for enterprise customers which need a higher quota than what we offer in the standard plans.

To make a educated guess about the expected log file size volume, Qloudstat peeks into the logging targets of your configured endpoints to predict the estimated quota requirement and suggest a plan

Containers in Rackspace Cloudfiles have both a download and streaming CDN URL to distribute content. Log events from the HTTP streaming URL for both US and UK containers are now supported in Qloudstat and give an overview of streaming events triggered by clients (such as connect, play, stop and disconnect) for a given URI with hits and bandwidth metrics. No configuration change is required for your existing endpoints.

We have added support for both the new S3 bucket location in the AWS Sydney region following the announcement. This allows to add buckets from ap-southeast-2 to your S3 configurations and read the logging configuration and download log files.

The regular subscription plans in Qloudstat cover analytics as of the signup date onward. We now offer additional one-time payment subscriptions to fetch log files from prior months.

From your account settings, choose the months backwards from your signup date with unprocessed log files you want to fetch.

A preview will be displayed summarizing the total byte count to be fetched for the selected period with the corresponding price plan. The calculation is based on all unprocessed log files of all your configured endpoints.

Constantly adding refinements and new features based on user feedback, the latest version of Qloudstat now allows to invert filters. Select not equals from the Filter dropdown menu to select all values not matching. The exclusion of certain dimension values is certainly handy when wanting to drill down on minor events.

Our recommended way to grant Qloudstat access to your Amazon S3 & CloudFront log files is to create an IAM user with a read-only policy attached. This post is derived from the AWS S3 documentation and CloudFront documentation on IAM policies.

When setting up a new AWS configuration in Qloudstat, you are asked to enter a valid Access Key and Secret Key. This could be your main AWS credentials but this is discouraged. Instead we recommend you to login to the IAM console and create a new user with its dedicated access key.

S3
You can attach the IAM Read Only Policy Template which should suit most needs. A further restricted custom policy with the least grants would be edited like

{
    "Statement":[
        {
             "Effect":"Allow",
             "Action":[
                "s3:GetObject",
                "s3:ListBucket"
             ],
             "Resource":"arn:aws:s3:::logging-target-bucket/*",
             "Condition":{
                "Bool":{
                    "aws:SecureTransport":"true"
                }
             }
        },
        {
            "Effect":"Allow",
            "Action":[
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation",
                "s3:GetBucketLogging"
            ],
            "Resource":"arn:aws:s3:::*",
            "Condition":{
                "Bool":{
                   "aws:SecureTransport":"true"
                }
            }
        }
    ]
}

• To facilitate handling of your buckets in Qloudstat, we recommend to grant the s3:ListAllMyBuckets to the user.
• Grant reading the logging status and location of every bucket.
• Grant listing and fetching files in the target logging bucket named logging-target-bucket. You must repeat this statement for all your logging target buckets or use the wildcard resource name arn:aws:s3:::*
• All communication must be secured using HTTPS.

You can find additional information in the Qloudstat FAQ.

CloudFront
A policy to fetch log files for CloudFront distributions must allow to read your CloudFront distribution status plus fetching the log files from the S3 logging target bucket.

{
    "Statement": [
        {
          "Action": [
            "s3:Get*",
            "s3:List*"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:s3:::logging-target-bucket/*"
        },
        {
          "Action": [
            "cloudfront:Get*",
            "cloudfront:List*"
          ],
          "Effect": "Allow",
          "Resource": "*"
        }
    ]
}

• An asterisk (*) is used as the resource when writing a policy to control access to CloudFront distributions. There are no CloudFront resource ARNs (Amazon Resource Names) for you to use in an IAM policy, because IAM cannot control access to specific CloudFront distributions.
• To facilitate handling of your distributions in Qloudstat, we recommend to grant the cloudfront:ListDistributions to the user. We use a cloudfront:List* wildcard to include both download and streaming (cloudfront:ListStreamingDistributions) API actions.

You can find additional information in the Qloudstat FAQ.

Support was added two days ago for the new Madrid (MAD50) Amazon CloudFront edge location announced by AWS. This affects the request and data transfer pricing metrics and the report for requests split by edge location.

We aim with Qloudstat to provide much better insights than what static reports of access logs can give. We are thrilled to announced today a major milestone with support for two dimensional queries that allow to drill down on a dimension value and plot these against a second dimension to visualize the combined metrics.

The query interface allows to specify filters on both the primary and drill down dimensions:

To illustrate this we select all URIs with Hits and Completed Download metrics that that had a 404 HTTP status code returned.

Other use cases that come into mind are to plot the combinations of URIs and operating systems in a table and compare the user agents accessing the resource over time in a line chart. Or for CloudFront, you can analyze which edge locations were accessed from which country. Or to get technical, you can plot the HTTP operations against the HTTP response code sent to the client.

Open an account and try it with our own data set!

With the latest update to Amazon CloudFront it cache result types from edge locations are now logged and available for analytics in Qloudstat as of today. The new edge result types reported are as follows:

As most demanded by enterprise users, you can now add more users to your Qloudstat account to allow these to access analytics and configurations. You can assign either role to  edit configurations or limit a user to view analytics only. We anticipate this makes the use of Qloudstat inside larger organization easier. Only the initial owner of the account is allowed to add users and invitations are sent to confirm the email address and request a password change for new users.

We have just added a new metric to Qloudstat that allows to track incomplete transfers. The new metrics Partial Transfers and Complete Transfers work for S3 and CloudFiles (Akamai) endpoints.

We have updated the dashboard and reports for analytics of the different dimensions and metrics with a simpler user interface.

• Topmost you find the totals for all metrics for the selected time span including the change in percent to the previous period prominently placed below the endpoint name.

• The redesigned look has much less navigation overhead. Instead of a navigation menu taking screen real estate, a dropdown selector allows to quickly select the dimension you are interested in such as the URI, country or user agent. Use the tabs to switch between metrics.

• Further down above the table listing the dimension values you can select the maximum size of the query result set. To search for specific dimension values, enter a glob pattern (Such as  *.jpg for the URI dimension or 20? for the HTTP Status Code) to only include rows with dimension values that match the input. As always, all charts have an export option to download the full result set as a spreadsheet.

• Scroll down for the line chart with the metric values by time (day, week or month) or a column chart showing the distribution by hours over the queried time period. Select dimension values in the table to compare these with the total.

Stay tuned for more updates in the coming weeks. We will always refine the user interface based on customer feedback. And most notably we have drill down support coming to give insights by combining queries with a second dimension.