Query Strings in URIs

It is a common practice to include query strings in URIs pointing to CloudFront or S3 resources to get a better understanding of the popularity of different sources for your content. Qloudstat visualizes these query strings by hits or bandwidth usage over time.

For CloudFront, any query string appended to an URI is logged and then stripped when fetching the origin resource. AWS had a blog post when introducing this feature. Arbitrary query strings are supported. The same applies for Rackspace Cloudfiles with a Akamai CDN setup.

For S3, you must prepend query strings in URIs with “x-“. More details from the S3 documentation:

You can include custom information to be stored in the access log record for a request by adding a custom query-string parameter to the URL for the request. Amazon S3 will ignore query-string parameters that begin with “x-“, but will include those parameters in the access log record for the request, as part of the Request-URI field of the log record. For example, a GET request for “s3.amazonaws.com/mybucket/photos/2006/08/puppy.jpg?x-user=johndoe” will work the same as the same request for “s3.amazonaws.com/mybucket/photos/2006/08/puppy.jpg”, except that the “x-user=johndoe” string will be included in the Request-URI field for the associated log record. This functionality is available in the REST interface only.

Purge log files

To enable purging log files after analytics you have to setup IAM (Identity and Access Management) in AWS to allow Qloudstat to delete log files in your account. It is best to restrict such write application to the least minimum required.

Below a code snippet with an example policy to attach to the user setup in IAM in addition to the existing read-only policy that you have configured to allow Qloudstat to fetch log files.

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:DeleteObject",
      "Resource": "arn:aws:s3:::bucket/loggingprefix*"
    }
  ]
}

Login to your Qloudstat account to edit your Amazon S3 or CloudFront configuration, where we display this policy preconfigured for your bucket names next to the option to toggle purge for log files.

Choose a Logging Target

We want to share the following best practice when configuring a S3 bucket or CloudFront distribution for logging.

  • The logging target should be another S3 bucket than the origin. Otherwise your report will include requests for log files delivered by Amazon and download requests for log files.
  • Use a different logging target bucket for every S3 bucket and CloudFront distribution. Or at least choose a different logging target prefix. This will help Qloudstat to fetch your log files more efficiently.

You can use Cyberduck or the AWS Console to set up the logging configuration.

Tool of choice for Cloudfiles Akamai CDN

Qloudstat is part of the cloud tools ecosystem supporting the Rackspace cloud beside Amazon (AWS) and Google Storage and today prominently featured as a cloud tool application of choice.

When adding a configuration for Rackspace CloudfilesQloudstat supports both the US and UK API endpoints. Just choose from the drop down menu where your account is set up.

 

Our Price Plans

Qloudstat is available with five different price plans to choose from because one size doesn’t fit all. The free plan allows to use Qloudstat at zero costs for sites with less than 100’000 hits per month. The large plan allows as much as 5GB of log data per month. If you need even more, choose the enterprise plan or contact us for specific needs.

The price plans differ based on the following criteria:

  • Endpoints. An endpoint refers to a bucket (S3), distribution (CloudFront) or container (Rackspace). We have just updated our plans to allow an  unlimited (∞) number of configurable endpoints except the free plan. You can freely organize your data in as many endpoints you want without any restriction.
  • Volume per month. Each plan has an upper limit for the volume (number of bytes) of your log files we process. We sum up the number of uncompressed bytes of each log file processed on a monthly base. After a month we reset our counters. If you hit the limit of your plan we temporarily suspend the fetching of new log files until you upgrade your subscription plan. With that said you can always start with the free plan and align your subscription with your changing needs.
  • Retention. We store all your processed log files for at least 36 months. That gives you the possibility to gain a deep insight on how your files have been accessed  over a long time period. Please contact us if you are interested in an even higher retention.
  • Resolution. Log files are processed with a sampling rate of 1 day with a summed up value for each dimension, metric and day.

Note that there are no long term contracts. Once subscribed to a plan, you can cancel, downgrade and upgrade at any time.

Contact us if you are confused which plan to choose. We will help you figure out the best option for you.

Real Artists Ship

After a private beta phase of three months with intensive testing and refinements, we are pleased to announce Qloudstat is now available to the general public. Sign up today! You will be up and running within minutes. Qloudstat does all the work behind the scenes without any administration, maintenance and infrastructure required by you. We have a zero cost plan to get you started.

Download the Press Kit.

We are pleased to announce the immediate availability of Qloudstat, the premier service for analytics of server side access logs of cloud hosted content.

Qloudstat is doing analytics for not only hits but other crucial metrics such as bandwidth and transfer costs. Data is plotted in an interactive chart, timeline, table or geographical map split up by different dimensions in a user friendly web interface. Gain insight of URIs and filetypes used. List referring sites and search keywords. Analyze HTTP user agents, operations and status codes. Compare CDN edge locations. Visualize requests on a country, region or city map. The analytics provided are not static but queries can be made for for any given daily time period with custom filters applied. Dynamic reports are rendered instantaneously not dependent on the time period chosen to be visualized.

Reports are updated continuously around the clock to give instant and always up to date statistics no matter of access volume or the number and size of log files. Qloudstat copes with the rapidly growing traffic using highly optimized scalable systems without any installation, administration, maintenance and infrastructure required by clients. During the 3 months private beta the service was already operational with customers facing up to 25 million hits per day.

Qloudstat integrates with the market leaders in cloud storage and content delivery networks (CDN) currently supporting AWS S3, AWS CloudFront, Google Cloud Storage and Rackspace CloudFiles (Akamai CDN).

Security is a pivotal requirement for cloud based applications. Qloudstat accesses log files in third party accounts using dedicated security credentials either using OAuth authentication for Google Cloud Storage or a user managed under the Identity and Access Management (IAM) for integration with Amazon Web Services. Log files are fetched using a TLS secured connection and the website to access reports is only available with HTTPS as well.

Pricing is based on raw log data volume with four different monthly subscription plans offered. Additionally a free plan allows to use the service at zero costs for sites with less than 100’000 hits per month. With data liberation in mind, export formats for further external processing are provided.

Qloudstat is Swiss made software.

CloudFront Streaming Distributions

Qloudstat now supports analyzing CloudFront streaming distributions. Beside the regular dimensions supported such as geolocation and CDN edge locations for metrics the two additional streaming server specifics are plotted in reports:

  • Streaming Events. Streaming events such as Connect, Disconnect, Play, Stop, Pause, Unpause, 
  • Client IDs. The identifier can be used to differentiate clients. This value is unique for each connection.

Fast Results

The aim of Qloudstat is to give you instant and always up to date statistics of your cloud accounts. The update frequency depends on the interval your cloud provider writes log files that we can analyze.

  • Rackspace Cloudfiles CDN writes access logs multiple times per hour depending on the traffic and delivery of log files by Akamai.
  • AWS S3 log files are delivered within a few hours of the time that they were recorded.
  • AWS CloudFront access logs are delivered multiple times per hour depending on the traffic.
  • Google Storage currently only provides access logs in 24 hours intervals as their support is still labeled experimental provides hourly log files as of 2012/06/27.

In the list of active configurations and on the left hand below the navigation menu you always see the last update to your report and if new files are currently fetched and analyzed.

 

Qloudstat copes with the rapidly growing traffic using highly optimized scalable systems. Your report will be updated in no time as soon as log files are delivered by the cloud storage provider.

Keep your data safe

We want to emphasize that Qloudstat is engineered to allow access to your log files with dedicated access tokens that are under your control and can be revoked independently from your personal access credentials.

For AWS S3 and AWS CloudFront configurations instead of using your personal access and secret key the best option is to set up an IAM (AWS Identity and Access Management) user in the AWS console. When setting up a new configuration in Qloudstat, you can follow the step-by-step instructions recommended.

Adding a Google Cloud Storage configuration will prompt you for the x-goog-project-id and a token to authorize access for Qloudstat issued with OAuth 2.0 authentication with the permission to read log files from your account without revealing personal access credentials. The token issued is under your control and can always be revoked using your Google Account settings.

We are working together with Rackspace to offer a similar technical solution to access Cloudfiles Akamai CDN access logs. Currently this configuration requires the shared credentials for API access to Cloudfiles.

Private beta now open!

We are now accepting users to the private beta. Signup and give it a try. Configuration is a snap and you will be ready within minutes with plotted graphs of your access logs.

Update: We are currently not enforcing any quota of the different subscription plans while in beta. You are more than welcome to try Qloudstat with big data. Log files created up to two month prior your account setup are processed.

Update (20/03/2012): Following our announcement of public availability we are now enforcing quotas as per our plans.

By example: cdn.cyberduck.ch

Tracking downloads either from a S3 bucket or CDN enabled container is a typical use case for Qloudstat. The open source software Cyberduck is distributed using Rackspace Cloudfiles with the Akamai content delivery option enabled. Followed some graphs plotted from the access logs that show some of the possible analytics.

  • Example 1. URIs accessed over time. Comparison of selected files with overall number of requests.

  • Example 2. Costs of ZIP files compared to total cost of data transfer.


 

Enable Logging

While waiting for Qloudstat to get ready to open the doors, you can prepare as well. To have the raw log data available when we launch, enable logging for your AWS, Google or Rackspace accounts.

Cyberduck for Mac & Windows allows you to configure logging for your cloud storage containers with a click of a button.

What is Qloudstat?

The aim of Qloudstat is to make it simple and cost effective to provide access logs analytics of your content distributed in the cloud. The cloud in numbers. plotted.

Qloudstat gathers statistics from access logs of your cloud storage and CDN accounts. Whether using Amazon S3, Amazon CloudFront, Google Cloud Storage or Rackspace CloudFiles with Akamai CDN, Qloudstat gives you instant access to statistics by days, months or years plotted in graphs and maps.

Analyzing access log statistics from cloud providers is a tedious task due to the vast numbers involved and the raw format of the access logs provided. Qloudstat solves this for you with software as a service working behind the scenes without any administration, maintenance and infrastructure required by you.

A typical use case would be to break down downloads hosted in S3 by country, split up transfer costs by referring sites or map the geographical location of visitors.

Stay tuned and request an invitation for the private beta opening very soon!