IAM Policies in Depth

Our recommended way to grant Qloudstat access to your Amazon S3 & CloudFront log files is to create an IAM user with a read-only policy attached. This post is derived from the AWS S3 documentation and CloudFront documentation on IAM policies.

When setting up a new AWS configuration in Qloudstat, you are asked to enter a valid Access Key and Secret Key. This could be your main AWS credentials but this is discouraged. Instead we recommend you to login to the IAM console and create a new user with its dedicated access key.

S3
You can attach the IAM Read Only Policy Template which should suit most needs. A further restricted custom policy with the least grants would be edited like

{
    "Statement":[
        {
             "Effect":"Allow",
             "Action":[
                "s3:GetObject",
                "s3:ListBucket"
             ],
             "Resource":"arn:aws:s3:::logging-target-bucket/*",
             "Condition":{
                "Bool":{
                    "aws:SecureTransport":"true"
                }
             }
        },
        {
            "Effect":"Allow",
            "Action":[
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation",
                "s3:GetBucketLogging"
            ],
            "Resource":"arn:aws:s3:::*",
            "Condition":{
                "Bool":{
                   "aws:SecureTransport":"true"
                }
            }
        }
    ]
}
  • To facilitate handling of your buckets in Qloudstat, we recommend to grant the s3:ListAllMyBuckets to the user.
  • Grant reading the logging status and location of every bucket.
  • Grant listing and fetching files in the target logging bucket named logging-target-bucket. You must repeat this statement for all your logging target buckets or use the wildcard resource name arn:aws:s3:::*
  • All communication must be secured using HTTPS.

You can find additional information in the Qloudstat FAQ.

CloudFront
A policy to fetch log files for CloudFront distributions must allow to read your CloudFront distribution status plus fetching the log files from the S3 logging target bucket.

{
    "Statement": [
        {
          "Action": [
            "s3:Get*",
            "s3:List*"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:s3:::logging-target-bucket/*"
        },
        {
          "Action": [
            "cloudfront:Get*",
            "cloudfront:List*"
          ],
          "Effect": "Allow",
          "Resource": "*"
        }
    ]
}
  • An asterisk (*) is used as the resource when writing a policy to control access to CloudFront distributions. There are no CloudFront resource ARNs (Amazon Resource Names) for you to use in an IAM policy, because IAM cannot control access to specific CloudFront distributions.
  • To facilitate handling of your distributions in Qloudstat, we recommend to grant the cloudfront:ListDistributions to the user. We use a cloudfront:List* wildcard to include both download and streaming (cloudfront:ListStreamingDistributions) API actions.

You can find additional information in the Qloudstat FAQ.

Tracking CloudFront Cache Results

With the latest update to Amazon CloudFront it cache result types from edge locations are now logged and available for analytics in Qloudstat as of today. The new edge result types reported are as follows:

  • Hit: CloudFront served the object to the viewer from the edge cache.
  • Refresh Hit: CloudFront found the object in the edge cache but it had expired, so CloudFront contacted the origin to verify that the cache has the latest version of the object.
  • Miss: The object wasn’t in the edge cache, so CloudFront requested the object from the origin and served it to the viewer.
  • Limit Exceeded: The request was denied because a CloudFront limit was exceeded.
  • Capacity Exceeded: CloudFront returned a 503 error because the edge location didn’t have enough capacity at the time of the request to serve the object.
  • Error: The request resulted in a client error (4xx) or server error (5xx).

You can drilldown these result types by edge locations and URI and vice versa for more insight.

Note: This will be effective September 12th, 2012 when the new log file format includes these changes.

 

Distribution of metrics per hour

Qloudstat offers a resolution of all metrics by day. To give you an idea how these are distributed over a given day we now additionally plot a graph of accumulated hits per hour to give you a better insight of access characteristics over the day.

Additionally you can now also display the timeline with a resolution of weeks or months if you want a more generic view of time data.

Monitoring with threshold alerts

We are constantly working on the feature set of Qloudstat and today we are introducing new monitoring capabilities that are very useful when you need to track the usage of resources served by Amazon S3 or a CDN.

Qloudstat now allows to configure alerts for a given lower or upper threshold triggered by a dimension and metric. Optionally a filter for the dimension value can be set. Once the threshold for the given period is reached you will get notified by email. The next alert will only be sent again after the period has been elapsed. The accumulation for the configured dimension and metric is reset after each period.

Monitoring threshold with alerts

 

Consider the following usage scenarios where this becomes very useful:

  • Alert when the cost per month reaches a certain limit imposed for a country.
  • Get an alert for 404 HTTP error codes to get notified as soon as possible when you tried to serve a broken link.
  • Get notified when a file is no more downloaded with a lower limit and a filter for the URI dimension.

There is no limit for the number of alerts and they can be configured for daily, weekly or monthly periods.

 

Query Strings in URIs

It is a common practice to include query strings in URIs pointing to CloudFront or S3 resources to get a better understanding of the popularity of different sources for your content. Qloudstat visualizes these query strings by hits or bandwidth usage over time.

For CloudFront, any query string appended to an URI is logged and then stripped when fetching the origin resource. AWS had a blog post when introducing this feature. Arbitrary query strings are supported. The same applies for Rackspace Cloudfiles with a Akamai CDN setup.

For S3, you must prepend query strings in URIs with “x-“. More details from the S3 documentation:

You can include custom information to be stored in the access log record for a request by adding a custom query-string parameter to the URL for the request. Amazon S3 will ignore query-string parameters that begin with “x-“, but will include those parameters in the access log record for the request, as part of the Request-URI field of the log record. For example, a GET request for “s3.amazonaws.com/mybucket/photos/2006/08/puppy.jpg?x-user=johndoe” will work the same as the same request for “s3.amazonaws.com/mybucket/photos/2006/08/puppy.jpg”, except that the “x-user=johndoe” string will be included in the Request-URI field for the associated log record. This functionality is available in the REST interface only.

Purge log files

To enable purging log files after analytics you have to setup IAM (Identity and Access Management) in AWS to allow Qloudstat to delete log files in your account. It is best to restrict such write application to the least minimum required.

Below a code snippet with an example policy to attach to the user setup in IAM in addition to the existing read-only policy that you have configured to allow Qloudstat to fetch log files.

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:DeleteObject",
      "Resource": "arn:aws:s3:::bucket/loggingprefix*"
    }
  ]
}

Login to your Qloudstat account to edit your Amazon S3 or CloudFront configuration, where we display this policy preconfigured for your bucket names next to the option to toggle purge for log files.

Choose a Logging Target

We want to share the following best practice when configuring a S3 bucket or CloudFront distribution for logging.

  • The logging target should be another S3 bucket than the origin. Otherwise your report will include requests for log files delivered by Amazon and download requests for log files.
  • Use a different logging target bucket for every S3 bucket and CloudFront distribution. Or at least choose a different logging target prefix. This will help Qloudstat to fetch your log files more efficiently.

You can use Cyberduck or the AWS Console to set up the logging configuration.

Our Price Plans

Qloudstat is available with five different price plans to choose from because one size doesn’t fit all. The free plan allows to use Qloudstat at zero costs for sites with less than 100’000 hits per month. The large plan allows as much as 5GB of log data per month. If you need even more, choose the enterprise plan or contact us for specific needs.

The price plans differ based on the following criteria:

  • Endpoints. An endpoint refers to a bucket (S3), distribution (CloudFront) or container (Rackspace). We have just updated our plans to allow an  unlimited (∞) number of configurable endpoints except the free plan. You can freely organize your data in as many endpoints you want without any restriction.
  • Volume per month. Each plan has an upper limit for the volume (number of bytes) of your log files we process. We sum up the number of uncompressed bytes of each log file processed on a monthly base. After a month we reset our counters. If you hit the limit of your plan we temporarily suspend the fetching of new log files until you upgrade your subscription plan. With that said you can always start with the free plan and align your subscription with your changing needs.
  • Retention. We store all your processed log files for at least 36 months. That gives you the possibility to gain a deep insight on how your files have been accessed  over a long time period. Please contact us if you are interested in an even higher retention.
  • Resolution. Log files are processed with a sampling rate of 1 day with a summed up value for each dimension, metric and day.

Note that there are no long term contracts. Once subscribed to a plan, you can cancel, downgrade and upgrade at any time.

Contact us if you are confused which plan to choose. We will help you figure out the best option for you.

Real Artists Ship

After a private beta phase of three months with intensive testing and refinements, we are pleased to announce Qloudstat is now available to the general public. Sign up today! You will be up and running within minutes. Qloudstat does all the work behind the scenes without any administration, maintenance and infrastructure required by you. We have a zero cost plan to get you started.

Download the Press Kit.

We are pleased to announce the immediate availability of Qloudstat, the premier service for analytics of server side access logs of cloud hosted content.

Qloudstat is doing analytics for not only hits but other crucial metrics such as bandwidth and transfer costs. Data is plotted in an interactive chart, timeline, table or geographical map split up by different dimensions in a user friendly web interface. Gain insight of URIs and filetypes used. List referring sites and search keywords. Analyze HTTP user agents, operations and status codes. Compare CDN edge locations. Visualize requests on a country, region or city map. The analytics provided are not static but queries can be made for for any given daily time period with custom filters applied. Dynamic reports are rendered instantaneously not dependent on the time period chosen to be visualized.

Reports are updated continuously around the clock to give instant and always up to date statistics no matter of access volume or the number and size of log files. Qloudstat copes with the rapidly growing traffic using highly optimized scalable systems without any installation, administration, maintenance and infrastructure required by clients. During the 3 months private beta the service was already operational with customers facing up to 25 million hits per day.

Qloudstat integrates with the market leaders in cloud storage and content delivery networks (CDN) currently supporting AWS S3, AWS CloudFront, Google Cloud Storage and Rackspace CloudFiles (Akamai CDN).

Security is a pivotal requirement for cloud based applications. Qloudstat accesses log files in third party accounts using dedicated security credentials either using OAuth authentication for Google Cloud Storage or a user managed under the Identity and Access Management (IAM) for integration with Amazon Web Services. Log files are fetched using a TLS secured connection and the website to access reports is only available with HTTPS as well.

Pricing is based on raw log data volume with four different monthly subscription plans offered. Additionally a free plan allows to use the service at zero costs for sites with less than 100’000 hits per month. With data liberation in mind, export formats for further external processing are provided.

Qloudstat is Swiss made software.

CloudFront Streaming Distributions

Qloudstat now supports analyzing CloudFront streaming distributions. Beside the regular dimensions supported such as geolocation and CDN edge locations for metrics the two additional streaming server specifics are plotted in reports:

  • Streaming Events. Streaming events such as Connect, Disconnect, Play, Stop, Pause, Unpause, 
  • Client IDs. The identifier can be used to differentiate clients. This value is unique for each connection.

Fast Results

The aim of Qloudstat is to give you instant and always up to date statistics of your cloud accounts. The update frequency depends on the interval your cloud provider writes log files that we can analyze.

  • Rackspace Cloudfiles CDN writes access logs multiple times per hour depending on the traffic and delivery of log files by Akamai.
  • AWS S3 log files are delivered within a few hours of the time that they were recorded.
  • AWS CloudFront access logs are delivered multiple times per hour depending on the traffic.
  • Google Storage currently only provides access logs in 24 hours intervals as their support is still labeled experimental provides hourly log files as of 2012/06/27.

In the list of active configurations and on the left hand below the navigation menu you always see the last update to your report and if new files are currently fetched and analyzed.

 

Qloudstat copes with the rapidly growing traffic using highly optimized scalable systems. Your report will be updated in no time as soon as log files are delivered by the cloud storage provider.

Private beta now open!

We are now accepting users to the private beta. Signup and give it a try. Configuration is a snap and you will be ready within minutes with plotted graphs of your access logs.

Update: We are currently not enforcing any quota of the different subscription plans while in beta. You are more than welcome to try Qloudstat with big data. Log files created up to two month prior your account setup are processed.

Update (20/03/2012): Following our announcement of public availability we are now enforcing quotas as per our plans.

Enable Logging

While waiting for Qloudstat to get ready to open the doors, you can prepare as well. To have the raw log data available when we launch, enable logging for your AWS, Google or Rackspace accounts.

Cyberduck for Mac & Windows allows you to configure logging for your cloud storage containers with a click of a button.