The edit screen for Rackspace Cloudfiles configurations now allows to toggle the delivery of access logs per container. This makes setting up logging analytics for your Akamai content distribution even easier without having to use a thirdparty tool or console.
We are constantly working on the feature set of Qloudstat and today we are introducing new monitoring capabilities that are very useful when you need to track the usage of resources served by Amazon S3 or a CDN.
Qloudstat now allows to configure alerts for a given lower or upper threshold triggered by a dimension and metric. Optionally a filter for the dimension value can be set. Once the threshold for the given period is reached you will get notified by email. The next alert will only be sent again after the period has been elapsed. The accumulation for the configured dimension and metric is reset after each period.
Consider the following usage scenarios where this becomes very useful:
- Alert when the cost per month reaches a certain limit imposed for a country.
- Get an alert for 404 HTTP error codes to get notified as soon as possible when you tried to serve a broken link.
- Get notified when a file is no more downloaded with a lower limit and a filter for the URI dimension.
There is no limit for the number of alerts and they can be configured for daily, weekly or monthly periods.
Qloudstat allows you to draw quick conclusions about individual dimension values (such as a single resource in your container) by selecting an entry in the value result list. We now also allow to filter these table results with a custom input supporting the ? and * glob patterns to match dimension values that fit the input pattern.
To give an example consider you want to have the metrics for all files that are stored in a container with a given prefix or suffix only. Navigate to the URI dimension analytics page of your report and enter a glob pattern such as */5902/*.png to only list URIs that contain this pattern in their path. This is quite useful to filter versioned resources or limit the result list to files that are served from a specific location in your bucket or CDN origin.
The dimension values list supports multi-selection which gives you a simple way to bring the selected values in relation. When selecting multiple values a pie chart next to the table shows up with the percentage of each value compared to all selected ones.
Update: The filter with search patterns now also works for enumerated dimension values such as HTTP status code.
It is a common practice to include query strings in URIs pointing to CloudFront or S3 resources to get a better understanding of the popularity of different sources for your content. Qloudstat visualizes these query strings by hits or bandwidth usage over time.
For CloudFront, any query string appended to an URI is logged and then stripped when fetching the origin resource. AWS had a blog post when introducing this feature. Arbitrary query strings are supported. The same applies for Rackspace Cloudfiles with a Akamai CDN setup.
For S3, you must prepend query strings in URIs with “x-“. More details from the S3 documentation:
You can include custom information to be stored in the access log record for a request by adding a custom query-string parameter to the URL for the request. Amazon S3 will ignore query-string parameters that begin with “x-”, but will include those parameters in the access log record for the request, as part of the
Request-URIfield of the log record. For example, a
GETrequest for “s3.amazonaws.com/mybucket/photos/2006/08/puppy.jpg?x-user=johndoe” will work the same as the same request for “s3.amazonaws.com/mybucket/photos/2006/08/puppy.jpg”, except that the “x-user=johndoe” string will be included in the
Request-URIfield for the associated log record. This functionality is available in the REST interface only.
Qloudstat is available with five different price plans to choose from because one size doesn’t fit all. The free plan allows to use Qloudstat at zero costs for sites with less than 100’000 hits per month. The large plan allows as much as 5GB of log data per month. If you need even more, choose the enterprise plan or contact us for specific needs.
The price plans differ based on the following criteria:
- Endpoints. An endpoint refers to a bucket (S3), distribution (CloudFront) or container (Rackspace). We have just updated our plans to allow an unlimited (∞) number of configurable endpoints except the free plan. You can freely organize your data in as many endpoints you want without any restriction.
- Volume per month. Each plan has an upper limit for the volume (number of bytes) of your log files we process. We sum up the number of uncompressed bytes of each log file processed on a monthly base. After a month we reset our counters. If you hit the limit of your plan we temporarily suspend the fetching of new log files until you upgrade your subscription plan. With that said you can always start with the free plan and align your subscription with your changing needs.
- Retention. We store all your processed log files for at least 36 months. That gives you the possibility to gain a deep insight on how your files have been accessed over a long time period. Please contact us if you are interested in an even higher retention.
- Resolution. Log files are processed with a sampling rate of 1 day with a summed up value for each dimension, metric and day.
Note that there are no long term contracts. Once subscribed to a plan, you can cancel, downgrade and upgrade at any time.
Contact us if you are confused which plan to choose. We will help you figure out the best option for you.
After a private beta phase of three months with intensive testing and refinements, we are pleased to announce Qloudstat is now available to the general public. Sign up today! You will be up and running within minutes. Qloudstat does all the work behind the scenes without any administration, maintenance and infrastructure required by you. We have a zero cost plan to get you started.
Download the Press Kit.
We are pleased to announce the immediate availability of Qloudstat, the premier service for analytics of server side access logs of cloud hosted content.
Qloudstat is doing analytics for not only hits but other crucial metrics such as bandwidth and transfer costs. Data is plotted in an interactive chart, timeline, table or geographical map split up by different dimensions in a user friendly web interface. Gain insight of URIs and filetypes used. List referring sites and search keywords. Analyze HTTP user agents, operations and status codes. Compare CDN edge locations. Visualize requests on a country, region or city map. The analytics provided are not static but queries can be made for for any given daily time period with custom filters applied. Dynamic reports are rendered instantaneously not dependent on the time period chosen to be visualized.
Reports are updated continuously around the clock to give instant and always up to date statistics no matter of access volume or the number and size of log files. Qloudstat copes with the rapidly growing traffic using highly optimized scalable systems without any installation, administration, maintenance and infrastructure required by clients. During the 3 months private beta the service was already operational with customers facing up to 25 million hits per day.
Qloudstat integrates with the market leaders in cloud storage and content delivery networks (CDN) currently supporting AWS S3, AWS CloudFront, Google Cloud Storage and Rackspace CloudFiles (Akamai CDN).
Security is a pivotal requirement for cloud based applications. Qloudstat accesses log files in third party accounts using dedicated security credentials either using OAuth authentication for Google Cloud Storage or a user managed under the Identity and Access Management (IAM) for integration with Amazon Web Services. Log files are fetched using a TLS secured connection and the website to access reports is only available with HTTPS as well.
Pricing is based on raw log data volume with four different monthly subscription plans offered. Additionally a free plan allows to use the service at zero costs for sites with less than 100’000 hits per month. With data liberation in mind, export formats for further external processing are provided.
Qloudstat is Swiss made software.
The aim of Qloudstat is to give you instant and always up to date statistics of your cloud accounts. The update frequency depends on the interval your cloud provider writes log files that we can analyze.
- Rackspace Cloudfiles CDN writes access logs multiple times per hour depending on the traffic and delivery of log files by Akamai.
- AWS S3 log files are delivered within a few hours of the time that they were recorded.
- AWS CloudFront access logs are delivered multiple times per hour depending on the traffic.
- Google Storage
currently only provides access logs in 24 hours intervals as their support is still labeled experimentalprovides hourly log files as of 2012/06/27.
In the list of active configurations and on the left hand below the navigation menu you always see the last update to your report and if new files are currently fetched and analyzed.
Qloudstat copes with the rapidly growing traffic using highly optimized scalable systems. Your report will be updated in no time as soon as log files are delivered by the cloud storage provider.
We are now accepting users to the private beta. Signup and give it a try. Configuration is a snap and you will be ready within minutes with plotted graphs of your access logs.
Update: We are currently not enforcing any quota of the different subscription plans while in beta. You are more than welcome to try Qloudstat with big data. Log files created up to two month prior your account setup are processed.
Tracking downloads either from a S3 bucket or CDN enabled container is a typical use case for Qloudstat. The open source software Cyberduck is distributed using Rackspace Cloudfiles with the Akamai content delivery option enabled. Followed some graphs plotted from the access logs that show some of the possible analytics.
- Example 1. URIs accessed over time. Comparison of selected files with overall number of requests.
- Example 2. Costs of ZIP files compared to total cost of data transfer.
While waiting for Qloudstat to get ready to open the doors, you can prepare as well. To have the raw log data available when we launch, enable logging for your AWS, Google or Rackspace accounts.
Cyberduck for Mac & Windows allows you to configure logging for your cloud storage containers with a click of a button.
The aim of Qloudstat is to make it simple and cost effective to provide access logs analytics of your content distributed in the cloud. The cloud in numbers. plotted.
Qloudstat gathers statistics from access logs of your cloud storage and CDN accounts. Whether using Amazon S3, Amazon CloudFront, Google Cloud Storage or Rackspace CloudFiles with Akamai CDN, Qloudstat gives you instant access to statistics by days, months or years plotted in graphs and maps.
Analyzing access log statistics from cloud providers is a tedious task due to the vast numbers involved and the raw format of the access logs provided. Qloudstat solves this for you with software as a service working behind the scenes without any administration, maintenance and infrastructure required by you.
A typical use case would be to break down downloads hosted in S3 by country, split up transfer costs by referring sites or map the geographical location of visitors.
Stay tuned and request an invitation for the private beta opening very soon!