The latest release of Cyberduck, a cloud storage browser to access S3 or Rackspace accounts, simplifies the setup of access log analytics with Qloudstat with a click of a button. If you already use Cyberduck to configure access logging for CloudFront distributions, S3 buckets or containers in Rackspace Cloudfiles, getting reports in Qloudstat just got a lot easier with the new Read Access for Qloudstat checkbox in the Info panel tab for CDN or S3 buckets.
For Amazon S3 buckets or CloudFront distributions, this will create a dedicated IAM user with a read only IAM policy for Qloudstat to fetch log files. Unchecking the Read Access for Qloudstat checkbox will remove the IAM user again revoking all access for Qloudstat. When enabled, a clickable link is displayed that redirects to Qloudstat to confirm the new setup.
Constantly adding refinements and new features based on user feedback, the latest version of Qloudstat now allows to invert filters. Select not equals from the Filter dropdown menu to select all values not matching. The exclusion of certain dimension values is certainly handy when wanting to drill down on minor events.
We aim with Qloudstat to provide much better insights than what static reports of access logs can give. We are thrilled to announced today a major milestone with support for two dimensional queries that allow to drill down on a dimension value and plot these against a second dimension to visualize the combined metrics.
The query interface allows to specify filters on both the primary and drill down dimensions:
To illustrate this we select all URIs with Hits and Completed Download metrics that that had a 404 HTTP status code returned.
Other use cases that come into mind are to plot the combinations of URIs and operating systems in a table and compare the user agents accessing the resource over time in a line chart. Or for CloudFront, you can analyze which edge locations were accessed from which country. Or to get technical, you can plot the HTTP operations against the HTTP response code sent to the client.
We have updated the dashboard and reports for analytics of the different dimensions and metrics with a simpler user interface.
Topmost you find the totals for all metrics for the selected time span including the change in percent to the previous period prominently placed below the endpoint name.
The redesigned look has much less navigation overhead. Instead of a navigation menu taking screen real estate, a dropdown selector allows to quickly select the dimension you are interested in such as the URI, country or user agent. Use the tabs to switch between metrics.
Further down above the table listing the dimension values you can select the maximum size of the query result set. To search for specific dimension values, enter a glob pattern (Such as *.jpg for the URI dimension or 20? for the HTTP Status Code) to only include rows with dimension values that match the input. As always, all charts have an export option to download the full result set as a spreadsheet.
Scroll down for the line chart with the metric values by time (day, week or month) or a column chart showing the distribution by hours over the queried time period. Select dimension values in the table to compare these with the total.
Stay tuned for more updates in the coming weeks. We will always refine the user interface based on customer feedback. And most notably we have drill down support coming to give insights by combining queries with a second dimension.
Qloudstat offers a resolution of all metrics by day. To give you an idea how these are distributed over a given day we now additionally plot a graph of accumulated hits per hour to give you a better insight of access characteristics over the day.
Additionally you can now also display the timeline with a resolution of weeks or months if you want a more generic view of time data.
We are constantly working on the feature set of Qloudstat and today we are introducing new monitoring capabilities that are very useful when you need to track the usage of resources served by Amazon S3 or a CDN.
Qloudstat now allows to configure alerts for a given lower or upper threshold triggered by a dimension and metric. Optionally a filter for the dimension value can be set. Once the threshold for the given period is reached you will get notified by email. The next alert will only be sent again after the period has been elapsed. The accumulation for the configured dimension and metric is reset after each period.
Consider the following usage scenarios where this becomes very useful:
Alert when the cost per month reaches a certain limit imposed for a country.
Get an alert for 404 HTTP error codes to get notified as soon as possible when you tried to serve a broken link.
Get notified when a file is no more downloaded with a lower limit and a filter for the URI dimension.
There is no limit for the number of alerts and they can be configured for daily, weekly or monthly periods.
Qloudstat allows you to draw quick conclusions about individual dimension values (such as a single resource in your container) by selecting an entry in the value result list. We now also allow to filter these table results with a custom input supporting the ? and *glob patterns to match dimension values that fit the input pattern.
To give an example consider you want to have the metrics for all files that are stored in a container with a given prefix or suffix only. Navigate to the URI dimension analytics page of your report and enter a glob pattern such as */5902/*.png to only list URIs that contain this pattern in their path. This is quite useful to filter versioned resources or limit the result list to files that are served from a specific location in your bucket or CDN origin.
The dimension values list supports multi-selection which gives you a simple way to bring the selected values in relation. When selecting multiple values a pie chart next to the table shows up with the percentage of each value compared to all selected ones.
Update: The filter with search patterns now also works for enumerated dimension values such as HTTP status code.
After a private beta phase of three months with intensive testing and refinements, we are pleased to announce Qloudstat is now available to the general public. Sign up today! You will be up and running within minutes. Qloudstat does all the work behind the scenes without any administration, maintenance and infrastructure required by you. We have a zero cost plan to get you started.
We are pleased to announce the immediate availability of Qloudstat, the premier service for analytics of server side access logs of cloud hosted content.
Qloudstat is doing analytics for not only hits but other crucial metrics such as bandwidth and transfer costs. Data is plotted in an interactive chart, timeline, table or geographical map split up by different dimensions in a user friendly web interface. Gain insight of URIs and filetypes used. List referring sites and search keywords. Analyze HTTP user agents, operations and status codes. Compare CDN edge locations. Visualize requests on a country, region or city map. The analytics provided are not static but queries can be made for for any given daily time period with custom filters applied. Dynamic reports are rendered instantaneously not dependent on the time period chosen to be visualized.
Reports are updated continuously around the clock to give instant and always up to date statistics no matter of access volume or the number and size of log files. Qloudstat copes with the rapidly growing traffic using highly optimized scalable systems without any installation, administration, maintenance and infrastructure required by clients. During the 3 months private beta the service was already operational with customers facing up to 25 million hits per day.
Qloudstat integrates with the market leaders in cloud storage and content delivery networks (CDN) currently supporting AWS S3, AWS CloudFront, Google Cloud Storage and Rackspace CloudFiles (Akamai CDN).
Security is a pivotal requirement for cloud based applications. Qloudstat accesses log files in third party accounts using dedicated security credentials either using OAuth authentication for Google Cloud Storage or a user managed under the Identity and Access Management (IAM) for integration with Amazon Web Services. Log files are fetched using a TLS secured connection and the website to access reports is only available with HTTPS as well.
Pricing is based on raw log data volume with four different monthly subscription plans offered. Additionally a free plan allows to use the service at zero costs for sites with less than 100’000 hits per month. With data liberation in mind, export formats for further external processing are provided.
We are now accepting users to the private beta. Signup and give it a try. Configuration is a snap and you will be ready within minutes with plotted graphs of your access logs.
Update: We are currently not enforcing any quota of the different subscription plans while in beta. You are more than welcome to try Qloudstat with big data. Log files created up to two month prior your account setup are processed.
Update (20/03/2012): Following our announcement of public availability we are now enforcing quotas as per our plans.
Tracking downloads either from a S3 bucket or CDN enabled container is a typical use case for Qloudstat. The open source software Cyberduck is distributed using Rackspace Cloudfiles with the Akamai content delivery option enabled. Followed some graphs plotted from the access logs that show some of the possible analytics.
Example 1. URIs accessed over time. Comparison of selected files with overall number of requests.
Example 2. Costs of ZIP files compared to total cost of data transfer.