ENTRIES TAGGED "visualization"

The Power of a Private HTTP Archive Instance: Finding a Representative Performance Baseline

Velocity 2013 Speaker Series

Be honest, have you ever wanted to play Steve Souders for a day and pull some revealing stats or trends about some web sites of your choice? Or maybe dig around the HTTP archive? You can do that and more by setting up your own HTTP Archive.

httparchive.org is a fantastic tool to track, monitor, and review how the web is built. You can dig into trends around page size, page load time, content delivery network (CDN) usage, distribution of different mimetypes, and many other stats. With the integration of WebPagetest, it’s a great tool for synthetic testing as well.

You can download an HTTP Archive MySQL dump (warning: it’s quite large) and the source code from the download page and dissect a snapshot of the data yourself.  Once you’ve set up the database, you can easily query anything you want.

Setup

You need MySQL, PHP, and your own webserver running. As I mentioned above, HTTP Archive relies on WebPagetest—if you choose to run your own private instance of WebPagetest, you won’t have to request an API key. I decided to ask Patrick Meenan for an API key with limited query access. That was sufficient for me at the time. If I ever wanted to use more than 200 page loads per day, I would probably want to set up a private instance of WebPagetest.

To find more details on how to set up an HTTP Archive instance yourself and any further advice, please check out my blog post.

Benefits

Going back to the scenario I described above: the real motivation is that often you don’t want to throw your website(s) in a pile of other websites (e.g. not related to your business) to compare or define trends. Our digital property at the Canadian Broadcasting Corporation’s (CBC) spans over dozens of URLs that have different purposes and audiences. For example, CBC Radio covers most of the Canadian radio landscape, CBC News offers the latest breaking news, CBC Hockey Night in Canada offers great insights on anything related to hockey, and CBC Video is the home for any video available on CBC. It’s valuable for us to not only compare cbc.ca to the top 100K Alexa sites but also to verify stats and data against our own pool of web sites.

In this case, we want to use a set of predefined URLs that we can collect HTTP Archive stats for. Hence a private instance can come in handy—we can run tests every day, or every week, or just every month to gather information about the performance of the sites we’ve selected. From there, it’s easy to not only compare trends from httparchive.org to our own instance as a performance baseline, but also have a great amount of data in our local database to run queries against and to do proper performance monitoring and investigation.

Visualizing Data

The beautiful thing about having your own instance is that you can be your own master of data visualization: you can now create more charts in addition to the ones that came out of the box with the default HTTP Archive setup. And if you don’t like Google chart tools, you may even want to check out D3.js or Highcharts instead.

The image below shows all mime types used by CBC web properties that are captured in our HTTP archive database, using D3.js bubble charts for visualization.

Mime types distribution for CBC web properties using D3.js bubble visualization. The data were taken from the requests table of our private HTTP Archive database.

Mime types distribution for CBC web properties using D3.js bubble visualization. The data were taken from the requests table of our private HTTP Archive database.


Read more…

Comment |

In-Memory Databases, Burning Silos, Reintroducing Dart, and Google Glass Gatherings

Weekly Highlights and Insights: May 19-25

In-memory Databases: A discussion of near real-time manipulation of massive datasets

Burning the Silos: Minimize boundaries to reduce product cycle times.

Dart Is Not the Language You Think It Is: Seth Ladd’s enthusiastic reintroduction of Dart prompts a voluminous discussion on Slashdot.

Is that the Google Glass?: The anthropology of an always-on society

Tracing the Disappeared: An interactive visualization of CIA rendition flights

Comment |

Google I/O, Big Data Adolescence, Visualization, and the Future of Open Source

Weekly Highlights and Insights: May 13-17

Google I/O: O’Reilly Editor Rachel Roumeliotis reports from the conference floor.

Big Data, Cool Kids: Fumbling toward the adolescence of big data tools.

Code as Art: Interactive Data Visualization for the Web author Scott Murray on becoming a code artist.

Real-time World-wide Wikipedia Edits: Stephen LaPorte and Mahmoud Hashemi’s addictive visualization.

Future of Open Source: The quality, security, and community driving open source adoption.

Comment |

Weekly Highlights and Insights: April 29-May 3

Leading indicators, CSS selectors, medical data sharing, DDoS visualization, and a new jQuery class

Leading Indicators: Over on O’Reilly Radar, Mike Loukides and Q Ethan McCallum come up with a few ideas for evaluating an organization’s data science program from the “outside.”

CSS Selectors as Superpowers: Simon St. Laurent hopes that “the success of CSS selectors will bring developers to look for other ways to apply pattern-matching to their markup.”

Data sharing drives diagnoses and cures, if we can get there (Parts 1 & 2): Andy Oram explores the take-aways from this year’s Sage Congress.

Visualization of the Week: Jenn Webb shows us Ludovic Fauvet’s Logstalgia visualization of a recent VideoLAN DDoS attack.

jQuery for Advanced Front-End Development: New jQuery class from O’Reilly School of Technology.

Comment |
Developer Week in Review

Developer Week in Review

Special Jeopardy edition featuring Nokia, MacBook Pro rumors, and Google's Public Data Explorer.

Tired of everyone making "Terminator" or "Matrix" references to Watson's domination of its pitiful human rivals? Well, we go old school with our media references, as we look at Nokia's fickleness, new toys for geeks, and Google's campaign for pretty data.

Comments Off |