URLs are a type of data that is particularly easy to detect. The summarise tool does a reasonable job of displaying them anyway, but a couple of tricks make it even better.

The first insight is that just as time can be grouped by days or months or years, URLs can – in theory – be grouped by protocol, domain, partial paths or query parameters. We found the domain was the most useful, as it shows which website the URLs are from.

For example, I use a tool called Pocket to bookmark things to read or watch later on my phone. This is the list of the most common websites I bookmark that way:

Images are another common kind of URL that is easy to detect. A regular expression catches most of them automatically based on the file extension. (Although see this bug, at some point we’ll need a semantic layer…)

Here you can see the top artwork tracks from a Last.fm scraper, which gets the data of all the music that I’ve ever listened to:

You can immediately see that Yann Tiersen features heavily in the most replayed, with both Goodbye Lenin! and Amelie.

As explained in Summarise #2: Pies and facts, the summarise tool generates lots of different “facts” about the data. It then has a ranking algorithm to decide which are the best to show.

When the data is a URL, extra facts as described above (tables of website domains, collages of images) are generated. The tool then selects whether those are more interesting than the basic facts about the URLs.

For example, if there were only a few different URLs, they might all be shown in a pie. But if there were a lot more, but from only a few domains, then the domains would be shown in a pie.

Try it yourself! Use “Create a dataset” to get some data into new ScraperWiki. Then pick “Summarise this data” from the tools menu and see what it tells you.

Next time – words and countries! And then, a final post to round it up, about data that is (nearly) always one type, and other interesting tidbits.

For now it is daily, as that covers most people’s uses. Please ask if you need something else! Or have a look at the tool’s source code.

Want to know how to use the new ScraperWiki? There’s a quick start guide to coding in your browser on it.

We’ve been teasing and tempting you with blog posts about the first few tools on the new ScraperWiki Beta for a while now. It’s time to let you try them out first-hand.

As of right now, the new ScraperWiki Beta is open for you, your aunt, anyone, to sign up for a free community account: Check out beta.scraperwiki.com.

We’re really excited. Not only does this mean all of our Classic Premium Account holders, and our new private beta applicants, have been settled into the new platform, but now regular Classic users get to try the new ScraperWiki out, for free.

The new ScraperWiki beta is a little rough around the edges, but it can already do everything ScraperWiki Classic did, and more. As we (and you!) develop and share new tools on the platform, it’s only going to get more powerful and more exciting.

The Code in your browser tool will let you copy and paste your scrapers from ScraperWiki Classic, while Search for Tweets, Summarise Automatically and Query with SQL should give you an idea of how simple and focussed ScraperWiki tools are meant to be. The new ScraperWiki isn’t one monolithic app – it’s an ever-expanding collection of tools that interact and plug into each other to help you get your job done. I can’t wait to see more tools appear in the near future!

To find out more about how the new Beta is different from ScraperWiki Classic, check out our “What’s new” guide. And to report any bugs or missing features, raise an issue on our Github repo or email Zach, our community manager, at zach@scraperwiki.com.

Free ScraperWiki Community accounts. Come try us out: beta.scraperwiki.com.

The book is largely aimed at web designers who are looking to include interactive data visualisations in their work. It includes some introductory material on JavaScript, HTML, and CSS, so has some value for programmers moving into web visualisation. I quite liked the repetition of this relatively basic material, and the conceptual introduction to the d3 library.

I found the book rather slow: on page 197 – approaching the final fifth of the book – we were still making a bar chart. A smaller effort was expended in that period on scatter graphs. As a data scientist, I expect to have several dozen plot types in that number of pages! This is something of which Scott warns us, though. d3 is a visualisation framework built for explanatory presentation (i.e. you know the story you want to tell) rather than being an exploratory tool (i.e. you want to find out about your data). To be clear: this “slowness” is not a fault of the book, rather a disjunction between the book and my expectations.

From a technical point of view, d3 works by binding data to elements in the DOM for a webpage. It’s possible to do this for any element type, but practically speaking only Scaleable Vector Graphics (SVG) elements make real sense. This restriction means that d3 will only work for more recent browsers. This may be a possible problem for those trapped in some corporate environments. The library contains a lot of helper functions for generating scales, loading up data, selecting and modifying elements, animation and so forth. d3 is low-level library; there is no PlotBarChart function.

Achieving the static effects demonstrated in this book using other tools such as R, Matlab, or Python would be a relatively straightforward task. The animations, transitions and interactivity would be more difficult to do. More widely, the d3 library supports the creation of hierarchical visualisations which I would struggle to create using other tools.

This book is quite a basic introduction, you can get a much better overview of what is possible with d3 by looking at the API documentation and the Gallery. Scott lists quite a few other resources including a wide range for the d3 library itself, systems built on d3, and alternatives for d3 if it were not the library you were looking for.

I can see myself using d3 in the future, perhaps not for building generic tools but for custom visualisations where the data is known and the aim is to best explain that data. Scott quotes Ben Schniederman on this regarding the structure of such visualisations:

overview first, zoom and filter, then details on demand

It’s an open source tool – geeks can play along on Github, or use the SSH button to log into the tool and see the code running in action.

After adding pie charts, I realised that even for general data lots of columns had basic types in them which could be easily detected, and special visualisations shown. For example, dates and times or image URLs.

For a while now, I’ve been using Pocket (formerly Read It Later) to bookmark articles and videos to read or watch later. I’ve got a scraper that calls the Pocket API (get in touch if you’d use it and want me to release it as a tool!) and saves all my bookmarks as a dataset.

This is a histogram the “Summarise automatically” tool made of when I added bookmarks.

You can immediately see I first tried out Pocket a tiny bit in November/December 2011, but then stopped using it for four months. Then in May 2012, I start again in earnest. That was because I’d got a new smartphone with a larger screen, and wanted to read articles on the train. You can also see I went on holiday in August 2012, and didn’t bookmark much then.

The code that automatically made this chart is in the “fact_time_charts” function in facts.js. First of all it uses the fantastic moment.js library to parse every value in the column. If at least half of them appear to be dates/times, it goes ahead and makes a time histogram.

The interesting bit is when it tries various ways to “bucket” (or “bin”) the data. That is, count the number of times something in the columns is in a particular hour, data, month or year. It tries out all four, and only uses the chart that has fewer than 31 rows.

// try grouping into buckets at various granularities
_bucket_time_chart(col, group, "YYYY", "years", "YYYY", "time_chart_year", 90)
_bucket_time_chart(col, group, "YYYY-MM", "months", "MMM YYYY", "time_chart_month", 91)
_bucket_time_chart(col, group, "YYYY-MM-DD", "days", "D MMM YYYY", "time_chart_day", 92)
_bucket_time_chart(col, group, "YYYY-MM-DD HH", "hours", "ha D MMM YYYY", "time_chart_hour", 93)


This means if the data is spread out over lots of years, it will show it by year. If it all happened in one day, it’ll show a histogram by hour.

The “Summarise automatically” tool does a similar thing for columns of numbers. It shows a histogram so you can see how they are distributed. For example, this chart was made automatically for a Climate Code Foundation dataset of sea temperature station observations.

“Z” is the relative height of the sea. You can see there are two peaks in the histogram where lots of observations were made – one is for low tide and the other for high tide.

Once again this puts the data points into bins of different sizes to make the histogram. This time it does a logarithm to find the power of 10 to use as the width for each bar that means there are a reasonable number of bars – as near to 33 as possible. The code for this is in “fact_numbers_chart” in facts.js.

A notable bit works out if the histogram “looks interesting”. We had lots of them that only showed one bar, because a few outliers were far off the edge. The test in the end was to look at the second most common value, and see if it is at least the sqrt of the number of rows. This means the bar charts are at least slightly interesting – it falls back to tables of the top values if they’re not.

// if the second most common value is at least the sqrt of the number of rows
if (group[1].c < Math.sqrt(total) ) {
return
}


Try it yourself! Use “create a dataset” to get some data into ScraperWiki. Then pick “Summarise automatically” from the tools menu and see what it tells you.

Next time, I add fancy stuff to display URLs.

I say this here not as an advertisement (cough), but because I have almost accidentally ended up as a co-organiser of quite a big event. (It’s an accident, on my part, because Kirsty has put a hell of a lot of time and effort into it!)

Because of its twitter-heavy organisation, I would have a pretty good idea about what was going on, because @shropgeek would normally be included in conversations. With the nature of the beast evolving, I’ve had some pretty important questions about how people share, and what they’re saying about the conference, and I accidentally (I seem accident-prone when it comes to administrative tasks) discovered a very useful resource which sits under my nose at my day-job: the Summarise Automatically tool on the New ScraperWiki.

One of the first things I did, to test out the summariser was to search twitter for mentions of the “#revolutionconf” hashtag, then click on “summarise this data.” My expectations were to see some cool graphics, and mainly to test it out as a ScraperWiki tool. What I found, though, were some really valuable views on how people are tweeting.

Basically, the summariser tool tries to tell you some instant things about your data by, well, summarising the columns in your data. This can be a bit of a mixed bag, with some summaries making little sense (but we’ll get better at that). However, the really cool thing is the very high-level, dashboard-like information I could get on *this* data, which I know comprises tweets, and all of which are related to my hashtag.

1. The first win was a simple count of how many mentions there are. I saw that the hashtag hasn’t been used as much as it could be (with only 66 instances), and realised that I’ve tweeted several times without it. /me slaps own hand!

2. Next, for me, was the screen_name summary. I saw several people on that list who I didn’t realise were in-the-know, and was able to remind myself to thank them soon!

3. The pie-chart saying other hashtags was also interesting, because it included the word “#excited”. Although this doesn’t seem to have *every* other hashtag, it was good to see.

4. Finally, “url” column was summarised as a pie-chart, showing me which urls were included within tweets containing the conference hashtag. This is very interesting, because I can see if people are linking to the index page, or to the ticket page for the site. Also, I can see what *isn’t* being linked (e.g. the Lanyrd page for the event or direct to eventbrite, which I expected to happen.)

These were all interesting, and helped me instantly better understand how people are talking about the conference. I should also point out that the tool ran automatically: all I did was to install the tool on my search data, and it presents me with this information without any setup. Best of all, it also showed me some things I wasn’t planning for. The list of re-tweeters, for example, jogged my memory, and made me consider asking some specific people to mention the event, which is something I hadn’t thought of doing.

I’m pretty excited about this tool, not just because it’s geeky and has charts, but because it’s at a very early stage and *already* did something useful with my social data. As it improves, I hope we get some more instant-win effects from it, and I’d be keen to hear what we could do to make it better, too.

It quickly became clear that the big issues facing the industry are:

• how to grow revenue from advertising
• how to cut the cost of serving multiple platforms like tablets, mobile devices and PCs
• how to innovate.

Unsurprisingly HTML5 was also a popular topic and a number of ready-made products featured in the presentations.

Day 2 was focused on innovation and I had an opportunity to talk about what ScraperWiki has been doing to help in the sector.  I tried to feature stories that data scientists from our community created specifically for the media.

They Write for You

I also wanted to talk about some of the women doing great work, so I rolled back to the story that Anna Powell Smith (@darkgreener) helped craft at our very first Hacks/Hacker day in January 2010. The story was about the number of articles written by MPs for British newspapers – it is a simple and effective visualisation: ‘They Write for You”.

A load of bubbles!

I also talked about the data-driven stories that Nicola Hughes (@datamineruk), Francis Irving(@frabcus) and Julian Todd (@goatchurch) created for the awarding-winning Dispatches programme. These focused around the National Asset Register and English Brownfield Sites.

I finished on the rose visualisation that Julian and Zarino Zappia (@zarino) made to enliven Somerset and Devon Fire Incidents. It seemed like a good candidate to show how local government data can be used to make an interesting, evergreen story:

Rose visualisation from Devon

Dr Johnny Ryan (@johnnyryan), author of ‘A History of the Internet and the Digital Future’ and Chief Innovation Officer at The Irish Times followed up by introducing three new media startups.  This was interesting because the paper is a reasonably conservative publication that has taken the unusual step of acting as a technology accelerator in Dublin. So, hats off to its editor Kevin O’Sullivan!  It provides space, desks, access to management and a platform for the startups to introduce their offerings into the media market.

Here are the three Dr Ryan mentioned:

Oliver Mooney (@olivermooney)  told us how GetBulb allows you to make compelling infographics simply by copying and pasting your data into a template.  They also have a wacky introduction video which made me smile!

Paul Quigley (@paulyq) introduced NewsWhip, a technology that tracks all the news shared on Facebook and Twitter each day to find the fastest-spreading, most-shared, high-quality stuff.

Neil O’Connor from Blockmetrics showed his technology to detects ads being blocked by website visitors, and how to analyse how much ad revenue is being lost as a consequence how companies can do something about it.

The industry is very well aware of the challenges it faces, although there was a level of surprise by some delegates that mobile advertising would not be a panacea against falling revenues. This industry faces tough times ahead, but refreshingly it is proactively looking at innovation as both a defence mechanism and a route to growth and profitability.

200 Aldersgate, London

Friendly data scientists

Maurice the ScraperWiki marmot

Devoted developers

The last two summers, we had a really good intern (Aidan Hobson Sayers - thanks for finding him for us, John!).

We’d like to do it again this year. We’ve opportunities in three areas, depending on your skills and interests.

1. Platform team – CoffeeScript, Backbone, Unix. We use Extreme Programming.
2. Data science team – Python, R. Scraping, statistics, working with customers.
3. Tool making team – gorgeous user interfaces, with a mixture of the above skills.

The deal is…

• Work with a friendly, talented team in Liverpool, where a whole community is quietly growing the UK’s next big tech cluster.
• It’s at ScraperWiki’s offices. You need to be either based in commuting range, or prepared to move here for at least 6 weeks over the summer.
• We pay either travelling expenses, or if you’re more experienced, the standard student summer placement week rate.
• Oh, and you learn about startups, and changing the world of data analysis on the web.

If you’d like to apply, please send:

• Your CV
• A link to a scraper you’ve written of some kind, or an open source project you’ve made a large contribution to

To francis@scraperwiki.com with the word “swintern” in the subject. We’ll take applications from either students or non-students.

The French port of Brest has the longest historical tidal record. The Joint Archive for Sea Level has hourly readings from 1846. Those of you wanting to follow along at home should get the code:

    git clone git://github.com/drj11/sea-level-tool.git
    cd sea-level-tool
    virtualenv .
    . bin/activate
    pip install -r requirements.txt

After that lot (phew!), you can get the data for Brest by going:

    code/etl 822a

The sea level tool is written in Python and uses our scraperwiki library to store the sea level data in a sqlite database.

Tide data can be surprisingly complex (the 486 pages of [PUGH1987] are testimony to that), but in essence we have a time series of heights, z. Often even really simple analyses can tell us interesting facts about the data.

As Ian tells us, R is good for visualisations. And it turns out it has an installable RSQLite package that can load R dataframes from a sqlite file. And I feel like a grown-up data scientist when I use R. The relevant snippet of R is:

    library(RSQLite)
    db <- dbConnect(dbDriver('SQLite'), dbname='scraperwiki.sqlite', loadable.extensions=TRUE)
    bre <- dbGetQuery(db, 'SELECT*FROM obs WHERE jaslid=="h822a" ORDER BY t')

I'm sure you're all aware that the sea level goes up and down to make tides and some tides are bigger than others. Here’s a typical month at Brest (1999-01):

There are well over 1500 months of data for Brest. Can we summarise the data? A histogram works well:

Remember that this is a histogram of hourly sea level observations. So the two humps show the most frequent sea level heights that appear in the hourly series. These are clustered around two heights that are more commonly observed than all others. These are the mean low tide, and the mean high tide. The range, the distance between mean low tide and mean high tide, is about 2.5 metres (big tides, big data!).

This is a comparitively large range, certainly compared to a site like St Helena (where the British imprisoned Napoleon after his defeat at Waterloo). Let’s plot St Helena’s tides on the same histogram as Brest, for comparison:

Again we have a mean low tide and a mean high tide, but this time the range is about 0.4 metres, and the entire span of observed heights including extremes fits into 1.5 metres. St Helena is a rock in the middle of a large ocean, and this small range is typical of the oceanic tides. It’s the shallow waters of a continental shelf, and complex basin dynamics in northwest Europe (and Kelvin waves, see Lucy’s IgniteLiverpool talk for more details) that gives ports like Brest a high tidal range.

Notice that St Helena has some negative sea levels. Sea level is measured to a 0-point that is fixed for each station but varies from station to station. It is common to pick that point as being the lowest sea level (either observed or predicted) over some period, so that almost all actual observations are positive. Brest follows the usual convention, almost all the observations are positive (you can’t tell from the histogram but there are a few negative ones). It is not clear what the 0-point on the St Helena chart is (it’s clearly not a low low water, and doesn’t look like a mean water level either), and I have exhausted the budget for researching the matter.

Tides are a new subject for me, and when I was reading Pugh’s book, one of the first surprises was the existence of places that do not get two tides a day. An example is Fremantle, Australia, which instead of getting two tides a day (semi-diurnal) gets just one tide a day (diurnal):

The diurnal tides are produced predominantly by the effect of lunar declination. When the moon crosses the equator (twice a nodical month), its declination is zero, the effect is reduced to zero, and so are the diurnal tides. This is in contrast to the twice-daily tides which, while they exhibit large (spring) and small (neap) tides, we still get tides whatever time of the month it is. Because of the modulation of the diurnal tide there is no “mean low tide” and “mean high tide”, tides of all heights are produced, and we get a single hump in the distribution (adding the fremantle data in red):

So we’ve found something interesting about the Fremantle tides from the kind of histogram which we probably learnt to do in primary school.

Napoleon died on St Helena, but my investigations into St Helena’s tides will continue on the ScraperWiki data hub, using a mixture of standard platform tools, like the summarise tool, and custom tools, like a tidal analysis tool.

Image “Napoleon at Saint-Helene, by Francois-Joseph Sandmann,” in Public Domain from Wikipedia

I quickly realised that there were better ways of visualising the data than just showing tables. For example, if there are only a few possible values for a column, it makes better sense as a pie chart.

For example, these are the oceans from the Climate Code Foundation’s sea-level station data (the same dataset that appeared in the last blog post).

After playing with a few datasets, and with David’s help, we found that the pies are useful when there are more than two but fewer than eight values.

The code that makes the pie chart is in the “fact_groups_pie” function in the facts.js file. I’m calling each possible visualisation a “fact”. There’s a bunch of code in the “add_fact” function in code.js which, for each possible fact, decides which has the highest priority, and shows that one for each column. For example, a pie chart (if there are few enough values) overrides a table.

The pie is made using Google charts (code in charts.js) – I deliberately wanted to keep things simple for this tool. Because the visualisations are automatically chosen, it didn’t feel right to hand craft them in D3.

You can play too! If you are part of the Beta, you can use the “Summarise automatically” tool yourself now on your own dataset. Either upload a spreadsheet with the “Upload spreadsheet” tool, or use the “Twitter search tool” or one of the coding tools to get some data you care about into ScraperWiki. Then choose “Summarise automatically” from the tools menu and see what surprises there are.

You’ll probably see one of the visualisation type I haven’t talked about yet. Next time – all about showing time and numbers using buckets…