ScraperWiki

An outfit called UnLtd has these Better Net Awards of between £5,000 and £15,000 for “original and innovative projects” that increases education about the Internet and IT, or provides an internet based solution to social, educational, health and environmental issues, and so on. 

Your project is not supposed to be part of your paid employment, involve political campaigning, encourage commercial disharmony, employ people other than yourself, or fund living expenses. 

Well, that pretty much blows any of the ideas I tend to come up with.  Maybe I have the wrong attitude.  Or perhaps my attitude is a result of the ideas I happen to come up with.  

If these awards are your cup of tea, do look to their case studies for inspiration, as well as their ideas bank.

Update: If you have already have something working, then you can consider entering it into the Nominet Awards.

The relevance to us is that a proportion of the winning ideas will certainly rely on structured local public data — of the kind that ScraperWiki is intended to produce.  Suppose your idea is a creative way to encourage volunteers to visit their local old people’s home (eg to set up and maintain a free internet terminal in their common room).  Well, for that you’re probably going to need a database of old people’s homes.  And this can be scraped from the Care Quality Commission. 

The ScraperWiki technology is supposed to make it cheaper and easier to implement new and better ideas for these kinds awards.  So go for it.  Their online form is here.

(Posted by Richard)

Last Friday we organised The Hacks and Hackers Hack Day to see what happens when you put journalists and developers in the same room and ask them to come up with a data-driven story in one day. We also wanted to see how ScraperWiki would fare being used in anger in one of the areas we designed it for - data-driven journalism*.

We were a bit worried beforehand about whether the whole event would work, whether journalists and developers - who generally have different working practices - would work well together, and whether the hack day format would translate well to journalism or not. But it turned out to be a fantastic day and a great learning experience, with 30 top developers and journalists from organisations like the BBC, Financial Times, Times Labs and mySociety, as well as freelancers and members of the ScraperWiki team.

Groups formed quickly as people started discussing their ideas, and by the end of the day we all sat down to watch presentations from nine projects, who used everything from screen-scraping to old-fashioned cold-calling to get at data and turn it into something meaningful.

Here’s a list of what the teams built - or scroll down to the bottom to see a video of who won.

The Lazy Commuter

Shiva Kumar-Naspuri from the BBC, Francis Irving of mySociety, Simon Briscoe from the Financial Times and Simon Willison (who, as you will see below, managed to clone himself for the day) of the Guardian set out to find “where the laziest people in Britain live”. Using data from data.gov.uk they mapped and profiled the places where people make the most really short journeys in the UK.

Conservative Safe Seats

Developer Edmund van der Burg, freelance journalist Anne Marie Cumiskey, Charlie Duff from HRzone.co.uk, Ian McDonald of the BBC and Dafydd Vaughn munged a whole host of datasets together to produce an analysis of the new Conservative candidates in the 12 safest Tory seats in Britain.

Their conclusions: British white and male, average age 53, Oxford-educated, rarely on Facebook or Twitter.

r.

Data.gov.uk Format Verifier

After a few initial conversations about how journalists could work what is and is not useful on the new data.gov.uk website, Tom Morris came up with the idea for a tool to crowdsource information on the format of the datasets available - and built the data.gov.uk format verifier.

At the time of writing, of the 2905 datasets listed on data.gov.uk, all but 405 had already been classified using the tool, and there is talk of importing the data created back into data.gov.uk!

This wasn’t remotely the kind of project we thought would come out of the hack day, but is a lovely example of what can happen when people from one discipline explain a problem to someone from another.

They Write For You

Jonathan Richards from Times Labs and developers Anna Powell-Smith, Premasagar Rose, Simon Willison and Julian Burgess took a look at which MPs write for which newspapers.

Using ScraperWiki to get at the data contained on the online news archive Journalisted and the list of MPs from the TheyWorkForYou.com API, they came up with this visulisation:

One surprising finding was that the Guardian has twice as many articles by MPs as any other newspaper. Full version

Lords Interests

Rob McKinnon from Who’s Lobbying did a comparison on which members claim what from the House of Lords.

City Hall Goes to Lunch

For my own little hack, I started thinking what I could do with a dataset I’d recently added to ScraperWiki that listed gifts and freebies that the Mayor of London and London Assembly members have received.

With a bit of data jiggling, I worked out that you can map where they have been taken out to dinner, as well as calculating the porkiest AM. (More in this in a future blog post.)

Who Pays Who (Enterprise Ireland)

Gavin Sheridan from TheStory.ie and Duncan Parkes of mySociety used ScraperWiki to combine a list of grants made by Enterprise Ireland (which Gavin had aquired via an FOI request) with the profile data listed on the Enterprise Ireland website. This will no doubt be a source for stories in the near future.

Metascopes

Writer and designer David McCandless and Simon Willison from the Guardian set about turning a year’s worth of horoscopes, screen-scraped from the Yahoo astrology website, into a beautiful, tongue-firmly-in-cheek visualisation that shows the bunkum of star signs in their full glory.

Bank Holidays

As the presentations were taking place, there was a last-minute hack from Julian Burgess and Chris Taggart in the form of a reusable screen-scraper for grabbing the offical list of UK bank holidays (the government provides these on a web page hidden away on direct.gov.uk).

The winners…

A huge thanks to Tom Loosemore from 4iP and Tom Steinberg from mySociety for judging the projects. Here’s the video of them announcing the winners, along with the winning presentations.

All in all it was a great day, and the quality of the projects was first-rate given the limited time. We’d love to organise anther Hacks and Hackers Hack Day, so please get in touch with us if you are interested!

* ScraperWiki wasn’t compulsory, but it was lovely to see it used in some of the hacks and we got some great feedback.

We think great things can happen when people who understand what makes a great story and top developers with an eye for data get together. So, in the New Year we’re hosting a hackday for journalists and developers.

The idea is to team people up from different disciplines for the day to rummage through public datasets - the newly opened-up government data, stuff in ScraperWiki and lots of other sources - to come up with stories, apps and visulisations that show what the future of data-driven journalism will look like.

If you are interested, you can grab a ticket here.

(ps there will be prizes)

We’ve been putting the alpha release of ScraperWiki though its paces by grabbing a few datasets and poking about with them.

One of the things we have been taking a look at is police priorities across London, as set by local people.

Under the Safer Neighbourhoods scheme, each council ward has a dedicated team of police officers and a panel made up of local people, who help set three or four priorities for the police in the area. The idea is to let locals tell the police what’s most affecting their community.

Most police forces publish these priorities somewhere on the web, but there’s no central register where you can compare them, and no way of checking when they change. So, just the sort of thing we are building ScraperWiki for then!

We set about scraping the data for all 624 wards in the Metropolitan Police area, like this one, and came out with somewhere near 1,800 priorities. The top ten (tidied up a bit*) is here:

1. Burglary (364)
2. Groups of youths (359)
3. Motor vehicle crime (193)
4. Drugs (187)
5. General anti-social behaviour (124)
6. Alcohol (66)
7. Road safety (63)
8. Youth engagement (53)
9. Motorists (40)
10. Robbery - personal property (37)

The first thing that jumps out is the lack of variation: the top five priorities account for around 70% of the total of all priorities. Does that say anything about how well the system picks up local concerns? Or is it just that the problems facing communities across London are more or less the same?

Secondly, ‘burglary’ and ‘groups of youths’ are the run-away winners in the priority stakes. The latter raises questions about how many younger people sit on the panels that set local priorities. What is the demographic make-up of each panel - and does it reflect the fact that 40% of Londoners are under 30? Unfortunately, that data isn’t available to scrape.

Finally, it’d be fascinating to compare the data by area - do Islingtonites worry about the same things as natives of Kensington? How do the concerns of well-heeled Harrow differ from deprived Hackney? We haven’t made these comparisons, but then we’re programmers, not journalists.

Anyway, if you want to explore the data yourself we’ve uploaded it to a Google spreadsheet. Please let us know if you do anything fun with it (and journalists, please give us a credit if you use it anywhere ;) ).

If you are interested in scraping the priorities for your local police force (it would be great to get every force in the country), please get in touch with us and we’ll send you an invite to our alpha site.

——————————————

* Lots of priorities are prefixed by “anti-social behaviour”, e.g. “anti-social behaviour by youths loitering” or “anti-social behaviour related to alcohol”. We removed the term for clarity, so the above terms became “youths loitering” and “alcohol”.

We also merged similar or overlapping priorities, e.g. “prostitutes” and “prostitution”, or “groups of youths” and “youths loitering”. The original terms can all be seen in the “raw data” tab of the Google spreadsheet.

image (cc) http://www.wordle.net