Tag Archives: scraping

Digging Olympic Data at Londinium MMXII

Posted on July 24, 2012 by

This is a guest post by Makoto Inoue, one of the organisers of this weekend’s Londinium MMXII hackathon. The Olympics! Only a few days to go until seemingly every news camera on the planet is pointed at the East End … Continue reading →

Posted in events | Tagged , , , | Leave a comment

Three hundred thousand tonnes of gold

Posted on July 4, 2012 by

On 2 July 2012, the US Government debt to the penny was quoted at $15,888,741,858,820.66. So I wrote this scraper to read the daily US government debt for every day back to 1996. Unfortunately such a large number overflows the … Continue reading →

Posted in developer | Tagged , , , , , , , | 3 Comments

Software Archaeology and the ScraperWiki Data Challenge at #europython

Posted on June 29, 2012 by

There’s a term in technical circles called “software archaeology” – it’s when you spend time studying and reverse-engineering badly documented code, to make it work, or make it better. Scraper writing involves a lot of this stuff. ScraperWiki’s data scientists … Continue reading →

Posted in developer, events, opendata | Tagged , , , , | 2 Comments

Local ScraperWiki Library

Posted on June 7, 2012 by

It quite annoyed me that you can only use the scraperwiki library on a ScraperWiki instance; most of it could work fine elsewhere. So I’ve pulled it out (well, for Python at least) so you can use it offline. How … Continue reading →

Posted in developer | Tagged , , | 4 Comments

Fine set of graphs at the Office of National Statistics

Posted on March 22, 2012 by

It’s difficult to keep up. I’ve just noticed a set of interesting interactive graphs over at the Office of National Statistics (UK). If the world is about people, then the most fundamental dataset of all must be: Where are the … Continue reading →

Posted in opendata | Tagged , , , , | Leave a comment

How to get along with an ASP webpage

Posted on November 9, 2011 by

Fingal County Council of Ireland recently published a number of sets of Open Data, in nice clean CSV, XML and KML formats. Unfortunately, the one set of Open Data that was difficult to obtain, was the list of sets of … Continue reading →

Posted in developer, Scrapers | Tagged , , , , | 6 Comments

Scraping guides: Excel spreadsheets

Posted on September 14, 2011 by

Following on from the CSV scraping guide, we’ve now added one about scraping Excel spreadsheets. You can get to them from the documentation page. The Excel scraping guide is available in Ruby, Python and PHP. Just as with all documentation, you can … Continue reading →

Posted in developer | Tagged , , , | Leave a comment

Access government in a way that makes sense to you? Surely not!

Posted on May 11, 2011 by

alpha.gov.uk uses Scraperwiki, a cutting edge data-gathering tool, to deliver the results that citizens want. And radically for government, rather than tossing a finished product out onto the web with a team of defenders, this is an experiment in customer … Continue reading →

Posted in news, users | Tagged , , , , , , | 5 Comments

ScraperWiki: A story about two boys, web scraping and a worm

Posted on May 5, 2011 by

“It’s like a buddy movie.” she said. Not quite the kind of story lead I’m used to. But what do you expect if you employ journalists in a tech startup? “Tell them about that computer game of his that you … Continue reading →

Posted in developer, journalism, news, Scrapers | Tagged , , , , | Leave a comment

Scrape it – Save it – Get it

Posted on April 11, 2011 by

I imagine I’m talking to a load of developers. Which is odd seeing as I’m not a developer. In fact, I decided to lose my coding virginity by riding the ScraperWiki digger! I’m a journalist interested in data as a … Continue reading →

Posted in developer, users | Tagged , , , , , | Leave a comment