-
Recent Posts
Our Tweets
- RT @CMCLRN: @ScraperWiki Give your neighbours a RT? Less than a week to go until the ICTD Event in Liverpool!! goo.gl/qTsEh #N… 3 days ago
- Nice to see Google Doodle celebrating a Liverpudlian: bit.ly/13xf3mX 4 days ago
- The basic "View in a table" tool on beta.scraperwiki.com is lots faster after lots of work today from the team 5 days ago
- Images and Domains – @frabcus' 4th piece on automatically summarising data: bit.ly/YT40VF 5 days ago
- An interesting question on @quora – "How do I become a data scientist?" b.qr.ae/17Z5a2b 5 days ago
Find us on Facebook
Archives
Categories
Meta
Category Archives: developer
Scheduling! Keep your data fresh
We’ve added scheduling to the “Code in your browser” tool on beta.scraperwiki.com. For now it is daily, as that covers most people’s uses. Please ask if you need something else! Or have a look at the tool’s source code. Want … Continue reading
Posted in developer
Leave a comment
Free community accounts on the ScraperWiki Beta
We’ve been teasing and tempting you with blog posts about the first few tools on the new ScraperWiki Beta for a while now. It’s time to let you try them out first-hand. As of right now, the new ScraperWiki Beta … Continue reading
Summarise #1: Grouping automatically for you
Late at night, after a long conversation in a bar (after Social Media Cafe), Zach mentioned one feature that everyone loved about Kasabi. It had an overview page, which automatically summarised each dataset. Of course, Kasabi did it using linked … Continue reading
Posted in beta, developer
5 Comments
From future import x.scraperwiki.com
Time flies when you’re building a platform. At the start of the year, we announced the beginnings of a new, more powerful, more flexible ScraperWiki. More powerful because it exposes industry standards like SQL, SSH, and a persistent filesystem to … Continue reading
The next evolution of ScraperWiki
Quietly, over the last few months, we’ve been rebuilding both the backend and the frontend of ScraperWiki. The new ScraperWiki has been built from the ground up to be more powerful for data scientists, and easier to use for everyone … Continue reading
How to test shell scripts
Extreme hipster superheroes like me need tests for their shell. Here’s what’s available. YOLO: No automated testing Few shell scripts have any automated testing because shell programmers live life on the edge. Inevitably, this results in tedious manual ‘testing’. Loads … Continue reading
Posted in developer
2 Comments
We’re hiring: the world’s best data scientists!
If you’re a ScraperWiki coder with great communication skills and a passion for data, then you should probably bookmark our new Jobs page. We’ll be hiring for a few different roles over the coming months, and we’d love to hear … Continue reading
DumpTruck 0.0.3
I’ve added some new features to DumpTruck. Changes Dictionary case sensitivity I removed the dictionaries with case-insensitive keys because that just seemed to be delaying the conversion to case sensitivity. Ordered Dictionaries DumpTruck.execute now returns a collections.OrderedDict for each row … Continue reading
Posted in developer
Leave a comment
The state of Twitter: Mitt Romney and Indonesian Politics
It’s no secret that a lot of people use ScraperWiki to search the Twitter API or download their own timelines. Our “basic_twitter_scraper” is a great starting point for anyone interested in writing code that makes data do stuff across the … Continue reading
Three hundred thousand tonnes of gold
On 2 July 2012, the US Government debt to the penny was quoted at $15,888,741,858,820.66. So I wrote this scraper to read the daily US government debt for every day back to 1996. Unfortunately such a large number overflows the … Continue reading
