Date: Fri, 27 Jan 2012 14:29:57 +0000
From: Francis Irving <francis@scraperwiki.com>
To: team@scraperwiki.com
Subject: Capital!

Today we closed our round of investment from Enterprise Ventures and Blue Fountain.

In total, provided we hit certain milestones next August, and with the Knight Foundation money, this means we have a cool $1,000,000 of capital.

Many many thanks to Aidan who has done most of the work, and now has no hair left to keep him warm in his old age (it generally takes, and took, total one person full time for 6 months to get investment).

And to everyone else for making a company that someone would want to invest in.

Short FAQ

1. What’s this money for? It’s for us to create a viable business, by helping coders make data do things across the web. To do that, we have to reach clear product/market fit, with paying developers, and/or with corporations. So please continue to ask of everything you/we do “is this testing, in as lean a way as possible, how we can get to product/market fit?”.

2. How long will it last for? If we hit the revenue in our business plan, in theory until August 2014. Even in worst cases, it’ll last a year.

3. Can I tell the world? Not just yet. We’re not press releasing it immediately, it’s embargoed for writing about, blogging about or tweeting about. But feel free to tell friends and family.

4. When’s the party? Not sure, but Monday night in Liverpool looks best to me. Who isn’t free then? (At least for an early evening drink. Now the hard (and fun) part starts! Francis

For full details, read the press release. As you can see, we also have a new board member – I’ll write about her in a separate blog post. Any questions? Please ask in the comments!

During the final hours of the Columbia University hack day, I got to work on a corker in the form of the New York State Joint Committee on Public Ethics Lobbying filing system.

This is an aspx website which is truly shocking. The programmer who made it should be fired — except it looks like he probably got it to a visibly working stage, and then simply walked away from the mess he created without finding out why it was running so slowly.

Directions:
1. Start on this page.
2. Click on 2. Client Query – Click here to execute Client Query.
3. Select Registration Year: 2011
4. Click the [Search] button

[ Don't try to use the browser's back button as there is a piece of code on the starting page that reads: <script language="javascript">history.forward();</script> ]

A page called LB_QReports.aspx will be downloaded, which is the same as the previous page, except it is 1.05Mbs long and renders a very small table which looks like this:

If you are able to look at the page source you will find thousands of lines of the form:

<div id="DisplayGrid_0_14_2367">
	<a id="DisplayGrid_0_14_2367_ViewBTN"
href="javascript:__doPostBack('DisplayGrid_0_14_2367$ViewBTN','')">View</a>
	</div>

Followed by a very long section which begins like:

window.DisplayGrid = new ComponentArt_Grid('DisplayGrid');
DisplayGrid.Data = [[5400,2011,'N','11-17 ASSOCIATES, LLC','11-17
ASSOCIATES, LLC','','11-17 ASSOCIATES, LLC','NEW YORK','NY',10000,
'APR',40805,'January - June','11201',],[6596,2011,'N','114 KENMARE
ASSOCIATES, LLC','114 KENMARE ASSOCIATES, LLC','','114 KENMARE
ASSOCIATES, LLC','NEW YORK','NY',11961,'APR',41521,'January -
June','10012',],[4097,2011,'N','1199 SEIU UNITED HEALTHCARE
WORKERS EAST','1199 SEIU UNITED HEALTHCARE WORKERS EAST','','1199
SEIU UNITED HEALTHCARE WORKERS EAST','NEW YORK','NY',252081,'APR',
40344,'January - June','10036',],...

This DisplayGrid object is thousands of lists long. So although you only get 15 records in the table at a time, your browser has been given the complete set of data at once for the javascript to pagenate.

Great, I thought. This is easy. I simply have to parse out this gigantic array as json and poke it into the database.

Unfortunately, although it can be interpreted by the javascript machine, it’s not valid json. The quotes are of the wrong type, there are trailing commas, and we need to deal with the escaped apostrophes.

mtable = re.search("(?s)DisplayGrid.Data =\s*(\[\[.*?\]\])", html)
jtable = mtable.group(1)
jtable = jtable.replace("\\'", ";;;APOS;;;")
jtable = jtable.replace("'", '"')
jtable = jtable.replace(";;;APOS;;;", "'")
jtable = jtable.replace(",]", "]")
jdata = json.loads(jtable)

Then it’s a matter of working out the headers of the table and storing it into the database.

(Un)Fortunately, there’s more data about the lobbying disclosure than is present in this table if you click on those View links on each line, such as person names, addresses, amounts of money, and what was lobbied.

If you hover your mouse above one of these links you will see it’s of the form: javascript:__doPostBack(‘DisplayGrid_0_14_2134$ViewBTN’,”).

At this point it’s worth a recap on how to get along with an asp webpage, because that is what this is.

[The scraper I am working on is ny_state_lobby, if you want to take a look.]

Here is the code for getting this far, to the point where we can click on these View links:

cj = mechanize.CookieJar()
br = mechanize.Browser()
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
br.set_cookiejar(cj)

# get to the form (with its cookies)
response = br.open("https://apps.jcope.ny.gov/lrr/Menu_reports_public.aspx")
for a in br.links():
    if a.text == 'Click here to execute Client Query':
        link = a
response = br.follow_link(link)

# fill in the form
br.select_form("form1")
br["ddlQYear"] = ["2011"]
response = br.submit()
print response.read()  # this gives massive sets of data
br.select_form("form1")
br.set_all_readonly(False)

The way to do those clicks onto “DisplayGrid_0_14_%d$ViewBTN” (View) buttons is with the following function that does the appropriate __doPostBack action.

def GetLobbyGrid(d, year):
    dt = 'DisplayGrid_0_14_%d$ViewBTN' % d
    br["__EVENTTARGET"] = dt
    br["__EVENTARGUMENT"] = ''
    br.find_control("btnSearch").disabled = True
    request = br.click()
    response1 = br1.open(request)
    print response1.read()

…And you will find you will have got exactly the same page as before — including that 1Mb fake json data blob.

Except it’s not quite exactly the same. There is a tiny new little section of javascript in the page, right at the bottom. (I believe I discovered it by watching the network traffic on the browser when following the link.)

<script language=javascript>var myWin;myWin=window.open(
'LB_HtmlCSR.aspx?x=EOv...QOn','test','width=900,height=450,toolbar
=no,titlebar=no,location=center,directories=no, status=no,menubar=
yes,scrollbars=yes,resizable=yes');myWin.focus();</script>

This contains the secret new link you have to click on to get the lobbyist information.

    html1 = response1.read()
    root1 = lxml.html.fromstring(html1)
    for s in root1.cssselect("script"):
        if s.text:
            ms = re.match("var myWin;myWin=window.open\('(LB_HtmlCSR.aspx\?.*?)',", s.text)
            if ms:
                loblink = ms.group(1)
    uloblink = urlparse.urljoin(br1.geturl(), loblink)
    response2 = br1.open(uloblink)
    print response2.read()   # this is the page you want

So, anyway, that’s where I’m up to. I’ve started the ny_state_lobby_parse scraper to work on these pages, but I don’t have time to carry it on right now (too much blogging).

The scraper itself is going to operate very slowly because for each record it needs to download 1Mb of uselessly generated data to get the individual link to the lobbyist. And I don’t have reliable unique keys for it yet. It’s possible I could make them by associating the button name with the corresponding record from that DisplayGrid table, but that’s for later.

For now I’ve got to go and do other things. But at least we’re a little closer to having the picture of what is being disclosed into this database. The big deal, as always, is finishing it off.

Journalism School Columbia University

Columbia University, arguably the best Journalism school in the world is giving us the opportunity of a lifetime. We are hosting our first ever US event (Journalism Data Camp #jdcny)  and its their first hackathon in a proud 100 year history. It is scary but very exciting! The campus is lovely and welcoming.  Located in uptown Manhattan just above Central Park, the ‘J’ School building is one of the oldest in the complex.

Joseph Pulitzer

The entrance hallway has a bust of Joseph Pulizer the Hungarian-American Publisher who established Columbia as the world’s first school of Journalism.  We also know him for the Pulitzer Prize which is synonymous with excellence in journalism and the arts since 1917.

Our event is designed as an attempt to marry highly skilled journalists with data scientists, coders, statisticians and technology!  As he was such an innovator we would hope that Joseph would have approved! Our mission is to ‘Liberate Data’ to allow the professionals to hold power and money to account and to perform the exemplary role of being guardian of freedom of speech and to ask hard questions.

Emily Bell and Francis Irving will do a short introduction on their thoughts on digital journalism and the world of data.

Agenda…here it is…

…this is a fairly crude cropped PDF which we created with the ScraperWiki cropper tool and  it works beautifully.

So what exactly are we going to do tomorrow  and Saturday?  We think that we have packed the event with stuff!

Project Data Derby

The project data derby is where people will work together in teams to create ‘data driven’ stories and applications. This is a facilitated session where people will learn and understand the various techniques and skills to work with data. We will look to have multiple skill sets in a team and they will be encouraged to follow a process that will give the best outcome at the end of the two days!

Liberate the Data

We have been asking people to nominate data sets for the past few weeks and we already have a list!  We will put these on index cards and ask our ‘Data Liberators’ to dig up the data, get it into a structured format and publish it for the project teams and the world to see and reuse!

Learn to Scrape

We are also running two three hour tutorials on Friday morning (Python) and afternoon (Ruby).  Our chief data scientist Julian Todd and Thomas Levine data advocate, will run the sessions with the assistance of Michelle Koeth (Code for America Fellow) supporting and assisting the students.   They will cover things like identifying good targets for webscraping.  Navigating the complexity of different types of web pages.  Attendees will create their own scrapers to get and analyse the Department of Labour’s Unionreports.gov (Collective bargaining agreement listings).  The objective will be to get the data into a structured format, and join it with data from the US census in order to establish the number and order of union employees across the US by state.  If time allows we will also try to encourage people to do further analysis.

We will have prizes for the best projects, most daring data liberation and the craziest constructed data scrapers.   Our judges will be Aron Pilhofer from the New York Times and Susan E McGregor Assistant Professor of Journalism at Columbia.

And there’s more…

Hear from the pros!  On Friday and Saturday we are running some lunchtime lightening sessions in the plenary room and you will hear from Tom Lee from the Sunlight Foundation who will talk about a joint project between ScraperWiki and Sunlight and Jake Porway – Data without Borders who will talk about some of the fabulous projects that they are currently running.

A big big thanks to Tahiat Mahboob and Sam Guzik our two Digital Media Associates at Columbia University Graduate School of Journalism who have very generously offered their time to film the event!  Hats off to the facilities team at Columbia for all their help with the logistics.

We look forward to seeing everyone who has signed up and a million thanks for supporting us.

This list is only for backup in case not enough ideas show up with people on the day (never happens, but it’s always a fear).

1. Iowa accident reports

The site http://accidentreports.iowa.gov contains all the police roadside reports of accidents. It’s easy to scrape because the database ids are consecutive numbers:

http://accidentreports.iowa.gov/index.php?pgname=IDOT_IOR_MV_Accident_details&id=50070

And it contains thousands of rinky-dink diagrams of the incidents.

First step is to copy all the html from each page into one database. Second step is to scan through all these pages and progressively extract more and more data from them.

Contrast with dataset of accidents available for the UK.

2. South Dakota state budget information

Apparently complete set of expenditures, contracts and revenues disclosed on http://open.sd.gov/ in a form that is easy to scrape (some datasets even allow CSV download). Many states do this, with varying degrees of success.

Use this case to learn how to restructure and analyse financial accountancy flow information. Can you find any contracts that have suddenly been dropped in favour of another supplier?

3. New York School budgets

The site schools.nyc.gov/AboutUs/funding/schoolbudgets/GalaxyAllocationFY2012 requires a school code. Try “M411″.

Apparently there is this spreadsheet of school codes.

Is there anything interesting to plot across all schools, such as the PSAL SNAPPLE FUNDS?

4. New York Lobbying registers

Lobbying at the state and city level. Some of this is challenging.

Is there a cross-over between the jurisdictions? Can you uniquely identify the corporate interests and relate them to the legislative or regulatory program?

5. Court case information

Go to https://www.dccourts.gov/cco/ (Try “Lockheed”). Not obvious where the information is.

The New York City courts are behind a captcha. Maybe better luck with the New York State courts.

Court datasets are usually very difficult to obtain and jealously protected. The legal process resists modernization and is universally paper based. Electronic documents (contracts, settlements, filings) almost always turn out to be image scans of papers.

6. New York City Police crime data

There are weekly PDFs for each police precinct. These are taken down and replaced by the next one, so there is no historical record.

Luckily someone has scraped the data since 2010, though the numbers may need some processing before you map them.

7. New York State gas drilling permits

These are available but don’t seem to have been updated recently. What’s going on?

Wouldn’t it be nice to make another twitterbot to be friends with NorthSeaOil1?

Don’t forget to read the Well ownership transfers.

Click the image to see the PDF!

Thank you

The Tow Center for Digital Journalism at Columbia University Graduate School of Journalism is proud to be partnering with Knight News Challenge winner ScraperWiki this Friday and Saturday for their first Journalism Data Camp in the U.S. This event provides us with an opportunity to host a wide range of programmers, journalists and educators interested in expanding access to essential data sets, while connecting those communities to one another. We are also looking forward to extending the impact of this weekend’s activities by working in conjunction with our colleagues at the Stabile Center for Investigative Journalism and The New York World to further pursue those stories related to New York accountability issues that may be touched on during this weekend’s data “liberation” activities.

As an online tool, ScraperWiki is an innovative technical platform that allows users to build, test, and execute programmatic “scrapers” that transform web pages and pdfs into more accessible, usable data formats. As an online archive and repository, ScraperWiki helps improve access to scraped data sets by making them collectively available on their website. Finally, as a web-based collaboration space, ScraperWiki helps convene journalists and programmers around projects of shared interest, in addition to fostering peer-to-peer training and support.

Each of the above features of the ScraperWiki platform resonates closely with the Tow Center’s own priorities for data journalism. Making data available in formats that can be easily parsed, analyzed, and distributed is an essential part of data transparency, and the accountability journalism it serves. Providing a public access point for that data allows both journalists and their audiences to fact-check and elaborate upon the work that their peers have done, leveraging it against future projects and creating more comprehensive resources. And of course, the knowledge sharing and collaboration that takes place between programmers and journalists through ScraperWiki echoes the Tow Center’s mandate to educate and innovate at the intersection of computer science and journalism, both through its own dual-degree program in computer science and journalism, and through such public events as this one.

While we are certain that ScraperWiki will find ready adoption in cities and newsrooms throughout the country in the months to come, we look forward to growing an ongoing relationship with ScraperWiki and its contributors here in the New York area. By hosting this event we hope to introduce many of our students and colleagues to a truly remarkable tool, one whose impact on our industry only begins this weekend.

Luckily, help is at hand in the form of the ScraperWiki email alert system.

As you may have noticed, when you do any work on ScraperWiki, you start to receive daily emails that go:

Dear Julian_Todd,

Welcome to your personal ScraperWiki email update.

Of the 320 scrapers you own, and 157 scrapers you have edited, we
have the following news since 2011-12-01T14:51:34:

Histparl MP list - https://scraperwiki.com/scrapers/histparl_mp_list :
  * ran 1 times producing 0 records from 2 pages
  * with 1 exceptions, (XLRDError: Unsupported format, or corrupt file: Expected BOF record; found '<!DOCTYP')

...Lots more of the same

This concludes your ScraperWiki email update till next time.

Please follow this link to change how often you get these emails,
or to unsubscribe: https://scraperwiki.com/profiles/edit/#alerts

The idea behind this is to attract your attention to matters you may be interested in — such as fixing those poor dear scrapers you have worked on in the past and are now neglecting.

As with all good features, this was implemented as a quick hack.

I thought: why design a whole email alert system, with special options for daily and weekly emails, when we already have a scraper scheduling system which can do just that?

With the addition of a single flag to designate a scraper as an emailer (plus a further 20 lines of code), a new fully fledged extensible feature was born.

Of course, this is not counting the code that is in the Wiki part of ScraperWiki.

The default code in your emailer looks roughly like so:

import scraperwiki
emaillibrary = scraperwiki.utils.swimport("general-emails-on-scrapers")
subjectline, headerlines, bodylines, footerlines = emaillibrary.EmailMessageParts("onlyexceptions")
if bodylines:
    print "\n".join([subjectline] + headerlines + bodylines + footerlines)

As you can see, it imports the 138 lines of Python from general-emails-on-scrapers, which I am not here to talk about right now.

Using ScraperWiki emails to watch the weather

Instead, what I want to explain is how I inserted my Good Weather Weekend Watcher by polling the weather forecast for Holyhead.

My extra code goes like this:

weatherlines = [ ]
if datetime.date.today().weekday() == 2:  # Wednesday
    url = "http://www.metoffice.gov.uk/weather/uk/wl/holyhead_forecast_weather.html"
    html = urllib.urlopen(url).read()
    root = lxml.html.fromstring(html)
    rows = root.cssselect("div.tableWrapper table tr")
    for row in rows:
        #print lxml.html.tostring(row)
        metweatherline = row.text_content().strip()
        if metweatherline[:3] == "Sat":
            subjectline += " With added weather"
            weatherlines.append("*** Weather warning for the weekend:")
            weatherlines.append("   " + metweatherline)
            weatherlines.append("")

What this does is check if today is Wednesday (day of the week #2 in Python land), then it parses through the Met Office Weather Report table for my chosen location, and pulls out the row for Saturday.

Finally we have to handle producing the combined email message, the one which can contain either a set of broken scraper alerts, or the weather forecast, or both.

if bodylines or weatherlines:
    if not bodylines:
        headerlines, footerlines = [ ], [ ]   # kill off cruft surrounding no message
    print "\n".join([subjectline] + weatherlines + headerlines + bodylines + footerlines)

The current state of the result is:

*** Weather warning for the weekend:
  Mon 5Dec
  Day

  7 °C
  W
  33 mph
  47 mph
  Very Good

This was a very quick low-level implementation of the idea with no formatting and no filtering yet.

Email alerts can quickly become sophisticated and complex. Maybe I should only send a message out if the wind is below a certain speed. Should I monitor previous days’ weather to predict whether the sea will be calm? Or I could check the wave heights on the off-shore buoys? Perhaps my calendar should be consulted for prior engagements so I don’t get frustrated by being told I am missing out on a good weekend when I had promised to go to a wedding.

The possibilities are endless and so much more interesting than if we’d implemented this email alert feature in the traditional way, rather than taking advantage of the utterly unique platform that we happened to already have in ScraperWiki.

The way in which Information Technology is taught in England is so dull and harmful it should be scrapped – that’s the view of the Education Secretary Michael Gove.
‘A nation of digital illiterates’ (BBC)

Many years ago there was a total corporate take-over of the computer software sector in the UK. Big money was to be made out of controlling the profits generated by software applications, which were protected from competition by incompatible and inoperable standards and the force of law. (An attempt by the UK government to establish a very modest requirement for open standards was successfully killed off last week.)

One of the most painful aspects of this take-over was the way in which the same corporations managed to deform the entire education system into serving their purposes. All things resembling actual computer programming were cleansed from the curriculum, which was instead packed with dire, tedious training modules for drilling students in how to use those self-same corporations’ big software suites.

Back in the 1980s, before this take-over, I learned to program on the BBC Micro, which was widespread throughout the UK at the time. There is good evidence that this was the reason there has been such a strong software industry in the UK over the last three decades.

Let’s just use the words of computer games pioneer Ian Livingston from his February 2011 report:

“Given that the new online world is being transformed by creative technology companies like Facebook, Twitter, Google and video games companies, it seems incredible that there is an absence of computer programming in schools. The UK has gone backwards at a time when the requirement for computer science as a core skill is more essential than ever before. When Sir Clive Sinclair launched the ZX Spectrum in 1982, affordable computers were eagerly purchased for the homes of a creative nation. At the same time, the BBC Micro was adopted as the computer platform of choice for most schools and became the cornerstone of computing in British education in the 1980s. There was a thirst for creative computing both in the home and in schools creating a further demand at universities for courses in computer science. This certainly contributed to the rapid growth of the UK computer games industry.

“But instead of building on the BBC’s Computer Literacy Project in the 1980s, schools turned away from programming in favour of ICT. Whilst useful in teaching various proprietary office software packages, ICT fails to inspire children to study computer programming. It is certainly not much help for a career in games. In a world where technology affects everything in our daily lives, so few children are taught such an essential STEM skill as programming. Bored by ICT, young people do not see the potential of the digital creative industries. It is hardly surprising that the games industry keeps complaining about the lack of industry-ready computer programmers and digital artists.”

The official Government response was mostly public relations, mentioning developments that have nothing to do them such as Raspberry Pi, which, incidentally, appears to be an attempt by Livingston’s generation to recreate the 1980s when we once watched computers on TV (instead of watching TV on computers).

While this change in Government policy is absolutely vital, it is odd the way the people lobbying for it haven’t branched outside of their narrow fields of computer games and computer graphics – which are ultimately little more than a game of shifting pixels around on a VDU, utilizing very mature technologies based on software applications developed by large corporations.

They’re missing the point: This is the era of smart energy monitors that need to be coupled to microcontrollers that do something with appliances over the internet in response to data. Or robotrading bank accounts that use live data feeds to monitor and execute your share portfolio while you’re watching Corrie. The focus should be on Arduinos and simple robotics to control the energy use in your house, or on creative programming to trump the hedgefunders.

Recently I have been doing experiments through the interface of a home-built 3D printer, completely bypassing their UI application to drive it directly through the serial port.

Here is my first result:

Come on guys. I know it takes years to mastering the ability to render the perfect series of CGI frames of a spoon stirring the brown liquid in a cup of tea. But get a computer controlled robot to actually make me a cup of tea — that would be something!

はじめに

みなさんスクレイプ（Scrape）という単語はご存知でしょうか？

ウェッブページから特定のデータを引っこ抜く作業のことをスクレイピング（Scraping）と呼びます。

昨今のホームページではデータを簡単に提供するためのAPI（Application Programming Interface）というしくみが多いので「なんで今更そんなの必要なの」と思われる方>も多いかもしれません。しかしながら前回起きた東日本大地震の際、地震や電力の速報や、各地の被害状況を把握するために必要な政府の統計情報などがAPIとして提供されておらず、開発者の中には自分でスクレイパー（Scraper）用のプログラムを書いた人も多いのではないのでしょうか？　ただそういった多くの開発者の善意でつくられたプログラムがいろいろなサイトに散らばっていたり、やがてメンテナンスされなくなるのは非常に残念なことです。

そういうときにScraperWikiの出番です。

ScraperWikiとは

ScraperWikiはイギリスのスタートアップ企業で、スクレイパーコードを共有するサイトを提供しています。開発者達はサイト上から直接コード（Ruby, PHP, Python）を編集、実行することができます。スクレイプを定期的に実行することも可能で、取得されたデータはScraperWikiに保存されますが、ScraperWikiはAPIを用意しているので、このAPIを通して、他のサイトでデータを再利用することが可能です。

「Wiki」といっているだけあって、一般公開されているコードは他の人も編集したり、またコードをコピーして他のスクレイピングに利用することもできます。定期的に実>行されているスクレイパーがエラーを起こしていないかをチェックする仕組みがあり「みんなでスクレイピングを管理」するための仕組みがいたるところにあります。

ScraperWikiは、もともとイギリスで、どの議員がどの法案に賛成または反対票を投じたかを議会のサイトから創業者の一人が2003年頃にスクレイプしたことを起源に持ちます。

日本であればちょうどこういったページでしょうか？

現在ではGuardian社といった大手報道機関が企業ロビイストの議会での影響力を調べるのにつかったり、イギリス政府自身がalpha.gov.ukというプロトタイプサ>イトで、各省庁に点在したデータを一元的にアクセスするための仕組みとしてScraperWikiを使っているそうです。

ScraperWikiのビジネスモデルですが、一般公開するコードに関しては無料ですが、非公開にしたり、定期的にスクレイプする量などに応じて課金するようになっています。

前置きが長くなってきましたが、実際に使ってみましょう。

既存のスクレイパーを眺めてみる

「ScraperWiki」でGoogle検索すると、すでにScraperWikiを使用している日本人の方がいらっしゃいました。

• 「スクレイピングするなら ScraperWiki 使うといいよ 」

ここでは衆議院議員のデータをスクレイプするのに使用しています。

• Members of the House of Representatives of Japan

データは月一回走るように設定されていたり、複数のcontributorがいるのがわかります。

ページの下の方にはスプレッドシート形式でデータを閲覧できるようになっていますが、これだけだと他のサイトで再利用とか難しいですよね。そういうときは”Explorer with API”ボタンをクリックしてみて下さい。そこのページの最後に以下のようなurlがあると思います。

https://api.scraperwiki.com/api/1.0/datastore/sqlite?format=jsondict&name=members_of_the_house_of_representatives_of_japan&query=select%20*%20from%20%60swdata%60%20limit%2010

このurlにアクセスすると、先ほどのデータをJSON(Javascript Object Notation)で返してくれます。出力フォーマットは CSV, RSS,HTMLテーブルといった他の形式にも対応している上sql文をつかってフィルタリングなどをかけることも可能です。

select * from `swdata` where party = ‘民主’

COPYして独自のスクレイパーを作ってみる

ブラウザの”バック”ボタンを押して先ほどのページのスプレッドシートの下の方に目を通してい見て下さい。”This Scraper in Context”というところ”Copied To”という項>目があります。これはこのソースコードがコピーされ、他の用途に利用されていることを示しています。

そこに「makoto / Members of the House of Councillors of Japan」とあるの>でクリックしてみて下さい。実はこれは私が参議院議員の名簿を抜き出すために作ったスクレイパーです。衆議院と参議院はそれぞれ別にホームページを持っているのです>が、それぞれの議員名簿のページが結構似ていたので簡単に流用できるのではと思っていました。

• 衆議院議員一覧
• 参議院議員一覧

作り方は簡単でCopyリンクをクリックするだけです。ログインしていなくてもコピーをとれますが、これを機にアカウントを取得するのをお勧めします。

”Edit”ページを開くとその場でコードを編集するためのオンラインエディタが現れます。下にある”Run”ボタンを押すと実際にサイトからデータをとってきている模様が見て取れます。

http://www.screenr.com/embed/BgQs

以下がオリジナルのコードと私のコードの差異です。

衆議院と参議院のページ異なっていたため変更した点はは４つほどありました。

• エンコーディング（文字の表示形式）がUTFとShift-JISでことなる
• 衆議院のページは複数ページにまたがっているが参議院ページ１ページのみ
• 衆議院のページで議員名は「くん」づけ。参議院のページは芸名と本名の両方が載っている

他にもHTMLのページの文法が微妙に違っていたので、XPathという、HTMLに構造的にアクセスするための書式を少し換えました。

もちろんこれらの変更をするのにはある程度のプログラミング知識が必要なのですが。動くサンプルを少し自分用にカスタマイズするScraperWikiはプログラミングを勉強したい人にとっても絶好の教材なのではないでしょうか？　私自身XPathはあまり使ったことがなかったのですが、このもとプログラムを参考にすることで比較的簡単に学習できました。

最後に

今回のScraperWikiの簡単なチュートリアルで概要がわかっていただけたでしょうか？

公共機関、メディアや政府機関の中でインターネットを通じた情報公開は進んできていますが、「マッシュアップを前提としたデータの再利用」を考慮したサイトが十分で>ないのが現状です。そういった状態に一石と投じるべくScraperWikiは活動しており、ヨーロッパのジャーナリストや政府関係者の間では徐々に認知度があがってきております。　現在ScraperWikiでは米国でのワークショップを予定していますが、日本でもワークショップを始めるべく準備をしている所です。　もし興味のある方はコンタクトページより気軽にご連絡下さい。

We have been working with Emily Bell @emilybell, Director of the Tow Center and Susan McGregor @SusanEMcG, Assistant Professor at the Columbia J School to plan the event. The main objective is to liberate and use New York data for the purposes of keeping business and power accountable.

After a short introduction on the first day, we will split the event into three parallel streams; journalism data projects; liberating New York data; and ‘learn to scrape’. We plan to inject some fun by running a derby for the project stream and also by awarding prizes in all of the streams.  We hope to make the event engaging and enjoyable.

We need journalists, media professionals, students of journalism, political science or  information technology, coders, statisticians and public data boffs to dig up the data!

Please pick a stream and sign-up to help us to make New York a data driven city!

Our thanks to Columbia University, Civic Commons, The New York Times, and CUNY for allowing us to use their premises as we sojourned in the big apple

Zarino has created a map with our US events which we will update with additional events as we add locations. https://scraperwiki.com/events/