Mainly Data

Blogging for the Yoga Bear Cause

I’ve been lucky to watch Yoga Bear grow from an idea to a nationwide organization of hundreds of volunteers, yoga studios, and participants recovering from cancer. The Blogging for a Cause campaign from Zemanta is looking for nonprofits to help out, and I can’t think of one more worthy than Yoga Bear. From events like Yogapalooza and the Meeting of the Minds to their commitment to hilighting research on the effectiveness of alternative medicine in the treatment of cancer, Yoga Bear is enhancing the lives of many on a shoestring budget. Let’s hope Zemanta recognizes the great work Yoga Bear is doing and awards them the grant!

August in San Francisco

zachklein:

I’m renting a place near Alamo Sqaure for a month in order to be closer to a handful of friends I’ve been distant from for too long, to recruit developers, to learn the geography of the Bay Area, and to have a convenient jumpoff point for a few backpacking trips in the Cascades.

yay.

MMDS 2008 and CIM

Last week the second Workshop on Algorithms for Modern Massive Data Sets was held at Stanford University.  This workshop had an incredible density of prestigious speakers from the field of machine learning; I guess spending a weekend in California during the summer is an easy sell.

Each of the four days of the conference had a theme.  The official themes:

1. Data Analysis and Data Applications
2. Networked Data and Algorithmic Tools
3. Statistical, Geometric, and Topological Methods
4. Machine Learning and Dimensionality Reduction

After attending several talks by these machine learning luminaries over the course of four days, I tried to pull together a few common themes of my own for further exploration:

1. Incomplete Dyadic Data
2. Distributing Data and Computation
3. Manifolds with Noise
   

I’ll try to put a detailed post up on each of these three topics this week.

Over the past few years I’ve become a regular conference attendee.  It seems plausible that I will continue to attend KDD, VLDB, SIGMOD, and MMDS for years to come, leaving me with a major question: where can I find a longitudinal analysis of the content of these conferences?

It seems that these conferences provide an excellent yearly cross-section of the state of their respective fields, but I’m much more interested in the deltas.  What are the new topics, which topics are making forward progress, and which topics are losing steam?  Taking time to collect this information would provide some fascinating insight into the progress of science.

The KDD community has done some analysis of the DBLP data set, but not with the aims proposed above.  From the VLDB community has come some work that is a bit closer in intent: AnHai Doan’s Cimple project.  So far, they’ve produced the moderately useful DBLife, and they’ve outlined a promising research direction.

I’ve recently started a conference type on Freebase and have a simple tool to monitor conference progress using their API.  I’d like to make this tool more general and robust.  Any CIM students or conference geeks looking for a summer project?

REPL

Python is an amazing language for many reasons, but code prototyping via the interactive interpreter has crippled my development speed in languages that do not have a REPL.

For this reason, I was ecstatic when I joined Facebook and learned they had developed their own interactive shell for PHP, phpsh.  My Javascript development was hastened when I found Mozilla’s spidermonkey.

I also enjoyed learning Ruby and Erlang, because in Ruby, you have irb, and in Erlang, you have the Erlang shell, Eshell.

I recently came across CERN’s CINT, a REPL for C/C++.  Finally!  I have come to dread coding in these languages, especially C++, but I’m now looking forward to my next big C/C++ project.  If you’ve used CINT, drop me a line.

I’ve also started looking for a nice Java REPL.  There’s DynamicJava, and the Groovy interactive shell.  Anything else I should try?

Wired Magazine: (Brief) Dispatches from the "Petabyte Age"

I tend to geek hard when a publication drops a series of articles about petabyte-scale data analysis.  Frequent culprits include the SIGMOD Record or Teradata Magazine, but this week, Wired Magazine splashed into the pool.

Chris Anderson (the long tail one) starts things off by declaring that the data deluge spells trouble for the scientific method.  Clearly this claim is false; Google’s success is almost entirely due to rapid product iterations via thousands of hypothesis tests every month.  To give Chris the benefit of the doubt, he seems to be asserting instead that the practice of model development should be done in close concert with the collection of empirical data.  I couldn’t agree more.

Near the end of the article, Chris makes a common (if frustrating) mistake: he claims that the cluster used for the NSF’s CluE program will be running the Google File System, as well as crediting Google and IBM for building the software that will power this cluster.  In actuality, Yahoo deserves almost all of the credit, as they have done an incredible job scaling the Hadoop project to thousands of nodes and making the NSF program possible.  Just another testament to the effectiveness of Google’s PR machine.

The rest of the articles cover many application areas of large scale data analysis in brief, including agriculture, astronomy, high-energy physics, politics, epidemiology, and insurance.  There’s also a startlingly incoherent attempt to describe how MapReduce works that probably should have been left out.

Wired’s intentions were noble but their execution was not up to expectations.  I’m hoping that tomorrow’s MMDS Workshop will have a bit more substance.

Bloomberg for the Web? We Need Real-Time News, Data, and Analytics.

The NYT had an article today about the battle for efficient financial services information delivery currently heating up between Bloomberg and Thomson Reuters.  A few weeks ago, there were articles about the NASDAQ and NYSE making their stock quotes available in real time to multiple information outlets.  And just last week, Tibco, once a wholly owned subsidiary of Reuters, purchased Insightful.  Insightful makes the S-Plus statistical analysis software on which the R project is based.  Tibco already owns Spotfire, a firm that makes excellent software for data exploration.

These developments made me think back to my time in financial services.  Trading floors have the highest concentration of numerate people I’ve ever been around.  They also have ready access to superb information manipulation software.

Now consider the current state of analytics software for the web.  I won’t bother to list the major competitors, as they are all pretty mediocre.  As the online advertising space grows in sophistication and mechanisms to promote a more efficient market are introduced (see, for example, Right Media and ContextWeb’s ADSDAQ), a workbench for the real-time exploration of news and data related to the web will be a necessary tool for many quantitative marketers.

As Hal Varian points out, marketing is the next field to be overrun with quants, and I expect that the tools most useful in finance will be brought along for the invasion.