The Practical Quant

When evaluating Investment Funds, use Dollar-weighted Returns

2012-02-01T09:00:00.000-08:00

Investors, particulary ones who employ investment professionals (e.g., institutional investors & pension funds), should be using dollar-weighted returns in their fund/manager selection process.

Sarah Palin and machine-translation

2012-01-25T09:11:00.000-08:00

I just came across this incomprehensible quote from Sarah Palin, that at first blush I thought was a misquote:

“I think what Rick Perry having dropped out and that patriot having done well for the front-runner, whom I will call Newt Gingrich now, being the front-runner, having endorsed him, was a good smart move,” Palin said on Fox News. “He kind of took one for the team there, the conservative team, when he dropped out.”

But then the same article contains an embedded video that shows the above quote was verbatim:

You would think that a politician of her stature should at least be able to put together simple sentences. Hopefully the above example was an exception, but I have a sense it isn't. In any case I think this is what the Governor meant to say:

The patriot Rick Perry dropped out and did a favor to the person whom I now consider the front-runner: Newt Gingrich. By dropping out, Perry helped the conservative cause.

What would be cool is for someone to train a machine-translation (or text summarization) algorithm, to convert Palin quotes into simple prose.

2011 Hedge Fund Performance

2012-01-18T09:00:00.000-08:00

In a year when the S&P 500 finished flat, many hedge fund managers once again failed to produce -- so much for absolute returns. 2011 was a tough year, but three styles stood out: Global Macro, Fixed Income Arb, Equity Market Neutral managers were able to eke out modest end-to-end returns.

(Click to enlarge)

In addition, when viewed over a longer time-horizon only two styles (Global Macro, Fixed Income Arb) produced impressive end-to-end performance numbers when compared to the S&P 500.

(Click to enlarge)

Once again many investors placed money with funds that produced stellar returns over the previous year. Unfortunately simple momentum strategies never end well for investors, and 2011 may have finally brought that message home. While hedge funds have suffered through terrible years in the past, one gets the sense that investors are waking up to the fact that there just aren't that many great trading ideas and consistent managers around. The release of Simon Lack's book is sure to give investors pause about hedge funds as an asset class.

"Investors anoint a new hedge fund demi-god all the time," said professor Jim Liew, who teaches hedge fund strategies at New York University's Stern School of Business. "But they are bound to be disappointed because the rule of thumb is that a manager who can be up 40 percent one year can be down 40 percent the next. They are absolutely human."
... "After 20 years of investing in hedge funds, I finally realize they are not the holy grail but an asset class with enormous fees, illiquidity, high leverage and hidden risk given their lack of transparency," said Bradley Alford, chief investment officer at Alpha Capital Management.

Below is the relative illiquidity of the different DJ/Credit Suisse indices. Note that Global Macro, the top performer over the last 12/24 months, is also the most liquid style:

(Click to enlarge)

MAXIMUM DRAWDOWN
Here is the graph comparing the riskiness of the top 2 performers in 2011 (Global Macro, Fixed Income Arb), and the S&P 500. For visual purposes I normalized the time-series so all three had a common starting value. Note the low volatility of Fixed Income Arb, didn't shield it from a drawdown that was even larger than the maximum drawdown observed from Global Macro:

(Click to enlarge)

See my earlier post for graphs highlighting the Maximum Drawdowns of other trading styles.

[Hedge Fund performance from previous months can be found here. More details, including sector/style descriptions and methodology, are available here.]

Related posts:

Long-term performance of Venture Capital & Hedge Funds

The inside scoop on how investors pick Hedge Funds: A simple momentum strategy for picking hedge funds leads to suboptimal returns.

Long-term performance of Venture Capital & Hedge Funds

2011-12-30T11:26:00.000-08:00

With the 1990's dotcom period no longer part of the most recent 10-year window, VC returns have come crashing down. In a recent article over at Verisi Data Studio, I compare VC's to Hedgies and a series of financial benchmarks. See the complete article for details:

(Click to enlarge)

Data Mining with the Maximal Information Coefficient

2011-12-18T16:00:00.000-08:00

I have a new post up on the Maximal Information Coefficient, over at Verisi Data Studio.

Go to this page for details.

Hedge Fund Performance (Nov/2011) & Maximum Drawdown

2011-12-16T09:00:00.000-08:00

Here are the latest performance numbers for the Dow Jones Credit Suisse Hedge Fund Indices: November was a tough month for all trading styles. The top performer (Fixed Income Arb) returned half a percent. On the other hand, given the turmoil in the markets due to uncertainty in the EU, the worst performer (Emerging Markets) was only down 2.7%. The top performer in 2011 has been Global Macro (up 6% for the year, and flat in November):

(Click to enlarge)

Global Macro also leads the way over the most recent 24-month period:

(Click to enlarge)

MAXIMUM DRAWDOWN

This month I put together charts to highlight the Maximum Drawdown, at least for the period covered by my data. Recall that drawdown measures the percentage decline from a peak to a trough. What I did was calculate the maximum such decline and highlight the month when it occurred.

Here is the graph for Equity Market Neutral, Dedicated Short Bias, and the S&P 500. For visual purposes I normalized the time-series so all three had a common starting value. All three series suffered their Maximum Drawdowns in Feb/2009. The most dramatic graph is the line for Equity Market Neutral: after a steady, non-volatile upward trend, the index fell of the cliff in late 2008 and bottomed out on Feb/2009! What a scary ride for investors:

(Click to enlarge)

Here are the equivalent graphs for Global Macro and Managed Futures: much less dramatic than the Equity Market Neutral chart. Note that over the time period covered (1994 - Nov/2011), the Global Macro index blows away Managed Futures.

(Click to enlarge)

Finally here is the equivalent chart for Fixed-income, Convertible, and Risk Arbitrage. All three produced steady gains until plunging at the end of 2008. They have since recovered and surpassed their peaks in 2007/2008 (especially the Convertible Arb index).

(Click to enlarge)

Related post:

Quantifying Losses with Maximum Drawdowns

On Elite Schools, and Soft vs. Hard Skills

2011-12-13T09:36:00.000-08:00

Steve Hsu pointed me to a recent discussion on recruiting practices of elite professional service firms (investment banking, consulting and law firms), nicely summarized by Robert Teitelman.

I have two quick comments:

The results from Lauren Rivera's survey of elite professional service firms fits nicely with what Karen Ho documented in her ethnographic study of investment banking firms. See here for my review of Karen's book.
I am a technical quant (nowadays I'm more fashionably referred to as a data scientist¹), so I definitely appreciate the importance of math/analytic/technical skills. However, I think that the current fascination with all things Big Data is not without its problems. There is always a danger that companies become hostage² to data-driven³ decision making. Google (machine-learning & big data) and Apple (blending computer science and the humanities) demonstrate that there is more than one approach to creating a results-oriented company culture. I think too many companies think that being data-driven (like Google) is the only road to success. Even companies who compete in low-margin industries should value "soft" skills (such as elegant design, and empathy with users/customers). I am pleased to report that among technology startups there is a noticeable rise in the emphasis on design and usability.

(1) I personally think the title Data Alchemists is a lot more accurate, and a lot less pretentious.

(2) Don't get me wrong, I appreciate the sentiment behind Andreas Schleicher's famous quote: "Without data, you are just another person with an opinion ...". But many times decisions have to be made when data is insufficient, or far from convincing.

(3) Here I favor the notion of being data informed over being data driven.

2011 Web 2.0 Summit: By the Numbers

2011-10-20T07:40:00.000-07:00

Here's a sampling of factoids I heard at the sessions I attended this week.

Sean Parker:
Recorded music annual revenue has gone from about $48B to about $12B annually
Validation for Secondmarket prices: The Forbes list of the richest people in the US, contains several early Facebook investors.
Retail industry & ebay stats from John Donohoe:
Customers use their phones to buy 2,600 cars on ebay each week (mostly via ebay's mobile app).
Paypal has over 100M customers with at least two financial accounts linked to their Paypal accounts. (In comparison Visa has 2B card holders & 30M merchants in its network, according to John Partridge of Visa.)
In over half of offline retail transactions, customers access the web (increasingly with mobile devices while they're in the store).
$325B annual total global eCommerce revenue ($10T for all retail). A giant like Walmart only has single-digit market share globally.

VMWare will soon reach $1B in revenue in a single quarter (Marc Benioff)

Stats from Intel's from Kirk Skaugen:
There will be 15B connected devices by 2015, there are currently over 4B.
Annual global IP traffic exceeds 300 exabytes. In fact more data traveled across the Internet in 2010 than during all the preceding years combined.
Annual data center spend is about $450B.
By 2018 Intel hopes to produce a supercomputer that has 100X more computing power than the current fastest supercomputer, at only 2X more energy.
Stats from Dick Costolo:
From 100M tweets/day (start of the year), now approaching 250M tweets/day.
Twitter now has 100 million active global users, about half log in every day
25-30% of Twitter employees are Xooglers.
40% increase in mobile users growth (year-over-year)
Mary Meeker (slides):
81% of the traffic to the most popular web sites are from outside the US.
Less than 50% of internet enabled devices run Windows.
More people have cellular signals (85%) than are connected to an electric grid (80%).
Smartphones with "made in the US" operating systems have 64% share, up from 5% five years ago

Dennis Crowley: 50% of Foursquare usage is outside the US ; there are now over 10M accounts

Amazon, Amazon Web Services, & other stats from Alyssa Henry:
560B objects on S3
1.5B items on Amazon's retail catalog; 50M items are updated every week
Yelp stats: 18M reviews; 50M visitors/month
Khan academy stats from Sal Khan:
They now have about 2,600 videos.
Last month: 39M pageviews (up 495% year-over-year); 3.5M unique users (up 309% year-over-year)
Total spent on the site: about $1M (now that's a Lean Startup!)
Wordpress stats from Toni Schneider:
25M total bloggers and 300M unique visitors per month on wordpress.com
15% of all sites on the internet powered by Wordpress software
Google, Google+ and Android stats from Vic Gundotra and Sergey Brin:
4M businesses have signed up for Google Apps
There were 3.4B photos uploaded onto Google+ in the first 100 days (mostly using the auto photo upload on Android devices)
There are over 200M Android device out there, and recently there have been about 550K activations per day.
Keep them memos short: In reference to Steve Yegge's "Platforms rant", Sergey Brin jokingly remarked that he didn't make it "beyond the first 1,000 pages".

Related posts:

2010 Web 2.0 Summit: By the Numbers

Factoids from Twitter's Engineering Open House

Chirp 2010: Twitter By The Numbers

Today's Occupy SF March

2011-10-15T20:05:00.000-07:00

We've spent a few evenings hanging out at the OccupySF encampment in front of the Federal Reserve Building. Today we went to a march that went from the base camp to City Hall. While not as big as some recent SF marches (particularly right before the Iraq invasion) the energy level and enthusiasm was high. I estimate the crowd at a few thousand. As a veteran of street protests in the Philippines I've always believed that real change requires a parliament of the streets.

Related Resources:

Income and Wealth Inequality in the U.S.

Wall Street Bonuses by the Numbers

Campaign Finance & Lobbying by the Numbers

The Best of @BigData Sep/2011

2011-10-01T09:30:00.000-07:00

Here are the most popular tweets/links shared through my BigData twitter stream:

Case for accurate metrics: Realizing that the nation's economics statistics could use an overhaul, the Federal Reserve puts out an RFP.

Large-Scale Classiﬁcation (video events on Youtube): Using text-mining to learn optimal taxonomies.

The World Bank's open data repository: Easy to navigate site filled with loads of interesting data.

Is Teaching MapReduce Healthy for Students?: Berkeley DB Professor, Joe Hellerstein's blog post elicits lots of interesting response.

Realistic (asymmetric) Information propagation model for social media: Instead of peers, identify and study "authority" figures.

Twitter's Front-end Summit

2011-09-29T10:05:00.000-07:00

Twitter had another well-attended engineering open house last Tuesday and I was able to attend the first two presentations. Here are a few observations:

pdf.js: Mozilla's Andreas Gal gave an overview of a relatively new project to develop a faithful and efficient pdf reader in HTML5 and Javascript. I've been waiting for Firefox to come with a pdf reader for a long time, and at some point (I hope) in the not-so-distant pdf.js will ship with it. What began as an experiment in HTML5 and Javascript will soon result in a pdf reader that does NOT rely on native code (great news for security). They still have issues to work out, notably (type 2) fonts and printing (which turns out to be most in-demand once a year, for tax returns). Speaking of fonts, font conversion was the main challenge encountered by Slideshare's engineers in moving from Flash to HTML5.
The traditional approach to rendering PDFs in a browser is to use a native-code plugin, either Adobe’s own PDF Reader or other commercial renderers, or some open source alternative (e.g. poppler). From a security perspective, this enlarges the trusted code base, and because of that Google’s Chrome browser goes through quite some pain to sandbox the PDF renderer to avoid code injection attacks. An HTML5-based implementation is completely immune to this class of problems.
Andreas confessed that he knew very little Javascript when they started working on pdf.js. Well he picked it up quickly! One of the benefits of listening to his presentation was that I learned a few useful Javascript performance tips that I plan to start using right away.

Bootstrap: Designer Mark Otto of Twitter gave a quick overview of Bootstrap
... a toolkit from Twitter designed to kickstart front-end development of web apps and sites. It's a solid and constantly growing library that includes all the essential and well-designed HTML, CSS and javascript you need for any project.
I haven't played around with, but from what I saw it's a pretty impressive suite of tools. Dave Winer has, and he seems to like it a lot. It comes with many, many design components: templates, grids, forms, layouts, buttons, navigation, ... Most of the things you need to get your web design done quickly.

Ender: I had to leave early so I missed Jacob Thornton's presentation on a new "package manager for the front-end".
In the browser - small, loosely coupled modules are the future and large, tightly-bound monolithic libraries are the past! Ender capitalizes on this by offering a unique way to bring together the exciting work happening in javascript packages and allows you to mix, match, and customize your own build, suited to your individual needs, without all the extra cruft that comes with larger libraries. With Ender, if one library goes bad or unmaintained, it can be replaced with another.

Related posts:

Factoids from Twitter's Engineering Open House

Chirp 2010: Twitter By The Numbers

Text Mining and Twitter III- LDA Code on Hadoop

Proposed U.S. Tax Hikes in a few charts

2011-09-20T09:08:00.000-07:00

Over the past few weeks I've put together a few charts and infographics that are relevant to current tax discussions in the U.S. A day after the proposed millionaires tax was announced, critics have already cast doubts on its central premise: that millionaires pay taxes at a lower rate than middle class families. I created these charts to help me place the proposals in (historical) context and hopefully it will be of value to others.

[Unfortunately you have to follow the links to the charts which I've created on another site.]

Historical values of Federal Income & Capital Gains Tax Rates: When viewed in historical context, a return to Clinton era marginal rates for the top bracket doesn't seem extreme. And as the Economist points out, taxing capital gains & dividends as ordinary income might be the more efficient & progressive mechanism.

Concentration of Wealth: The phrase class warfare is routinely used by opponents of tax increases on the wealthy. A quick glance at data from Piketty and Saenz shows that inequality is a real and growing problem.

Income Distributions by Ethnicity and Gender: The 2010 Census Survey revealed poverty rates have soared in most U.S. states. In a series of two inforgraphics you can see that the poverty rate varies widely by income & gender.

Hedge Fund Performance - August/2011

2011-09-15T11:50:00.000-07:00

It's been a while since I've posted anything about Hedge Fund performance. I figured that with the struggles of the markets last August that I should do a brief update. First off, here are the year-to-date performance numbers along with returns for the most recent month:

(Click to enlarge)
The S&P 500 was down close to 6% in August so it's not surprising that the Shorts did well last month. In fact Shorts are right up there with Global Macro managers through the first eight months of 2011. Markets have been down through the first half of September so Shorts may end up with two consecutive months of strong gains. Managed Futures have been up-and-down through the first eight months of 2011, and are essentially flat for the year. Over the most recent 12 & 24 months, most Hedge Fund styles still trail the S&P 500 although Global Macro managers are closing the gap. Continuing their miserable 2008/2009 performance, Equity Market Neutral funds have fared poorly over the last 24 months (see 2008 and 2009 results).

(Click to enlarge)

To close this post, here's an updated view of volatility (of monthly returns) since Jan/2006 -- the year prior to the start of the global financial crisis. Over the past 5 years, Global Macro funds continue to provide decent returns with relatively low volatility, especially when compared to the wild swings one associates with Managed Futures.

(Click to enlarge)

The Best of @BigData Aug/2011

2011-09-01T09:21:00.000-07:00

Here are the most popular tweets/links shared through my BigData twitter stream:

Hadoop prototype for incremental realtime analytics with MapReduce: a hash-based analytics framework from @UMassAmherst

Probability & Graphs: Estimating Sizes of Social Networks, with a Biased Sampling method dramatically reduce required samples

Extracting common sense facts (causal reasoning): Simple, scalable methods for tapping Web-scale data

Mining Massive Data Sets: Free Stanford CS lecture notes & outline, from a Winter/2011 course

HaLoop: A Hadoop extension for large-scale, *iterative* data analysis applications

twitInfo: MIT CSAIL (prototype) web app for investigating topics on Twitter

The luckiest woman on earth: Stanford PhD wins TX lottery 4 times

Bits from Scifoo 2011

2011-08-14T20:00:00.000-07:00

Just got back from Scifoo 2011 and wanted to share a few observations:

Economics and AI: I had an interesting discussion with Alex Wissner-Gross, who has been using mathematics and machine-learning techniques in finance and other settings. I was particularly interested in what he and his colleagues have been doing in helping us understand (the limits of) high frequency trading. Alex isn't just a researcher, he also has experience founding startups.

Citizen Science: Chris Lintott of Zooniverse shared a funny story of how Hanny’s Voorwerp (Dutch for "thing") was discovered. Dr. Brian May, guitarist for Queen blogged about Galaxy Zoo, a Citizen Science platform for astronomy. Apparently this led to a surge of participation from fans of Queen, and voila, Hanny’s Voorwerp was discovered. The newly launched platform Zooniverse is actively looking for projects, so if your a scientist with loads of data that can use processing/crowdsourcing, definitely check them out.

Clinical Trials (the Information Architecture of Medicine is broken): Ben Goldacre, who writes the Guardian's Bad Science column, gave an overview of the state of clinical trials data. In a nutshell, the news isn't great for consumers. Ben and his colleagues estimate that about half of all the trials that fail don't share their raw data. In essence consumers and medical professionals are left with data from cherry-picked studies. Data transparency is essential for science to thrive, especially when results have the potential of causing harm to so many patients.

Aging: Over the past few Scifoos, I've had the good fortune of attending sessions run by leading researchers on aging. This year was no exception. In particular, I had the chance to spend time with one of the best researchers in the field: Jay Olshansky, here featured on Charlie Rose and 60 Minutes. In keeping with Scifoo's reputation for non-conventional ideas, South Korean genomicist Jong Bhak speculated on how computing can lead to breakthroughs in cancer and aging research.

Theoretical Physics: As always there was a contingent of brilliant theorists, and I was able to attend sessions by Frank Wilczek, Martin Rees, David Tong, Eva Silverstein, Max Tegmark, Lisa Randall, and Joe Polchinski.

Contagion the movie: Larry Brilliant and several friends dreamed up the idea for this movie at Scifoo a few years ago. We were treated to several clips from the movie, as well as the inside story of how they developed the script. Several Scifoo attendees were consultants to the film, and according to their comments, the science behind the movie is solid.

Large-scale Named Entity Recognition in the Cloud

2011-08-04T12:46:00.000-07:00

Below is an amazing factoid shared by Carlos Guestrin during his GraphLab overview:

CoEM is an entity-extraction algorithm introduced by Rosie Jones in 2005. In the slide above, the data set was a graph with 2M vertices and 200M edges. A team at CMU applied CoEM using Hadoop and found it took about 7.5 hours. GraphLab on far fewer cores (from 95 to 16 cores) took 30 minutes: 6X less cores, but still 30X faster. The same problem on 32 EC2 machines (with 256 processors) took 80 seconds, or 0.3% of the time it took to complete the same task in Hadoop!

If you're interested in large-scale machine-learning, do watch Carlos' GraphLab overview. What's intriguing is that the GraphLab team has put together a precompiled GraphLab EC2 AMI, making it much easier to get the GraphLab system up and running on EC2. Carlos describes many more experiments that they conducted on EC2: the total cost of running many large, compute intensive experiments was a mere $4,000. Compare that to the cost of purchasing and maintaining a similar cluster on your own.

Update: See comment below from Aapo Kyrola. It turns out the multicore version of CoEM on GraphLab was even more impressive. (Instead of 30 mins, it actually runs in 2-3 mins on 16 core. In EC2, in maybe 1 mins. For three full iterations.).

Related posts:

Text Mining and Twitter III- LDA Code on Hadoop

Some thoughts on (natural language) Search

Compressed Sensing and Big Data

Rebranding Data Scientists as Data Alchemists

2011-08-03T08:48:00.000-07:00

The more I think about it, maybe the term Data Alchemists is more accurate than Data Scientists. For one thing Alchemy itself is regarded as a protoscience, which aptly describes the body of skills under the heading of Data Science. Secondly Alchemy describes what good Data Scientists do: they turn data into insights

a power or process of transforming something common into something special

The title Data Scientist itself is a misnomer: most Data Science practitioners are actually either engineers or artists by training, so Alchemist is the more descriptive title.

Finally, the book The Alchemy of Data is just waiting to be written :-)

Related posts:

How to nurture Data Scientists

Computer Vision & Compressed Sensing: Recovery of signals, images, & video from meager data

2011-06-28T09:29:00.000-07:00

An interesting new paper (The Split Bregman Method for L1-Regularized Problems) proposes a general set of tools applicable to a broad range of problems for which compressed sensing makes sense:

The class of L1-regularized optimization problems has received much attention recently because of the introduction of “compressed sensing,” which allows images and signals to be reconstructed from small amounts of data. Despite this recent attention, many L1-regularized problems still remain difficult to solve, or require techniques that are very problem-specific. In this paper, we show that Bregman iteration can be used to solve a wide variety of constrained optimization problems. Using this technique, we propose a “split Bregman” method, which can solve a very broad class of L1-regularized problems. We apply this technique to the Rudin–Osher–Fatemi functional for image denoising and to a compressed sensing problem that arises in magnetic resonance imaging.

Making compressed sensing more accessible will make it likely that applications beyond imaging will happen in short order. To me security and surveillance are the natural next domains: with so many surveillance systems around, denoising data becomes critical. (As with any Big Data problem, it goes without saying that summarization is also important. An interesting recent example is the video summarization startup Briefcam.)

The research has relevance in various areas such as medical imaging, intelligence surveillance, statistics, and data mining. “This is a method which is useful in predicting what movies Netflix customers might prefer, finding objects on the ground from satellite observations, enabling patients to spend less time in MRI machines, reducing radiation exposure from CT scans, allowing doctors to track needles in ultrasound machines, and more,” the authors wrote.
New challenges and future directions include figuring out ways to further speed up methods based on these algorithms, optimizing geometries, enabling imaging capabilities through turbulence, and enhancing filtering methods for dynamical systems.

RELATED POST:

Compressed Sensing and Big Data

Writing code that makes your data sing: Auditory Data Display Tools and Resources

2011-06-18T09:30:00.000-07:00

I was listening to Dr. Joshua A. Miele being interviewed by my local public radio station, when I overheard him mention tools for the auditory display of data. Dr. Miele, who happens to be blind, was trained in Psychoacoustics. His educational training and present work requires data analysis, which for those of us who can see, usually involves data visualization. In order to understand his data sets in graduate school, Dr. Miele ended up building a set of tools for the auditory display of data in MATLAB.

While auditory data display tools are aimed primarily at the visually impaired, there might be contexts when such tools can help the rest of us detect patterns/clues, we otherwise might miss. Imagine working on a problem, stepping away from your computer for a change-of-pace. Behold, the data-rich soundcheck: pattern reconition with your iPod! Or as Barry Manilow might put it: "Write code that makes your data sing."

I was intrigued enough that I researched some of the available tools. Below are a few of the notable things I found:

Miele's Auditory/Tactile Data Representation Toolbox For Matlab

Auditory Graphs and Sonification Sandbox Software from Georgia Tech's Sonification Lab

SONIPY (An Extensive Open-Source Python Framework For Data Sonification Research And Auditory Display): For the accompanying paper, click here.

Data Sonification from the Desktop: Should Sound Be Part of Standard Data Analysis Software? (2005 paper)
The design of auditory formats for data display is presently focused on applications for blind or visually impaired users, specialized displays for use when visual attention must be devoted to other tasks, and some innovative work in revealing properties of complex data that may not be effectively rendered by traditional visual means. With the availability of high-quality and flexible sound production hardware in standard desktop computers, the potential exists for using sound to represent characteristics of typical “small and simple” samples of data in routine data inspection and analysis. Our research has shown that basic properties of simple functions, distribution properties of data samples, and patterns of covariation between two variables can be effectively displayed by simple auditory graphs involving patterns of pitch variation over time. While such developments have implications for specialized applications and populations of users, these displays are easily comprehended by normal users with minimal practice. Providing further software enhancement to encourage exploration of data representation by sound may lead to a variety of useful creative developments in data display technology
An alternative to auditory displays are Tactile Graphics. Here is a list of links about Tactile Maps

Factoids from Twitter's Engineering Open House

2011-06-16T22:30:00.000-07:00

I just got back from the first-ever (?) Twitter Engineering open house: three good talks, and a chance to mingle with Twitter engineers over lots of free food and drinks.

Like any scrappy startup that has gone on to amass millions of users, Twitter has had to overhaul key portions of its technical infrastructure. The event was a chance to hear (publicly for the first time) about some of those changes. Twitter like many companies relies heavily on open source technologies. But as with most large-scale sites, they customize open source tools to meet their specific needs. So when their engineers say "we use Tool X", what they really mean is "we use a heavily customized version of Tool X".

Real-time search
Early on Twitter bought the search engine Summize, and for its first three years, search.twitter.com was powered by Summize's MySQL based search tool¹. Michael Busch gave an overview of the new real-time search architecture that has been in the works for about 18 months. He opened with some interesting factoids:

185 million tweets are posted daily

Update (6/30/2011): Twitter just announced they are up to 200 million Tweets per day. The growth over the past two years has been astounding. Consider that they were at 2 million tweets/day in Jan/2009, and 65 million tweets/day in Jun/2010.

they handle about 1.6 billion search queries daily

average response time for a search on Twitter is 50 milliseconds

indexing latency is less than 10 seconds (from the time you post a tweet, link, photo, within 10 seconds they'll start showing up on search.twitter.com)

The new search infrastructure is built on a customized version of Lucene. Michael went through the technical reasons why they had to build custom data storage/structures, caching, garbage collection, and compression strategies. In particular they needed to use a word-level inverted index, because freshness² is an important relevance criterion for most searches on Twitter. The resulting Posting List (internally know as EarlyBird) is traversible in reverse order. Given the amount of customization they had to do, they needed a technical lead who was familiar with Lucene's internals: Michael Busch has been a Lucene committer for the last 5 years.

Private Cloud Computing with Mesos
Xoogler Bill Farner gave a talk about Twitter's internal cloud computing infrastructure. I was unaware that Twitter has been running their clusters with Mesos. Given Bill's experience working on Google's infrastructure, I'm sure they looked at multiple options. Mesos comes witha lot of features: resource discovery & accounting, failure detection, messaging, service discovery via Zookeeper ... But most importantly, it comes with an API that makes it extensible using familiar languages like Java, Python and C++.

Client-side Architecture of #NewTwitter
Marcus Philipps and Sarah Brown discussed #NewTwitter -- their move from Rails to Javascript on the client. As always, they used a lot of open source tools, in particular jQuery and Mustache templates. Unfortunately I had to step out to take a call and missed most of the talk. Judging from the number of questions from the audience, it clearly was a subject that many of the external attendees were interested in.

Text Mining and Twitter III- LDA Code on Hadoop

2011-06-09T15:21:00.000-07:00

Yahoo Research just released LDA code that runs on Hadoop. They claim that this is the fastest implementation of LDA to-date:

It’s seriously fast and scales very well to 1000 machines or more (don’t worry, it runs on a single machine, too). We believe that at present this is the fastest implementation you can find, in particular if you want to have a) 1000s of topics, b) a large dictionary, c) a large number of documents, and d) Gibbs sampling. It handles quite comfortably a billion documents. Shravan Narayanamurthy deserves all the credit for the code. The paper describing an earlier version of the system appeared in VLDB 2010.

In earlier posts (see below), I mentioned my fondness for using LDA, particularly during the exploratory phase of a text mining project. With the parallelized code released by Yahoo, LDA via Gibbs Sampling becomes an option for tackling larger corpuses.

Related posts:

Text Mining and Twitter

Text Mining and Twitter II - Python code for Analyzing Streaming Data Sets

2010 Census: Ethnic & Racial breakdown within the major Metro Areas

2011-06-05T09:00:00.000-07:00

In trying to familiarize myself with the 2010 U.S. Census results, I find drawing simple graphics goes a long way. I looked at the share of Asians and a diversity index in an earlier post. Here are related charts that contain more detail: this time I took the 32 largest metropolitan areas (MSA's) and looked at their ethnic/racial decomposition. (For an interactive heat map, go to this Verisi Data Studio page.)

This first chart has distributions for large metro areas on the West Coast. Note that the SF Bay area is broken up into two MSA's (San Francisco-Oakland-Fremont and San Jose-Sunnyvale-Santa Clara).

(Click to enlarge)
Here are large metro areas on the East Coast:

(Click to enlarge)

Next, some metro areas in the middle of the country:

(Click to enlarge)

Finally, metro areas in the South & Southwest, with a few California metro areas thrown in to fill in the 3 by 3 grid:

(Click to enlarge)
Finally a chart that gives the relative size of the metro areas shown in the previous charts. Once again note that the SF Bay area is broken up into two MSA's

(Click to enlarge)

Feeling right at home in the SF Bay Area

2011-06-01T08:59:00.000-07:00

Having grown up in Southeast Asia, I've always felt at home in my adopted residence of San Francisco. Besides the exciting technology scene, the quick access to outdoor recreation, a vibrant and affordable cultural scene, and rich history, I love the diversity of San Francisco in particular. As the city becomes more expensive, I worry that things will change as highly-compensated knowledge workers displace many of our current residents.

Among the most populous counties in the U.S., the Bay Area has several of the most ethnically diverse counties. I also grew up in a mega city, so dense urban areas are to my liking. (NOTE: For an interactive heat map, go to this Verisi Data Studio page.)

(Click to enlarge)
The city/county of San Francisco ranks 5th in terms of population density (pop. per sq. mile), 18th in terms of USA Today's Diversity Index, and 2nd in terms of the relative size of its Asian population.

[If you expand to the 100 most populous counties, another Bay Area county (San Mateo) enters the picture. See here for details.]

Tweaking Press Releases

2011-05-31T08:09:00.000-07:00

I receive my share of press releases, and while I don't read all of them, I try to peruse many of the ones that I receive. The objective of many of these emails is to get you to broadcast (via blogging, tweeting, ...) to your audience/network. Like any (email) marketing campaign, I imagine the response rates are extremely low. Since a majority of press releases are written by PR specialists, I assume that majority of the effort is directed towards the text (or "copy"). The more sophisticated outfits likely even conduct tests, where they send out different versions of the text to determine which messaging leads to higher response rates.

But if the objective is to boost "response" rates, why not play around with other factors that can do that? Here are a few suggestions:

Targeting: I'm assuming that most outfits do some form of targeting prior to an email blast. But there are still many that don't. A great side-effect of targeting is you'll end up not spamming people on your lists: they won't receive every single email you send out, only the ones that are of interest to their audience. There are tools for targeting members of traditional media, as well as new tools for targeting bloggers and micro-bloggers. There are even a slew of tools that are tailored for the needs of PR professionals. Make sure the tools you use account for influence across a variety social media sites.

Rightsizing + Dataizing your copy: Email recipients are drowning in email, plus many content producers have enough reading to do. A simple Who/What/Why email is more than enough. I would experiment with the length and format, but in our short-attention-span era nothing kills your response rates more than a long-winded delivery (albeit a world-class subject line does help).

A note about the Why component of PR emails: People love factoids! Nothing is more compelling than data, especially in a simple-to-understand and elegant (static) infographic. Even if recipients don't immediately write or tweet about your subject, the data/graphic might be something they reference in the future. The companies you work with may have raw data you can use, but it might be up to you polish their data for PR usage. Imbuing your PR releases with data means investing in data scientists within your agency.

Create a compelling landing page: Most press releases end with a short description of the companies and links to their web sites. I rarely see companies who think about constructing landing pages tied to their press releases. If you look at successful e-commerce campaigns (especially those that involve SEO or search marketing), many optimize the landing pages for conversion. Why not adopt the same mindset in PR? At the very least, the landing pages tied to a press release may help future SEO efforts. Sadly in many cases the landing page is the actual press release! The technology is available so that isn't the problem. There are collaborative tools that make it easy to design landing pages.

Measure, test, and learn: Besides measuring the number of stories, tweets, clicks (of shortened URL's), you can start tracking other things. The influence/effectiveness of landing pages can be measured through standard web analytics (nowadays you measure things in real-time and in-page). If you start using infographics and data, you can track their citations and uses.

More importantly you have many other knobs, besides your copy, to tweak and experiment with. A bit of experimentation can't hurt. Nowadays familiarity with the modern tools used in web analytics and search marketing is a must for any successful marketing campaign.

Hedge Fund Performance - April/2011

2011-05-16T18:45:00.000-07:00

Trend followers were quick to exploit recent price movements in commodities: Managed Futures gained 5.4% in April, while Global Macro was up 2.5%. In keeping with its reputation for being volatile, Managed Futures was up-and-down through the first four months of 2011: Jan (-0.8%), Feb (+1.8%), Mar (-2.8%), Apr (+5.4%). If price movements in commodities are due to short-term demand (as some market watchers believe), then I wouldn't bet on Managed Futures having a big year.

(Click to enlarge)
Over the most recent twelve months (April 2010 to April 2011), Managed Futures & Global Macro led all the other hedge fund indices. In the case of Managed Futures, gains over the last month bolted it to first place, after being in 8th place through the end of March.

(Click to enlarge)
Which of these two styles (Managed Futures & Global Macro) is less volatile? Using the interquartile range as my volatility metric, I sorted the different indices from the most to the least volatile. The start date Jan/2006 is meant to capture the year prior to the recent global financial crisis (thus the relatively high volatility of the S&P 500). Global Macro is not only far less volatile, it has a higher average monthly return over the given period.

(Click to enlarge)
Similarly, Global Macro scores better on my Illiquidity metric than Managed Futures:

(Click to enlarge)
[Hedge Fund performance from previous months can be found here. More details, including sector/style descriptions and methodology, are available here.]

Semi-regular Dimes:

JP Morgan and valuing credit derivatives: You hear of stories about how Wall Street struggles with complex computations that involve massive data sets. In this talk (slides & video), two senior managers at JP Morgan (a quant and a technology manager) give an overview of how they were able to dramatically reduce computation time. Their secret? They partnered with a Palo Alto startup (Maxeler) that specializes in acceleration. JP Morgan was so happy with the results of their project with Maxeler, it ended up buying 20% of the company.

The People vs. Goldman Sachs: You may quibble with some points raised by Matt Taibbi, but as I said in a recent tweet, at least he raised examples of possible criminal behavior, that the major newspapers haven't.

What triggered oil's greatest rout: Interesting "special report" from Reuters.

How to Make Money in Microseconds: If you've been following the high-frequency trading space closely, nothing in this article will surprise you. But it's still a great read.

Has Social Media in Financial Services Arrived?: A roundup of articles on companies who use status updates (such as tweets) as signals for trading. I'm skeptical that text mining tweets can be the primary basis of a trading strategy, especially when results are judged over several years. But it wouldn't surprise me if such signals become one of many that investors begin using.

You know there's a Tech Bubble

2011-05-14T09:00:00.000-07:00

when

... apartment rental rates in San Francisco are skyrocketing (Updated: from 8/7/2011 NYTimes & 8/11/2011 SF Chronicle)

... data analysts (and other data geeks) now refer to themselves as data scientists¹

... examples of Big Data sets cited by the media, keep getting Bigger

... besides the public stock markets, we need SecondMarkets

... Skype gets bought again, this time for 3X its previous price

... beneficiaries of the previous bubble become some of the most influential and bullish angels and VC's

... the moniker Web 2.0 is in on the verge of being replaced by Web 3.0

... many insiders are confidently proclaiming: "things are different this time"

(1) That includes yours truly. On a positive note: At least we haven't gotten to the point, in this iteration of fancy & clever titles, where Wall Street investment analysts are referred to as investment scientists. I've since come to the conclusion that Data Alchemists is more descriptive.

Hedge Fund Performance - Q1/2011

2011-04-16T07:59:00.000-07:00

Only two trading styles generated more than 1% in March: Emerging Markets (2.1%) and Equity Market Neutral (1.1%). The other trading styles lost or gained less than 1%, save for Managed Futures and Shorts which both lost over 1%. Following gains of 1.8% in February, the Managed Futures index was down 2.8%. In fact the first three months of 2011 highlight the volatility in Managed Futures returns: Jan (-0.8%), Feb (+1.8%), Mar (-2.8%).

(Click to enlarge)
The S&P 500 is up 13% over the last twelve months, as it continues to outperform all the trading styles tracked by DJ/Credit Suisse:

(Click to enlarge)
Next we examine recent performance using the middle 50% of the distribution of monthly returns: I chose Jan/2006 as a starting point to include months before the global financial crisis of 2007/2008. Emerging Markets has the highest median return but was relatively volatile (volatility =: interquartile range). Managed Futures was the second most volatile style and also produced one of the lowest median monthly returns. Global Macro produced the 3rd highest median monthly return, while being the 4th least volatile style:

(Click to enlarge)
Finally here are updated Illiquidity scores through Q1/2011: the top performing index through the first three months of 2011 (convertible arb) is also the most illiquid.

(Click to enlarge)
[Hedge Fund performance from previous months can be found here. More details, including sector/style descriptions and methodology, are available here.]

Semi-regular Dimes:

Why market-neutral funds may come up short: Buyer beware. A year ago, the DJ/Credit Suisse equity market neutral index was down 40% over the previous 24 months (Mar/2008 to Mar/2010). In addition, consider recent drawdowns and research studies indicating market-neutral funds may not be a good fit for retail investors.

Momentum and mutual fund investors: Not surprising that many mutual fund investors chase recent winners. As I pointed out in a previous post, hedge fund investors are just as prone to momentum fades. For a recent example, the best-performing index in Mar/2011 (Emerging Markets) is where momentum (hedge fund) investors are putting their money at the moment.

Roy Niederhoffer Hedge Fund Lost 10% In March:
"We have had periods like this before in which exogenous negative or positive events have caught our strategy positioned incorrectly both during the event and afterwards," firm founder Roy Niederhoffer explained. "March was particularly disappointed, given our strong start to the month. Unfortunately, the vents at the Fukushima reactor in the two days following the tsunami and the subsequent three-week 'risk-on' rally that followed caused many parts of our strategy to incur significant losses."

The Carry trade and Central Banks on divergent paths: (For previous posts on currency carry trades, see here and here.)
Many argue that the European Central Bank’s decision on April 7th to raise the policy rate in the euro area, and the prospect of further rises to come, has reinvigorated the carry trade. An interest-rate gap is opening between currencies like the dollar and the yen on the one hand, where monetary policy is likely to remain ultra-loose, and higher-yielding ones like the euro on the other. This gap may explain the strength of the euro, which has risen against the dollar in recent weeks despite endless euro-zone sovereign-debt worries

Knowns and unknowns in the dazzling world of derivatives: I'm finally getting around to reading Satyajit Das 2006 book on derivatives. What a fun and informative read, Das has many great stories compiled from his 30+ years in the industry.

Hedge Fund Performance - Feb/2011

2011-03-15T13:00:00.000-07:00

All of the DJ/Credit Suisse Hedge Fund indices trailed the S&P 500 in each of the last three months. Through the first two months of 2011, all but Shorts and Emerging Markets posted modestly positive returns, with Global Macro essentially flat over Jan/Feb 2011. The catastrophic earthquake in Japan will likely slow down the S&P 500 in March (from 1/1/2011 through 3/14, the S&P 500 was up 3.08%), so it will be interesting to see which of the Hedge Fund indices surge in March:

(Click to enlarge)
The S&P 500 has been on a tear over the most recent 12 and 24 months, and it has dominated the DJ/Credit Suisse Hedge Fund indices. How much longer can equities deliver outsized returns? The S&P 500 is likely to enter a period of more modest returns:

(Click to enlarge)
I previously introduced the use of the middle 50% of a distribution (the median, 25th, and 75% percentiles) as a potential tool for comparing recent volatility and performance. In the chart below, after plotting the DJ/Credit Suisse Hedge Fund Index and the S&P 500, I arranged the rest of the indices in terms of volatility: Dedicated Short Bias being the most volatile, and Equity Market Neutral the least volatile. Using this particular metric, Managed Futures funds and the S&P 500 exhibited quite a bit of volatility during the Jan/2006 to Feb/2011 time frame:

(Click to enlarge)
Finally, an update on liquidity. I tracked down the mysterious rise in the Illiquidity of Managed Futures to an error on my end. While I was calculating my Illiquidity metric properly, I was inadvertently assigning the wrong score to certain indices (a labeling mixup). Below are updated (and error-free) Illiquidity scores: the top-performing index through, the first two months of 2011 (Convertible Arbitrage) remains the most Illiquid style.

(Click to enlarge)

[Hedge Fund performance from previous months can be found here. More details, including sector/style descriptions and methodology, are available here.]

Semi-regular Dimes:

2 Finance-related documentaries: I finally got around to seeing Client 9 (The Rise and Fall of Eliot Spitzer) and Inside Job. I liked both films and recommend you see Inside Job after watching Client 9.

Hedge funds seen taking in $210 bln in 2011:
The bulk of the new money is expected to come from institutional investors, the bank wrote, noting that 83 percent of sovereign wealth funds and 83 percent of pension funds raised their allocations to hedge funds last year.
These types of investors are readying themselves for new investments by beefing up their own teams while also turning more to consultants for help in selecting managers.
While tastes for hedge funds has historically favored the industry's biggest players, the bank's survey found that investors are now ready to give smaller players a try with 65 percent of the investors, saying they expect to put money with funds that have less $1 billion in assets.

Scathing report alleges corruption at CalPERS: I'm not a big fan of investors who rely heavily on consultants. If you do decide to use Alternative Investments, make sure you have an in-house team to guide you.

Yale may not have the key (When diversification doesn’t work): I've long suspected that the Ivy League schools may not be the appropriate role model for other (University & College) Endowments.
THERE can be fashions in investing as well as in the arts. Over the past 25 years many university endowments have moved over to the “Yale model”, an investment strategy adopted by the Connecticut-based university in the 1980s. Under the leadership of David Swensen, Yale has invested across a wide range of “alternative assets”, from private equity and hedge funds to timber.
One idea behind the Yale strategy was that endowments have the luxury of time, since their liabilities (paying for new buildings, academic salaries and so on) stretch far into the future. They can thus afford to invest in illiquid asset classes. Such asset classes may offer a better return simply because other investors (mutual funds, for example) are unwilling or unable to deal with illiquidity.
In addition, the traditional dominance of domestic equities within institutional portfolios put lots of eggs in one basket. There were other sources of return—the management skills of private-equity houses, the market for distressed bonds—where the trade-off between risk and reward might be better than could be obtained from the S&P 500 index.
But the model came into question in 2008 and 2009, when the financial crisis hit. In the year to June 30th 2009, the Yale endowment fell by 24.6% and Harvard’s portfolio fell by 27.3%, losing the latter a whopping $10 billion (there has been a modest recovery since then). The illiquidity of the portfolios counted against them.
The broader question is whether what worked for Yale and Harvard, for a time, can be a successful strategy for a much broader range of universities over a much longer period. Yale, in particular, has a first-mover advantage; by being an early investor in private equity it was able to get access to the best-managed funds. Later endowments found that some of these funds were closed to new money and were forced to settle for less talented managers and more modest returns.
… Martin Leibowitz of Morgan Stanley has analysed the characteristics of endowment portfolios over the past ten years. He looked at three portfolios: a classic 60/40 US equity/Treasury bonds split; a Yale-like portfolio with seven separate asset classes; and a portfolio with international diversification but without the illiquid private-equity, hedge-fund and real-estate portions. What is remarkable about these portfolios is how closely correlated they all are with the S&P 500. Even the Yale-like portfolio had a correlation of more than 0.9 (where 1 is a perfect fit).
Another way to measure relationship with the market is “beta”. An asset with a beta of more than 1 rises more than the market in bull phases (or falls more when the market drops); an asset with a beta of less than 1 moves less violently.
Sadly for the Yale-like portfolio, its beta rose sharply in the third quarter of 2008 when the market was in turmoil as Lehman collapsed. The beta then fell again as the market recovered in 2009 and 2010. In short, the benefits of diversification were highly diluted. In beta terms, endowment portfolios traded like a traditional 60/40 fund in the boom and then were more volatile in the bust.

Tulip Mania redux?: Interested in an alternative asset class? Supply is limited and the bubble may burst at any moment. Head over to China -- the asset in question is Dogs (Tibetan Mastiff worth over $1.5M)! (via Paviavio)

Intellectual Ventures licensing revenue hits $700 million in 2010: Speaking of an alternative asset class, Nathan Myhrvold’s patent aggregator is starting to generate serious revenue.

Social Networks evolve into two opposing camps, or into a single all-friendly group

2011-02-24T09:01:00.000-08:00

A recent short paper (Continuous-time model of Structural Balance) by a group of mathematicians and computer scientists at Cornell, uses ordinary differential equations to explain how relationships within social networks evolve.

It is not uncommon for certain social networks to divide into two opposing camps in response to stress. This happens, for example, in networks of political parties during winner-takes-all elections, in networks of companies competing to establish technical standards, and in networks of nations faced with mounting threats of war. A simple model for these two-sided separations is the dynamical system dX/dt = X², where X is a matrix of the friendliness or unfriendliness between pairs of nodes in the network. Previous simulations suggested that only two types of behavior were possible for this system: Either all relationships become friendly or two hostile factions emerge. Here we prove that for generic initial conditions, these are indeed the only possible outcomes.

The math needed to understand their proof is accessible, but the technical details do require advanced Real Analysis and Matrix Theory. In this post, I'll describe the main result of this concise and elegant paper.

Structural Balance is a concept from social psychology, and can best be explained using a three-node graph. In the graph below, edges are signed ( + = friends, - = rivals):

In a three-node network, balanced graphs are ones with an odd number of friendly edges. Conceptually, it is easy to see why unbalanced graphs might be prone to transitioning into having an odd number of friendly edges. Consider (G3) above: either Alice and Bob can approach Eve to try to get her to terminate her relationship with the other person (resulting in a graph with 1 friendly edge). Alternatively, Eve can try to bring Alice and Bob together (resulting in a graph with all 3 edges being friendly). In (G4), there is incentive for the two nodes with the weakest rivalry to form an alliance against the third node (resulting in a graph with 1 friendly edge).

An n-node graph is complete if it contains edges between all pairs of nodes, and a complete graph with signed edges is balanced if all its triangles are balanced (all triangles have an odd number of positive edges). Only complete graphs are considered in the paper.

The relationships in a complete n-node graph can be represented by an (n x n) symmetric matrix:

X = (x_ij), where x_ij encodes the type of relationship (+/- = friends/rivals) and strength

X is balanced if the graph that corresponds to it is balanced. Previous researchers settled on the following differential equation to describe the evolution of relationships over time ("matrix Riccati equations"):

dX/dt = X²

The authors show that for a given random initial matrix, the above dynamical system produces a balanced X in finite time, with probability 1: in the end either all the relationships are friendly or two hostile factions emerge (within each faction all the edges are friendly). When all the relationships are friendly, the graph is clearly balanced. Consider the case of two hostile factions: any given triangle will be balanced because a triangle is either entirely in one faction (3 friendly edges) or is a mix of nodes from the two factions (1 friendly edge).

In essence, this paper provides a mathematical proof of what earlier simulations revealed: for a generic initial matrix X, the dynamical system results in a balanced graph in finite time. They actually derive the "final sign configuration" of a social network, given the initial matrix of relationships.

Our analysis yields a closed-form expression for faction membership as a function of the initial conditions and implies that the initial amount of friendliness in large social networks (started from random initial conditions) determines whether they will end up in intractable conflict or global harmony.

Additional Links:

The authors put together a short movie to illustrate how Structural Balance occurs.

One of the authors, Jon Kleinberg, gave a talk on the results described above (and much more), recently at Stanford. See my recent tweet for details.

Hedge Fund Performance - Jan/2011

2011-02-19T13:30:00.000-08:00

For the second consecutive month, the S&P 500 out-gained all the DJ/Credit Suisse Hedge Fund indices. The hedge fund styles that did produce positive returns in January, eked out very modest gains. Managed Futures followed large gains in Dec/2010 with -0.8% decline in the opening month of January. Dedicated Shorts continue to swim upstream, opening 2011 with modest losses. Based on what Ben Bernanke wrote in Nov/2010

"... “higher stock prices will boost consumer wealth and help increase confidence, which can also spur spending. Increased spending will lead to higher incomes and profits that, in a virtuous circle, will further support economic expansion.”

Short Bias funds have to once again contend with central bankers moving to prop up equities. On the other hand, the cyclically adjusted price-earnings ratio is 45% above historical average, suggesting that real stock market returns will be lower over the next few months.

(Click to enlarge)
In addition, the S&P 500 gained 20% and 56% over the last 12 and 24 months, ending Jan/2011. It just isn't likely that gains like these can continue for much longer.

(Click to enlarge)
So with equity indices likely to produce smaller returns, which hedge fund styles look promising? Rather than offer specific recommendations, let me review the performance of the different styles over the last 5 years, ending Jan/2011. I previously introduced the use of the middle 50% of a distribution (the median, 25th, and 75% percentiles) as a potential tool for comparing recent volatility and performance. Based on MEDIAN monthly return over the 5-year period, the top 4 styles have been: Emerging Markets, Event Driven - Multi Strategy, Global Macro, Event Driven - Distressed. Using the Interquartile range as a measure of volatility, the least volatile styles over the same 5-year period have been Equity Market Neutral, Event Driven - Risk Arbitrage, Fixed Income Arbitrage, Global Macro.

(Click to enlarge)
So at least over the most recent 5-year period, Global Macro has done well compared to the other styles. How about over a longer time period? In the past I've plotted the various Hedge Fund indices using the standard Risk vs. Reward framework. Desirable funds are those that have low risk and high return -- you want funds that are high on the y-axis, but low on risk (i.e. the upper left corner). Using these measures to evaluate risk/reward over a longer period (17 years, from Jan/1994-Jan/2011), Global Macro is below average when it comes to volatility of returns.

(Click to enlarge)
I close with my Illiquidity metric: note that Managed Futures has surprisingly exhibited signs of increased illiquidity over the last 6 months.

(Click to enlarge)

Semi-regular Dimes:

What do burger prices tell us about the reliability of official inflation figures?: With inflation worries on the rise, official government measures may be subject to more massaging than usual.

John Paulson Racks Up a Second Epic Gain:
Hedge-fund manager John Paulson personally netted more than $5 billion in profits in 2010—likely the largest one-year haul in investing history, trumping the nearly $4 billion he made with his "short" bets against subprime mortgages in 2007.

Hedge fund D.E. Shaw cuts fees: In contrast, funds that are struggling have to adjust their fee structures. In the case of D.E. Shaw here are the rumored reductions: Management fees cut to 2.5% from 3%, Performance fees cut to 25% from 30%. Not exactly a huge sacrifice!

Performance of the 50 stocks that most frequently appear among the largest 10 holdings of fundamentally-driven equity hedge funds: A Goldman Sachs report, examines the stock-picking ability of equity hedge funds. Holding these stocks is a decent strategy, but it works only during upward trending markets!
... The basket of the 50 stocks that “matter most” has outperformed the S&P500 by 75 basis points on a quarterly basis since 2001. The VIP list outperformed the S&P500 by 127 basis points in the fourth quarter, and this year (to February 15), the list has outperformed the S&P500 by 46 basis points.

... It shows that the strategy of buying the 20 most concentrated stocks has a strong track record over more than nine years. The report said: "Since 2001, the strategy has outperformed the market 72% of the time by an average of 298 basis points per quarter (not annualised). The back test suggests that this strategy works in an upward trending market but tends to perform poorly during choppy or flat markets."

The close ties between governments and pension funds: Pension funds support governments by owning a sizable chunk of government bonds. OTOH, governments partially guarantee the same pension funds that finance their spending!
Governments rescued banks from the threat of failure in 2008, but banks are also big buyers of government bonds—so much so that a sovereign default in Europe might cause a banking crisis. Who is supporting whom?
The same question can be asked of pension funds. European countries would struggle to fund themselves without demand from the pensions industry. A survey by Mercer, an actuarial consultancy, suggests that around half the assets of continental European funds are invested in government bonds. Yet governments also stand behind their domestic pension funds.
The commitment is explicit in the case of public-sector schemes. But in America and Britain there is surely implicit backing for the insurance funds (respectively the Pension Benefit Guaranty Corporation and the Pension Protection Fund) that backstop the private sector’s schemes. It is hard to imagine politicians standing idly by if pensioners were at risk of losing their benefits.
Related topics
An example of this tangled relationship can be seen in the efforts of individual American states to deal with budget shortfalls. Some states are launching special bond issues to get the funds needed to finance their pension contributions. And who are likely to be significant buyers of those bonds? State pension funds.

Hedge Funds May Pose Systemic Risk in Crisis: With the official report making the rounds, let the lobbying begin. The Alternative Investments industry employ some of the most skilled lobbyists around, so I'm skeptical about the prospects for meaningful regulation and oversight.

[Hedge Fund performance from previous months can be found here. More details, including sector/style descriptions and methodology, are available here.]

A quick overview of the App Economy, using data from the iTunes app store

2011-02-18T11:46:00.000-08:00

Had fun at this year's TOC conference in New York City, the weather was great, not too cold, no rain and snow. Walking through Central Park on Valentine's evening is hard to top!

I also had to do some work, including this keynote presentation on Wednesday afternoon:

James Simons reminisces about Math

2011-02-08T09:28:00.000-08:00

Co-inventor of the Chern-Simons invariants, Jim Simons is known primarily for being the founder and CEO of Renaissance Technologies, whose track record remains the gold standard in the hedge fund industry. Not as well-known to the general public, Simons was a first-rate mathematician at the time he decided to leave academia for finance.

This talk, particularly the introduction by I.M. Singer, gives an overview of Simons' life in mathematics (and beyond). Simons also discusses the history of Renaissance Technologies, and hints at some of the things that makes it so successful.

Supporting old-fashioned Journalism

2011-01-31T19:45:00.000-08:00

With print and broadcast journalism constantly under attack, I just wanted to highlight a recent example of investigative journalism at its best: This American Life's episode dedicated to the NUMMI car plant in Fremont, California. I recently heard host Ira Glass recount that it took them months to put this episode together. The team of reporters would go out and talk to a few people, come back to the studio and listen to what they gathered, and decide who else they needed to talk to. This went on and on, for months on end.

The resulting episode is probably the most entertaining business case study you'll ever hear. One minute I'd be laughing out loud, a few minutes later I would be shaking my head in frustration. There are some touching and emotional moments as well, especially when the first set of American workers finished their training program in Japan.

Unfortunately, labor-intensive journalism of this sort is becoming the exception. But for a mere 99 cents you can own this amazing episode and learn a ton about the car industry in particular, and management in general. I recommend it highly, I can't think of anything more informative to purchase for a mere 99 cents!

You can purchase it at either:

Amazon , or

iTunes

In 1984, General Motors and Toyota opened NUMMI as a joint venture. Toyota showed GM the secrets of its production system: How it made cars of much higher quality and much lower cost than GM achieved. But today, GM cars still don't have the quality of Japanese imports, GM is bankrupt and on March 31, NUMMI will be closed, sending thousands of car workers looking for jobs. In this hour-long story, NPR Automotive Correspondent Frank Langfitt tells the story of NUMMI and why GM — and the rest of the American car business — wasn't able to learn from it more quickly.

Hedge Fund Performance - 2010

2011-01-18T18:00:00.000-08:00

The S&P 500 gained close to 13% in 2011 (6.5% in Dec/2011), making it more difficult for active managers of all stripes to shine. Hedgies were no exception. Nevertheless 9 out of 14 style indices gained 10% or more in 2011. But clearly once you factor in (performance and management) fees, gains likely trailed popular index funds and ETF's.

(Click to enlarge)
Over a the most recent 24 months (2008 to 2010) the S&P 500 returned 39%. Luckily hedgies are graded on their ability to deliver absolute (& not relative) return, and many were able to just that: 9 out of 14 style indices gained 20% or more over the same period.

(Click to enlarge)
Last month, I introduced the use of the middle 50% of a distribution (the median, 25th, and 75% percentiles) as a potential tool for comparing recent volatility and performance. I updated the chart from last month to include Dec/2010: data used are monthly returns from Jan/2007 to Dec/2010 -- the period immediately preceding the financial crisis, to the present. The chart provides a visual for the average monthly return (the median) during the period, as well as a measure of volatility (the interquartile range, or length of the bar). [As an added bonus you can even glean the type of skewness, if any, possessed by the distribution of returns.]

Judging from this simple chart, Global Macro has been the best style through the recent crisis. The 4 least volatile styles have been Event Driven - Risk Arbitrage, Equity Market Neutral, Global Macro, Fixed Income Arb. The most consistent (as measured by the median monthly return) have been Global Macro, Event Driven - Multi Strategy, Emerging Markets, and Convertible Arb.

(Click to enlarge)
In the past I've plotted the various Hedge Fund indices using the standard Risk vs. Reward framework. Desirable funds are those that have low risk and high return -- you want funds that are high on the y-axis, but low on risk (i.e. the upper left corner). A few event-driven funds seem to fit the profile, at least for the limited time period under consideration. Global macro has the best average monthly return, but is smack in the middle of the pack when it comes to volatility. (Note that the graph below is based on fairly recent data: Jan/1994 to Dec/2010.)

(Click to enlarge)
Before you rush into Event Driven funds, bear in mind that they are relatively illiquid styles:

(Click to enlarge)
Semi-regular Dimes:

Pack mentality grips hedge funds: I've noted several times that hedge fund investors tend to chase recent winners, recent studies suggest that hedgies themselves suffer from herd mentality. For one, there aren't that many original money-making trading styles out there -- at least ones that profit over long periods. So funds have long used similar signals and indicators. More recently, there have been many stories about information sharing (dinners) among hedgies. Finally, the added pressure stemming from the recent financial crisis has probably meant more managers are being cautious. The tendency to flock to similar trades, means that liquidity will be even more important in event of another market panic.
An analysis of hedge-fund returns by Andrew W. Lo, a Massachusetts Institute of Technology researcher and fund manager, shows that funds have become more likely to lose and gain money together over the past five years. There is a roughly 79% chance any randomly selected pair of hedge funds will move up and down in tandem in a given month from 2006 to 2010, compared with a roughly 67% likelihood from 2001 to 2005, according to his analysis.

One explanation is that they are focusing increasingly on the same stocks. Last year, for example, stock in Apple Inc. was held by 55 of the nearly 200 large hedge funds tracked by AlphaClone LLC. In 2008, by comparison, the favorite hedge-fund stock, Microsoft Corp., was held by 34 funds.

"The whole hedge-fund industry is a series of crowded trades," says Mr. Lo.

Speaking of herd mentality, here are 2 lists of stocks: companies that hedgies are shorting and ones they own.

Commodities are surging: Managed Futures had a strong December, a respectable 2010, but a disappointing 2-year run. Let's see if Managed Futures managers can go on a sustained run over the next quarter.

Global macro fund Clarium slumps 90% from peak: Because of Peter Thiel, Clarium gets a lot of media coverage. The challenge for Global Macro managers has always been how to convert macro-economic forecasts and trends, into profitable trades. A task that can be very difficult for the larger funds.

Global Macro Managers must remain nimble: As I noted above, since 2007 Global Macro funds have outperformed other hedge fund styles. But things are getting tougher:
Last year also showed the importance of being quick at entering and exiting trades. Typically macro funds build their positions slowly. However, “In last year’s choppy market, trades had a shorter lifespan,” says Tomas Jelf of Prologue Capital, a London-based macro fund. “You had to go in closer to the full size early on and then be willing to take it off quicker. You had to be tactically more aggressive.”

Despite tepid performance in 2010 investors have remained enthusiastic about macro funds. They are liquid and diverse, since they trade in many asset classes, and received around $9.5 billion in investor allocations in the first 11 months of last year, according to BarclayHedge, a data firm. That made macro funds one of the most popular types of hedge fund. Investors are fans “because there’s still a nervousness about the future”, says Michael Novogratz, a principal of Fortress, an investment firm. “Investors see macro as a hedge to the world shifting,” reckons Mr Novogratz, who oversees his firm’s macro funds.

That enthusiasm means other kinds of hedge fund are busy buffing up their expertise in macro. Steven Drobny of Drobny Global Advisors, a research and strategy firm for macro funds, says he is seeing a surge in firms venturing into the area for the first time—or what he calls “macro-tourists”.

[More details, including sector/style descriptions and methodology, are available here. Hedge Fund performance from previous months can be found here.]
.

John Sayles' Amigo

2010-12-25T12:01:00.000-08:00

I recently learned that one my favorite alt film directors completed a period piece about the Philippine-American war at the end of the end of the 19th century. With a shoestring budget of $1.5M, John Sayles outsourced essentially the entire production to the Philippine film industry. This translated to much lower production costs, and allowed him the luxury of six weeks of filming. It helped that actors such as Sayles favorite, Academy Award winner Chris Cooper, agreed to lower their fees substantially.

I wonder if many more movies will start getting made this way. When it makes sense, large English speaking countries like the Philippines could serve as locales for ambitious period films. In addition, their favorable cost structures allow directors like Sayles to tackle controversial (and/or less well-known) topics like the brutal Philippine-American war.

Below are a few interviews with Sayles where he elaborates on the making of Amigo, and the current state of alternative films:

John Sayles 'Amigo' Q&A -54th London Film Festival- from Artefacto Magazine -Aculco Radio on Vimeo.

Ranking U.S. Universities using Research Ouput and Impact

2010-12-22T07:51:00.000-08:00

I recently came across a Nov/2010 study from the American Association for the Advancement of Science, and the periodical Science. The goal was to measure research output (in U.S. universities) using research publications. At least for the summary report I read, the results weren't broken down by discipline ("math" vs. "biology") or even field ("health sciences" vs. "engineering"). It would have been interesting to see if regions that attract startups, are even more dominant in certain areas.

The first metric was volume ("publish or perish") with 5 U.C. campuses finishing in the Top 24. I'm not a big fan of volume, since it favors certain disciplines over others.

... Two dozen universities hold a combined 42% share of the overall U.S. output for the years 2005 to '09, the report finds. That's up from 31% during the 1981–85 period. That increased concentration has occurred at the same time the size of the overall pie has doubled, to roughly 1.6 million papers. Harvard University tops both lists, with a 4.2 share of that output, and its margin over second-place University of Michigan has widened in the past 30 years. The 61 U.S. members of the Association of American Universities (AAU) claim an outsized 56% share, up eight points.

(Click to enlarge)
The second metric measured the impact of papers by counting the number of citations, 4 U.C. campuses finished in the Top 19.

... Similarly, 19 universities received 47% of all citations to U.S. papers for 2005 to '09. Papers from the Massachusetts Institute of Technology, which has been ranked first or second during the past 3 decades, have more than twice the impact as the world average. In addition, a handful of universities have maintained their dominance: Only six universities have held one of the top five places in the impact rankings since the 1980s.

(Click to enlarge)

Hedge Fund Performance - Nov/2010

2010-12-16T07:19:00.000-08:00

November was uneventful for most of the DJ/Credit Suisse Hedge Fund indices, most finished flat (as did the S&P 500). Managed Futures was down 4.1%, essentially giving back the 4.3% it gained the previous month.

(Click to enlarge)
Looking over a 12 and 24-month time horizon, the results are much more to the liking of investors. The S&P 500 gained close to 8% over the most recent 12 months, a period when 7 indices gained around 9% or more (including Global Macro at 8.9%). However, the 32% gained by the S&P 500 over the last 24 months was exceed by only 4 indices (Convertible Arb led the way with a whopping 60% return, followed by Emerging Markets, and Fixed Income Arb).

(Click to enlarge)
The recent financial crisis started in the Fall of 2007, and I thought it might be useful to examine the volatility of these indices over the period spanning the crisis. There are many possible ways to measure volatility, I chose one that would deliver a simple graph. The interquartile range (IQR) is usually used graphically as part of a boxplot, but I find the combination of the IQR and the median to be effective in visualizing and comparing dispersion/volatility.

In the chart below, the length of the bar is the IQR. Note that the S&P 500 was fairly volatile during the period (Jan/2007 to Nov/2010), we did have a pretty serious financial crisis! The top-performers over the most recent 24 months (Convertible and Fixed Income Arb, along with Emerging Markets) were much less volatile. Global Macro (up 25% over the last 24 months) was much less volatile than Managed Futures (up a mere 2% over the last 24 months).

(Click to enlarge)
[Here's the same chart minus the legend.]

I close with my standard graph which measures the Illiquidity of the various trading styles: in case we have an unexpected crash over the last few weeks of the year, the top year-to-date performers (Fixed Income Arb and Global Macro) are relatively liquid.

(Click to enlarge)
Notable links:

Just when you thought investors have pulled back from startups, 250 HedgeFunds launched in Q3/2010!
Many of them are finding life is not like it used to be.

Being that we are close to the end of the year, here's the first of many more lists of the Worst Hedge Fund Managers of 2010. Lots of smallish funds ($200 million or less in assets) on the list, and I wouldn't be surprised if recent startup funds show up on the list next year. I've been reading that Clarium has been struggling, but I don't see them on this list.

Buyside wants more transparency: It's about time buyside traders organize and demand end-of-day dark pool trade disclosures.

Muni market update: A couple of long posts to help you get up-to-speed on the recent sell-off. (1) Making sense of the correction, (2) Default and Bankruptcy explained
One of the more frustrating aspects of muni market coverage in the news and blogosphere is the tendency to talk about municipal debt as if only one type of bond is issued and traded. There is actually considerable diversity among borrowers in the muni market (e.g., they are not all government entities), and by extension, the types of commitments that are made for the repayment of the debt. Although the relative health of the muni market has macroeconomic consequences, this is in many ways a market that defies generalization. (That’s one reason I find the muni market unusually interesting…)

[More details, including sector/style descriptions and methodology, are available here. Hedge Fund performance from previous months can be found here.]
.

Recent data analytics & visualization contests

2010-12-02T08:00:00.000-08:00

I've been sharing the results of several data-related contests in my BigData twitter feed. Given the increasing demand for data scientists, it might not be a bad idea for companies to reach out to the top contestants in some of these competitions. Below are a few contests that caught my eye over the past few weeks. I'm not surprised that direct marketing is a popular source of contest topics -- the direct marketing industry has held analytics contest for years. It is interesting to see geo-location, navigation, and traffic prediction emerge as another popular domain. (If you know other interesting contests (on Kaggle or other sites), let me know and I'll update this post.)

2010 UC San Diego Data Mining Contest: Winning entries from the U.S., India, Canada, South Korea, and Russia.
All current undergraduate students, graduate students, and postdoctoral researchers studying full-time, in residence at an accredited university or college may compete for prizes.

In this year's contest, an online retailer is interested identifying potential new customers from a population of consumers. Your task is to rank ordering consumer pool according to who is most likely to become customers of the retailer.

World Economic Forum Data Visualization Challenge: The contestants faced a pretty challenging task, and unless you're familiar with the different bureaucracies involved, the resulting visualizations are still hard to grok.
Making decisions in the 21st century means swimming through data — drinking from the proverbial fire hydrant. The World Economic Forum's Global Agenda Councils are tasked with tackling the world's thorniest issues. To do this, they must navigate a flood of interconnected, complex and interdependent information that can all too often confuse and obscure rather than clarify and expose. We are asking the world's design community to help us solve the problem of information overload.

The Hearst Challenge (optimize distribution to newstands): Co-sponsored by the Hearst Corporation, the Direct Marketing Association, and EXL, this contest "has closed to new registrations", but registered contestants can continue to submit entries until December 4th. I'm looking forward to seeing who bags the $25,000 prize. [Update (1/10/2011): Winning entry was from US-based A^3.]
Over the years, the magazine publishing industry has made significant strides in improving subscription based circulation by developing analytic frameworks that better predict customer response to acquisition and renewal offers. The objective of this contest is to apply the same analytic discipline and effectively predict newsstand locations "response". Specifically the objective is to predict the number of copies to be placed in each newsstand location to optimize the overall contribution of the newsstand location typically referred to as draw.
2010 Direct Marketing Association Analytic challenge: The contest rules and details should be familiar to anyone who's ever worked in marketing analytics. Consulting company DATALAB, using data mining (and decision tree) software from Salford Systems, produced the winning entry.
Build the best targeting solution for lapsed donor segments.

Each participant in the challenge will receive an identical set of information for building their model.

PARTICIPANTS MAY NOT INCLUDE ANY OTHER EXTERNAL VARIABLES FOR THE CHALLENGE.

IEEE/TomTom/ICDM Contest (Traffic Prediction for Intelligent GPS Navigation): The contestants devised algorithms for three tasks: (1) Traffic congestion prediction, (2) Modeling the process of traffic jams formation during morning peak in the presence of roadworks, and (3) Traffic reconstruction and prediction based on real-time information from individual drivers. Winning entries were from the U.S. and Europe.
It attracted 575 participating teams, of whom over 100 submitted solutions, most of them many times: the total number of solutions was close to 5000. Best algorithms achieved nearly 3-fold improvement over baseline solutions in predicting traffic congestion and jams.

RTA Freeway Travel Time Prediction: This is an ongoing contest with a $10,000 prize.
Forecasting travel times helps improve road safety and efficiency. Accurate predictions help commuters make informed decisions about when to travel and on what routes. This helps to lower intensity on problem arterials by encouraging motorists to use underutilised parts of the grid, and where possible, by having them select alternative times and modes of travel. This competition requires participants to predict travel time on Sydney's M4 freeway from past travel time observations. ... Participants in this competition are required to forecast the travel time on the M4 freeway for 15 mins, 30 mins, 45 mins, one hour, 90mins, two hours, six hours, 12 hours, 18 hours and 24 hours ahead.

The Google AI Challenge (co-sponsored by the University of Waterloo): More than 4600 entries from contestants in 112 countries. Winning entries were announced on December 1st.
The objective is to create a computer program that plays the game of Planet Wars as intelligently as possible. ... Planet Wars is a strategy game set in outer space. The objective is to take over all the planets on the map, or alternatively eliminate all of your opponents ships.

... Whereas the Google AI Challenge has traditionally been dominated by entries written in compiled languages like C++, Gábor has solidly taken the top spot with an innovative algorithm. His entry is one of just 33 submissions written in Lisp

KDD Cup 2010 : Sponsored by the ACM Special Interest Group on Knowledge Discovery and Data Mining (KDD), this annual contest uses challenging problems from a variety of domains. The 2010 challenge was from Mathematics Education, and the winning entry was from National Taiwan University:
... is it possible to infer the knowledge requirements of problems directly from student performance data, without human analysis of the tasks? This year's challenge asks you to predict student performance on mathematical problems from logs of student interaction with Intelligent Tutoring Systems. ... At the start of the competition, we will provide 5 data sets: 3 development data sets and 2 challenge data sets. Each of the data sets will be divided into a training portion and a test portion. Student performance labels will be withheld for the test portion of the challenge data sets but available for the development data sets. The competition task will be to develop a learning model based on the challenge and/or development data sets, use this algorithm to learn from the training portion of the challenge data sets, and then accurately predict student performance in the test sections. At the end of the competition, the actual winner will be determined based on their model's performance on an unseen portion of the challenge test sets.

The 2010 Analytics X Prize (Predicting Homicides in Philadelphia): The goal is to predict homicides in 47 Zip codes, and results are announced at the end of each month. Two users appear to be dominating the contest, although it's unclear how many entries/teams are actively involved.
Analytics X is an ongoing contest to apply analytics, modeling, and statistics to solve the social problems that affect our cities. It combines the fields of statistics, mathematics, and social science to understand the root causes of dysfunction in our neighborhoods. Understanding these relationships and discovering the most highly correlated variables allows us to deploy our limited resources more effectively and target the variables that will have the greatest positive impact on improvement.

Contestants predict the fraction of all 2010 Philadelphia homicides that will occur in each of the 47 major zip codes. These predictions are uploaded (using the Submit a Prediction button) and are then automatically compared to actual homicides that occur throughout the year. The best performing models are given awards at the end of each month.

2010 International Olympiad in Informatics: I didn't know that there is a CS/Informatics version of the Math Olympiad. The first one was held in Bulgaria in 1989, and the 2011 contest is slated for July in Thailand.
The International Olympiad in Informatics (IOI) is one of the most recognized computer science competitions in the world. The competition tasks are of algorithmic nature; however, the contestants have to show such basic IT skills as problem analysis, design of algorithms and data structures, programming and testing. The winners of the IOI belong to the best young computer scientists in the world.
Gold medals awarded at the International Olympiad in Informatics (1999-2009)
China’s approach to IOI is proof of just how serious the contest has become and how tied up it is in notions of national prestige and economic competitiveness. To earn a spot on the Chinese team, a coder has to beat 80,000 of his compatriots in a series of provincial elimination rounds that last an entire year. Then he—the competitors are almost all male—has to sit on a stage in front of hundreds of onlookers and answer questions in English like “How will you show the traditional culture of China to the foreign friends?” says Yi Wu, a member of China’s team.

... Universities and corporations have taken notice, treating the IOI—competitive programming’s crowning event—as the equivalent of the NFL Scouting Combine. This year’s primary sponsor is BlackBerry maker Research in Motion, whose CTO spoke at yesterday’s glitzy opening ceremony featuring the Waterloo Warriors cheerleading squad. Other sponsors include Lenovo, IBM, Amazon.com, and Google. It’s understandable that so many tech companies would want to nurture IOI: Google is famously built atop an algorithm called PageRank; Twitter’s “Who to Follow” feature is a set of algorithms; Amazon’s entire site, from sales rank to product recommendations, is a system of interlocking algorithms. IOI may be an academic competition, but its purpose is to advance a discipline that has driven the growth of the Internet and turned geeks into billionaires. Here in Waterloo, what’s at stake for the kids is a college scholarship, but what’s at stake for the industry is something larger: the ability to keep innovating new algorithms to make sense of an increasingly complex world.

2010 International Math Olympiad: Speaking of the IMO, I grabbed the 2010 results and graphed the medalists by country. China and the US each had 6 individual medalists, but China had 6 Gold medals to the US' 3.

(Click to enlarge)

Hedge Fund Performance - Oct/2010

2010-11-21T16:56:00.000-08:00

[I was at a conference this week, so this post has been delayed several days.]

Through the first ten months of 2010, trading styles that can take advantage of movement in currencies, commodities, and (sovereign) have done quite well. Managed futures funds led the way in October, and managers are enjoying inflows as true-to-form, investors chase after hot trading styles. Shorts continued to get clobbered. Profiting consistently from shorting stocks when governments are printing money is simply put, extremely difficult.

(Click to enlarge)

Over the most recent 12 months the S&P 500 outperformed all trading style, but over the most recent 2-year period several styles outperformed the S&P 500 by at least 5%:

(Click to enlarge)

Of the top-performers in 2010 (Global Macro, Managed Futures, Fixed Income Arb, Emerging Markets, and Convertible Arb), two remain among the more illiquid styles (Convertible Arb, Emerging Markets):

(Click to enlarge)

I close by plotting the various Hedge Fund indices using the standard Risk vs. Reward framework. Desirable funds are those that have low risk and high return -- you want funds that are high on the y-axis, but low on risk (i.e. the upper left corner). A few event-driven funds seem to fit the profile, at least for the limited time period under consideration. (Note that the graph below is based on fairly recent data: Jan/1994 to Oct/2010.) A major drawback, judging from the previous chart, is that event-driven funds are relatively illiquid.

(Click to enlarge)
Recommended reading:

Junk Bond stars: Drexel Burnham Lambert's legacy

Some thoughts on the recent sharp correction in the muni market
... The pending expiration of the Build America Bond (BAB) program has pulled supply forward, and this is going to seesaw over the next several weeks. Since the BAB program was initiated, most issuers have structured their new issues with the sense that they will go to either the tax-exempt or taxable market, whichever is more advantageous at the time. It has been almost completely a supply management game since the market for these bonds was established and munis became truly bifurcated.

... What is going on now is that muni issuers are scrambling to get deals done to take advantage of the program before it expires, and this is pulling the number of new issues that would ordinarily be coming to market forward. So the looming expiration of the BAB program is creating the very conditions it was created to alleviate. Issuers are very conscious of this fact, and that is why a large number of deals are getting pulled. As more issues get pulled and supply is reduced, there will be some relief on rates, which I think is what happened today. But you can expect that muni issuers will be dancing around this until the program expires at the end of the year, so there will likely be significant volatility. There is also considerable uncertainty as to how supply issues will play out in the first quarter of 2011. The BAB program may expire, but the social-economic-political realities behind why it was established in the first place have not materially changed. It is also difficult to gauge investor demand for tax-exempt bonds when no one knows how investors will be taxed. Since our esteemed policymakers are almost guaranteed to twiddle their thumbs and “study” their options, I doubt that the muni market will reflect fundamentals any time soon (although that is pretty much true for the financial markets as a whole).

[More details, including sector/style descriptions and methodology, are available here. Hedge Fund performance from previous months can be found here.]

Bad Girls Book Club

2010-11-19T10:42:00.000-08:00

My wife brought home this magnet from our neighborhood (women's) book club, which really does describe that particular group!

2010 Web 2.0 Summit: By the Numbers

2010-11-18T13:30:00.000-08:00

I'll let the pundits do the analysis and stick to factoids and statistics. Here's a sampling of I what heard at the sessions I attended this week.

Google CEO Eric Schmidt:

AdMob serves 1 billion impressions daily
Google hires 200 people per week
Amount of video uploaded onto Youtube: 35 hours of video per minute

Zynga CEO Marc Pincus

Zynga currently has 1,300 employees
320 million users have played a Zynga game
There are 650,000 same-sex marriages in Frontierville
On average there are 3 million peak concurrent users each day
On average 4 of your (social network) "friends" are playing a Zynga game with you

Chief Globablization Officer at Cisco, Wim Elfrink

By 2020 there will be 1 trillion connected devices
700 million people will move to urban centers over the next decade
A math tutor in South Korea earns 4 million U.S. dollars per year

Baidu CEO, Robin Li

There are a mere 420 million internet users in China, so there is still quite a bit of growth available. Broadband access is even lower.
99% of Chinese internet users are Baidu users
100 million questions have been posed and answered on Baidu's Q&A service (Baidu Notes).
There are 800 million mobile phone users in China
There are 150 million mobile internet users in China, most use 2G phones which are slow and have expensive data plans

Yahoo! CEO Carol Bartz

Yahoo! serves 18 billion ads daily
On any given month 600 million people land on a Yahoo! domain
Yahoo! serves 6 million different versions of its homepage each day. Machine-learning is used to match users to an optimal home page.

Facebook CEO Mark Zuckerberg

On any given day 50% of active Facebook members use the service
200 million access Facebook with mobile devices
Zuckerberg's Ratio to measure the speed and agility of the company: Ratio of active users (500+ million) to engineers (a few hundred) remains over a million users per engineer.

Automattic CEO Toni Schneider

There are over 30 million web sites that run on Wordpress software; 50% of such sites are hosted on wordpress.com
8.5% of all internet sites run on Wordpress software
Growth is accelerating: in recent months, over a million Wordpress sites were created monthly.
The sites hosted on wordpress.com attract 250 million visitors each month, one of the largest audiences online.
The company only has 72 employees, reminiscent of another small company, high-traffic site based in SF.
As of 2010, 22% of content published on wordpress.com is published/posted from a mobile device
Domain name registration is the fastest-selling upgrade on wordpress.com: for 2010, they are on track to sell 60,000 domain registrations.
The rise of Social Media Optimization (below are some web sites and the share of their traffic that come from a given social media site):

GigaOM => 30% of traffic comes from Twitter
Bandcamp => traffic from Facebook surpasses traffic Google
wordpress.com => Ratio of (traffic coming from social)/ ((traffic coming from search) has gone from 3% to 15%, a fivefold increase.
In general, Automattic is seeing that traffic from Google continues to grow, but traffic from social sites is growing faster.

Update (7/10/2011): The following stats were just released by Wordpress: 50B blogs on Wordpress, 287M unique readers and 2.5B page views/month, plus engagement metrics (500K posts and 400K comments daily).

Linkedin CEO Jeff Weiner

Linkedin currently has 85 million members
It took 477 days to add their first million members, it took just nine days to add their most recent million
Linkedin adds 1 new member per second: half of all new members are based outside the U.S.
Close to 60 million unique visitors per month
John Battelle asked Weiner who he considered to be direct competitors and he somehow forgot to mention two other large professional networking sites: Xing and Viadeo.

Susan Wojcicki of Google

30X growth in mobile "shopping" queries

Active Facebook users by region: November, 2010

2010-11-16T15:00:00.000-08:00

With Facebook unveiling an integrated messaging system for its more than 500 million users, I decided to update a few charts that breakdown its users by region.

I. Percentage share of active users (weekly): note the steady rise in the share of users from Asia.

(Click to enlarge)

For details, read my Radar post.

Hiring trends among the major platform players

2010-11-15T15:45:00.000-08:00

The battle for the Internet's points of control requires amassing talent.

(Click to enlarge)For details, read my Radar post.

Inside Facebook's infrastructure: Hive

2010-11-14T08:38:00.000-08:00

Following up on my previous post on Google, here are some interesting stats on Facebook's data warehouse.

(Click to enlarge)

More details contained in John Sichi's recent presentation:

Hive Evolution: ApacheCon NA 2010

View more presentations from John Sichi.

Inside Google's Infrastructure: MapReduce Statistics Over Time

2010-11-11T12:58:00.000-08:00

Google uses more than 39,000 servers to process roughly an exabyte per month, with MapReduce (data is for May/2010 shortly before Google switched over to Caffeine):

(Click to enlarge)
For more, see my recent tweet.

.

Windows Mobile apps are more expensive than iPhone Apps

2010-11-05T06:50:00.000-07:00

Update: Several readers have correctly pointed out that the source of the data I used for the post, was for Windows Mobile apps, so I decided to tweak the title to reflect that. The goal of this post is to examine the marketplace for Windows smartphone apps prior to the much-anticipated launch of Windows Phone 7. I hope to do an update a few months post-launch, where I can use this baseline data, to demonstrate positive effect of the launch of WP7.

(Click to enlarge)For details check my recent Radar post.

Crowdsourcing Specific Microtasks

2010-10-25T15:14:00.000-07:00

Since the first-ever Mechanical Turk meetup a year ago, there has been an explosion in crowdsourcing services and a well-attended conference in San Francisco. I remain enthusiastic about crowdsourcing, but the number of companies has me worried about quality of work. Fortunately specialization is already occurring, so for particular tasks there are companies out there ready to provide high-quality service.

For more, see my Radar post.

Semi-regular Dimes

2010-10-25T08:00:00.000-07:00

A selection of links recently shared through my two twitter accounts (@dliman and @bigdata)

Can computers learn to read? NELL (Never-Ending Language Learner) has acquired a knowledge base of nearly 440,000 beliefs read from a variety of web pages.

DARPA's deep learning program: A new program to award grants for the creation of machines that use layers of processing similar to that used by biological brains.

Two python modules: (1) PyBrain a modular Python Machine Learning Library, (2) Whoosh is a fast, featureful full-text indexing and searching library implemented in pure Python.

Geospatial Forecasting with an Agent-based, Bayesian method for Geographic profiling: This research project out of UCLA has had glowing press coverage for its potential applications in policing and crime prevention (see the LA Times and the Economist). Looking forward to seeing more details about how the system works.

Tools from Google: (1) Pregel is a tool used to mine network graphs and scales to billions of vertices and edges, (2) Dremel is an ad-hoc query system for analysis of read-only nested data, aggregate queries over trillions of rows in seconds

Compressed Sensing and Big Data

2010-10-21T17:30:00.000-07:00

Every discussion on Big Data usually includes a passing reference to the ubiquity of sensors: as we collect more data through sensors, we'll need tools to take advantage of data. The focus has been primarily on the tools that help you store & manage, visualize & summarize, and finally analyze and make sense of Big Data collections. Since the leading Big Data innovators have tended to be web companies, data acquisition has often involved data exhaust or crawled data. The mechanics and details of collecting data from the physical world has garnered less attention.

Consider common examples such as digital images¹, video, or audio. The standard approach usually involves three steps:

Collect enough signals to measure "everything": This typically entails a lot of measurements (say 1 million) to generate a RAW image.
(JPEG) Compression: Computed wavelet coefficients decay fast enough, that only the largest 100 coefficients are retained (1% of 1 million).
Invert the (wavelet) transformation to recover original: The rest of the wavelet coefficients are discarded making the data collection/sampling step somewhat wasteful.

Rather than store and discard many signals/measurements, it would be more efficient if one can acquire only the necessary signals to begin with. Compressed sensing goes against the long-established tradition outlined above. It is an attempt to simultaneously perform measurement and compression. Efficient data acquisition has obvious benefits, and they all point to faster-growth in the number of big data collections.

Reduction in data collection time: If you measure less signals, you'll be done faster. This is very important in medicine, particularly pediatric MRI. Wired had a recent article describing how researchers in Stanford were able to significantly reduce MRI scan times (I hear the state-of-the-art is by a factor of 6, so from holding your breath and remaining still for 120 seconds, to a more manageable 20 seconds). Further improvements can be expected once compressed sensing is combined with knowledge about the structure of what is being measured.
Reduction in energy consumption: If you measure less signals, you most likely will consume less energy. Anything that extends battery life is automatically worth looking at.
Data deluge becomes even more likely: The combination of time reduction and energy efficiency will make sensors even more popular. Pictures, images, audio, location (common manifestations of compression and sensor technology) may not be where this technology takes off. But what compressed sensing may do is make other forms of sensors more ubiquitous. Compressed sensing is an active area of research, with the focus primarily on efficient algorithms and concrete applications. Areas where companies are forming include optical systems (e.g., single-pixel camera), analog-to-digital conversion, and (Dynamic) MRI.

It's one thing to propose a magical system that combines compression and measurement in one step, but is it possible? Compressed sensing pioneer, Emanuel Candes defines Idealized Sampling as "collecting information to measure the most important components". Unfortunately, one doesn't know a priori which of the components are important! But through a series of numerical experiments, Candes suspected that incoherent/random samples may be the answer.

Candes' algorithm looked great on a few examples, so the next step was to determine theoretical reasons for why it might work in general. In a series of papers, Candes and Fields Medalist Terence Tao³ set out to establish the mathematical foundations that underpin Compressed Sensing. The mathematical details are complex, and the efficacy of the algorithms rely on deep results in harmonic analysis. But once you establish the fundamental results, the algorithms proceed roughly along these lines:

Take m random measurements: One can think of random sensing as a sampling using a different vector space basis. The new basis has minimal properties (isotropy, incoherence), and some common examples include Gausian vectors, random binary vectors, randoms rows of a fixed unitary matrix, etc.
Reconstruct original signal, f, using linear programming: Minimize the L₁ norm (sum of the inner products between f & the new basis vectors) instead of the usual energy metric (the L₂ norm). Call the L₁ minimizer f^#.
L₁magic and exact reconstruction: Unlike the L₂ norm, L₁ promotes sparsity. Moreover, it turns out that f^# also minimizes MSE⁴.

A recent article from Wired has an example of Compressed Sensing in action, see here. Interest picked up once the requisite theorems⁵ proving the universality of this new approach were published. Since the actual algorithms rely on familiar things like convex optimization, many groups could quickly start experimenting with Compressed Sensing.

Finally, a slight detour into mathematical analysis. Why L₁ works isn't immediately intuitive, but it lies at the heart of Compressed Sensing. Terence Tao explains L₁magic as follows:

The key difference between this minimisation problem and the L₀ problem is that the L₁ norm is convex, and so this minimisation problem is no longer NP-hard but can be solved in reasonable (though not utterly trivial) time by convex programming techniques such as the simplex method. So the method is computationally feasible; the only question is whether the method actually works to recover the original S-sparse function f.

Before we reveal the answer, we can at least give an informal geometric argument as to why L₁ minimisation is more likely to recover a sparse solution than L₂ minimisation. The set of all f whose Fourier coefficients match the observed data forms an affine subspace of the space of all functions. The L₂ minimiser can then be viewed geometrically by taking L₂ balls (i.e. Euclidean balls) centred at the origin, and gradually increasing the radius of the ball until the first point of contact with the affine subspace. In general, there is no reason to expect this point of contact to be sparse (i.e. to lie on a high-codimension coordinate subspace). If however we replace L₂ with L₁, then the Euclidean balls are replaced by octahedra, which are much “pointier” (especially in high dimensions) and whose corners lie on coordinate subspaces. So the point of first contact is now much more likely to be sparse. The idea of using L₁ as a “convex relaxation” of L₀ is a powerful one in applied mathematics; see for instance this article on the topic.

(1) Image is from Roman Vershynin's slides.
(2) Recent talk at Stanford.
(3) Tao is an example of (what my friend Steve Hsu would refer to as) a Big Brain.
(4) Not only is f^# optimal from in terms of MSE, Candes mentioned initial tests indicate Radiologists prefer the results produced by compressed sensing techniques.
(5) I dig the theoretical math behind compressed sensing, as much as the actual algorithms and applications. (To the mathematically inclined, I apologize for using the symbol L₁ instead of l¹ - the latter is more accurate but less readable.)

Apple's Q4-2010 Earnings: Strong revenue from the iPhone and iPad

2010-10-18T17:01:00.000-07:00

In the meantime, the iPod continues to plod along. The iPod does sell more during the Christmas season, so we should see a spike in iPod revenue for the next earnings statement. But you definitely get the sense from the revenue chart that the standalone iPod hasn't been a growing category since the iPhone launched.

Below are some simple charts to help bring to life the tabular data that accompanies Apple's quarterly release:

Apple earnings q4-2010

View more presentations from O’Reilly Media.

Hedge Fund Performance - Q3/2010

2010-10-16T08:00:00.000-07:00

In a month when the S&P 500 gained close to 9%, the DJ/CS Hedge Fund indices (except for Shorts) generated positive returns ranging from 1% to 5%. Following a month in which it gained 5%, Short Bias Index got hammered hard (the index was down 11.3% in Sept.). With so much movement in the currency markets, Global Macro and Managed Futures funds could be poised to add to their respectable returns through the first three quarters of the year. Looking ahead, one thing to consider is that compared to Global Macro funds, Managed Futures tend to struggle when volatility is low.

The fact that the Fixed Income Arb Index is up 10% through Q3/2010, combined with hedge fund investors propensity to chase winning styles, I'm not surprised that fixed-income arb pioneer John Meriwether is launching yet another fund!

(Click to enlarge)
Over the most recent 24 months (Sep/2008 to Sep/2010), the S&P 500 returns slightly negative (-2%), while all but two of the indices are up double-digits during the same period. The two that suffered losses, Shorts and Equity Market Neutral, were down big:

(Click to enlarge)

Replaying a pattern from earlier this year, Managed Futures has inched up my Illiquidity rankings. Through the first three quarters of 2010, the top two performing indices (Fixed Income Arb, Global Macro) are relatively more liquid than some of the other styles.

(Click to enlarge)

To measure absolute return, I group months by quintiles of the S&P 500: Q1 is comprised of months in the top 20% of monthly returns, Q5 is comprised of months in the bottom 20%. Computing average monthly returns across the quintiles is one way to gauge whether strategies make money irrespective of conditions in the equity markets. Some of the indices produced positive mean returns across all five quintiles, but not after investors factor in management and performance. (Data is from from Jan/1994 to Sep/2010.)

(Click to enlarge)

[A similar chart for MEDIAN returns across the 5 quintiles can be found HERE.]

I close by plotting the various Hedge Fund indices using the standard Risk vs. Reward framework. Desirable funds are those that have low risk and high return -- you want funds that are high on the y-axis, but low on risk (i.e. the upper left corner). A few event-driven funds seem to fit the profile, at least for the limited time period under consideration. (Note that the graph below is based on fairly recent data: Jan/1994 to Sep/2010.) A major drawback, judging from the previous chart, is that event-driven funds are relatively illiquid.

(Click to enlarge)

[More details, including sector/style descriptions and methodology, are available here. Hedge Fund performance from previous months can be found here.]

Partha Niyogi, Computer Science Professor, 1967-2010: Manifold (Machine) Learning Pioneer

2010-10-12T10:08:00.000-07:00

One of the best talks I've attended this year was Partha Niyogi's brief introduction to Manifold Learning, an hour-long survey at this year's joint meetings of the AMS/MAA/SIAM. I was struck by how natural and elegant it is to cast machine-learning tasks such as classification/categorization, in a differential geometric framework. My brief write-up, which contains links to key papers, can be found over at the O'Reilly Radar.

While reveling in his talk in January, I was unaware that Professor Niyogi was battling brain cancer. He was fully engaged and enthusiastic while giving his presentation, but in retrospect there might have been signs that he was battling a serious illness even back then.

University of Chicago Professor Partha Niyogi, who applied computation to the analysis of speech recognition, language evolution and data analysis, died Friday, Oct. 1, at his home in Chicago’s Hyde Park neighborhood, of brain cancer. He was 43.

Niyogi, who was the Louis Block Professor in Computer Science and Statistics, leaves a deep personal and professional void among his friends and colleagues. “Partha was one of the most beloved and respected people in the department,” said John Goldsmith, the Edward Carson Waller Distinguished Service Professor in Computer Science and Linguistics at UChicago.

... Niyogi was known among his colleagues as a master of the mathematical and computational techniques that he applied to his specialty of machine cognition and learning. “But what distinguished him was that he was not content with the mastery over the techniques,” Mulmuley said. “He also understood their limitations and tried to transcend them. Most importantly, he had a vision.”

Colleagues remarked as well about Niyogi’s wide-ranging intellectual interests, which he pursued with an equally broad, open mind. One of Niyogi’s first doctoral students, Mikhail Belkin, PhD’03, learned this about his mentor-to-be early on. Belkin, now an assistant professor of computer science and engineering at Ohio State University, met Niyogi in 2000, when both men were new to UChicago. Belkin sought out Niyogi after encountering a convoluted mathematical argument in a paper on pattern recognition.

“I asked him some questions about the paper, and even though the topic was new to him, he had put serious thought into it and we started regular meetings,” Belkin said. “We made significant progress and developed a line of research stemming initially just from trying to understand that one paper and to simplify one derivation. I think this was typical of Partha, showing both his intellectual curiosity and his intuition for the serendipitous.”

... Niyogi was born July 31, 1967, in Calcutta (Kolkata), India. As a child he lived in Bombay (Mumbai), where he developed a passionate interest in cricket. He received his bachelor’s degree in electrical engineering from the Indian Institute of Technology in New Delhi, India, in 1989. There, Niyogi already began demonstrating his wide-ranging interests, writing a thesis on the automatic recognition of beats on the tabla, a percussion instrument of northern India.

His undergraduate thesis then led him to the study of perception, recognition and learning, as well as acoustics, music and language. He earned his master’s degree in 1992 and his doctorate in 1995 from the Massachusetts Institute of Technology, where he interacted with computer scientists, linguists and cognitive scientists more generally.

... Surviving Niyogi are his wife, an economist at the National Opinion Research Center at UChicago; their twin sons, Nikhil and Kabir Niyogi; his parents, Ranjit and Prabhati Niyogi; and a brother, Siddhartha Niyogi.

Smartphone stats on the day Windows Phone 7 Launches

2010-10-11T11:50:00.000-07:00

On the day Windows Phone 7 (WP7) launches, it's good to review its market share in smartphones. I'm very interested to see what Microsoft's marketing machine will accomplish over the next few months - it is widely reported that they are planning to spend $400-$500 million to help boost WP7. The early reviews are going to be important and can make or break WP7's launch.

“Microsoft is way behind the competition and the chances of it catching up are fairly low,” says Brent Thill of UBS, an investment bank. He points out that it will take time to persuade developers to create a rich range of software applications for Microsoft’s operating system. IDC, another research company, reckons that by 2014 Microsoft will still trail both Apple and Android, as well as Research in Motion (RIM), the maker of the BlackBerry.

I personally think that Microsoft is going to exceed¹ (the current low) expectations. If they launch with a few essential apps, and continue to woo developers and carriers, they will probably do enough to remain relevant. Except for iPhone users, many consumers still purchase their phones through their carriers retail stores. Microsoft will surely try to get the retailers to push some WP7 models.

Number of apps is something the media hypes too much. It's an indicator of interest in a platform, but few consumers use and install more than a handful of apps. For developers, Microsoft brings some important pieces to WP7 -- Outlook/Exchange, Bing, Speech Technologies, and Maps -- they'll need to provide tools that will let developers easily² integrate these technologies. In addition, Microsoft already has relationships with game developers through the Windows and Xbox platforms. The latter is especially important given Apple's experience - Games remain the most popular apps on the iPhone.

(1) My guess is Windows phones will ~~have close to~~ exceed 10% market share worldwide 12 months from now.

(2) Apple's continued use of Objective-C prove developers are a patient lot. I'm sure many will write WP7 apps if Microsoft makes it easy for them to do so, and especially if WP7 can reach double-digit market share.

Failcon 2010 Fails (at least temporarily)

2010-10-11T10:05:00.000-07:00

Here's what I got when I tried to connect to the Failcon conference site:

(Click to enlarge)

A Taste of Home in San Francisco

2010-10-04T09:17:00.000-07:00

Taken outside the Asian Art Museum yesterday. Just happened to walk by and found a bunch of events commemorating Filipino American History month. I've been hoping to track down Senor Sisig at some point, it was great to just accidentally stumble upon their truck.

(Click to enlarge)

Manufacturing in China: Two Recent Perspectives

2010-10-01T08:00:00.000-07:00

I just finished Paul Midler's engaging book (Poorly Made in China) which chronicles his experiences assisting U.S. importers navigate the confusing world of Chinese manufacturing. While Midler seems to have gotten jaded towards the end of his tenure in China, the business examples and cultural observations he raises resonated with me. Growing up in Southeast Asia, the importance of "saving face" and the reluctance to be direct is something I can identify with.

from p. 214: "It was important for the game to be won, and it mattered less how the score was achieved. Appearance over substance was a cultural theme in China, it quite likely weighed on the reporting of population statistics"

Using personal stories and interviews, Midler repeatedly illustrates that "low prices" aren't really a bargain, especially given the poor attention to quality and consumer safety:

from p. 198: "When politicians pushed for free trade with China, though, none of them had ever suggested that the American public would be involved in any sort of trade-off. If anyone had joked about how sending manufacturing in China came with a jump back to 1849 - a time that predated consumer safety protection - for only modest savings, most consumers would have been against it.

While based primarily on the experience of one person (Midler himself), the stories are eerily similar to those I've heard from other importers. As an added bonus, the book is filled with incredibly funny anecdotes.

About a year and half ago, I raved about another book on Chinese manufacturing, Leslie Chang's Factory Girls. Chang's book is filled with numerous stories she collected during her tenure as a WSJ reporter in China. The impact of large-scale factories on China's social fabric and families are the subject of Factory Girls. Like Midler, Chang is a gifted story-teller and her book is filled with many moving stories from factory workers and their families.

If you want to get up-to-speed on the state of manufacturing in China, reading both Factory Girls and Poorly Made in China should do the trick. One is told using stories from (young) women in Southern China's factory zones, the other cites the challenges faced by small to mid-size U.S. importers. I highly recommend both books.

Text Mining and Twitter II - Python code for Analyzing Streaming Data Sets

2010-09-29T22:08:00.000-07:00

Updating my previous post on the use of LDA methods to analyze Twitter data, I just came across tools, from David Blei and his collaborators, for performing LDA Inference on Streaming Data Sets.

The python code is HERE and requires both NumPy and SciPy. It includes

"... a script that repeatedly downloads random wikipedia articles and updates an LDA posterior. (in our forthcoming paper, we have fit such a model to 3.3M articles from wikipedia."

The accompanying paper is forthcoming and I'll update this post as soon as it's available.

Related posts:

Text Mining and Twitter

Text Mining and Twitter III - LDA Code on Hadoop

The most dangerous job in America

2010-09-27T14:53:00.000-07:00

Via @jstogdill: the workers in the video below are scaling a tower 1,768 feet high. According to OSHA

"Tower climbing remains the most dangerous job in America. The majority of fatalities are the result of climbers not being tied off to a safe anchorage point at all times or relying upon faulty personal protection equipment. Many fatalities have occurred during the erection, retrofitting or dismantling of a tower. "Tie or Die!" has become synonymous with the requirement for 100 percent fall protection."

Some thoughts on (natural language) Search

2010-09-21T08:50:00.000-07:00

Google Instant motivated me to write down some thoughts about the state of web search. What follows are a few observations in some rapidly changing areas within search:

Natural Language
A few signs point toward the growing importance of natural language techniques.

Default is to search using sentences: When search interface researchers study consumers who aren't heavy users of their services, they notice they formulate sentence-like queries. Only over time do they start routinely submitting queries that involve only a few words. In other words, newbie users prefer precise rather than broad queries.
Trend is towards longer search queries: Google and other search engines have noticed an upward trend in average search-query length (i.e., searches involve more words). One would expect that as users iterate through a search task, they end up submitting longer queries in the process. Google Instant could lead to users submitting longer and more precisely formed queries. Interestingly Google recently found that longer queries tend to appear in the middle of sessions involving difficult search tasks.
A sure sign that users want to move beyond keywords is that many no longer hesitate to submit questions and sentences to search engines. One of the fastest-growing areas is "Knowledge Search": pioneered by the South Korean search engine Naver (the inspiration behind Yahoo! Answers) there are many new tools along these lines (eHow, Stack Overflow, O'Reilly Answers, IBM's Watson, and Quora in particular is generating buzz). Google Instant may further accelerate the trend towards longer and more refined queries involving natural language.
Factual Searches: Tools that are great at answering Who, What, Where, and When, but aren't great for Why. Two recent entries are Wolfram Alpha, which relies on curated structured data mainly from Science and Technology, and Swingly, which crawls the web for answers to "factual questions". Both work best when users submit well-formed, natural language queries.
At some point, the search engines will direct their crawlers to harvest even morenatural language (syntax, semantics) structure: MSFT bought Powerset, Google bought Metaweb and Aardvark. The search engines have to be noticing the popularity of Knowledge/Factual search tools, and their search logs are probably showing an uptick in natural language queries.

Content

Rich Web Apps: Google/Bing/Yahoo continue to improve their ability to crawl Flash and Javascript. Google announced that they just started crawling SVG graphics at the end of August. HTML 5 will make some things easier for crawlers, but I'm glad the search engines continue to improve their crawlers in this important area.
Expanding the search domain with Machine Translation: While Google Instant got most of the attention, the more interesting thing they announced is the ability to search pages in all languages (the true WORLD wide web) and return ranked results automatically translated: see example here. I'm looking forward to seeing if Bing and Yahoo! will copy this feature before they relase their version of Instant. OTOH, Google has placed things like this on their interface only to remove them in future releases. I hope Translate stays and becomes a part of regular search results -- things will get interesting fast for SEO's!! Users don't care what the original language on a page is, as long as it answers their search query.
Multimedia search: Have you ever tried to do a reverse image search? There are some tools available, notably TinEye and Gazopa, but I'm waiting for one of the major search engines to include this as a feature. Google recently bought Like.com, which has long allowed shoppers to do reverse image searches to find similar products. However, computer algorithms can only do so much and I haven't heard of any recent major breakthrough in this area. Additional meta-data would go a long way towards improving reverse image search (note that for a long time Google didn't even use image processing for their search results, they just analyzed nearby text). Like.com recognized early on the limits of automated image search and employed teams of fashionistas to help their search algorithms.
Meta-data enhances multimedia search: Systems that can automatically harvest meta-data from multimedia content have a natural advantage. TalkMiner uses OCR to parse slides from video presentations, giving their algorithms a leg up over video search tools that rely primarily on the title/abstract/description.
Virtual Goods: As social gaming and virtual goods become more popular, search tools need to improve. Try searching for virtual items within Second Life! The lack of interoperability and standards will make it hard to build a search tool capable of wading through multiple virtual worlds.
Specialized content (Math/Science): I'm always on the lookout for quirky tools, and as a nod to some of you more quantitatively inclined readers, let me highlight search engines for finding Formulas and Equations: LaTex Search and (uniq)uation.

User Interface

Wonder Wheel: How many of you ever used Google's wonder wheel? It was a great visual tool for learning about related searches. In many cases Wonder Wheel let you drill-down while maintaining context. Sadly Wonder Wheel disappeared the day Instant went live.
The growing number of search interfaces: Google, Bing, Yahoo all have some form of social and real-time search. In the case of real-time Twitter/Facebook searches, the average user either has to go to another URL (Bing, Google) or know how to enable it (Google's real-time search is called "Updates"). Many users probably don't know either option exists. Working within the simple interface that users have come to embrace (the search box), while introducing different modalities is a difficult problem.
Show me the Deep Web: It can take a while for a page to organically generate links to finish within the first page of search results. Not only do publishers have to contend with SEO's, but real-time linking from social media services are becoming more important. I know the search engines take pride in their ranking algorithms, but I continue to discover great content way past the first few pages of search results. We need instant re-ranking tools as much as instant display tools! Luckily a startup is looking into this problem: Surf Canyon is an Oakland,CA startup that is useful for discovering content buried deep down SERP's. Surf Canyon is a great first step, but I hope real-time re-ranking gets more attention from the major search engines.

Hedge Fund Performance - August/2010

2010-09-16T07:57:00.000-07:00

With the S&P 500 down close to 6% through the first 8 months of 2010, there were a few hedge fund strategies that produced decent returns during the same period. Most notably, Fixed Income and Convertible Arb funds joined Global Macro traders as funds that have delivered more than 5% through the end of August. Benefiting from a widespread rally in commodities, Managed Futures returned close to 5% in August, momentarily erasing middling performance numbers through the first half of 2010.

(Click to enlarge)

Over the most recent 24 month Convertible Arb was the only strategy with double-digit returns, as Convertible Arb managers continued to add to their stellar returns in 2009. Global Macro (8%) and Managed Futures (6.7%) also provided decent diversification from slumping equities (S&P 500 was down 18% during the same period):

(Click to enlarge)

Looking ahead, momentum investors who are likely to pile into Fixed Income and Convertible Arb funds should note that they are investing in relatively illiquid strategies:

(Click to enlarge)

To measure absolute returns, I group months by quintiles of the S&P 500: Q1 is comprised of months in the top 20% of monthly returns, Q5 is comprised of months in the bottom 20%. Computing average monthly returns across the quintiles is one way to gauge whether strategies make money irrespective of conditions in the equity markets. Some of the indices produced positive mean returns across all five quintiles, but not after investors factor in management and performance. (Data is from from Jan/1994 to Aug/2010.)

(Click to enlarge)

[A similar chart for MEDIAN returns across the 5 quintiles can be found HERE.]

As usual, I close by plotting the various Hedge Fund indices using the standard Risk vs. Reward framework. Desirable funds are those that have low risk and high return -- you want funds that are high on the y-axis, but low on risk (i.e. the upper left corner). Note that the graph below is based on fairly recent data (Jan/1994 to Aug/2010).

(Click to enlarge)

[More details, including sector/style descriptions and methodology, are available here. Hedge Fund performance from previous months can be found here.]

Power to the Peaceful 2010

2010-09-11T18:28:00.000-07:00

As I wrote last year, we've been going to the Power to the Peaceful festival since Sept/2001, back when the event was at Precita Park. Sadly this year may have been our last. The crowds have changed (e.g., way more alcohol) and the event has gotten too big for us. We bailed around 2:30 and left long before Michael Franti and Spearhead took the stage.

We stumbled upon a crew from SF Parkour and marveled at the graceful and fluid movements they were doing. I wasn't aware that organized parkour¹ crews existed in the Bay Area.

(1) According to Wikipedia, parkour is "... the physical discipline of training to overcome any obstacle within one's path by adapting one's movements to the environment. It is a non-competitive, physical discipline of French origin in which participants run along a route, attempting to negotiate obstacles in the most efficient way possible. Skills such as jumping and climbing, or the more specific parkour moves are employed. The goal of a practitioner of parkour, called a traceur if male, or traceuse if female, is to get from one place to another using only the human body and the objects in the environment. The obstacles can be anything in one's environment, but parkour is often seen practiced in urban areas because of the many suitable public structures available such as buildings and rails."

Too Smart

2010-09-06T13:46:00.000-07:00

Is Apple making a mistake by relying too much on iTunes? Playing around with the Ping social network, it's clear that while iTunes may be great for certain tasks, navigation is horrible compared to web browsers. Even for managing phones and phone apps, Google's approach is inherently more portable. Switching between Android phones is a breeze as data and apps get synced instantly with Google's cloud.

2 Smart
But perhaps I'm overreacting and being too smart about which apps are best suited for certain tasks. Jobs and the team at Apple have made iTunes the app for buying/managing audio. Ping clearly fits nicely into that and it wouldn't be surprising to see Ping users start recommending video, books, AND apps to each other. So their first stab at social networking involves convincing users to view iTunes as a social network browser. Ping looks empty now, but I wouldn't count Apple out just yet.

Amazon's cloud platform still the largest, but others are closing the gap

2010-08-31T15:39:00.000-07:00

Tim's recent tweet on the growing demand for Google App Engine skills inspired me to measure the popularity of the major cloud computing platforms. Elance is one of many job boards in our data warehouse of U.S. job postings, and I wanted to measure demand across many more job sites.

Measured in terms of (U.S.) job postings, Amazon's Cloud Computing platform is still larger than Google's App Engine. What's interesting is that the gap has closed over the past year.

For details, see my recent Radar post.

Don't Panic, It's Organic

2010-08-23T20:50:00.000-07:00

We bought white nectarines last Saturday at our neighborhood farmers market. This morning we pulled one from the pile and discovered it grew dreadlocks:

"Don't Panic, It's Organic!"
(SF Activist, Diamond Dave Whitaker)
We sliced around the hairy bits, and it was actually pretty good!

The number of Hadoop jobs continue to rise

2010-08-08T14:44:00.000-07:00

While still a small fraction of data management job postings, the number of job posts that mention "hadoop" continue to grow steadily. Year-over-year, there were 300% more such job posts in the first seven months of 2010 compared to the same period in 2009:

For details, see my O'Reilly Radar post.

Which Social Gaming companies are Hiring

2010-07-29T16:40:00.000-07:00

Disney's announced purchase of Mountain View gaming startup Playdom, follows on the heels of EA's purchase of London-based Playfish last November. Based on active users Zynga remains by far the biggest online social gaming company, but what other independent companies are growing?

(Click to enlarge)
For details see my Radar post.

Absolute Return

2010-07-27T08:30:00.000-07:00

Models and Theories, Event-driven Hedge Fund set to capture Inflows, High-frequency trading and futures markets, Opening up CalPERS.(Recent news from the world of Alternative Investments)

Emanuel Derman remembers Fischer Black: Derman worked with Fischer Black at Goldman Sachs and wrote about their collaboration in his book. In a recent talk, Derman expounded on the difference between models and theories. A distinction that Black understood well.

... Fischer understood how models work and their limitations, that models are not gospel, not the world itself, but idols that we make to try to mimic it.

... Models are fundamentally metaphors. They compare something you don’t understand very well to something you understand better, in order to add insight. Calling a computer an electronic brain, for example, is a metaphor that once cast light on the function of computers. Nevertheless, a computer is not an electronic brain. Conversely, calling the brain a computer is a model too – the brain is a long way from a computer. In tackling the mysterious world with models we do our best to explain the thus-far incomprehensible by describing it in terms of the things we already partially comprehend. Models, like metaphors, take the properties of something rich and project them onto something strange.

... If models are metaphors, shedding light by analogy, then theories are the real thing. They don’t compare, they describe and explain. ... Theories tell you what something is. Models tell you only what something is more or less like. Unless you constantly remember that, therein lies their danger.

Event-driven funds set to capture inflows: I highlighted the popularity of event-driven funds in a post at the end of March. Event-driven funds haven't delivered strong returns through the first half of 2010, and given that many hedge fund investors favor momentum strategies, I wonder if the headlines will truly result in significant allocations.

CFTC Weighs Crack-Down on High-Frequency Trades: A lot of the discussion and academic research pertaining to high-frequency trading (HFT) tend to be about stocks, particularly the most heavily traded stocks in the NYSE and Nasdaq. Fewer articles have been written on the fact that HFT has become significant in other markets. Regulators have started to take notice:

... High-frequency trading (HFT) accounts for about 35 percent of U.S. futures volumes -- a number that some expect will reach 60 percent by the end of the year.

Probes may open books at CalPERS: The largest pension fund in the U.S. is a huge investor in alternative assets. How it goes about picking its investments has always been shrouded in secrecy. Recent scandals may result in (slightly) more transparency.

Where Facebook's half a billion users reside

2010-07-21T14:07:00.000-07:00

Facebook just announced that they now have half a billion users, but where do they reside? 17% are from Asia:

(Click to enlarge)

For details, see my Radar post.

Absolute Return

2010-07-20T08:30:00.000-07:00

Waning popularity of funds of funds, Bridgewater Associates and Radical Truth, Betting big on cocoa beans, Too many ETF's
(Recent news from the world of Alternative Investments)

Survey highlights waning popularity of funds of funds: I've harped on hedge fund-of-funds in multiple posts (see [1] and , [2]). To my surprise lots of investors continued to rely on fund-of-funds even after the crisis of 2007/2008. But things are slowly changing:

Enthusiasm for funds of hedge funds (FoHF) is diminishing as institutional investors re-evaluate their investment strategies in favour of direct allocation to single manager funds.The results of a survey of institutional investors carried out by Preqin show that the most significant motivation behind the move to direct allocation was the extra layer of fees introduced by the FoHF structure. This was the primary reason why 60% of respondents moved towards direct investment in hedge funds.In addition to this 54% moved to direct investments due to a desire for greater control over their investments and 13% cited more in-house resources as reason for the adjustment.The survey suggested that the move was also a reflection of the development of institutional investors. Almost two-thirds (64%) of respondents made their first hedge fund investment through FoHFs but only 36% continue to invest solely through funds of funds. No respondent started with direct investment and moved to investment through funds of funds alone.

Bridgewater Associates's Management Philosophy: Continuing the long tradition of hedge funds employing strange management principles, founder Ray Dalio subscribes to what he refers to as Radical Truth. Even saying it feels cultish! I'm not sure how "public shaming" helps employee morale and retention, both of which are important over the long-term.

… Dalio believes that grasping the reality is essential to achieving excellence. Therefore, he advocates the controversial practice of radical truth and radical openness, which involves directly criticizing people and pointing out their weakness. The draw back, said Dalio, is hurt egos. “Most people like compliments and agreement, and they dislike criticisms and conflict,” he admitted. However, he believes the benefits of radical truth allows Bridgewater and its employees to "evolve….at a much faster pace than they would otherwise." He believes the practice of inviting criticisms and challenges to both individuals and processes, and then debating them openly and logically, facilitates evolution and drives Bridgewater towards excellence.
… However, online publication Dealbreaker quoted an ex-employee of Bridgewater who called the principles "pretty cult-ish, as is the culture of the whole company." This person also described Bridgewater's meetings, often recorded and then distributed company-wide, as the "public shaming and the demolition of people." At one meeting in particular, COO Hope Woodhouse was “shredded in front of the management committee” and “brought to the point of crying.” The recording was then sent out to the whole company. This individual also said that 360-degree feedback meetings have become "everyone’s chance to totally dig on and destroy other individuals and say whatever things all year you’ve hated about people."

Is it time to go long cocoa?: Armajaro Holdings, a London hedge fund seems to think so.

… Anthony Ward, the manager of the Armajaro Holdings fund, last week paid £658m to buy 241,000 tons of cocoa beans, equal to the whole supply for Europe and enough to make 5.3bn standard chocolate bars. In an even more unusual move, the fund has taken physical delivery of the beans, which are sitting in warehouses in the UK, Netherlands and Germany. Actual delivery only normally occurs in a fraction of commodity deals.
Ward’s dealing echoes a similar move eight years ago when he made a fortune buying 204,000 tons of cocoa when supply was limited. The price subsequently soared netting him an estimated £40m in two months. Analysts suggested conditions could be similar now to 2002, with weak harvests in the key growing areas of Ghana and the Ivory Coast and the possibility of floods to exacerbate the situation, especially in Ivory Coast.

There are too many ETF's: Just like any industry, financial services tends to flood the market with what may have originally been sound concepts. The usual pattern is to recognize an opportunity, educate and market to investors, and if successful the opportunity is seized by other firms and becomes commoditized. In finance good ideas that have been commoditized tend to misapplied and pushed towards inappropriate settings. In addition, some companies inevitably re-package things as fund-of-funds:

... Bolivian magnate R. Marcelo Claure had been looking for a way to make a broad-based bet on lithium. A hedge fund in which he invests found him one. The fund, MC Capital Advisors, this year turned to a company that creates exchange-traded funds. The result was Global X Lithium, an ETF that tracks lithium producers and battery makers. It is expected to launch this week. MC Capital provided seed money to New York-based Global X Management Co., to start the ETF, and will receive half the ETF's profits.

Global X Lithium is an unusual case of ETF creation. But it also is part of a trend in which ETF firms are joining forces with other companies to launch funds that track obscure parts of the financial markets. ETFs typically are tradable securities designed to track an index of stocks, bonds or commodities. Among ETFs soon to be released are funds designed to track small-cap companies in Thailand, producers of rare metals such as gallium and selenium, and shares of fishing-related companies.

"It's harder and harder to come up with simple, obvious and good ideas," said John Hyland, chief investment officer of U.S. Commodity Funds, which runs eight energy ETFs, including the U.S. Oil Fund and U.S. Natural Gas Fund. Mr. Hyland's company is about to launch an ETF designed to track commodities whose current contract is trading at a premium to future contracts, generally a sign of tight supplies, along with long-term trends. The idea was pushed by SummerHaven Investment Management, a Connecticut investment firm, Mr. Hyland said.

This month, AdvisorShares, a Maryland ETF firm, launched the Mars Hill Global Relative Value ETF, which uses a hedge-fund-like strategy to invest in stock ETFs. It was initiated by, and named after, Colorado money manager Mars Hill Partners. SummerHaven and Mars Hill don't have a financial investment in the ETFs, but will manage the funds and earn fees, the firms said.

A quick tour through next-gen Payments Systems

2010-07-19T08:30:00.000-07:00

Over the last few years, alternative payment systems and technologies have emerged to capture large amounts of online and mobile phone transactions. Since interesting companies and technologies launched recently, I thought I'd introduce quants and data scientists to a few of the new payment technologies. If nothing else, this is an area where employment opportunities will be plentiful. These companies (if they haven't yet) should be hiring teams of quants and data scientists to mimick and go beyond what conventional financial services companies have done in the areas of fraud detection, risk modeling, direct marketing, and personalization.

Mobile Banks in the developing world:
I've written about this topic in other forums, and even wrote a short O'Reilly Radar report on it last year. These platforms started out by allowing users to transfer minutes and cash between phones (remittance service), and evolved to point-of-sale payment systems (like credit cards) once merchants came on board. More recently alliances between mobile bank providers and traditional banks have led to more sophisticated financial products such as savings accounts and micro-insurance. There are many systems available in many developing countries ranging from the rudimentary to those with world class technology. The most famous and largest examples include GCASH and SmartMoney in the Philippines, and M-PESA in Kenya.

In many countries, the initial goal of financial inclusion (financial services for the unbanked) has led to large financial institutions that make established consumer banks nervous. A recent survey of mobile banks spanning 10 developing countries found that mobile bank (transaction) fees are much cheaper than traditional banks. The combination of (Reach + Convenience + Low costs) have traditional banks rushing to form alliances and joint ventures with mobile banks.

Other mobile payment systems:
Unless you track the space, you may not be aware that there are many startups dedicated to mobile payment systems. It's worth noting a few significant differences between these companies/technologies and mobile banks in developing countries:

1. Mobile banks in developing countries were launched primarily by mobile carriers (remember most started out as means of transferring air time).
2. Carriers in developing countries designed their mobile payment services to work on simple (even rudimentary) hand sets, whereas the examples below tend to require apps and smartphones.
3. Payment systems require financial infrastructure, most carriers in developing countries developed their own infrastructure and distribution networks. Most payment companies in the U.S. rely on systems that support credit or debit card payments. (As I'll detail in later examples, other companies use the carrier billing system while others have introduced their own virtual currencies.)

It's clear from how things have played out in developing countries, that the mobile carriers have the upper hand in the mobile payment systems battle. But they have to actually be interested in developing mobile payment platforms. While U.S. carriers get their act together, several companies have released interesting solutions:

• SF-based Square, a credit card payment system for smartphones, is probably the most familiar since it has been in the news recently. Swiping cards with a small reader attached to a smartphone is catchy, but other startups built on the debit (or carrier billing) platform provide the same benefits without requiring a card "reader". (ROAMpay from ROAMdata is similar to Square, but relies on a much larger card reader.)
• A small Houston startup called cMoney is working with banks to offer a platform that can serve as a point-of-sale payment system and a money-transfer service between mobile phones.
• With Bling Nation, consumers can make point-of-sale purchases with their mobile phones by placing a sticker (equipped with a microchip) on their phones. The sticker can be read by devices that look like normal credit card readers. So unlike Square, (SF Bay area based) Bling Nation requires the cooperation of both consumers and merchants². (OTOH, Bling Nation doesn't require a smartphone app, just a phone number and email address.)
• Debit payment systems: Chicago-based startup mPayy.com has rolled out an interesting mobile debit card and money transfer service, where payments are sent to a person's (or merchant's) mobile phone number. Since mPayy uses the debit card platform, fees are lower than credit card transactions. mPayy is similar to the octopus smart card payment system in Hong Kong.
• A Redwood City startup called Obopay has a service similar to mPayy (point-of-sale payment system and a money-transfer service between mobile phones) but is based on a custom payment network. Users can "fund" their Obopay account with their credit cards or bank accounts.
• Alliance of major U.S. mobile carriers: There are reports that the major U.S. carriers will soon be testing a smartphone payment system. Details are unclear but it appears that the system will use Discover's payments network, an odd choice given the power of leveraging carrier billing. (As we noted above, mobile banks in the developing world are primarily built on new payment networks, and we've all seen how disruptive they've become.) The focus seems to be on the payment technology itself as reports describe the system as "... contactless technology to complete purchases in stores". Not to be outdone, Visa has announced and NFC -based payment system that it plans to test with Bank of America and a few other banks.

Having observed what happened to slow moving financial services companies in the developing world, large companies in the U.S. haven't ceded mobile payments to startups -- see for example PayPal Mobile, Intuit GoPayment, Amazon payments.

Most of the technologies in the previous paragraph rely on apps installed on smartphones. Some companies in East Asia and Western Europe are starting to offer swipe-to-pay systems that use Near Field Communications. I don't expect NFC based payment systems to take off anytime soon.

Web Services and (Virtual) Currencies for Social Networks/Games:
Web services: SF startup Boku is actually a mobile phone payment system, that web site owners can deploy through API's. Content providers can concentrate on creating virtual goods while Boku handles the requisite payment technologies. For consumers, paying with Boku is as simple as entering their mobile phone numbers. Boku and another Silicon Valley startup (Zong) use carrier billing¹, which means transactions end up on consumers mobile phone bills. Both companies have relationships with mobile carriers in many countries. If their technology becomes accepted by retailers, Boku and Zong can be used by carriers to offer money transfer and point-of-sale payment systems similar to those in the developing world. (Related company: Mopay, which has services similar to those offered by Boku and Zong, recently announced an extension that lets users/merchants bill landline accounts.)

Virtual Economies, Virtual Goods: For some web sites and virtual worlds, built-in payment systems are central to the experience. Early examples include Second Life (over $500M in user-to-user transactions in 2009) and IMVU. More recently, social gaming powerhouse Zynga ($300M in estimated 2009 revenues) agreed to start testing a virtual currency system from Facebook ("Facebook credit"). Google is positioning Google Checkout as the default payment system for Google Games. China's largest instant messaging provider Tencent QQ, has long had a virtual currency (the "Q coin") that enables users to buy virtual goods within QQ. At some point, other Chinese web sites started to accept Q coins and they became popular enough to attract the attention of Chinese regulators who worried that the Q Coin could be used for money laundering or even "destabilize" the Chinese economy.

Security focus:
Another set of (startup) companies emphasize the security angle:

• PassWindow: Based in Singapore, PassWindow is an online payment system that uses special cards to dynamically generate passwords. It's currently being tested by banks in Malaysia, Chile, Turkey, Indonesia and Australia.
• Visa CodeSure: A credit card with a screen and 12-button keypad, used to generate one-time codes for online transactions. (Continuing this theme, the emergence of a mobile phone the size of a credit card, might lead to credit cards that are phones.)
• FaceCash: A mobile payment system for smartphones, FaceCash users install an app that serves as a point-of-sale payment system. It's like having a credit card stamped with a photo, so merchants can compare your face with the photo in their system.
• Arcot: A cloud-based authentication provider that has a line of products and services for mobile phones.

Other examples:
Miscellaneous examples, some are targeted at specific types of transactions and businesses:

• Bitcoin: A peer-to-peer network based digital currency.
• Oomph (Office on my phone): Processes mobile deposit/receipts using smartphones with cameras.
• eLayaway: A web service that simplifies layaway (or lay-by) purchases.
• PayLease: A web service that streamlines lease payments and collection.
• WePay: A web service that helps groups "... collect, manage, and spend money." WePay is based on the same architecture that manages prepaid debit cards.

Technology providers:
There are also companies that provide key technologies that can be used to put together (mobile) payment systems. They include companies that have products in security (e.g., Gemalto), messaging (e.g., Syniverse), transaction management (e.g. Mozido, eServGlobal), and even entire billing and payment systems (e.g. Fundamo, INSIDE , mBlox). Fundamo is a South African company that provides technology to carriers interested in deploying mobile payment systems and mobile wallets.

(1) With Android users not embracing Google Checkout, Google is hoping that carrier billing will increase the sales of Android apps.
(2) Interestingly, Paypal is outfitting its employees phones with Bling Nation's stickers and all its on-campus cafeterias with Bling Nation readers.

Hedge Fund Performance - Q2/2010

2010-07-16T08:30:00.000-07:00

First off, the CS/Tremont indices have been renamed to the Dow Jones Credit Suisse Hedge Fund Indices. It may have happened earlier, but it's the first time I noticed! Through the first half of 2010, the S&P 500 declined close to 8%, while some of the styles represented in the DJ/CS hedge fund database managed to eke out low positive returns. Fixed Income specialists led the way returning 5.5% through the first half of 2010. For the second straight month, Shorts outperformed all the other styles. In June, four other styles (Fixed Income Arb, Global Macro, Managed Futures, and Risk Arb) managed to finish in positive territory, but barely did so. (In May/2010, Shorts was the only style that finished positive.) Except for Short Bias funds, we've had two straight months of negative to (essentially) flat returns.

(Click to enlarge)
I recently cited a study that concluded that hedge funds suffered net withdrawals in Q1/2010. Compared to the end of Q1/2010, the hedge fund indices generally outperformed the S&P 500 through the first half of 2010. That might be enough to keep investors happy for the next few months.

Over the most recent two years, Shorts and Equity Market Neutral managers trail all the other styles. Over the past year, Convertible and Fixed Income Arb managers outperformed, while Global Macro traders were in the middle of the pack returning 12.4% from Jun/2009 to Jun/2010:

(Click to enlarge)
Revisiting my liquidity metric, the top performers over the last 12 months (Convertible and Fixed Income Arb, Event Driven - Distressed) happen to be the most illiquid styles. A major cause of concern given current psychology:

(Click to enlarge)
Fund managers routinely claim that they can deliver returns regardless of how traditional asset classes like equities are doing. Two straight negative (or flat) months and some investors are again questioning their claims of being able to produce absolute return. Simply outperforming equities isn't enough to justify their management fees. Anecdotal reports indicate that many managers have cut back on trading, and are sitting on piles of cash. In order to place large bets you have to have strong convictions as to where markets are headed, and given their struggles over the last two months, few hedgies are feeling confident. In addition high vol and low trading volumes have hedgies spooked:

... Prime brokers such as Credit Suisse Group AG and JPMorgan Chase & Co. that service hedge funds report that managers are borrowing less money and are sitting on more cash. Credit Suisse’s hedge-fund clients held 24 percent of their assets in cash in June, compared with 19 percent three months earlier, according to the Zurich-based bank’s prime brokerage unit.
... U.S. stock market trading last month had its steepest June decline in at least 13 years. Daily trading volume for the Standard & Poor’s 500 Index of the largest U.S. companies averaged 1.09 billion shares in June, 20 percent less than in May. The 15 percent decrease last year was the second-biggest slump between May and June in Bloomberg data going back to 1997.

To measure absolute returns, I group months by quintiles of the S&P 500: Q1 is comprised of months in the top 20% of monthly returns, Q5 is comprised of months in the bottom 20%. Computing average monthly returns across the quintiles is one way to gauge whether strategies make money irrespective of conditions in the equity markets. While some of the indices produced positive mean returns across all five quintiles, investors need to factor management and performance fees to get a true picture. (Data is from from Jan/1994 to Jun/2010.)

(Click to enlarge)
[A similar chart for MEDIAN returns across the 5 quintiles can be found HERE.]

As usual, I close by plotting the various Hedge Fund indices using the standard Risk vs. Reward framework. Desirable funds are those that have low risk and high return -- you want funds that are high on the y-axis, but low on risk (i.e. the upper left corner). Note that the graph below is based on fairly recent data (Jan/1994 to Jun/2010).

(Click to enlarge)
For recent news from the world of Alternative Investments, check out my weekly roundup.

[More details, including sector/style descriptions and methodology, are available here. Hedge Fund performance from previous months can be found here.]

Absolute Return

2010-07-13T08:30:00.000-07:00

Sports betting Hedge Fund off to a modest start, Absolute return mutual funds continue to struggle, Applied Finance with R, New Hedgies continue to raise money
(Recent news from the world of Alternative Investments)

Sports betting Hedge Fund has 20 investors: Recently launched Galileo Managed Sports Fund has about 20 investors, mostly from Europe and the Middle East. As I suspected, the depth of the sports betting market would make it difficult to absorb large amounts of money: the managers are targeting a small sized fund (roughly 50 million euros). At a mere 100K euros, the minimum buy in might increase its appeal.

Absolute return mutual funds continue to struggle: In previous posts (see here and here) I cited studies that indicated that hedge fund-like and absolute return mutual funds were failing to live up to the marketing hype. Not much has changed according to a recent WSJ article:

... But be warned: Returns vary widely among the 21 funds with the words "absolute return" in their names. On average, they gained 0.17% during the first six months of 2010, according to Morningstar, compared with a loss of 6.65% in the Standard & Poor's 500-stock index.

... Another concern: fees, which can take a big bite out of absolute-return funds' results. Average fees for long-short funds—which include many absolute-return funds—are 2.06%, compared with 1.11% and 1.01% for large-blend and moderate-allocation funds, respectively, according to Morningstar.

And most absolute-return funds are only a few years old, and are untested over the long term—a classic red flag for investors.

For these reasons, Mr. Tuttle, the financial adviser, says that while he uses absolute-return funds for some conservative clients, "you just have to be careful that absolute return doesn't turn into no return."

Applied Finance with R: Back when I was a hedge fund quant, I was a heavy user of S-Plus. S-PLUS' descendant R remains a popular tool in quantitative finance, and this recent conference in Chicago has some interesting examples.

New Hedgies continue to raise money: Read the fine print and you find that most of the money went to spin-offs or new funds launched by established funds. There's much more money from institutional investors wanting to jump back into alternative investments, some of which might end up seeding genuinely new funds.

Twice as many new hedge funds raised three times as much money from investors in the first half of this year as compared to the first half of last year.

All told, funds launching between January and June netted $10.9 billion, according to AR magazine’s biannual New Funds Survey. In the first half of last year, new hedge funds managed to raise just $3.9 billion, an all-time low.

Most of the money was raised by a pair of spin-offs and three new funds launched by established funds.

Fundraising by VC's: Venture capitalists aren't raising boatloads of money anymore:

... A report out today from Thomson Reutuers and the National Venture Capital Association indicates that dollar commitments to VC funds dropped by 49 percent during the first quarter as just 38 funds raised $1.9 billion. That marks the lowest dollar total since the third quarter of 2003.

How to nurture Data Scientists

2010-07-12T08:30:00.000-07:00

Among technology startups, data scientist is an increasingly common term used to refer to data geeks able to bridge traditionally separate functional areas of data intelligence. A data scientist is someone who's comfortable performing several (if not all) aspects of data intelligence projects:

1. data acquisition: this might entail writing custom parsers and web crawlers, or scripts that target specific web services or API's for non-traditional data sources.

2. data management: ETL, manipulate, query, and maintain data in databases, key-value stores, or Hadoop.

3. information visualization: uncovering patterns through the use of static visualization toolkits and/or interactive platforms based on Flash, Javascript, or Processing.

4. analytics: this can range from simple to complex techniques in multivariate statistics, machine-learning, and NLP.

5. insight: extract, summarize, and present key findings to a broad audience.

There are many tools, skills, and technical details, and one can spend years mastering each of the items listed above. While a data scientist may not possess true expert knowledge in any of the areas, he/she is comfortable skipping back and forth and performing basic tasks in all of them. The result is a data geek nimble enough to quickly investigate a data project and produce answers to (high-level) questions from management.

To nurture data scientists, companies need to focus more on culture and organizational structure. Many data workers have enough skills and training to quickly become productive in multiple areas of data intelligence. The problem is that most don't work in environments that encourage them to become data scientists. They're stuck in silos and limited to one or two areas of data intelligence. Often, they're restricted to use tools "approved" by their managers.

After working in companies both large and small, it's clear to me that the strict separation of tasks is the major obstacle faced by data scientists. The most common manifestation is the separation between data analysis and data management. In many large companies, most analysts/statisticians have to wait for data from a designated data warehousing team, and in a lot of cases they wait for data from multiple owners of different data warehouses.

For the moment, data scientists thrive in smaller startups, internet companies, and other organizations where there is less emphasis on defined roles and tasks. But there really is no reason why large and mature organizations can't join the fun. (There's no reason why your statisticians can't learn how to write simple web scrapers and why your database people can't learn simple statistics and visualization.) Here are a few suggestions on how to make it happen:

Embrace non-traditional data sources: One way to get people to think beyond their traditional roles is to use data sources outside those controlled by existing data warehouse groups. Many companies limit data intelligence to data from ERP systems or data vendors (or a variety of "log" files). The web is awash with data, much of which might be useful for your business analysis if you had a team of data scientists.

Start with a small team: Once you commit to forming a team of data scientists, you can start by identifying current employees who might fit the profile. They have to be open-minded, team-oriented, and have some programming skills in one of the areas described above. Ideally you would have mix of people from computer science, statistics/quantitative, or data oriented backgrounds. Team members need to be willing to share simple tools, hacks, and techniques with one another. Cross-fertilization will happen naturally if team members get excited about learning from each other. Employees who are reluctant to share techniques, tools, and ideas would hinder progress.

Allow the use of new tools and techniques: Many I.T. departments are quite strict on what employees can install and use. Many of the favorite tools used by data scientists are free and/or open source, and might be unfamiliar to the I.T. department. (Many come from very recent work done by academics.) New data sources may also require the use of web crawlers and services that may not be to the liking of those who maintain your existing firewalls and filters. Vendors will start offering tools that cover multiple areas of data intelligence, thus reducing context switching and enabling flow. But for the moment, data scientists use a variety of tools, and in any one of the areas described above, one can avail of simple to advanced tools. Simple tools are great ways to introduce basic skills that can form the basis for more advanced learning.

Start with simple projects, and experiment: Rapid iteration and experimentation is important as you're starting out. Pose simple and concrete hypotheses. Start slowly, perhaps leveraging simple tools, web services, and free data sources. Instead of crawling large web sites or taking on complex text parsing and NLP tasks, take advantage of semi-structured data available through web services and API's, while slowly expanding your set of non-traditional data sources. Rather than jumping into Hadoop or a NoSQL database, it might be wise to go with more familiar SQL databases: e.g., Greenplum has a free, single-node version of their MPP SQL database. Static visualization toolkits like R, and free interactive visualization tools from Google docs (or the Google Viz API) offer a variety of infoviz choices.

Shield your data scientists from (middle) managers: Once managers get a whiff that there's a team playing with new data sources, they might try to put obstacles up ("What about data integrity? They're not using the proper machine-learning/statistical techniques! The experimental design is wrong! How can they combine that with our data?"). Without political support, your team of data scientists is going to encounter (non) friendly fire. New things tend to be perceived as threats, so it's best to reassure managers quickly that the data scientists complement what they do. Insights uncovered by your small team of data scientists can be used to inform more formal data/analytic projects. Data scientists aren't going to eliminate the need for statisticians, but they may point them towards different data sets and questions.

Use your initial team of data scientists as evangelists: If you picked your initial team of data scientists correctly, they should be comfortable presenting their findings to others in your company. Better yet, they would be enthusiastic about it! Use them to influence how the rest of the company views data intelligence and to slowly knock down those silos.

I'm not saying that new training and enterprise tools won't be eventually needed as you form your in-house team of data scientists. But I think that by addressing cultural and organizational structures, many companies can use their own employees along with free tools, to seed a small team of data scientists. I speak from experience having worked for large companies -- the talent is there and the techniques aren't that hard to learn, but the organizational silos are hard to overcome. Their ranks already include a pool of talent ready to shine, if not for the rigid corporate structures that limit what they can do.

Absolute Return

2010-07-06T08:30:00.000-07:00

Hedge funds and syndicated loans, Lessons from US Endowments, B2A: Back to Academia, War for Talent in Financial Services, China's property bubble.
(Recent news from the world of Alternative Investments)

Hedge Fund lending and short sales: Two recent studies by teams of researchers from York University and NYU, and Harvard and UC Irvine sought to determine whether or not hedge funds involved in syndicated loans traded on private information. While the researchers didn't have direct evidence that hedge funds privy to these loan arrangements were shorting stocks of the companies seeking loans, they produced enough statistical evidence to warrant further studies by regulators.

... Over the past decade, hedge funds have made significant inroads into the syndicated loan market. In particular, anecdotal evidence suggests that hedge funds are willing to lend to borrowers that commercial banks may be unwilling to lend to. In 2005, hedge funds and other institutional investors provided almost 50% of the $509 billion loans made in the “highly leveraged” segment of the syndicated loan market.

... Specifically, we find evidence consistent with the equity of hedge fund borrowers being short-sold, prior to both loan origination and loan amendment dates, to a greater degree than that of bank borrowers. These results appear stronger when a borrower’s loan amendment is unfavorable, or the borrower’s credit quality deteriorates. Further, for hedge fund borrowers, we find evidence that short-selling can be profitable around both loan origination and amendment announcement dates. Overall, our evidence is consistent with a link between hedge funds’ generation of private information during the syndicated lending process, and the profitable abnormal short-selling of the equity of hedge fund borrowers. Thus, our results raise important issues regarding the regulation of hedge funds involved in both syndicated lending and short-selling.

Can everyone learn from US endowments?: This Financial Times op-ed fails to point out that much of the success of the large (university) endowments can be traced to their organizational structure and their teams of seasoned investment professionals. If anything, endowments who mimicked Yale and Harvard's asset allocation fared worse than those who stuck with what they were most familiar with. A major commitment to alternative assets isn't something many endowments should just rush into.

Baruch's Masters of Financial Engineering program: As quants tire of Wall Street, many have returned to academia to help train the next generation of financial engineers. I enjoyed this recent interview with theoretical physicist and ex-quant Jim Gatheral, a new faculty member at Baruch's respected MFE program.

Hiring in the Financial Sector: As financial services companies start hiring again, recruiters are signaling that demand for talent (at least in certain areas) is intense. Let's see what happens over the next few months. A recent survey of hedge funds found that most expected 2010 to be a challenging year, and I doubt if traditional asset managers are much more optimistic.

China's property bubble: Updating a post I did a few months ago, at least one of the leading voices warning about China's property market is seeing nascent signs of a collapse. Ken Rogoff is now saying the bubble is starting to burst.

Popular iPhone games stay highly-ranked only for a few weeks

2010-07-01T08:30:00.000-07:00

It's been over a month since I last posted on the O'Reilly Radar, and since I've been digging around the Games category, I decided to do a short post on the Top 100 Free and Paid games. Among other things, I looked at how long it takes a game app to break into the Top 100. Judging by the median age at chart debut, Top 100 game apps tend to crash the charts within a few days of their appearance in iTunes.

For more details, see my O'Reilly Radar post.

Absolute Return

2010-06-29T08:30:00.000-07:00

Private Equity 2008/2009 performance, Commodity consumers and the rise of commodity prices, Institutional investors and alternative assets, Hedge funds net in/out flows
(Recent news from the world of Alternative Investments)

Private Equity performance and idle cash: A few months ago, I cited work by academic researchers who concluded that "... after fees, many private equity investments just about match or even trail the returns of the broad stock market between 1980 and 2001. Additional research ... shows that private equity funds underperformed the Standard & Poor’s 500-stock index by 3 percent annually from 1980 to 2003, after accounting for fees." Cambridge Associates just released highlights from their index of private equity and venture capital funds and at least for the 2008/2009 period, buyout shops outperformed VC's. (Both trailed the S&P 500 in 2009 and outperformed the S&P 500 in 2008.)

... U.S. buyouts returned 14.8% for the year, up from -22.7% for the year ended Dec. 31, 2008. Venture capital's return was 3% for the year, up from -16.4% for the year ended Dec. 31, 2008.

With $500 billion in cash, private equity funds are under pressure to find deals or else they risk foregoing (or even returning) lucrative management fees. But the rush to land deals will likely affect the performance of private equity funds for the next few years.

... Private equity funds generally tie up investors’ money for 10 years. But they typically must invest all the money within the first three to five years of the funds’ life. For giant buyout funds raised in 2006 and 2007, at the height of the bubble, time is short. They must invest their money soon or return it to clients — presumably along with some of the management fees the firms have already collected.

Some buyout firms are asking their clients for more time to search for companies to buy. Many more are rushing to invest their cash as quickly as possible, whatever the price. Many in the industry are getting caught in bidding wars. Firms are assigning surprisingly high valuations to companies they are acquiring, even though the lofty prices will in all likelihood reduce profits for their investors. A big drop in returns would be particularly vexing for pension funds, which are counting on private equity, hedge funds and other so-called alternative investments to help them meet their mounting liabilities.

The rise in commodity prices: A recent OECD report concludes that investors did not cause the sudden rise in the value of index funds which track the price of commodities. While the OECD report did not identify who was responsible for the rise in commodity prices, some economists speculate that commodity consumers deserve a significant chunk of the blame.

... After analysing data on both prices and individual holdings from America’s Commodity Futures Trading Commission, the OECD study found that there was “no convincing evidence that positions held by index traders…impact market returns”; indeed, the OECD reckons that larger positions led to lower market volatility (although data issues mean the finding is more convincing for agricultural products than for energy markets).

... In his book, “The Logic of Life: Uncovering the New Economics of Everything”, Tim Harford postulates a “marriage supermarket” in which men and women who agree to wed can present themselves at the checkout and share $100. Given equal numbers of men and women, say 20 apiece, one would expect couples to split the $100 down the middle. But assume there are 20 women and just 19 men. If the women are determined to get married, this shortage of males will cause a bidding war. None will want to be left in the aisles so they will agree to cut their share of the pot. In theory, the lack of just one male will mean the men get to keep $99.99 of the proceeds, leaving the women with one cent.

Similarly, in boom conditions commodity consumers will be so desperate to get their hands on raw materials that they will drive prices up to absurd levels, at least for a short time. When demand falters, or new supply shows up, the price bubble can evaporate as quickly as it arose, without a speculator in sight.

Institutional investors to increase their investments in alternative assets: A recent survey of more than a hundred of the world's largest institutional investors contains good news for hedge fund and other alternative asset managers.

... found that fund managers expected to increase their investments to 19% from 14% of their total investment portfolio by 2012.

The survey found that types of preferred alternative investments included real estate, private equity and hedge funds while “meaningful gains” are expected in commodities and infrastructure.

North American respondents said they expected the allocations to private equity in their total portfolios to increase to 6.8% from 4.3% by 2012 while in Europe a 3.7% share is expected and in Japan, 2.5%. Little change is expected in Australia.

Respondents across the board said they planned to increase the proportion of their portfolios committed to hedge funds to 5.7% in 2012 up from 4.2% in 2009.

Hedge funds net in/out flows: Through the first 5 months of 2010, global macro funds have performed reasonably well compared to other styles. Given that hedge fund investors tend to employ momentum based selection strategies, combined with continuing concerns about the global economy, it isn't surprising that global macro funds raised money in April 2010.

... The hedge fund industry has recouped most of the assets it lost during the financial crisis, according to a new report.

Net inflows into hedge funds totaled $23.7 billion in April, according to research firm BarclayHedge. That puts the industry at $1.65 trillion, their best figure in 18 months, or since before the crippling outflows at the end of 2008.

Macro hedge funds enjoyed the biggest vote of confidence from both investors and the markets, with inflows of $2.5 billion in April. The strategy, which took in a total of just $4 billion last year, now boasts $94.9 billion in assets.

A separate study (using data from data from Thomson Reuters Lipper TASS) came to a different conclusion: hedge funds suffered net withdrawals in Q1/2010. As with the previous study, we see the momentum strategy at work, with investors withdrawing funds from market neutral, managed futures, and multi-strategy funds.

... There was a relatively large $11 billion net outflow of investor cash from hedge funds in the first quarter, fully erasing the $7.5 billion net inflow experienced in the fourth quarter of 2009, according to data from Thomson Reuters Lipper TASS.

The bulk of net outflows in the first quarter were concentrated in selected hedge fund strategies, namely equity market-neutral, event-driven, managed futures and multi-strategies. The drawdown came even as hedge funds produced a positive 3 percent return in the first quarter, which helped to keep the industry’s total net assets at $1.39 trillion, an increase of 2.6 percent from the previous quarter.

The drawdown seemed to be concentrated in the smaller hedge funds, with larger funds posting relatively larger and positive money flows for the quarter.

Absolute Return

2010-06-22T08:30:00.000-07:00

Hedge Funds & Institutional Investors, the SEC's first Quants, Regulatory Arbitrage, Regulating hedge funds, Growing interest in UCITS
(Recent news from the world of Alternative Investments)

Hedge Funds rely mainly on Institutional Investors: A recent survey by Preqin concludes that on average, 73% of a hedge fund's assets comes from institutional investors. Moreover, the larger the hedge fund the greater the dependence: for funds with over a billion dollars in assets, 88% of those assets come from institutional investors.

(Click to enlarge)
Preqin touts a database of 2,500+ "... institutional investors worldwide that are either actively investing, or considering investment, in hedge funds." I don't have access to the complete study, but I'm assuming they surveyed the investors in their database, and compared what they found to estimates for assets under management of individual hedge funds. (So their numbers might actually be underestimates.) Dig deeper and you find that they categorize the institutional investors in their database into the following groups:

(Click to enlarge)
The headline number ("73% of hedge funds assets come from institutional investors") include Insurance Companies, Banks, Investment Companies, and other investors I wouldn't have lumped under the "institutional investors" category.

The report has some interesting survey results on how specific types of investors select fund managers:

... It takes pension funds and endowments roughly 3 months to select a hedge fund, the survey shows, and they are very choosy, with 56 percent selecting only about 10 hedge fund investments a year.

... Just over half of the investors surveyed by Preqin said they rely on two sources to introduce them to hedge fund managers they might want to put money with. Especially in the United States, consultants are playing a bigger role in introducing hedge fund managers to pension funds.

... At the same time, 42 percent of the investors said they find managers by networking and through introductions at conferences, while 20 percent rely on so-called capital introduction teams at prime brokers and another 20 percent use databases. Investors acknowledge that they are often overwhelmed with proposals from managers -- Preqin put the number of proposals to actual investments at 60 to 1 -- but investors are also ready to give the rejected ones a second look.

Quants at the SEC: A few months ago the SEC was making noise about hiring former hedge fund managers and executives. Some of the first ex-hedgies are starting to make their presence felt at the SEC, and hopefully many more will join currently outgunned financial regulators. As the financial markets recover, the SEC will have a harder time luring and retaining talent from the private sector. If they can create a work environment that's less bureaucratic and stifling, they may at least have a chance at attracting those who've already made a bundle in the private sector.

Regulatory arbitrage: I'm the glad the SEC is making an effort to hire industry veterans, but we need all the agencies to do the same. Otherwise smart investors will structure trades and products that fall under the "weakest" agencies. Whoever ends up regulating derivatives (in the US there is lobbying to change regulators) will need to hire experienced traders and quants to prevent simple regulatory aribitrage.

... If regulation allows equity index swaps to be under the CFTC’s regime and the stocks to be under the SEC regime, there will be the same potential for regulatory arbitrage. I can already envision a thriving new market developing for what might be called Index Spread Total Return Swaps. A fund that wants to hold a long equity position in IBM and P&G, but wants to do it under the CFTC regime, will have a broker give them a total return swap that pays the difference between a position in an index that holds the S&P 500 and another index that holds all the stocks in the S&P 500 except for IBM and P&G. This is a swap on indexes, and so will be under the aegis of the CFTC. Whatever equity positions the fund wants to hold, a swap can be created to fulfill its needs. With the push of a button, voila, the fund is effectively trading stocks – securities – under the CFTC rather than SEC umbrella.

This is a simple example of a broader point: a financial engineer could just as easily construct a position drawn from the equity market that behaves like a commodity, or create a currency swap that looks like a bond. In other words, under the proposed OTC derivatives regime, traders will be permitted to choose their regulators. In my view, these provisions should seek to eliminate regulatory arbitrage, not create it.

Regulating hedge funds: After completing his book on the history of hedge funds, Sebastian Mallaby sees their survival during the recent financial crisis as evidence that they aren't completely evil. In fact, Mallaby argues that proposed financial reforms should encourage hedgies and other "entrepreneurial boutique financiers". Mallaby points out that unlike the large banks that almost brought down the world economy, hedge funds come and go all the time ("small enough to fail").

Some observations: (1) The rescue of LTCM didn't technically involve taxpayers money, but LTCM's potential failure did trigger a financial crisis. (2) Lawmakers feel the need to do something to address the Bernie Madoff fiasco. Investors who trusted fund-of-funds and other middlemen (to conduct due diligence) got burned badly. The resulting scandal is too sensational for lawmakers to resist.

UCITS gaining buzz: Speaking of tightening hedge fund regulations .... A few weeks ago I highlighted the growing interest in UCITS, among US asset managers interested in raising money from European investors. I just checked Google Trends and there is a slight upward trend in search volume, and a steep spike in news articles. A lot of the search traffic is coming from Luxembourg and the NY metro area.

Hedge Fund Performance - May/2010

2010-06-16T08:00:00.000-07:00

After reading multiple articles on how badly hedge funds performed in May, I was eager to look at the most recent data from CS/Tremont. On a month when the S&P 500 lost more than 8%, the only style in positive territory was Dedicated Short Bias. On a year-to-date basis, the returns have been paltry: while some styles remain in positive territory, none have returned more than 5%. Factor in fees and investors are left with even smaller returns. But what a difference a month makes. Just a month ago (through April 2010), all the styles (save for Shorts) were chugging along producing small positive returns.

(Click to enlarge)
Using data from investors, Bloomberg was able to glean returns for a few large hedge funds. These large hedge funds tend to employ a variety of strategies, but when possible I tried to identify the dominant strategy they use. The marquee funds listed have underperformed the CS/Tremont index for their corresponding styles and I'm sure investors are none too happy that they lag their smaller peers.

* Paulson & Co.'s Advantage fund dropped 6.9% through May 21 (-3.3% YTD) => Event Arbitrage
* Andreas Halvorsen’s Viking Global fund fell 3.4% over the same period span (-2.9% YTD) => Long/Short Equity
* Louis Bacon’s Moore Global fund declined 7.7% through May 20 (-4.8% YTD) => Global Macro
* Steve Cohen's SAC Capital Advisors reportedly dropped 2.9% last month through May 21 (up 4% YTD) => Multi-Strategy
* Ken Griffin's Citadel Investment Group lost about 2% in its biggest funds last month through May 21 => Multi-Strategy
* Europe's largest hedge fund Brevan Howard Asset Management remained almost flat, losing 0.1% for the month through May 21 (-0.3% YTD) => Global Macro

Over an extended period (1-2 years), the results are mixed. From May/2009 to May/2010, the S&P 500 gained more than 18%, and most of the hedge fund styles were also in double-digits during that period. But from May/2008 to May/2010, the S&P 500 lost over 20% while a few hedge fund styles eked out small positive returns.

(Click to enlarge)
In terms of liquidity, the odd pattern (of the previous two months) involving managed futures seems to have dissipated. The strongest performing style through the first 5 months of 2010 (Fixed Income Arb) is also one of the most illiquid. With so much uncertainty in the markets, liquidity is something to monitor closely in the months ahead.

(Click to enlarge)
To measure absolute returns, I group months by quintiles of the S&P 500: Q1 is comprised of months in the top 20% of monthly returns, Q5 is comprised of months in the bottom 20%. Fund managers routinely claim that they can deliver returns regardless of how traditional asset classes like equities are doing. Computing average monthly returns across the quintiles is one way to gauge whether strategies make money irrespective of conditions in the equity markets. While some of the indices produced positive mean returns across all five quintiles, investors need to factor management and performance fees to get a true picture. (Data is from from Jan/1994 to May/2010.)

(Click to enlarge)
[A similar chart for MEDIAN returns across the 5 quintiles can be found HERE.]

As usual, I close by plotting the various Hedge Fund indices using the standard Risk vs. Reward framework. Desirable funds are those that have low risk and high return -- you want funds that are high on the y-axis, but low on risk (i.e. the upper left corner). Note that the graph below is based on fairly recent data (Jan/1994 to May/2010).

(Click to enlarge)

For recent news from the world of Alternative Investments, check out my weekly roundup.

[More details, including sector/style descriptions and methodology, are available here. Hedge Fund performance from previous months can be found here.]

Absolute Return

2010-06-15T08:30:00.000-07:00

Renaissance man launches a fund-of-funds, Fund Raising news, Stemming Redemptions during Market Crises, Market conditions before and during crises.
(Recent news from the world of Alternative Investments)

Ex-Renaissance employee enters the fund-of-funds game: A glowing article about Renaissance alum Robert Frey, written by Robert Frey? (I'm not making it up, see the byline!) Anyone associated with Renaissance's flagship Medallion Fund will generate interest, and Frey is launching his multi-strategy fund with $350 million in capital. Many fund-of-funds saw their AUM plunge after 2008, but the industry still manages several hundred billions dollars. Amidst the wreckage, there's definitely an opening for someone with Frey's pedigree -- and he knows it. As readers of this blog know, I'm not a fan of funds-of-hedge funds because most investors use them to bypass the need to build strong and deep, active investment teams. If you aren't comfortable picking individual fund managers, what makes you sure you can correctly pick someone else to do it for you (i.e., fund-of-funds managers)? I'm not saying it's not possible, but it's a bad "shortcut" taken by too many alternative asset investors.

... The fund of hedge fund world is not what it was. Damaged by the Madoff affair, unexpectedly high losses in 2008, and the imposition of redemption restrictions, the industry has seen fund numbers decline and money disappear.

Outflows in 2009 continued into the first quarter of 2010, according to data from Hedge Fund Research. In contrast, the wider hedge fund industry saw outflows turn to inflows in the first quarter.

... There has been a view among fund of fund managers, and in the alternative investment world generally, “that alternative investments have no relationship to the larger economy, and are pure alpha players”. This led to the “ridiculous situation” where managers failed to do proper due diligence and simply chased performance, taking no account of whether strong performance in a particular strategy was due to manager skill or to systematic factors.

Fund of fund managers now say they are combining bottom-up qualitative and top-down quantitative research, but Mr Frey is not convinced much has really changed.

... “For example, CTAs [commodity trading advisers] have been under-utilised, whereas everybody was putting a lot of money into long-short equity. But if you do a good analysis, you realise that very often a CTA performs very well during periods when a long-short equity may not do so well.”

Blend the two together and the result is a fund that produces the holy grail of hedge fund investing: “steady, relatively low variance returns”.

... At Renaissance, models had to meet four principles, says Mr Frey. These were (and maybe still are): simplicity – “don’t make it more complicated than it needs to be”; commonality – “make it as broad as possible”; stability – “models you have to readjust constantly probably aren’t as good as ones that stand the test of time”; and rationality – “it can’t just be statistically valid”. You have to employ reason to identify a statistically significant but spurious pattern.

He is sympathetic to the concerns of people who say they do not like black box trading, claiming not to like it either. “But if I built it, it’s not a black box; I know everything that went in there.”

Fund raising by startup and established hedge funds: Startup hedge funds aren't raising tons of money anymore. After what happened in 2007/2008, investors are a bit more reticent handing money to funds without established track records, even if the founders have prior trading experience. A former star at Polar Capital raised a mere two hundred million dollars of the half billion he hoped to. It might be more effective to raise smaller chunks for specific trading strategies, and actually deliver some returns before raising more: a pair of experienced Canadian bond traders want to start a small fund focused on debt securities.

Giant hedge fund Citadel has seen its assets under management (AUM) drop from a peak of $20B to about $12B. More recently, founder Kenneth Griffin hit the road to drum up interest in a couple of new Citadel hedge funds. Citadel hopes to raise several billion dollars for these new funds, but has raised only $164M as of their last regulatory filing. A few weeks ago, I highlighted a study that confirmed that most hedge fund investors follow a naive momentum strategy of chasing after recent hot performers. In the same post, I noted that event-driven funds are benefiting from a recent wave of momentum investing. Another recent example involves Citadel and another giant fund: while Citadel has to work to lure investors back, the Baupost Group has seen its AUM double to more than $22B.

Investor-level Gates: Hedge fund managers were traumatized during the market crisis in 2008 when investors fled for the exits. As a consequence, some funds are introducing measures to dampen future stampedes. Institutional and other long-term investors seem to favor new investor-level gates. But to momentum investors burned by massive losses in 2008, the new terms mean they can't get out quickly when another market panic happens. Fund managers need investors willing to stick around during rough patches, and their policies to facilitate orderly withdrawals (or even fund closures) were seriously tested in 2008. Investor-level gates only work when investors and managers are long-term partners -- in that regard it may actually dissuade the harder-to-please momentum investors. Since these new policies mean investors will have less liquidity during market crises there are two immediate implications: (1) investor-level gates should motivate prospective investors to review their overall investment strategy before wading into alternative assets, and (2) investors should beef up their due diligence and manager selection processes.

... The new approach limits investors to withdrawing a portion of their money from hedge funds at any one time. A common investor-level gate limits redemptions to 25% of an investor's money each quarter over four quarters.

In contrast, fund-level gates were based on redemptions representing a certain percentage of the overall net asset value of hedge funds. For instance, if redemptions totaled more than 25% of a fund's assets, the manager could put up a gate limiting total withdrawals to 25%. That meant all investors in the fund would only get 25% of their money back.

During the 2008 financial crisis, fund-level gates contributed to redemption mayhem in the hedge fund industry. Many investors put in withdrawal requests just because they were worried other investors would redeem ahead of them and trigger an across-the-board gate.

Market conditions before and during crises: We've had enough panics since the mid-1990's to learn to expect a combination of the things in the first list below. For years some investors have routinely asked potential fund managers what they would do when such events occur. Unfortunately, not enough investors imagine the worst-case scenario of these things happening simultaneously! To further "stress test" your investment strategy, imagine all six conditions happening quickly (velocity) and persisting over long periods of time (duration). It's the added combination of velocity and duration that crushes many well-intentioned risk "controls".

During a crisis:
* Equities drop
* Volatility goes up
* Credit spreads widen
* Correlations rise
* Areas of low liquidity decline more than similar areas with high liquidity
* The yield curve flattens

... Knowing this, it is not hard to take steps to protect against a crisis. Just move away from equities, avoid being short volatility, stay away from credit-laden debt, focus on the liquid markets, and watch those carry trades. Also, don’t trust diversification, because those low correlations you are depending on will not be there when it matters.

... More to the point, it is exactly the opposite of what makes money as the market is building into a crisis.

... Before a crisis (I won’t say “during a bubble”):
* Equities are rising
* Volatility is low
* Credit spreads are narrow
* Correlations are low
* The opportunities are in the hinterland markets of low liquidity
* The yield curve is steep

The Practical Quant

When evaluating Investment Funds, use Dollar-weighted Returns

Sarah Palin and machine-translation

2011 Hedge Fund Performance

Long-term performance of Venture Capital & Hedge Funds

Data Mining with the Maximal Information Coefficient

Hedge Fund Performance (Nov/2011) & Maximum Drawdown

On Elite Schools, and Soft vs. Hard Skills

2011 Web 2.0 Summit: By the Numbers

Today's Occupy SF March

The Best of @BigData Sep/2011

Twitter's Front-end Summit

Proposed U.S. Tax Hikes in a few charts

Hedge Fund Performance - August/2011

Popular Links during Obama Jobs Speech & Republican Debate

The Best of @BigData Aug/2011

Bits from Scifoo 2011

Large-scale Named Entity Recognition in the Cloud

Rebranding Data Scientists as Data Alchemists

Computer Vision & Compressed Sensing: Recovery of signals, images, & video from meager data

Writing code that makes your data sing: Auditory Data Display Tools and Resources

Factoids from Twitter's Engineering Open House

Text Mining and Twitter III- LDA Code on Hadoop

2010 Census: Ethnic & Racial breakdown within the major Metro Areas

Feeling right at home in the SF Bay Area

Tweaking Press Releases

Hedge Fund Performance - April/2011

You know there's a Tech Bubble

Hedge Fund Performance - Q1/2011

Hedge Fund Performance - Feb/2011

Social Networks evolve into two opposing camps, or into a single all-friendly group

Hedge Fund Performance - Jan/2011

A quick overview of the App Economy, using data from the iTunes app store

James Simons reminisces about Math

Supporting old-fashioned Journalism

Hedge Fund Performance - 2010

John Sayles' Amigo

Ranking U.S. Universities using Research Ouput and Impact

Hedge Fund Performance - Nov/2010

Recent data analytics & visualization contests

Hedge Fund Performance - Oct/2010

Bad Girls Book Club

2010 Web 2.0 Summit: By the Numbers

Active Facebook users by region: November, 2010

Hiring trends among the major platform players

Inside Facebook's infrastructure: Hive

Inside Google's Infrastructure: MapReduce Statistics Over Time

Windows Mobile apps are more expensive than iPhone Apps

Crowdsourcing Specific Microtasks

Semi-regular Dimes

Compressed Sensing and Big Data

Apple's Q4-2010 Earnings: Strong revenue from the iPhone and iPad

Hedge Fund Performance - Q3/2010

Partha Niyogi, Computer Science Professor, 1967-2010: Manifold (Machine) Learning Pioneer

Smartphone stats on the day Windows Phone 7 Launches

Failcon 2010 Fails (at least temporarily)

A Taste of Home in San Francisco

Manufacturing in China: Two Recent Perspectives

Text Mining and Twitter II - Python code for Analyzing Streaming Data Sets

The most dangerous job in America

Some thoughts on (natural language) Search

Hedge Fund Performance - August/2010

Power to the Peaceful 2010

Too Smart

Amazon's cloud platform still the largest, but others are closing the gap

Don't Panic, It's Organic

The number of Hadoop jobs continue to rise

Which Social Gaming companies are Hiring

Absolute Return

Where Facebook's half a billion users reside

Absolute Return

A quick tour through next-gen Payments Systems

Hedge Fund Performance - Q2/2010

Absolute Return

How to nurture Data Scientists

Absolute Return

Popular iPhone games stay highly-ranked only for a few weeks

Absolute Return

Absolute Return

Hedge Fund Performance - May/2010