Four Ways of Looking at Twitter

Data visualization is cool. It’s also becoming ever more useful, as the vibrant online community of data visualizers (programmers, designers, artists, and statisticians — sometimes all in one person) grows and the tools to execute their visions improve.

Jeff Clark is part of this community. He, like many data visualization enthusiasts, fell into it after being inspired by pioneer Martin Wattenberg‘s landmark treemap that visualized the stock market.

Clark’s latest work shows much promise. He’s built four engines that visualize that giant pile of data known as Twitter. All four basically search words used in tweets, then look for relationships to other words or to other Tweeters. They function in almost real time.

“Twitter is an obvious data source for lots of text information,” says Clark. “It’s actually proven to be a great playground for testing out data visualization ideas.” Clark readily admits not all the visualizations are the product of his design genius. It’s his programming skills that allow him to build engines that drive the visualizations. “I spend a fair amount of time looking at what’s out there. I’ll take what someone did visually and use a different data source. Twitter Spectrum was based on things people search for on Google. Chris Harrison did interesting work that looks really great and I thought, I can do something like that that’s based on live data. So I brought it to Twitter.”

His tools are definitely early stages, but even now, it’s easy to imagine where they could be taken.

Take TwitterVenn. You enter three search terms and the app returns a venn diagram showing frequency of use of each term and frequency of overlap of the terms in a single tweet. As a bonus, it shows a small word map of the most common terms related to each search term; tweets per day for each term by itself and each combination of terms; and a recent tweet. I entered “apple, google, microsoft.” Here’s what a got:

Right away I see Apple tweets are dominating, not surprisingly. But notice the high frequency of unexpected words like “win” “free” and “capacitive” used with the term “apple.” That suggests marketing (spam?) of apple products via Twitter, i.e. “Win a free iPad…”.

I was shocked at the relative infrequency of “google” tweets. In fact there were on average more tweets that included both “microsoft” and “google” than ones that just mentioned “google.”

So then I went to Twitter Spectrum, a similar tool that compares two search terms and shows which words are most commonly associated with each term and which words are most commonly used in tweets with both terms. Here’s the “google, microsoft” Twitter Spectrum:

I love that the word “ugh” is dead center between Google and Microsoft. But the prominence of social media terms on the blue side versus search terms on the red side is fascinating. It looks like two armies marching at each other ready to fight different wars.

Clark has also created TwitArcs. This one, I feel, is still a work in progress and Clark says “visually I like it but it might be the least useful so far.” In this case, you type in a tweeter’s handle and it returns a stream of that person’s tweets with arcs that link common words between tweets (on the right) and common retweeters (on the left). Rolling your mouse over highlights the last tweet in the arc. Here’s a TwitArc of @timoreilly:

Finally, the Stream Graph. Enter a search term and Clark’s engine returns the frequency of the most common words found with your search term for the last 1,000 tweets. You see a literal flow of conversation. You can also highlight one term to see how its frequency changed over time and you’ll see the most recent tweets that include both your search term and that highlighted term.

Sometimes 1,000 tweets with your term may span weeks. For my search term, “Tiger Woods” which I entered yesterday afternoon right after news that he’d speak publicly broke, 1,000 tweets covered about 20 minutes. Here’s the “Tiger Woods” stream graph with “silence” highlighted:

It isn’t hard to imagine how this may be applicable to business. I can already see eager marketers watching the stream flow by as their commercial debuts during next year’s Super Bowl.

Clark, like many data visualizers, believes we’re on the front end of a revolution in information presentation. “There’s a lot of work done called scientific visualization or business intelligence graphics,” he says. “And it’s pragmatic, trying to solve practical problem. It’s all standard, a bar chart or pie. But those standard ways are not adequate when you’re trying to mine a richer data space. The world is full of complex data and we’re just starting to get the tools to make sense of it. We’re looking for new ways of presenting data.”

For the original file, please click here.

Margin Debt & Stock Market Returns

I ran a little experiment to see how the growth or decline of NYSE margin debt correlates with stock market returns. Before conducting the experiment, I expected that high rates of margin debt growth would mark periods of speculative excess, and therefore result in low future stock market returns.

The average 1yr rate of margin debt growth on the NYSE since 1959 is 11.18%.

For my experiment, I calculated rolling forward 2yr cumulative returns on the S&P 500 for all periods since 1959. Next, I divided the periods since 1959 into ‘above average’ and ‘below average’ margin debt growth groups.

Here are the results:

1. Average 2yr stock market return after all periods: 22.97%

2. Average 2yr return after periods with below average margin debt growth: 23.27%

3. Average 2yr return after periods with above average margin debt growth: 18.72%

Bottom Line: the results illustrate a moderate-to-weak relationship between above average margin debt growth and below average future stock returns.

Incidentally, 2yr returns after margin debt grew by 40% and 60% were 6.97% and 6.7% respectively. This supports the thesis that above average margin debt growth leads to below average stock market returns. However, it also shows that the relationship is not linear since the stock market returns stopped declining by a meaningful degree after margin debt growth surpassed 40%. Further clouding the relationship, there also were periods (e.g. 1983) with very high margin debt growth and double-digit stock market returns.

How fast is margin debt growing today? For the 12mths ending December 2009 NYSE margin debt grew by 23.66%. But given the results of my experiment, I wouldn’t rely on margin debt growth to anticipate future market returns.

This is not advice. None of this information is guaranteed to be accurate and should not be relied on. Investing involves risk and you could lose all your money. Consult a financial advisor before making any investing decisions.

For the original file, please click here

Copper/Gold Ratio

For the link, please click here.

(Internal) China exposure

China exposure for below.  They have bonds or perpetual outstanding:

WHB 0302.hk

BoC HK   2388.HK

BEA     0023.hk

Ka Wah Bank (CITIC Ka Wah?)

(Internal) CB Prospectus

go bloomberg ticker

say

2778 HK Equity

Go CVM

Pick the CB, do DES

Then right hand side RHS, go Prospectus

*************

Also

DBSSP 11       – CT2+125

UOBSP 19       – CT5+225

UOBSP13        245/235       3.30/3.20    103.80/104.12  135/125      Aa2 /*-/A-

OCBC 7.75% 2011  XS0132030759  offer 108.67

UOBSP 5.796% Perp  KYG9289K2003 offer 96.25

Dahsing 6.253%  Perp XS0287630932 offer 94.50

KaWah 9.125% Perp XS0148849390 offer 107.25

Winghan 6% Perp XS0296645012 offer 95.25

堅持用簡單量化模型

HKEJ

6th Feb 2010

堅持用簡單量化模型

接着，我跟她說了一個小故事：著名棒球隊波士頓紅襪（Boston Red Sox）的班主約翰．亨利（John W. Henry），就是堅持使用簡單的量化模型的基金經理之一。亨利很早就參與大豆和玉米的期貨交易；1976 年，他開始摸索使用量化投資方法；1981年，他的投資公司正式開張，並以自己名字的縮寫JWH來命名，是當時全球最大、面向散戶的另類投資公司。

JWH 使用的交易模型，主要是機械式地捕捉各種價格趨勢和逆向趨勢。以亨利自己的話說，就是「發現長期趨勢，不去理會短期的波動；堅持使用量化模型投資，將人工干預降到最低；積極採取風險管理策略，包括止蝕；全球分散投資。」JWH 的策略，在過去二十多年來鮮有變化，目光只盯着一個指標：「趨勢」，然後「在人們對資訊反應的過程中，尋找偏差帶來的機會」。誰說簡單說賺不到錢？

——————————————————

full article:

在交易大廳混了一段日子以後，我被分派到一個銷售小組，職銜稍稍更改，變成「戰地量化分析師」。

這銜頭聽起來嚇人，但工作內容其實甚為膚淺，至少用不着隨機微積分和偏微分方程等學問。說得得體一點，我是專門負責為銀行大客戶提供量化諮詢；說得坦白一點，這種諮詢服務只為討客人歡心，以後多跟我們銀行做生意。

不過，客戶的問題五花八門，接觸層面亦比較寬闊，正好適合我的沒有耐性的性格。

一天早上起來，我發現眼睛有些發紅，便在回公司的路上，隨意走進一家藥店找駐診的醫生看看。我知道也明白藥店裏的駐診醫生服務一般都是有傾向性的，例如大多建議客戶購買售價高昂、獨家銷售批發，或該店自家品牌的藥物。

可是，那個駐診醫生實在太過分，他只看了我一眼，便建議我購買先鋒黴素！我一向討厭醫生亂開抗生素，便指着他的鼻子，破口大罵他「缺乏醫德」，然後氣沖沖的返回公司。

名字created by me

經過見習分析員Anjaylia的位子時，她見我怒目圓睜，問發生什麼事，我便一五一十的告訴他。誰知這小妮子沒半點同情心，反而嫣然一笑謂：「這不是跟我們的工作很相似嗎？」

我一怔，想想也是的，但還是憤憤難平。「我們的服務自然也談不上公允，但這是雙方事先都知道的。」

「那人家給你看眼睛，也沒收你錢呀！」她的牙尖嘴利，有時真把人氣死。

Anjaylia是我新聘用的下屬。當初面試她自我介紹，我傻了眼，還以為自己聽錯。

「Angela？Angelina？Anjolie？」

「是A-N-J-A-Y-L-I-A，Anjaylia。」

「這名字是什麼意思？」

「就是created by me。」

我差點暈倒。當然，我不會因為她年輕貌美或名字古怪而輕易聘用，遂問道：「你為什麼想要這份工作？」「因為我想弄清楚什麼是量化投資。」她一臉認真的說。

我瞄瞄她的履歷表，嗯，祖籍四川，長春藤大學金融系畢業。於是，我搬出教科書的定義敷衍她：「量化投資，廣義來說，是指使用數學工具來計算和評估的投資決策方法。這跟憑藉判斷來投資的方法，是完全相對的。」

Anjaylia眨眨杏眼，「對呀，你不覺得很神奇嗎？量化的意思，是按照事先定好的規則來投資。舉個簡單的例子，如果有一個人每天都於早上10時30分把中石油（857）的股價，跟它前面三個交易日的收市價比較。假設他定好的規則是，如現價高於之前三天其中兩天的收市價，他即買入一萬股，然後在當天收市前15 分鐘沽售；否則，就按兵不動。

「以金融的術語來說，這個人建立了一個『模型』或『系統』來進行量化投資。而『10時30分』、『前三天中的兩天』、『一萬股』、『收市前15分鐘』就是模型的參數。問題是，為什麼這個人要選10時30分？為什麼是前三天中的兩天，而不是三天中的三天？再說，他是怎麼發現這個投資規則的呢？你不覺得很神奇嗎？」

「如果不是他自己想出來的，就是別人想出來的，或是根據數據研究出來的，有什麼神奇之處？模型中的參數，不過是隨便想出來罷了。這個人可以選10時35分、10時37分，甚至下午的交易時段；同樣，他也可以將現價與之前四天、五天、七天、三個月，甚至五年比較。參數的變化無窮無盡，交易模型的參數設定亦然。」我說。

肥尾惹的禍

「對呀，你剛才不是說量化投資是用數學工具來投資，而不涉及判斷嗎？可是，說到底，原來還是需要依靠人腦來設定咯！你不覺得很神奇嗎？」

當她一再重複「你不覺得很神奇嗎」這句話時，我感覺很詭異。我好像正在跟Hello Kitty討論尼采。

「要判定某種投資方法，到底是量化型還是判斷型，主要是看一個條件：如果資訊相同，同一個量化投資的模型會做出一樣的決定；但是判斷投資的結果卻因人而異，因為每個人的性格、經歷和判斷的方法都有所不同。」

Anjaylia把頭一側，轉轉眼珠，緩緩說道：「雖說每個蘋果都是獨特的，但味道其實也大同小異。我們只要吃過幾個，大概就知道蘋果是什麼味道了，不用每個都去嚐一口嘛！那大多數的所謂『判斷』，其實就是『跟風』、隨波逐流；量化投資也沒有什麼神秘可言，不就是有一條公式，然後按部就班地做就行了。所以兩類投資方法的變化，其實沒有想像中多咯！」

雖然蘋果的比喻非常怪異，但我不得不承認，Anjaylia的確頭腦敏捷、認真好學。

「其實，量化投資並不止於一個定義，有時候我們對這個概念的範圍會窄一些，有時候寬一些。狹義的量化投資，特別是指跟Black-Scholes-Merton定價模式有關，亦即所謂連續時間金融分析有關的投資方法。」

我侃侃而談，「所謂連續時間金融分析，背後有一個假設，就是股價或滙率等金融價格連續不斷地變動。變動的百分比（不是價格本身的變化），符合我們常說的鐘形正態分布（normal distribution），而上一個變化和下一個變化之間又沒有任何關係。在這樣條件下，各種金融工具都可以此方法定價，尤其是衍生工具。」

「喔，原來用這種定價模型來投資，也叫量化投資……」

「長期資本管理就是典型的例子，其模型用以估測各種金融產品的實際市價與理論價格的差異。如果兩者不符的話，就可以通過買賣各種產品，建立頭寸，然後坐等市價回到理論價格。」

「可是，長期資本管理一坐，就把46億美元搞沒了！」看不出這小妮子還關心財經新聞，而且記性甚好。我不禁笑道：「哈哈，這只怪肥尾（fat tail）惹的禍！」

簡單也可以美

要得到投行的一紙聘書，須過五關、斬六將。我不知道Anjaylia如何說服其他人聘請她，反正她輕輕鬆鬆就過了我這一關。正當我後悔自己面試新人過於衝動時，Anjaylia把一份文件遞給我說：「負責交易的『禿頭Albert』給你的。他發現有一個量化基金客戶交易非常頻繁，而且每次交易的貨幣組合比較類似，所以當中可能有些規律存在，但他又弄不清楚究竟是怎麼一回事。」

我故意不作聲，等她發表意見。「沒有什麼特別高深嘛！只要把這家客戶過去幾年所有的交易紀錄，從系統裏下載到指標軟件上面分析，不就行了嗎？」Anjaylia說。「那你去做吧，回頭給我一份報告。」我暗笑，沒有幾小時的測試，你休想得到什麼交易規律。

結果不出我所料，直到下班時間已過，Anjaylia才拖着累得半死的身軀，倚在我辦公室的門前說：「其實，只是一種比較簡單的趨勢模型。這客戶過去95%的交易，都符合這個規律。」

「進來，坐吧。我們研究一下那5%不能解釋的交易，看看是什麼原因。」說着說着，我們還談論到，萬一錯讀這家公司的交易方向，我們S銀行可能要賠多少錢，了解風險。

當一切處理好後，我就寫了一個很簡單的小程式，放在每個銷售同事的電腦系統內。以後這客戶打電話來問價，銷售同事立即便可以判斷他到底是要買、還是要賣，然後迅速將價格整體移位。就算每筆交易金額不算大，但是只要交易次數頻繁，每次多賺一點差價，一年下來就有上百萬美元的額外收入。

事後，Anjaylia問我：「我用不到一天的時間就能逆向推斷出來的簡單交易策略，怎麼會有用呢？」「是你聰明伶俐吧！」我故意逗她。

堅持用簡單量化模型

接着，我跟她說了一個小故事：著名棒球隊波士頓紅襪（Boston Red Sox）的班主約翰．亨利（John W. Henry），就是堅持使用簡單的量化模型的基金經理之一。亨利很早就參與大豆和玉米的期貨交易；1976 年，他開始摸索使用量化投資方法；1981年，他的投資公司正式開張，並以自己名字的縮寫JWH來命名，是當時全球最大、面向散戶的另類投資公司。

JWH 使用的交易模型，主要是機械式地捕捉各種價格趨勢和逆向趨勢。以亨利自己的話說，就是「發現長期趨勢，不去理會短期的波動；堅持使用量化模型投資，將人工干預降到最低；積極採取風險管理策略，包括止蝕；全球分散投資。」JWH 的策略，在過去二十多年來鮮有變化，目光只盯着一個指標：「趨勢」，然後「在人們對資訊反應的過程中，尋找偏差帶來的機會」。誰說簡單說賺不到錢？

Anjaylia淡然地走到我身後的書櫃，玉手一掃，竟揚出大堆灰塵。「這裏一大堆財經書，有關於畢非德的、索羅斯的、彼得林治（Peter Lynch）的、羅傑斯（Jim Rogers）的……但似乎用來裝飾居多。放在你案頭的，卻永遠都是西蒙斯的零碎資料。」果然心細如塵。

五之五

節錄自忻海著的《戰勝畢菲特的隱世股神》

February 4, 2010 Seven Days Each Month Beats the Market — By a Lot

http://www.crossingwallstreet.com/archives/2010/02/the_entire_stoc.html?

Since 1932, most of the S&P 500’s capital gain has come during a seven-day period at the turn of each month—specifically, the last four trading days and the first three trading days of each month. This represents about one-third of the total trading days. During the rest of the month, the stock market actually lost money.

Here are the numbers: Since the beginning of 1932, the S&P 500 has gained nearly 14,000% which is about 6.5% annualized. Investing in just the last four days and first three days of each month would have returned over 63,000% (not including trading costs). Annualized, that’s 8.6%. However, if you consider that it’s really only 32% of the time, the true annualized rate is over 28%.

The rest of the month — the other 68% of the time — has resulted in a combined loss of close to 78%.

Let me add some important caveats. First, I’m not offering this as trading advice. I’m merely showing that the market has historically experienced outsized gains at the turn of each month. Remember that trading in and out of the market is costly and these results don’t include taxes or commissions.

Secondly, this only refers to capital gains not dividends. A very large part of the market’s total return is due to dividends, and if you’re only invested one-third of the time, you’re going to lose out.

Having said that, here’s a graph showing what turn-of-the-month investing looks like. The S&P 500 is the red line. The blue line is performance during the seven-day period and the rest of the month is the black line.