Saturday, September 1, 2007 31 Comments

Statistical analysis moons the bull

There's an interesting article on statistical prediction in the FT this morning. The FT firewalls articles after a day, so I suspect this link will break, unless you are an FT subscriber.

I want to quote some chunks of the article, not because it's smart, but because it's one of the most embarrassing essays to ever appear in print. The trouble is that the author, Ian Ayres, is pimping a new book, and the FT article is adapted from that book. Since the book was not written in the last month, it says things about the financial markets that in the light of recent events are, at best, grimly hilarious. Presumably if you were a national-security pundit, you had a book release scheduled for October 2001, and your book observed that the threat of terrorism was overheated propaganda churned out by a cabal of neoconservative wingnuts, your publisher would have the sense to pull the book. Or at least hold it until 2005 or so.

You'd think the FT editor could at least have done a quick snip on this graf:
Since the 1950s, social scientists have been comparing the predictive accuracies of number crunchers and traditional experts - and finding that statistical models consistently outpredict experts. But now that revelation has become a revolution in which companies, investors and policymakers use analysis of huge datasets to discover empirical correlations between seemingly unrelated things. Want to hedge a large purchase of euros? Turns out you should sell a carefully balanced portfolio of 26 other stocks and commodities that might include some shares in Wal-Mart.
Um, sure. Want your hedge fund to go tits-up and explode like a rotting whale, raining putrid financial instruments all over Connecticut? Turns out you should assume financial markets are stochastic systems in which the future mirrors the past. Apparently some things that are "seemingly unrelated" are actually also actually unrelated.

As one guest on Brad Setser's blog (the Mos Eisley of the macro finance geek - but see also Macro Man, Nihon Cassandra, Winter, Plant, Waldman, and Hellasious) writes:
It seems like there are two points of view running through this blog:

(1) the securitization process was fundamentally flawed
(2) the process wasn’t flawed, but buyers weren’t prepared for predictable downgrades.

Twofish: “You have a CDO that's divided into three tranches. Let's call them high risk, medium risk, and low risk. Everyone knows that the high risk CDO will lose money, the question is how much. You do the calculation and it shows that the low risk tranche has a 2% chance of losing money.”

But the latter sentence is a prediction about the future. This prediction is based on assumptions (e.g. x percent of the underlying loans will behave like this population of loans on which we have historical data). The models only address correlations between the different cash flows to the degree that their assumptions about the behavior of the underlying mortgages are correct. The problem is times change, behavior changes. It’s hard for some of us to believe that models that process historical data can hope to give reliable predictions about future events.

If the models are only a little wrong – and the modelers would have taken this into account – you have a good CDO. If the models are a lot wrong, you have a lemon.

It seems that the people who believe (1) think that the models are likely to be wildly off the mark, while the people who believe (2) think the models are basically sound.

The problem I have with (2) is this: If in fact the process by which CDOs were created is fundamentally sound, why aren’t people in the know making a killing buying CDOs up right now?
Indeed. "Bueller? Bueller?"

Fortunately, at least the financial markets are not Professor Ayres' leading example. But instead he opens his article with this precious ort of automatist idiocy:
As the men talked, they decided to run a horse race, to create "a friendly interdisciplinary competition" to compare the accuracy of two different ways to predict the outcome of Supreme Court cases. In one corner stood the predictions of the political scientists and their flow charts, and in the other, the opinions of 83 legal experts - esteemed law professors, practitioners and pundits who would be called upon to predict the justices' votes for cases in their areas of expertise. The assignment was to predict in advance the votes of the individual justices for every case that was argued in the Supreme Court's 2002 term.
[...]
The experts lost. For every argued case during the 2002 term, the model predicted 75 per cent of the court's affirm/reverse results correctly, while the legal experts collectively got only 59.1 per cent right. The computer was particularly effective at predicting the crucial swing votes of Justices O'Connor and Anthony Kennedy. The model predicted O'Connor's vote correctly 70 per cent of the time while the experts' success rate was only 61 per cent.
Now, why is this not surprising?

It's not surprising because the good professor is testing a model against a survey. In other news, Professor Ayres is a world-class marathoner: in a 26.2-mile race, he can outperform my grandmother. Although considering the intellectual honesty that this exercise demonstrates, he'd probably feel the need to kick her walker as he shambled past.

If our esteemed professor were actually to establish a prediction market in which people could make actual money by predicting Supreme Court decisions, he could actually pit the best models against the best experts, as selected not by his secretary, but by an adaptive system which rewards success and punishes failure. Heck, if he has a good model, he could actually make some scratch in the game. Maybe he could even detach himself from the Official Tit.

Now, why hasn't anyone done this? (At least, I get no useful hits when I google "Supreme Court" and "prediction market.") Perhaps because, as I suspect, the market would have error rates well under 10%? Perhaps because, if you have a career as a crusader for "social justice," general-purpose media whore, and permanent Fedco flack, constructing an experiment designed to demonstrate the essentially comical nature of the rotary system's most revered agency - the Inspection Council - isn't the idea that leaps instantly to mind? I'm just guessing, here.

If you need a pop-science book about modeling and prediction, try David Orrell's The Future of Everything. Orrell, who actually is a mathematician, does not have a whole lot to say except that it's a really bad idea to trust a model. As Orrell's near-namesake once put it, "trust a snake before a Jew and a Jew before a Greek, but don't trust an Armenian." Hopefully, by the end of Orrell's book, you will trust a whole clan of snake-handling Greco-Jewish Armenians before you assume that "a carefully-balanced portfolio of 26 other stocks and commodities" will mirror the fluctuations of the euro.

If you're more interested in the financial end of this, I haven't finished Rick Bookstaber's book, but so far I like it. A little Austrian economics would do Bookstaber a large passel of good, but his Blowing Up The Lab essay in Time is as good an introduction to the CDO disaster as I've seen so far.

Gary Larson's first Far Side anthology has as its cover what I think is actually the best Far Side cartoon ever. Note, however, that the idiot photographers are only sticking their tongues out at the bull. They have not actually turned around and mooned it - as Professor Ayres just has. Hopefully he's already consigned his royalties to Goldman's Global Alpha fund, which, after "three 25-sigma events in three days," could certainly use the cash.

31 Comments:

Blogger Conrad H. Roth said...

Of course, most of this post is total gobbledegook to me, and I have nothing to say about economics macro, micro or nitro. But I will observe that 'ort' is one of my favourite Boggle words.

September 1, 2007 at 1:48 PM  
Blogger Mencius Moldbug said...

Nitroeconomics! That's it. I declare myself the first nitroeconomist.

The term is a clean Google miss. By the authority of the King of Spain (yes, since you ask; actually, I am a Carlist), I claim it. Lo, mens and womens, sons and bitches: nitroeconomics is what I say it is, and none other. He who shall to want it also, he it is must ask from me.

September 1, 2007 at 2:11 PM  
Anonymous George Weinberg said...

Predicting Supreme Court decisions may not be too hard, since the Justices don't mind being predictable. Predictable = consistent, for a judge that's a good thing.

But beating the market is something different. The market doesn't like being beaten.

Note, however, that the idiot photographers are only sticking their tongues out at the bull. They have not actually turned around and mooned it - as Professor Ayres just has.

Remember this one?

September 1, 2007 at 2:18 PM  
Blogger Mencius Moldbug said...

"Let's moon those Saxon dogs!" Perhaps Conrad can translate that into Latin for us. I feel my subtitle could use some refreshment.

For a judge, predictability is a good thing. For an Inspector-Councillor, I'm not so sure. It might highlight the difference a little too much.

An unpredictable judicial system is very good at making it look like there are deep, swirling pools of problematic complexity everywhere. If it's obvious to anyone that there is no complexity at all in the issues that the Inspectors decide, it is clear which side of the issues they systematically favor. The result is that the basic structure of violence in the society is far more obvious than it should be in a properly-operated liberal democracy.

September 1, 2007 at 2:29 PM  
Blogger Mencius Moldbug said...

Also, as for markets, no market likes getting beaten. At least not if there is real money involved.

September 1, 2007 at 2:31 PM  
Blogger Conrad H. Roth said...

On nitroeconomics, from Charles Webster, The Great Instauration: Science, Medicine and Reform 1626-1660:

"During the Civil War saltpetre was in very great demand for gunpowder manufacture and metallurgical works; chemists believed that if produced economically it might also become an important fertiliser. Such extension of its use was precluded by expensive and inefficient methods of production. For most purposes the trade depended on the unpopular breed of saltpetre men who excavated cellars, outhouses and chicken runs in search of the black nitrous earth, which was mixed with lime and ashes, and then leached for its salts. Finally the impure solution was boiled and partially purified, a process which consumed large quantities of wood or coal. This unsatisfactory method was partly circumvented after 1639 by various East India Company merchants who imported saltpetre from India."

As for your heading,

"istos canes Saxones luneamus"?

September 1, 2007 at 2:40 PM  
Blogger Conrad H. Roth said...

Or perhaps "istas caniculas Saxones luneamus" would be better, ie. 'little dogs, bitches'.

September 1, 2007 at 2:44 PM  
Anonymous TGGP said...

The problem is that it is a survey? Weak, Mencius. Point out some experts that can consistently outpredict the model. You have to actually present some comparisons in which the model comes out worse than actual people rather than an ideal oracle rather than just going "Algorithms, boo!"

September 1, 2007 at 4:31 PM  
Anonymous TGGP said...

Speaking of terrorism, Caplan still says the threat is overrated even after 9/11.

September 1, 2007 at 4:36 PM  
Blogger Michael said...

Mencius: If you are at all serious here you had better be able to answer TGGPs challenge. Stop expecting people to just accept your posit of perfect market efficiency.

September 1, 2007 at 9:54 PM  
Blogger drank said...

MM, you're certainly right about Ayres book. Nicholas Taleb has the financial book of the year, since he called shenanigans on the modeling of risk in financial markets long before they imploded this summer. And Ayres book is destined for obscurity.

On the other hand, I'm not really clear on your objections to models in general, either in this post or the one on the Baysians. Why is it surprising that in some problem domains an algorithm can reliably outperform expert human judgement? You certainly wouldn't object to this statement about an algorithm that plays chess. Why claim it's necessarily false for one that forecasts Supreme Court decisions?

Where are the experts who can predict supreme court decisions with greater accuracy than the model Ayers describes? His results are certainly consistent with Tetlock's studies of the political predictions of experts - you definitely should read his Expert Political Judgment. Basically, his assembled collection of political & foreign policy experts did little better than random chance in a dataset that he has been collecting for more than two decades. And, yes, they were outperformed by statistical models by a significant margin.

Perhaps a prediction market would perform better still. But, as you note, one doesn't seem to exist, and hence we have no idea what its success rate might be. So if one needed to forecast a supreme court decision today, Ayers' model would seem to be a useful tool, no?

September 2, 2007 at 10:02 AM  
Blogger Mencius Moldbug said...

tggp,

Perhaps you could think of a case in which a survey of authorized experts has been an effective device for predicting anything?

Let's take pro football games as a rough equivalent of Supreme Court decisions, without any of the statist cachet. Let's take sportswriters as a rough equivalent of Ayres' law professors.

Would you be terribly surprised if a computer model could beat a survey of sportswriters at predicting NFL results? Do you think the FT and its readers would have considered this a surprising fact? It's about the most boring and obvious fact I can think of.

But - there happens to exist a prediction market for the NFL. It's called Vegas. If you can build a computer model that can beat the latest line at Vegas, you can make some serious bank.

Markets are not magic. They work because they select for effective players and against ineffective ones. They are adaptive systems. They do not discriminate between human and machine players.

Perhaps someone is making a lot of money on pro football, with a computer model that can beat the best pro gamblers. But I kind of doubt it.

This is why the Ayres approach annoys me. He's saying that machines beat humans, but it's sort of like me saying I can beat Mike Tyson - at thumbwrestling. It's leaving the bar on the ground and simply stepping over it.

There is a well-defined way of finding out who, or what, can predict results best. It's called a prediction market. If Ayres or anyone can present a program that beats humans in a prediction market for Supreme Court decisions, pro football results, or anything of the like, I'd be genuinely impressed.

September 3, 2007 at 6:14 PM  
Blogger Mencius Moldbug said...

drank,

What I object to is the hyped tone of wizardry that these results are so often presented with.

A chess-playing algorithm is good at playing chess. A Bayesian inference engine is good at inferring probabilities from sequences of stochastic events.

None of these things are magic. Reason tells us that they should work and they do, indeed, work.

Reason also convinces me that if hardware technology continues to advance, at some point computers will be fast enough to simulate the human brain, and I'll have no objection to saying that they are thinking. And perhaps there are more efficient ways for computers to think than to simulate biological neural systems, so perhaps we will see thinking computers even sooner.

But a chess algorithm is not, in any human sense, thinking. Bayesian inference is not, in any human sense, thinking. Journalists are not, in general, philosophers, and they are easily persuaded to present this kind of work with a kind of Frankenstein spin. Giant mechanical brains! I don't think this kind of hype does anyone a service, and I note with distaste its tendency to propagate backward to the actual researchers, who should certainly know better.

September 3, 2007 at 6:21 PM  
Blogger Michael said...

The above comment didn't come out right. You can delete it, OK. I meant to say

http://www.cs.cmu.edu/~epxing/Class
/10701/project-reports/gimpel.pdf

but blogger doesn't seem to do page breaks for URLs automatically.

September 4, 2007 at 5:31 AM  
Anonymous Tobbic said...

I have some problems with your reservationist ideology.

"Reason also convinces me that if hardware technology continues to advance, at some point computers will be fast enough to simulate the human brain, and I'll have no objection to saying that they are thinking. And perhaps there are more efficient ways for computers to think than to simulate biological neural systems, so perhaps we will see thinking computers even sooner."

If human brains can be simulated by a computer, then that computer has human reason. Computer/AI is based on well-defined set of rules (such as Bayesin belief networks). If this set of rules can be used to replicate human reason, then reason can be "transcended or reduced". Which means the "automatists" were right.

So your reservationist ideology is falsified if a human brain is succesfully simulated/an powerful human-like AI is created.

You say you are convinced that at some point this happens ("Reason also convinces me that if hardware technology continues to advance, at some point computers will be fast enough to simulate the human brain"). Thus, you believe there exists a set of rules which can trancend "human reason". Which makes you an automatist.

September 4, 2007 at 7:29 AM  
Anonymous TGGP said...

Let's take sportswriters as a rough equivalent of Ayres' law professors.
I doubt most people would find that a plausible analogy.

Would you be terribly surprised if a computer model could beat a survey of sportswriters at predicting NFL results?
I don't know what models exist for the NFL and Tetlock didn't write a book about it either, so I don't know.

But - there happens to exist a prediction market for the NFL. It's called Vegas.
And there isn't one for the Supreme Court. Hence that model is the best thing going.

If you can build a computer model that can beat the latest line at Vegas, you can make some serious bank.
At least until the other players start using the same model!

Markets are not magic. They work because they select for effective players and against ineffective ones. They are adaptive systems. They do not discriminate between human and machine players.
So Ayres is wrong because markets work and he is also wrong because the financial markets just tanked. What?

Perhaps someone is making a lot of money on pro football, with a computer model that can beat the best pro gamblers. But I kind of doubt it.
A survey of pro gamblers perhaps?

He's saying that machines beat humans, but it's sort of like me saying I can beat Mike Tyson - at thumbwrestling.
I would actually be surprised by that. He's got meaty hands that could crush your frail little keyboard tappers.

What I object to is the hyped tone of wizardry that these results are so often presented with.
Ah, the tone. Perhaps you can join Hopefully Anonymous in objecting to Robin Hanson's libertarian vibes.

Journalists are not, in general, philosophers
I don't think it would be much better if journalists were more like philosophers, but perhaps it would be better if philosophers were more like journalists!

September 4, 2007 at 8:27 AM  
Blogger drank said...

What I object to is the hyped tone of wizardry that these results are so often presented with.

Well, I can agree with you there. And I also agree with you that neither a chess-playing algorithm nor a bayesian algorithm for predicting Supreme Court decisions are examples of "human thinking". Plainly they are not.

On the other hand, when you say...
A Bayesian inference engine is good at inferring probabilities from sequences of stochastic events.
I think you're underselling it. Justice Kennedy's decisions are not random (well, actually ...), but we can't easily know all of the causes of his decisions or how much weight those causes will receive in a particular case. In short, we lack the kind of perfect knowledge that would let us build a deterministic "opinion engine" that perfectly simulates Justice Kennedy.

But if we need to predict his decisions, we'll need some kind of tool that allows us to do so as accurately as possibly. Candidate A is the algorithm Ayers describes. Candidate B is a consensus of expert judgment. You are proposing candidate C, a prediction market where a group of motivated human traders invest according to their best judgments.

The choice is an easy one. A has an accuracy of about 75%. Tetlock's research tells us that B has an accuracy of around 50%. There is no example of C, so we have no idea how well it would do. If I have to make a forecast today, A wins hands down, right?

If such a prediction market existed, and was lucrative enough for anyone to care, we'd almost certainly see all the traders equip themselves with some variant of the model in A and try to derive competitive advantage from it. This is basically what has happened on Wall Street, right? And I think that not insignificant effort is put into developing models of NFL games (check out footballoutsiders.com sometime) that would enable one to outperform the Vegas sports books.

So when you say...
He's saying that machines beat humans, but it's sort of like me saying I can beat Mike Tyson - at thumbwrestling.
I think you're really grasping at straws here. Chess, Supreme Court decisions, medical diagnoses, weather forecasts, political election outcomes and financial markets are all areas where statistical models have proven capable of outperforming human judgment. Obviously these models aren't Frankenstein's Monster or some sort of Ghost in the CPU, nor are they infallible. They are tools that clever humans employ to obtain the best outcomes in these important areas.

Who cares if some of the quants making the models get an overblown Financial Times article once in a while?

September 4, 2007 at 12:56 PM  
Blogger Mencius Moldbug said...

drank,

Thank you for this very lucid presentation of the issue:

But if we need to predict his decisions, we'll need some kind of tool that allows us to do so as accurately as possibly. Candidate A is the algorithm Ayers describes. Candidate B is a consensus of expert judgment. You are proposing candidate C, a prediction market where a group of motivated human traders invest according to their best judgments.

The choice is an easy one. A has an accuracy of about 75%. Tetlock's research tells us that B has an accuracy of around 50%. There is no example of C, so we have no idea how well it would do. If I have to make a forecast today, A wins hands down, right?


The problem is that B is a strawman.

B and C are both methods of constructing a consensus of expert judgment.

In method B, we select our experts by asking some grad student to come up with a list of respected authorities in the field, who we email a form letter to. The set of people who reply to that letter are our sample, and we weight each of their votes equally. None of these people would even conceive of describing their professional expertise as the prediction of Supreme Court votes, and in fact all or almost all of them have emotional attachments to some result or other.

In method C, we select our human experts via an ongoing competition which adaptively retains winners and discards losers, weighting their votes according to who has more skin in the game.

Clearly, to anyone who was actually trying to solve the problem of predicting Supreme Court decisions - certainly an interesting and relevant problem - the option of B would not even occur, because C is clearly superior to B. Therefore, the choice is between C and A.

Which is why a study telling us that A is better than B is both (a) an extremely unsurprising result, and hence an uninteresting one; and (b) an active distraction from reality. This is why I find it pernicious, and this is why I jump up and down and call it names.

September 5, 2007 at 9:28 AM  
Blogger Mencius Moldbug said...

So Ayres is wrong because markets work and he is also wrong because the financial markets just tanked. What?

You're assuming that the financial markets are markets - ie, Fama's efficient market hypothesis. I happen to not believe in this hypothesis, although this is not the place to explain why.

Basically, the Burnhamian "real meaning" of the quants' activities is that they are all chasing each others' momentum. They are correlated with themselves. This is classic herd bubble behavior, and for the past three hundred years it's been well-known to be incredibly foolish. What math has done is given them a plausible excuse for bidding up the price of tulips. D'oh.

September 5, 2007 at 9:33 AM  
Blogger Mencius Moldbug said...

tobbic,

Your critique is a good one and exposes an ambiguity in my presentation.

When I say that reason can't be reduced to a list of rules, the implied meaning - which I should have stated explicitly - is that when I say "list of rules," I mean "list of rules that a human can understand as a list of rules."

So, for example, you can emulate a human brain by taking the positions of all the atoms in a human brain, uploading that into a physics simulator, and running the simulator.

That a human can understand the physics simulator, at least in terms of understanding the laws of physics and relating them to the algorithms used in the simulator, does not mean that he or she has reduced or transcended reason, because he or she cannot present a reduced or transcended definition of whether or not the simulated brain is thinking reasonably.

September 5, 2007 at 9:39 AM  
Blogger Mencius Moldbug said...

michael,

One, I don't trust "held-out" data. We have only the author's word that he didn't retune his algorithms after peeking at the 2000 and 2001 seasons. That information is not reproducible or verifiable. If you actually have a model that can beat the spread, the way to go about it is this: publish the model, or at least put it in escrow, and use it to bet on the current season.

The use of data this old is especially worrisome, because I suspect the NFL market has improved somewhat since 2001, due to things like Internet betting.

Two, after further review, I suspect that the NFL betting market has some structural inefficiencies due to bettors with emotional attachments. For example, look at how the trivial "always home" and "always away" strategies beat the spread for both of the two test years. This does not argue for a very efficient market.

Three, note that as compared to either Supreme Court decisions or the stock market, pro football is likely to be incredibly stochastic - as opposed to being dictated by processes that the human brain is designed to model effectively (eg, the thinking of Justice X). The NFL's ball is actually designed to bounce in a stochastic manner, for example! Although this is not to say that the statistical problem of predicting soccer results is likely to be any different...

September 5, 2007 at 9:58 AM  
Anonymous TGGP said...

You're assuming that the financial markets are markets - ie, Fama's efficient market hypothesis.
I didn't say what kind of markets they were, just that they were markets. Are you seriously claiming that the financial markets are NOT markets? Are they surveys? Are they computer simulations? Are they parliamentary procedure? Are they professional sports? Are they quarter pounders with cheese? Is it like a peanut or coconut that just happens to have nut/market in its name? You can't just go out and say "you are assuming financial markets are markets" while leaving it at that.

September 5, 2007 at 12:46 PM  
Blogger Mencius Moldbug said...

tggp,

Sorry, "are markets" was a facile way of putting it.

What I mean is that an efficient market will select for efficient predictors. If the market is not efficient - for example, if it demonstrates momentum effects - the result is quite different.

But, again, I will discuss this at some length when I get to the economics segment.

September 7, 2007 at 10:28 AM  
Anonymous Anonymous said...

tibia money tibia gold tibia item runescape accounts buy runescape accounts runescape money runescape gold runescape gp runescape power leveling runescape powerleveling cheap rs2 powerleveling runescape equipment buy rs equipment runescape runes cheap rs2 runes runescape logs cheap rs2 logs runescape items buy runescape items runescape quest point rs2 quest point cheap runescape questpoint runescape gold runescape items runescape power leveling runescape money runescape gold buy runescape gold buy runescape money runescape items runescape accounts runescape gp runescape accounts runescape money runescape power leveling runescape powerleveling tibia gold dofus kamas buy dofus kamas wow power leveling wow powerleveling runescape questpoint rs2 questpoint Warcraft PowerLeveling Warcraft Power Leveling World of Warcraft PowerLeveling World of Warcraft Power Leveling Hellgate money Hellgate gold buy runescape logs buy rs2 items cheap runescape items Hellgate London gold Guild Wars Gold buy Guild Wars Gold runescape items rs2 accounts cheap rs2 equipments lotro gold buy lotro gold buy runescape money buy runescape gold buy runescape runes lotro gold buy lotro gold runescape money runescape gold cheap rs2 powerleveling eve isk eve online isk buy runescape power leveling rs2 power leveling tibia gold tibia item runescape accounts Fiesta Silver Fiesta Gold SilkRoad Gold buy SilkRoad Gold Scions of Fate Gold Hellgate Palladium Hellgate London Palladium SOF Gold Age Of Conan Gold AOC Gold ArchLord gold tibia money tibia gold runescape accounts runescape gold cheap rs2 powerleveling buy ArchLord gold DDO Plat Dungeons and Dragons Online Plat

September 3, 2008 at 7:37 PM  
Anonymous Anonymous said...

情趣用品,情趣用品,情趣用品,情趣用品,情趣,情趣,情趣,情趣,充氣娃娃,免費A片,AV女優,美女視訊,情色交友,免費AV,色情網站,辣妹視訊,美女交友,色情影片,成人影片,成人網站,A片,H漫,18成人,成人圖片,成人漫畫,情色網,成人交友,嘟嘟成人網,成人電影,成人,成人貼圖,成人小說,成人文章,成人圖片區,免費成人影片,成人遊戲,微風成人,愛情公寓,情色,情色貼圖,情色文學,情色交友,色情聊天室,色情小說,一葉情貼圖片區,情色小說,色情,寄情築園小遊戲,色情遊戲,情色視訊,情色電影,aio交友愛情館,言情小說,愛情小說,色情A片,情色論壇,色情影片,視訊聊天室,免費視訊聊天,免費視訊,視訊美女,視訊交友,視訊聊天,免費視訊聊天室,AIO,a片下載,aV,av片,A漫,av dvd,av成人網,聊天室,成人論壇,本土自拍,自拍,A片,情境坊歡愉用品,情趣用品,情人節禮物,情人節,情惑用品性易購,生日禮物,保險套,A片,情色,情色交友,色情聊天室,一葉情貼圖片區,情色小說,情色視訊,情色電影,辣妹視訊,視訊聊天室,免費視訊聊天,免費視訊,,視訊聊天,免費視訊聊天室,情人視訊網,視訊交友90739,成人交友,美女交友

November 6, 2008 at 5:48 PM  
Blogger 信次 said...

情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,美國aneros,rudeboy,英國rudeboy,英國Rocksoff,德國Fun Factory,Fun Factory,英國甜筒造型按摩座,甜筒造型按摩座,英國Rock Chic ,瑞典 Lelo ,英國Emotional Bliss,英國 E.B,荷蘭 Natural Contours,荷蘭 N C,美國 OhMiBod,美國 OMB,Naughti Nano ,音樂按摩棒,ipod按摩棒,美國 The Screaming O,美國TSO,美國TOPCO,美國Doc Johnson,美國CA Exotic,美國CEN,美國Nasstoy,美國Tonguejoy,英國Je Joue,美國Pipe Dream,美國California Exotic,美國NassToys,美國Vibropod,美國Penthouse,仿真按摩棒,矽膠按摩棒,猛男倒模,真人倒模,仿真倒模,PJUR,Zestra,適趣液,穿戴套具,日本NPG,雙頭龍,FANCARNAL,日本NIPPORI,日本GEL,日本Aqua Style,美國WET,費洛蒙,費洛蒙香水,仿真名器,av女優,打炮,做愛,性愛,口交,吹喇叭,肛交,魔女訓練大師,無線跳蛋,有線跳蛋,震動棒,震動保險套,震動套,TOY-情趣用品,情趣用品網,情趣購物網,成人用品網,情趣用品討論,成人購物網,鎖精套,鎖精環,持久環,持久套,拉珠,逼真按摩棒,名器,超名器,逼真老二,電動自慰,自慰,打手槍,仿真女郎,SM道具,SM,性感內褲,仿真按摩棒,pornograph,hunter系列,h動畫,成人動畫,成人卡通,情色動畫,情色卡通,色情動畫,色情卡通,無修正,禁斷,人妻,極悪調教,姦淫,近親相姦,顏射,盜攝,偷拍,本土自拍,素人自拍,公園露出,街道露出,野外露出,誘姦,迷姦,輪姦,凌辱,痴漢,痴女,素人娘,中出,巨乳,調教,潮吹,av,a片,成人影片,成人影音,線上影片,成人光碟,成人無碼,成人dvd,情色影音,情色影片,情色dvd,情色光碟,航空版,薄碼,色情dvd,色情影音,色情光碟,線上A片,免費A片,A片下載,成人電影,色情電影,TOKYO HOT,SKY ANGEL,一本道,SOD,S1,ALICE JAPAN,皇冠系列,老虎系列,東京熱,亞熱,武士系列,新潮館,情趣用品,約定金生,約定金生,情趣,情趣商品,約定金生,情趣網站,跳蛋, 約定金生,按摩棒,充氣娃娃,約定金生,自慰套,G點,性感內衣,約定金生,情趣內衣,約定金生,角色扮演,生日禮物,生日精品,約定金生,自慰,打手槍,約定金生,潮吹,高潮,後庭,約定金生,情色論譠,影片下載,約定金生,遊戲下載,手機鈴聲,約定金生,音樂下載, 約定金生,約定金生,開獎號碼,統一發票號碼,夜市,統一發票對獎,保險套, 約定金生,約定金生,做愛,約定金生,減肥,美容,瘦身,約定金生,當舖,軟體下載,汽車,機車, 約定金生,手機,來電答鈴, 約定金生,週年慶,美食,約定金生,徵信社,網頁設計,網站設計, 約定金生,室內設計, 約定金生,靈異照片,約定金生,同志,約定金生,聊天室,運動彩券,大樂透,約定金生,威力彩,搬家公司,除蟲,偷拍,自拍, 約定金生,無名破解,av女優, 約定金生,小說,約定金生,民宿,大樂透開獎號碼,大樂透中獎號碼,威力彩開獎號碼,約定金生,討論區,痴漢,懷孕, 約定金生,約定金生,美女交友,約定金生,交友,日本av,日本,機票, 約定金生,香水,股市, 約定金生,股市行情, 股市分析,租房子,成人影片,約定金生,免費影片,醫學美容, 約定金生,免費算命,算命,約定金生,姓名配對,姓名學,約定金生,姓名學免費,遊戲, 約定金生,好玩遊戲,好玩遊戲區,約定金生,線上遊戲,新遊戲,漫畫,約定金生,線上漫畫,動畫,成人圖片, 約定金生,桌布,桌布下載,電視節目表, 約定金生,線上電視,約定金生,線上a片,約定金生,線上掃毒,線上翻譯,購物車,約定金生,身分證製造機,身分證產生器,手機,二手車,中古車, 約定金生,約定金生,法拍屋,約定金生,歌詞,音樂,音樂網,火車,房屋,情趣用品,約定金生,情趣,情趣商品,情趣網站,跳蛋,約定金生,按摩棒,充氣娃娃,自慰套, 約定金生, G點,性感內衣,約定金生,情趣內衣,約定金生,角色扮演,生日禮物,精品,禮品,約定金生,自慰,打手槍,潮吹,高潮,約定金生,後庭,情色論譠,約定金生,影片下載,約定金生,遊戲下載,手機鈴聲,音樂下載,開獎號碼,統一發票,夜市,保險套,做愛,約定金生,減肥,美容,瘦身,當舖,約定金生,軟體下載,約定金生,汽車,機車,手機,來電答鈴,約定金生,週年慶,美食,徵信社,網頁設計,網站設計,室內設計,靈異照片, 約定金生,同志,聊天室,約定金生,運動彩券,,大樂透,約定金生,威力彩,搬家公司,除蟲,偷拍,自拍, 約定金生,無名破解, av女優,小說,民宿,約定金生,大樂透開獎號碼,大樂透中獎號碼,威力彩開獎號碼,討論區,痴漢, 約定金生,懷孕,約定金生,美女交友,約定金生,交友,日本av ,日本,機票, 約定金生,香水,股市, 約定金生,股市行情,股市分析,租房子,約定金生,成人影片,免費影片,醫學美容,免費算命,算命, 約定金生,姓名配對,姓名學, 約定金生,姓名學免費,遊戲,約定金生,好玩遊戲,約定金生,好玩遊戲區,線上遊戲,新遊戲,漫畫,線上漫畫,動畫,成人圖片,桌布,約定金生,桌布下載,電視節目表,線上電視, 約定金生,線上a片,線上a片,線上翻譯, 約定金生,購物車,身分證製造機,約定金生,身分證產生器,手機,二手車,中古車,法拍屋,歌詞,音樂,音樂網, 約定金生,借錢,房屋,街頭籃球,找工作,旅行社,約定金生,六合彩,整型,水噹噹,貸款,貸款,信用貸款,宜蘭民宿,花蓮民宿,未婚聯誼,網路購物,珠海,下川島,常平,珠海,澳門機票,香港機票,婚友,婚友社,未婚聯誼,交友,婚友,婚友社,單身聯誼,未婚聯誼,未婚聯誼,婚友社,婚友,婚友社,單身聯誼,婚友,未婚聯誼,婚友社,未婚聯誼,單身聯誼,單身聯誼,婚友,單身聯誼,未婚聯誼,婚友,交友,交友,婚友社,婚友社,婚友社,大陸新娘,大陸新娘,大陸新娘,越南新娘,越南新娘,外籍新娘,外籍新娘,台中坐月子中心,搬家公司,搬家,搬家,搬家公司,線上客服,網頁設計,線上客服,網頁設計,網頁設計,土地貸款,免費資源,電腦教學,wordpress,人工植牙,關鍵字,關鍵字,seo,seo,網路排名,自然排序,網路排名軟體,

January 31, 2009 at 10:53 PM  
Anonymous Anonymous said...

徵信, 徵信社, 感情挽回, 婚姻挽回, 挽回婚姻, 挽回感情, 徵信, 徵信社, 徵信, 捉姦, 徵信公司, 通姦, 通姦罪, 抓姦, 抓猴, 捉猴, 捉姦, 監聽, 調查跟蹤, 反跟蹤, 外遇問題, 徵信, 捉姦, 女人徵信, 外遇問題, 女子徵信, 外遇, 徵信公司, 徵信網, 徵信, 徵信社, 外遇蒐證, 抓姦, 抓猴, 捉猴, 調查跟蹤, 反跟蹤, 感情挽回, 挽回感情, 婚姻挽回, 挽回婚姻, 感情挽回, 外遇沖開, 徵信, 徵信, 徵信社, 抓姦, 徵信, 徵信社, 外遇蒐證, 外遇, 通姦, 通姦罪, 贍養費, 徵信, 徵信社, 徵信社, 抓姦, 徵信社, 徵信社, 徵信, 徵信, 徵信公司, 徵信社, 徵信, 徵信公司, 徵信社, 徵信社, 徵信社, 徵信社, 徵信社, 徵信公司, 徵信社, 徵信, 徵信, 徵信公司, 女人徵信, 外遇, 外遇, 外遇, 外遇

徵信, 徵信網, 徵信社, 徵信網, 徵信, 徵信社, 外遇, 徵信, 徵信, 徵信社, 抓姦, 徵信, 徵信社, 外遇, 徵信社, 抓姦, 徵信社, 徵信公司, 徵信, 徵信社, 徵信公司, 徵信, 徵信社, 徵信公司, 徵信社, 徵信社, 徵信社, 徵信社, 徵信, 徵信社, 徵信社, 徵信社, 徵信,

February 12, 2009 at 2:52 AM  
Anonymous Anonymous said...

^^ nice blog!! ^@^

徵信, 徵信, 徵信, 徵信社, 徵信社, 徵信社, 感情挽回, 婚姻挽回, 挽回婚姻, 挽回感情, 徵信, 徵信社, 徵信, 徵信, 捉姦, 徵信公司, 通姦, 通姦罪, 抓姦, 抓猴, 捉猴, 捉姦, 監聽, 調查跟蹤, 反跟蹤, 外遇問題, 徵信, 捉姦, 女人徵信, 女子徵信, 外遇問題, 女子徵信, 徵信社, 外遇, 徵信公司, 徵信網, 外遇蒐證, 抓姦, 抓猴, 捉猴, 調查跟蹤, 反跟蹤, 感情挽回, 挽回感情, 婚姻挽回, 挽回婚姻, 外遇沖開, 抓姦, 女子徵信, 外遇蒐證, 外遇, 通姦, 通姦罪, 贍養費, 徵信, 徵信社, 抓姦, 徵信社, 徵信, 徵信, 徵信公司, 徵信社, 徵信, 徵信公司, 徵信社, 徵信公司, 徵信, 徵信公司, 女人徵信, 外遇

徵信, 徵信網, 徵信社, 徵信網, 外遇, 徵信, 徵信社, 抓姦, 徵信, 女人徵信, 徵信社, 女人徵信社, 外遇, 抓姦, 徵信公司, 徵信, 徵信社, 徵信公司, 徵信, 徵信社, 徵信公司, 徵信社, 徵信社, 徵信社, 徵信社, 徵信社, 徵信, 徵信社, 女人徵信社, 徵信社, 徵信, 徵信社, 徵信, 女子徵信社, 女子徵信社, 女子徵信社, 女子徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 征信, 征信, 徵信, 徵信社, 徵信, 徵信社, 征信, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社,

March 2, 2009 at 9:31 PM  
Anonymous Anonymous said...

^^ nice blog!! thanks a lot! ^^

徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社, 徵信, 徵信社

March 2, 2009 at 9:36 PM  
Anonymous Anonymous said...

~「朵語‧,最一件事,就。好,你西中瀟灑獨行。

March 6, 2009 at 4:55 AM  
Anonymous Anonymous said...

看房子,買房子,建商自售,自售,台北新成屋,台北豪宅,新成屋,豪宅,美髮儀器,美髮,儀器,髮型,EMBA,MBA,學位,EMBA,專業認證,認證課程,博士學位,DBA,PHD,在職進修,碩士學位,推廣教育,DBA,進修課程,碩士學位,網路廣告,關鍵字廣告,關鍵字,廣告,課程介紹,學分班,文憑,牛樟芝,段木,牛樟菇,日式料理, 台北居酒屋,燒肉,結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,台北住宿,國內訂房,台北HOTEL,台北婚宴,飯店優惠,台北結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,住宿,訂房,HOTEL,飯店,造型系列,學位,牛樟芝,腦磷脂,磷脂絲胺酸,SEO,婚宴,捷運,學區,美髮,儀器,髮型,牛樟芝,腦磷脂,磷脂絲胺酸,看房子,買房子,建商自售,自售,房子,捷運,學區,台北新成屋,台北豪宅,新成屋,豪宅,學位,碩士學位,進修,在職進修, 課程,教育,學位,證照,mba,文憑,學分班,網路廣告,關鍵字廣告,關鍵字,SEO,关键词,网络广告,关键词广告,SEO,关键词,网络广告,关键词广告,SEO,台北住宿,國內訂房,台北HOTEL,台北婚宴,飯店優惠,住宿,訂房,HOTEL,飯店,婚宴,台北住宿,國內訂房,台北HOTEL,台北婚宴,飯店優惠,住宿,訂房,HOTEL,飯店,婚宴,台北住宿,國內訂房,台北HOTEL,台北婚宴,飯店優惠,住宿,訂房,HOTEL,飯店,婚宴,結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,台北結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,台北結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,台北結婚,婚宴場地,推車飲茶,港式點心,尾牙春酒,居酒屋,燒烤,美髮,儀器,髮型,美髮,儀器,髮型,美髮,儀器,髮型,美髮,儀器,髮型,小套房,小套房,進修,在職進修,留學,證照,MBA,EMBA,留學,MBA,EMBA,留學,進修,在職進修,牛樟芝,段木,牛樟菇,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,住宿,民宿,飯宿,旅遊,美容,美髮,整形,造型,美容,美髮,整形,造型,美容,美髮,整形,造型,美容,美髮,整形,造型,美容,美髮,整形,造型,美容,美髮,整形,造型,美容,美髮,整形,造型,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,設計,室內設計,裝潢,房地產,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,進修,在職進修,MBA,EMBA,住宿,民宿,飯店,旅遊,美容,美髮,整形,造型,設計,室內設計,裝潢,房地產,進修,在職進修,MBA,EMBA,關鍵字排名,網路行銷,关键词排名,网络营销,網路行銷,關鍵字排名,关键词排名,网络营销,羅志祥,周杰倫,五月天,蔡依林,林志玲,羅志祥,周杰倫,五月天,蔡依林,林志玲,PMP,在職專班,研究所在職專班,碩士在職專班,PMP,證照,在職專班,研究所在職專班,碩士在職專班,網頁設計,網站設計,網頁設計,網站設計,网页设计,网站设计,网站设计,网页设计

March 18, 2009 at 10:00 AM  

Post a Comment

Subscribe to Post Comments [Atom]

<< Home