Thursday, March 11, 2010 41 Comments

The future of search

There are many reasons I'm grateful to Josh Hall for inviting me to the Foresight conference, but one of the main ones is that now, I get to advertise myself as a genuine expert on the Future.

I realized this when a small party, including both Professor Hanson and I, went to one of the Stanford strip-malls for a deli lunch. "So what are you guys speaking about?" asked a lady in the line. A fiftyish Palo Alto tennis wife. "What?" I said. "What? How about some ether?" Then I looked down, and noticed that someone had stuck a "Speaker" tag on my "Foresight 2010" badge. There was only one possible answer. "The Future," I said.

And when I think of the Future, I am of course reminded of one of the real speakers at Foresight - the incorrigible Paul Saffo, who is perhaps one of the five glibbest people on the planet. When Paul Saffo opens his mouth, you see hundred-dollar bills - rowed-up like a shark's jaw. Paul Saffo is not a dumb guy by any means, but his benjamin flow is definitely in rather than out. I keep telling my wife I'm working on it.

At the risk of sounding like Paul Saffo, I have a subject that I think will please the general UR community: the future of search. The Future of Search! Are we ready for that? Can we go there? The Future of Search? Or is this just too presumptuous?

While apparently a prominent expert on the Future, I actually know nothing at all about Search. Frankly, the problem has never really interested me. Nothing in software is really interesting except system software design. Search (at least, search as we know it) is a heuristic algorithm - a class of solution for which both education and career have taught me nothing but contempt.

In short, as an ancient system software guy (I started CS grad school in Berkeley the same year Sergey Brin started at Stanford), my attitude toward search is much that of a cavalry officer toward the machine-gun. A Victorian cavalry officer. I can tell you exactly what I thought the first time I heard about full-text search: it made me think of library science. In fact, it still does.

So in reality, I am anything but an expert on either Future or Search - let alone the Future of Search. Rather, I confess some expertise as to the Past; and I know a thing or two about Protocols, Kernels, Languages, etc, etc. On Past Kernels I am really at my best. With that said, let's look at the Future of Search.

There is only one real question about the Future of Search. Will it be Google, or won't it? And if it won't be, what will it be? In other words: who kills Google, and how? Or is the Google Age doomed to last forever?

Since I have been kicking around this town for almost twenty years, I have seen the colossi come and go. The Google of 20 years ago, for instance, was SGI. Where is SGI now? Managed into the ground. (As it so happens, Blogger videos were recently broken for over two weeks. Reading that thread, I see instantly that Google has no concept whatsoever of QA. And perhaps that works for you, Google! Or perhaps it's worked in the past. But...)

The Google Age could end just because Google grows old and sclerotic, despite its vast pool of brains, and starts regularly screwing up like this. In this case, it will be replaced by a younger, crisper Google. This matter does not really interest me - a mere corporate transition. (Anyway, despite little crap like this, Google is still so far as I can tell an extremely effective operation.)

As a pseudo-expert on the Future of Search, I cannot tell you when the Google Age will end, or who will end it. All I can tell you is what will end it. This is probably what you wanted to know, anyway.

The Google Age will end when the application we presently know as "search" is replaced by some other application, which does the same job for the user, but (a) does it much better, and (b) does it in a way that leaves no role for Google or anything so profitable.

Ancient geek that I am (my first tech bubble was the CD-ROM bubble), I have seen this process dozens of times. It's called commoditization. Google makes money because search is extremely difficult to implement, and just about impossible to implement well. It makes money in the only righteous way: solving a hard but necessary problem.

But once this problem becomes easy, such a company has a tough time of it - even if that company itself defined the problem and led the market in solving it. Even if that company itself makes the problem. For instance: if search becomes easy to implement, users start expecting ad-free search. Problem goes away; company goes away.

And Google - a collection of atomic individuals, really in fact among humanity's finest, who have at great profit to themselves clustered together on Earth's surface, in this place, under this name, eating this lunch, to solve the problem of ad-supported search - dissipates into air, like spirits, like smoke, like time itself. Where are the SGIs of yesteryear? There were so many good people at SGI.

Nonetheless, commoditization has not happened to Google. It is not about to happen. Because as anyone at Google (or any of its competitors, none of which is anywhere near killing it) can tell you - search is and will always be very, very, very hard. At least, search as we know it.

Commodity search, if there is any such thing, is clearly the Future of Search. But commodity search cannot be search as we know it. It cannot be the same technical problem that today we know as "search." That is, it cannot be the library-science problem which Google is solving. Rather, it must be a generic utility.

Commodity or utility search must be a solution to some different problem, which fulfills roughly the same user need as Google search. Clearly, utility search can only be system software: a platform, not an algorithm. At least, so my prejudices inform me!

What is search doing, anyway? The search experience I, the user, need: I type a line of text into a box. In response, I get a list of links relevant to that text, listed in order of importance.

Of course, producing this metric - importance - is the hard problem in search. The problem of crawling and indexing the Web, while unnecessarily annoying due to certain design mistakes by Tim B-L, is not a hard problem. Okay, it is a hard problem. But it is not a really hard problem, and it was solved well before Google.

Importance is a product of two factors: relevance and reputation. Relevance is nontrivial, but not hard. Reputation is hard. At least, as the problem is presently defined.

As everyone knows, the very hard problem that Google is solving is computing global reputation (ie, PageRank) from the graph of all HTML links on the Web. Its algorithms are now considerably more refined than the original PageRank, of course. But the problem is what it is.

In this problem as defined by the age of Google, just distinguishing between actual content and spam is a difficult problem. Google is not a good producer of reputation data. It is a competent producer of reputation data - at best. And given the problem that Google is solving, mere competence is almost a miracle.

The Google Age ends when the Internet migrates to some new global reputation algorithm, and users switch to it for their searches. To trigger any such switch, the new algorithm must suck less, maybe by an order of magnitude. There is only one way of beating Google this badly: change the problem.

The obvious such change is some systematic advance to some form of editorial reputation - ie, a reputation system in which reputation is generated not by passive algorithms, but by proactive human assessment. For example, consider one of the great achievements of Russian post-medievalism: Peter the Great's Table of Ranks. If we could hire Peter the Great to crawl the Web and assign a rank to every page, we could get rid of Google.

Feudal search is exactly this: a different way to compute global reputation. Which does not require either Google, or Peter the Great. If feudal is too strong a word for you, you can say hierarchical. Feudal search is search which uses, as its quality metric, hierarchical reputation.

We cannot hire Peter the Great to crawl the Web. We can, however, force everyone to join a community. We can ask that community what it thinks of you; and we can ask Peter the Great what he thinks of that community. This puts far less load on Peter the Great.

But wait - we still don't have Peter the Great. We can't actually force people to join a community. No, but we can create a general-purpose namespace of extremely consistent general quality, which will attract high traffic from the legacy Web and thus be highly searchable, even through Google.

In other words, feudal search posits a content namespace which, because ranked feudally, is a much more desirable neighborhood than the Internet. At least, if you don't want to wallow in the slums, you don't have to. You will not turn the digital corner and find yourself in a digital favela. Eventually, all desirable content will move out of the anarchic slums and into this new, happy gated community. And junkies will be shooting up in the old Google building.

A feudal search engine (Feudle, perhaps) separates the task of reputation assignment into two levels. Feudle assigns reputation not to pages, but to communities - a much smaller task. For pages within a community, it defers strictly to the community's own reputation system, connecting directly to it with an actual, standard API.

Thus we have two reputation values, perhaps on the unit interval, which multiplied produce another unit - global reputation. More generally, every search engine assigns every community a reputation transform, effectively grading its grades.

Thus, as a user, my map of global reputation gives high ratings to high-reputation pages at high-reputation communities, medium ratings to high-reputation pages at low-reputation communities, etc. Doesn't it seem to you that this makes sense?

Feudal search is feudal because, rather than computing a single democratic algorithm on the global, unstructured Web, it follows the natural hierarchical structure of all human institutions. Rather than passively computing rank from the random patterns of interaction in an atomized society, provide the institutions necessary for that society to recompose itself in an organized, aristocratic hierarchy. And stand back - the result will work a lot better.

Moreover, the analogy is historically correct. Just as it is not the king's business to involve himself in a dispute between two serfs of the same baron, it is not Feudle's business to decide which Blogger blogs are the best blogs. It is only the king's business if two of his barons quarrel. Likewise, Feudle must compute the value of a Blogger reputation versus a Wordpress reputation. Like an admissions department deciding that an Andover B is worth as much as a Montclair High A, it might decide that a Blogger A is worth a Wordpress B - or vice versa.

To continue the metaphor, Feudle's job is easy because, rather than computing the quality of every high-school student in America, it only needs to compute the quality of every high school in America. It still needs a quality-rating algorithm, but this algorithm rates communities rather than their members. A much smaller problem. Of course, Feudle cannot exist today, because neither Blogger nor Wordpress, like hippie high-schools, assign their bloggers grades.

If Feudle is such a great idea, why hasn't anyone built it? There is no way to grade the grades, because there are no grades to begin with. The trick about feudal search is that, since it's a platform, it faces a large chicken-egg problem. This is normal in system software. If your new language has no libraries, no one will use it. If no one uses the language, why write a library for it? Thus, since there is no local feudal reputation, there can be no global feudal reputation. Since there is no global feudal reputation, there is not much use for local feudal reputation.

The Web is just not optimized for feudal search. It is optimized for Google search. For one thing, feudal search requires a feudal search-reputation protocol - which doesn't exist. Even if the protocol existed, the information behind it is often absent. Even local reputation on the Internet is anything but a solved problem.

For instance, Blogger is not in any sense a community, and it has nothing like a communal reputation system. Or rather, its reputation system is Google PageRank. (Last time I checked, I got the impression Google hates UR for its long posts - it assumes a blog with 10K-word posts is a spamblog. Bing knows we're the real deal, and brings up UR as the first choice if you type "unq." I will always love Google, however, for the fact that "true history of the American Revolution" produces, as first match, the true history of the American Revolution.)

If Blogger had a reputation system, Feudle could exist - indeed, Google, which after all owns and operates Blogger, might even stop using PageRank on Blogger pages and rely on Blogger reputations. But, of course, Blogger has no reputation system and Feudle does not exist. Chicken-and-egg.

So nothing like Feudle exists or can exist. You cannot go to Paul Graham and apply to start up Feudle as a startup. The world is not even ready to begin to be ready for feudal search. What, therefore, convinces me that it is.. the Future?

While I am no Paul Saffo, I actually do have one test for whether I expect something to be around in the Future. Actually, it is two tests: one for things that exist, and one for things that don't.

For things that exist, I ask: if this didn't exist, would anyone invent it? For things that don't exist, I ask: if this existed, could anything kill it? Thus I conclude that newspapers as we know them will cease, sometime in the Future, to exist. And I conclude that feudal reputation systems are, somewhere out there, the Future of the Internet.

Another way to say this is that I'm convinced that, if these systems existed, they would grow stronger rather than weaker. Since they do not exist, the incentive to create them must be quite weak, which means they must be not that useful. However, they should experience a network effect: as they organize, they grow larger, stronger and more desirable, sucking in both traffic and content. Eventually, there will simply be no content worth searching which remains outside this network.

Let's narrow in on the Internet's feudal future by looking, again, at Blogger. Blogger is actually a microcosm of the Internet, because it is not in any sense a community. Rather, it is a general-purpose service. There is no sense in which blogs A and B are closer to each other because they both use Blogger, not Wordpress.

In the Feudal Age, Blogger as it stands today would do quite poorly as a community. It is not, of course, a community. It would have no way to assign its blogs high-quality local reputations, and therefore would not earn high global reputation. Therefore, any high-quality bloggers on it would promptly flee to other communities in which their talents would be recognized. This would further decrease the communal reputation of Blogger, and so on - basically a power-dive on flames into the Pacific.

What could Blogger do to avoid this fiery end? Up its game, of course. Specifically, before its bloggers flee to greener pastures, it would have to replicate the problem of feudalizing teh Internets, within its vast network of existing blogs.

It can only do this by replicating the feudal solution internally. Socially, Blogger is not and cannot be a single community. Or rather: as a single community, it could only be a totalitarian dictatorship. For instance, if Blogger was a single community, either left-wing political bloggers would dominate right-wing political bloggers, or vice versa. Expecting to integrate wingnuts and moonbats into a single community is like expecting to integrate alligators and peccaries into a single zoo exhibit. And this is just the start.

Rather, assessment gets better as it gets narrower. Basketball blogs can only be graded, presumably by other basketball bloggers, relative to other basketball blogs. They certainly cannot be compared to UR - which is not a very good basketball blog. (Disqus makes this mistake by assigning commenters global reputation across all Disqus comments; each forum should assign its own rank, or no one can possibly take the process seriously.)

To produce a high-quality reputation signal, Blogger must self-categorize. As a single community, it is a joke; for its reputations to matter, they must be accurate; for them to be accurate, they must be local; for them to be local, Blogger must fragment itself into hundreds, if not thousands, of self-rating communities. Each of these communities must perform its own quality control, and be rewarded for achieving quality by a communal grade across the whole service.

For instance - this suggestion, while nowhere near appropriate, is perhaps more appropriate than one might think - Blogger could adopt another content characterization, one into which enormous human effort was invested: the Usenet newsgroup hierarchy. It could say: blogs in Blogger are now organized into guilds, each of which is named after a newsgroup in the main Usenet hierarchy. Pick a newsgroup, and join its guild.

In each of these guilds, however organized, bloggers must be in some way ranked anonymously by their peers. If this devolves into a poisonous and corrupt travesty, as of course any such process can, it is time for a new guild. However, guilds are expected to function as aristocracies; it is in the interest of the entire guild to obtain a good reputation, and most of all in the interest of the guild's leaders.

Therefore, in the Future, if you are blogging for the public audience and want random strangers to find and read your blog (not every blogger does, of course), you will have to find some category or community in which you feel it belongs, and submit your blog for review by your peers in that category. Or, of course, create a new community if none suits you. Standard technology for this purpose must exist.

How is the reputation of an online clothing store determined? How is the reputation of an Internet poet determined? The poets must get together and evaluate each others' poetry. The online clothing stores must get together and evaluate each others' products and customer service. In short, they must form guilds. An elegant medieval social structure, which functioned beautifully for many centuries. Applicable to any form of content for which anyone might be searching - right down to commercial advertising.

In general, to have a reputation in the Feudal Age, you have to be part of something larger. You must either join some community which can assign you some reputation, or organize a new community of your own. As an isolated atom, you are scum by definition. You belong in the stocks, and you'll probably end up there. (We'll know teh Internets are dead when everyone who still uses them is a spammer.)

This effective communal coercion is good, not bad. Because the upside is: you don't have to be an isolated atom. You're a human being, a communal animal - you have to join, but you can join. Since these communities have to exist, they do exist. As a result, you can start producing content and acquire an accurate reputation very quickly and easily. If you're a poet, other poets will read your poetry. If you're a filmmaker, other filmmakers will watch your films. If you're a clothing designer, other clothing designers will try on your clothes. (If this costs money, you will have to pay for it.) And if all these someones are idiots, it is time to start your own guild!

And for the user - the poor schmuck behind the browser, who is, after all, the customer - the search experience is improved by roughly a bazillion. With Feudle, the reputation mechanism that orders his hits is not a heuristic algorithm, but a human process - facilitated, of course, by system software. Rank is not an algorithm, but a grade. To satisfy the customer's needs, all at all levels are working hard to get good grades.

The crucial fact about the Feudal Age is that, in that age, the Internet becomes deatomized. It does not get organized by Google. It is not passively organized. It actively organizes itself - which means ranking itself. The resulting ranks, since they follow the natural structure of human authority, are much more accurate than anything Google's algorithms can produce. As a result, Google dissipates in smoke etc. And Google, perhaps, is the least of it!

As for how it has to start: feudal reputation can only start from the bottom up. That is: communities must migrate from shitty tools, which don't support reputation, to good ones that do. Perhaps something like StackExchange is the beginning of the trend...

41 Comments:

Blogger G. M. Palmer said...

You want to replace Google with reddit?

March 11, 2010 at 4:43 AM  
Blogger newt0311 said...

This effective communal coercion is good, not bad. Because the upside is: you don't have to be an isolated atom. You're a human being, a communal animal - you have to join, but you can join. Since these communities have to exist, they do exist. As a result, you can start producing content and acquire an accurate reputation very quickly and easily.

I am not convinced by this. Guilds started out as useful organizations but they just became pointless monopolies stifling competition. I suspect new entrants will have similar problems. Will slashdot (or now HN) allow a new forum to get any traction if it had the ability to crush it with low ratings? Of not something like HN, what about a for-profit site like Wufoo and OkCupid or the hundreds others who suddenly have a way to crush their competition. The only way to prevent this is to assume that the per-community rating system is perfect which is obviously a bad joke.

There really is no justification for pervasive complete hierarchies (ie. ones which include all members or even anything approaching most) without significant externalities. The Internet by its very design, really doesn't have much in the way of externalities. I can perhaps see communities forming for small parts of it like Blogger and an API to link to these will be nice but the new stuff (some of which will be distinctly better). Google will never be out of fashion.

March 11, 2010 at 4:54 AM  
Blogger newt0311 said...

Update (to fix my bad editing): link to these will be nice but the new stuff (some of which will be distinctly better) will be in the ungated community. Google will never be out of fashion.

March 11, 2010 at 5:13 AM  
Anonymous josh said...

how to create the antiversity?

March 11, 2010 at 6:40 AM  
Anonymous Brent said...

Nice parable. Long live New Feudalism.

March 11, 2010 at 7:30 AM  
Blogger TGGP said...

You don't seem to fully recognize the two-valued function here. Desirability of a web-page is subjective. Peter's Table of Ranks is actually post (or anti) feudal, he wanted to shift from traditional hierarchy to one based on service to HIM. Having poets rate each others poetry is just what led to the circular credentialism problem the field has now. From the start the user can enter text they want to search, and a search algorithm can guess what kind of user it is and what they might want to see based on that text. As a user builds up a history (perhaps like Pandora) it should better be able to guess what they want. People want it personalized, not communalized. I know I'd much rather be an atomized individual than part of any community on the internet.

Perhaps more related to the last thread. I found this piece contrasting Walter Lippmann with Izzy Stone to be interesting, though the Izzy partisans that wrote it show complete contempt for the idea of attempting objectivity.

March 11, 2010 at 7:38 AM  
Anonymous Patrick said...

//How is the reputation of an Internet poet determined? The poets must get together and evaluate each others' poetry.//

Good lord, no, we've already got that. Besides that, poets are a vicious, jealous and petty breed. Any notion of another poet's reputation is intimately (some might say pornographically) tied to and dependent on their own. Get poets out of the way and let readers decide.

March 11, 2010 at 10:10 AM  
Blogger Aaron Davies said...

Donuts^WFeudalism. Is there anything they^Wit can't do?

March 11, 2010 at 10:42 AM  
Blogger Leonard said...

I second TGGP in thinking a "pandora search" would be just the ticket. That is, a search where my identity (including the history of all my previous searches), is taken into account by the search algorithm.

Of course, such a thing would not displace Google; google can run one just as well as anyone else. So in a sense, at least, MM is correct to ignore it. He is writing the Big Think piece about how to get away from google; Google can beat anyone in running any search-related computer algorithm, be it current google or not. And if Google won't do, some other company will; but it will be Google.

Let me also second... well, several people in mentioning that "reputation", at least as it currently works, is highly hackable. And thus, not an order-of-magnitude better than a computer algorithm. For example, even in the domain of political commentary (surely something where machines have a hard time rating things), who is on top? Tom "Flat" Friedman! Oh yeah, baby!

That said, in a sense Wikis in general, and Wikipedia in particular, are to some degree creating something like what MM is talking about. That is, a community which has its own reputation system, creating its own content, including its own search. And it is very successful.

I don't think it will drive google out of business, though. When I want to know about X, I often go direct to wikipedia if I just want a broad overview. But wikipedia doesn't have most content, and never will; Google is in no danger from it. And even if 10000 wikis captured most internet content, you'd still have Peter the Google sitting atop them all, making money from directing traffic to the right one.

March 11, 2010 at 11:22 AM  
Blogger G. M. Palmer said...

Patrick's right about poetry. Poets already control the academy & publication. Pobiz is the most incestuous and nepotistic biz there is. Ergo, I go to conferences and "network."

March 11, 2010 at 1:44 PM  
Blogger Malthus Bats Last said...

Good lord, no, we've already got that. Besides that, poets are a vicious, jealous and petty breed. Any notion of another poet's reputation is intimately (some might say pornographically) tied to and dependent on their own. Get poets out of the way and let readers decide.

The point is that the outside world (non-poets) can then go on to rate the community of poets as a whole, which doesn't look like it would go all that great for the poets. A poet's reputation would be a function of both what his peers think of him, and what non-poets think of the group grading him.

A tweak which would fix your problem is to have two separate ratings of a page- one by other members of the page's guild (poets rating other poets), and one by everyone else. For some topics, I couldn't care less about what experts in that area think (poetry), since I can make my own mind up, thank you very much. But for others I don't have the slightest idea of how to judge a page or blog, if I'm looking for information on some scientific or medical topic, its quite helpful to see what people with experience in the field think.

March 11, 2010 at 2:02 PM  
Anonymous Anonymous said...

The future of software is HyperCard.

March 11, 2010 at 2:53 PM  
Anonymous so long - from a boring jerk! said...

TGGP, to read that essay decrying aristocratic patronage of poetry, one would think that such patronage had been a disaster and only 'the public' had ever paid for the creation of great art, by buying poetry books and such. Obviously aristocratic patronage has in fact been as successful as public patronage.

But as for the author's other suggestion (get poetry out of the university): naturally. But you can't. Un-coerced students sign up for poetry writing classes and universities accordingly hire recursively-credentialed poets to teach them - whereas to an approximation no one else any hires poets to do anything poetry-related. Therefore, the poets aren't going to quit these posts and become accountants just because a bunch of savvy heads on the internet call for it. But, of course, universities and especially classes as impractical as poetry classes are not supported solely by market forces. We could yank the relevant government forces. But even then I suspect we will only make a modest dent in enrollments in poetry writing classes.

Poets supported by rich patrons or by selling books to a rather broad segment of the public are also somewhat recursively-credentialed. Everyone is. But some systems of recursive credentialing are better than others.

Surely it must be the advent of movies etc that has slain poetry. Today guys who would have written great poetry (if there are any today) may be in rock music or movies. Thus the poetry available largely does not measure up to the classics, even on the rare occasion when some culture consumer feels like giving poetry a chance instead of rock or a movie.

Most poetry was never that great, anyway. The number of poems which are highly rewarding to read is extremely small.

March 11, 2010 at 5:19 PM  
Anonymous Anonymous said...

There is already a quasi-feudal system of reputation and ranking; it's called academic publishing, which employs peer review, and citation analysis to get an effect quite like PageRank, but within small, highly selected communities of experts.

If you don't like the current guilds, feel free to start one of your own. Anyone can start and manage and online journal and apply whatever filtering criteria they like. PLoS (the leading open access science journal publisher) has open-sourced its content platform, so for the trivial costs of hosting you could put up the Journal of Reactionary Studies. There is even a large body of academic work on peer review itself.

March 11, 2010 at 5:35 PM  
Anonymous Anonymous said...

There is already a quasi-feudal system of reputation and ranking; it's called academic publishing, which employs peer review, and citation analysis to get an effect quite like PageRank, but within small, highly selected communities of experts.

If you don't like the current guilds, feel free to start one of your own. Anyone can start and manage and online journal and apply whatever sorting criteria they like. PLoS (the leading open access science journal publisher) has open-sourced its content platform, so for the trivial costs of hosting you could put up the Journal of Reactionary Studies. There is even a large body of academic work on peer review itself.

March 11, 2010 at 5:36 PM  
Anonymous Patrick said...

//to read that essay decrying aristocratic patronage of poetry, one would think that such patronage had been a disaster and only 'the public' had ever paid for the creation of great art, by buying poetry books and such.//

One could, but I never made that argument. What I wrote was that modern poets aren't any more liberated or independent that the artists of the 18th century. They've replaced one patronage system with another.

//But as for the author's other suggestion (get poetry out of the university): naturally. But you can't.//

I agree, but that doesn't mean we have to keep listening to them - by that I mean: who we should be reading and why.

//Un-coerced students sign up for poetry writing classes and universities accordingly hire recursively-credentialed poets to teach them//

Yes, it's a tremendous money-maker. And who am I to have a problem with it? But it ought to be pointed out that there is no evidence (to my knowledge) that such an education produces good poets or poetry (as opposed to the self-taught poet, let's say).

//Surely it must be the advent of movies etc that has slain poetry.//

Yeah, I hear that argument all the time and in any number of forms, but I don't buy it.

//Most poetry was never that great, anyway. The number of poems which are highly rewarding to read is extremely small.//

*That* is very true! - and is a point I frequently make. Not only that, but great poets & poetry can be separated by a hundred years or more. Contrary to all the hype, it should be expected that the baby boom generation would be a wasteland of mediocrity. (How old are you, by the way?) : |

March 11, 2010 at 5:43 PM  
Anonymous so long - from a boring jerk! said...

OK, so who is going to run Feudle? Some corp will. And that corp will sort of control our minds, no? What happens when they rate the neocon guild higher than the paleocon guild? Sages like ourselves will hardly be taken in, but the public will be bamboozled instantaneously.

Of course, this happens already I guess. Google's algorithms aren't transparent. They could easily dock reputation from stuff they don't like, such as Pat Buchanan. Sailer recently noticed some paleocon-type names and such that appeared to be edited out of that Google prompting thing. I guess people don't fret much about Google controlling your mind in general, because it seems like it's probably no worse than the content of TV or newspapers being determined by the people who run those - probably not as bad, actually.

It also seems like commercial sites could secretly pay Google, not to have "sponsored links," but rather to have their *regular* links come up higher in searches. I'm sure this possibility has somewhere been noted, but I haven't heard anything about it. I don't see how you would prevent this and I don't know if it's even illegal or not.

March 11, 2010 at 5:45 PM  
Anonymous so long - from a boring jerk! said...

PLoS-anon,
Yeah, but how would people *find* the Journal of Reactionary Studies and other alt-academies or interesting things? If the J-React gets lucky, it will build up a major visibility and people will know about it. But how would it get any attention at all in the first place, assuming it wasn't connected to any sort of traditional impressarios or celebrities? By people finding it on the internet of course, initially on Google (or via blog comments whose author name links to the journal's site). So MM's whole idea is to make the J-React and its like visible (if the thing in question is good) or not particularly visible (if it sucks).

March 11, 2010 at 5:58 PM  
Anonymous Anonymous said...

among blogs, there is already a peer rated system. UR readers are probably a fuzzy subset of MR readers, for instance. who found either through Google?

March 11, 2010 at 6:14 PM  
Anonymous so long - from a boring jerk! said...

Patrick,
I don't see why you think Ms. Lilly's money is replacing one system with another. The aristocrat patronage system never seems to have gone away, and I'm not sure its products over recent centuries haven't been better than the products of popular patronage. Didn't Rilke write the Duino Elegies in Duino Castle where he was enjoying a free vacation c/o some princess? Van Gogh had private patronage of a sort, from his brother - but I think his brother adored Vincent's art, rather than just supporting him for the simple reason that he was a sad sack. On the popular side you have Dickens and Dylan Thomas, and Shakespeare (in part or in total?). Cormac McCarthy got private patronage from the MacArthur foundation when he was unknown to the public, and didn't get rich (off lesser works) until 15 years after his magnum opus came out. Nietzsche, in addition to his disability pension from Basel University, scored dough from friends who recognized his greatness.

You have a high estimation of the public, speaking of its greatness. I have no scornful view of the public but in general I would expect slightly better taste from modern aristocrats like Ms. Lilly. I think only part of the public has greatness and that this part was larger in the old days, when members of the public depended, for their very survival, on currying favor with aristocrats whose power came largely from facing death in the course of military duty - a necessarily great act overcharged with the most elevated, proud, and terrifying feelings. The public imitated the modes of those men-at-arms, to please those whose favor they needed. Today one of the major determinants (but certainly not the sole one) of getting a high position in society is having the ability to flatter the mediocre, and the effect trickles down. I think this is the primary reason why old societies from over a century ago are more attractive than today's.

I'm 30, do you think that is too old to write poetry for more than a few years? I just got ahold of Hoelderlin's hymns which were written at that age, but of course he went insane right after, so it's not clear whether his powers would have descended by the age of 35 had he remained healthy. Rilke was 37 when he wrote the first Duino. I really get no value at all out of the enfant, Rimbaud.

I think Rilke, while still being great, could be adduced as an example of the aristo-decadence you describe in your essay. He can verge on the over-lavish, like Klimt and some other Art Nouveau. Schiele's "Agony" avoided this and is pretty much my favorite painting from that time.

March 11, 2010 at 7:00 PM  
Blogger Chris said...

Google has done well not just because it's a reasonable purveyor of reputation information, but increasingly because it is good at giving search results people want to see. They do ranking of search results based on the searcher's past browsing behavior (very easy to figure out: they show you a list, see what you click on, before long they learn what links you like), where you are geographically (someone should try your American Revolution query in the UK), and other user-specific factors. How do you see these other factors interacting with reputation in a world with a Feudal search / internet?

March 11, 2010 at 7:26 PM  
Anonymous Anonymous said...

Doesn't Google have ties with the NSA? I have a feeling that they would be "protected" through their links with fedgov from any potential upstart topplers.

March 11, 2010 at 10:40 PM  
Anonymous Pals said...

This is a very bad idea.

Whenever a system imposes high costs of exit on its members and consumers, it will be abused by the ones who control it. So, the guilds would fuck their members and consumers proportional to how hard it is to start a new guild to compete with them. A gated online community would allow plenty of abuse.

All you need to look at this is just to see the current peer-review mechanism in academia. It's the biggest source of bullshit and mendacity in the world.

The only good thing about the internet is precisely that eschews all of this garbage and puts everything out there for everyone to assess with their own evidence.

There was a time when I, and most people here, I presume, were full believers in the garbage of modern econ. A guild around that would've continued churning out Keynesian garbage and content forever, and there would never be a way out. The beauty of the internet is that as soon as anyone you know, regardless of their guild credentials, points you to mises.org, you start reading and realize how everything else is garbage.

What we have, with the internet, is that credentials are subjective. Whether through FB, Google, email, reddit, or any of the other sharing websites, people communicate to each other what they like. And based on how you value people's opinion, you follow their recommendations. With no centralized command, good ideas flourish and bad ideas die.

Just look at how much the internet has done to people's understanding of economics, politics and other stuff, and how much it has hurt the traditional scumbag gate-keepers of this knowledge: the mainstream media.

Teh Internets are FTW!!11!1!!

March 12, 2010 at 4:56 AM  
Blogger G. M. Palmer said...

Patrick and Solong,

We're all the same damn age. Patrick is in his early 30s if I remember correctly, Solong is 30 and I'm nearly 32. So the first question is why aren't we collaborating, since we're obviously some of the few poets not beholden to The Cathedral?

The second question is are either of you going to AWP?

The third thing is 30 is hardly "old." Eliot was in his 50s when he wrote Four Quartets, Milton was was in his 50s when he wrote Paradise Lost and in his 60s when he revised it, and Dante was in his 30s when he started LDC but in his 50s when he finished it. Crazy or no, some of Pound's best work on the Cantos came when he was in his 60s and 70s. Rilke and Keats were outliers, not the norm.

March 12, 2010 at 6:32 AM  
Anonymous Patrick said...

//I don't see why you think Ms. Lilly's money is replacing one system with another.//

Because it's not what I think. I never wrote that Lilly's money was replacing one system for another.


//The aristocrat patronage system never seems to have gone away//

*Aristocratic* patronage, as understood in the 19th century, has gone away. No American poet prior to 2oth century (to my knowledge) wrote for a patron (or was supported by a patron). (Or if there were, they aren't remembered.) During the course of the 20th Century, however, American poets and universities created a system of academic patronage that I find unhealthy - primarily because poets *themselves* have a hand in defining and perpetuating it (hence the term circular credentialism). In fact, the patronage of modern poets, arguably, far exceeds anything any historical *poet* could have dreamed of (even at the height of the European Patronage system). It's only comparable to the court composers of the 17th & 18th centuries.

//and I'm not sure its products over recent centuries haven't been better than the products of popular patronage.//

This goes much further afield than I went. But if we're limiting ourselves to *poets* - then there's nothing in the past that's comparable to the modern patronage system.

//in general I would expect slightly better taste from modern aristocrats like Ms. Lilly./

ok.

//I think only part of the public has greatness and that this part was larger in the old days//

I disagree. The modern public has no trouble recognizing greatness in other arts - like popular music and the entertainment industry in general.


//I'm 30, do you think that is too old to write poetry for more than a few years?//

Most poets don't really even mature until their mid thirties and alter. :-) Keats was an exception.


//I think Rilke, while still being great, could be adduced as an example of the aristo-decadence you describe in your essay. He can verge on the over-lavish, like Klimt and some other Art Nouveau. Schiele's "Agony" avoided this and is pretty much my favorite painting from that time.//

Ah, the visual arts. I'm the worst sort of dilettante when it comes to the visual arts...

March 12, 2010 at 7:33 AM  
Anonymous so long - from a boring jerk! said...

Patrick,
You aren't impressed with my grasp of your essay. Are you able to boil it down to a thesis of 20 or 40 words?

You say poets need to "answer to the public" and "make a living" from poetry, presumably by books and readings.

You seem to be, more or less, lumping in university-supported poets with other receivers of private means who have not answered to the public. I wouldn't ratify this.

First of all, who is it you think was great, who lived by selling his work to the public, *while* he was doing great work? Dylan Thomas?

Per Wik, Eliot was a graduate student when he wrote Prufrock. Later he taught French and Latin, then worked in business for a bank and then a publisher. Stevens famously practiced law. They were not much beholden to academe or the public (Eliot had some *supplemental* paid work of an academe bent, namely "extension" lectures and writing book reviews). Celine had his income as a medical doctor and a half-pension for his partial debility from getting blown up in World War I. Blake had a patron-friend and sold commercial illustrations which don't seem to be considered among his fine works.

Let's look at the problems with working in academe for an artist, its pathologies. One has to be many things. Reliable and presentable, so one is not a liability to the desirability of the institution to students. This would rule out Beethoven and Hoelderlin. Very likable and socially adept, since you are competing for just a few spots. Goodbye to Szukalsky. But the worst is that you have to praise so many of your fellow artists, especially those that are the most powerful (likable) rather than necessarily the most accomplished. A real great artist almost invariably has a laser perception for other great art, so he cannot perform these latter duties except by constant flat lying, which might be fun for a few but not for most. The problems are not insurmountable, just fairly high.

So this is how I disagree with you. First, unlike you I don't even know which people (if any) did great art while being supported (through the market) by the proceeds of their work. The main people I would think of there are the rock bands Portishead, Dirty Three, and Crosby Stills Nash & Young. Like you, I'm pessimistic about the system of university posts giving support to many greats or even people who have a shot at becoming greats. We don't disagree at all about the universities. We disagree about whether non-university great poets (and other artists) tended to "answer to the public" and make money on sales. I say most of them, maybe almost all, did not. Most of them held the Real Job: the Day Job. Szukalsky had it, Eliot and Stevens -- Hoelderlin, to the somewhat limited extent that he could hold on to one. Others had private dough from other job-holding people. I don't necessarily see a huge distinction as to whether their private patrons were truly rich and powerful aristocrats or not. Tallis' were, Nietzsche's weren't. In any case, the numerous greats of the Real Job and the Private Patron did not answer to the public or make a living from selling work. My instincts point me to the Real Job.

By the way, I was mistaken about Hoelderlin's chronology. He wrote the hymns at ages 32-6, not in the years around age 30. And he stopped doing great work only at the time when his madness finally deepened, age 36. He had already been deranged on-and-off before that, and was seen medically for deranged outbursts at 32.

March 12, 2010 at 11:53 AM  
Anonymous so long - from a boring jerk! said...

Mencken comes to mind as market supported. He was a heck of a lot of fun. But I don't think he can be counted as a great artist-thinker. In that aspect he was a recapitulator of Nietzsche, relying on Nietzsche to a somewhat embarrassing degree.

March 12, 2010 at 11:58 AM  
Blogger G. M. Palmer said...

So that's a no, then?

March 12, 2010 at 1:20 PM  
Anonymous Patrick said...

//You aren't impressed with my grasp of your essay.//

That's not how I would put it. So far, you've mostly brought your own preoccupations to the fore. That's OK. I just wanted to be clear about what I did and didn't say.

//First of all, who is it you think was great, who lived by selling his work to the public, *while* he was doing great work? Dylan Thomas?//

OK, but once again it sounds like you're ascribing, to me, an argument I didn't make. I didn't say that poets should expect to be successful, only that the value in trying might produce different and better poetry and different and better notions of what constitutes good or great poetry. However, to answer your question: all of the great Elizabethan dramatists lived off the proceeds of their work *while* producing great work. Pope was the first "poet", as opposed to a poet/dramatist, to make a living "as" a poet. Dryden wrote both plays and poetry - and was monetarily successful in both arenas. Byron *could* have lived off his poetry, but didn't need to. Wordsworth, Longfellow, Browning, Tennyson, Frost. I'm sure, if I made the effort, I could find any number of lesser, perhaps forgotten poets, who were successful in their own day. Did you know that Billy Collins was offered a six figure advance on his poetry? And I have been told by someone who knows her (so it is unverified hearsay) that Mary Oliver presently makes a living off her poetry.


//So this is how I disagree with you. First, unlike you I don't even know which people (if any) did great art while being supported (through the market) by the proceeds of their work.//

Well, now you knoow some people. :-)

//We disagree about whether non-university great poets (and other artists) tended to "answer to the public" and make money on sales.//

There's nothing to disagree about, that is, it's not a matter of opinion. The facts are available and speak for themselves. It *is* possible to make money on sales.

//By the way, I was mistaken about Hoelderlin's chronology.//

Not to worry. I only wish that my mistakes were as minor.

March 12, 2010 at 2:57 PM  
Anonymous Euro2cent said...

I say, comrade Sydney George Fisher seems to be a fellow with his head screwed on right, judging from the preface to The Struggle for American Independence he wrote. Link:

http://www.archive.org/stream/struggleforamer01fishgoog#page/n10/mode/1up

I could not find The true history of the American Revolution (Googlebooks gets coy about copyright on a work over a century old, with author dead for over eighty years ...), but the preface above says it's an update.

March 12, 2010 at 3:10 PM  
Anonymous Euro2cent said...

Ah, text of True History here: http://www.questia.com/PM.qst?a=o&d=6497342

March 12, 2010 at 3:27 PM  
Blogger TGGP said...

Leonard:
"He is writing the Big Think piece about how to get away from google; Google can beat anyone in running any search-related computer algorithm, be it current google or not. And if Google won't do, some other company will; but it will be Google."
So if it's Google rather some other company doing something innovative not currently done by Google, we don't care. But if it's some other company beating Google at what Google already does, we don't care about that either. It's only if some other company beats Google by being fundamentally different that we care. Why?

"a community which has its own reputation system"
There are some mods with lots of authority, but I think most substantive contributions come from people without much rep within Wikipedia (but perhaps more within their specialized interests). The majority of editors I would say have effectively no ranking. Even anonymous I.P addresses can make edits.


Malthus Bats Last:
"The point is that the outside world (non-poets) can then go on to rate the community of poets as a whole, which doesn't look like it would go all that great for the poets."
That doesn't seem very helpful for encouraging individual good poets. You could get that by having non-poets rate individual poets, and not bother with the first step of having the poet-guild rank its members. Your two-reputation system seems better, but then why just two? Mencius other ideas for Uberfact/some other go-nowhere One Wiki to Rule The World. There he wanted to replace Wikipedia, but most of us don't have that complaint about it (except the creators of the embarassing Conservapedia).


so long - from a boring jerk!:
I believe Patrick here is actually the author of that essay. I think the difference is that the nobility of the past were more like very rich consumers. The new patrons are non-profit bureaucracies, and we all know what happens to them over time.
It occurs to me that Jeffrey Friedman's "There Is No Substitute for Profit and Loss" is relevant here.

Higher education is quite massively subsidized. If it were not, I predict that the consumption aspects would significantly separate from the investment (even in mere signalling) aspects, so that the latter cease to subsidize the former. Furthermore, students are not simply "uncoerced", there are usually course requirements for graduation.

"Most poetry was never that great, anyway"
Sturgeon's Law.

March 12, 2010 at 8:24 PM  
Blogger TGGP said...

Anonymous March 11, 2010 5:35 PM/5:36:
Get a handle.
I hope somebody starts a competitor to PLoS which doesn't engage in this sort of one-sided skepticism.


so long:
I don't think they docked the actual search results though, and they quickly repaired it. Nor do I recall any other names mentioned. Some commenter theorized it was because of the country that feature was developed in.


Anonymous March 11, 2010 6:14 PM:
In case you're not the same anonymous as above, you get a handle too.
I don't think MR has ever linked to UR, but it has a very large number of readers and also discusses economics. I found both of them through Sailer, directly in the case of MR and indirectly through some others on his blog roll for UR. So Google gets no credit!


Chris:
Yup, that's just the point I was trying to make, and I thought Google does that, since different users seem to get different results for the same search. Mencius seems to want to create a bunch of round, square etc holes and then tell people "You've got to pick one of these pigeon-holes to fit yourself in!". Consumers don't want that, they want companies that bend over to fulfill every idiotic whim, even if it does nothing but make them feel special.


Pals:
Very good point on high exit costs, high entry costs are also important for restricting competition (this is Patri Friedman's main insight behind Seasteading/Thousand Nations).

I still am a believer in modern econ, and the majority of the economics I've read has probably been from Mises.org! Like I said before, you seem to overestimate how persuasive it is (and was a pre-convinced libertarian before first exposure who wanted to believe).

March 12, 2010 at 8:25 PM  
Blogger Mike Cane said...

Hey, print this out and mail it to the Authors Guild, will you? See their assertion here:

The Authors Guild Leadership: 21st Century Chamberlains

March 16, 2010 at 2:35 PM  
Blogger River Cocytus said...

The problem is that communities are not interest driven but proximity driven.

Consider 'Feudle', but apply it across Facebook's social networks instead of some usenet clubs.

Bazing! Google kryptonite.

March 16, 2010 at 6:04 PM  
Blogger River Cocytus said...

Also, consider Squidoo. The problem is complex, but simple guilds are impossible, mostly for the varying reasons posited by comments above, but I think MM is on to the right idea; delegated search. Google trusts nobody as far as I know, except maybe some global-level statistic provider, for information about reputation / relevance.

I would like to point out that the mechanism of delegation is, as Pals would likely point out, not to be so fixed as guilds, since the internet itself is a bit too radicalized for such fixed structures.

But, there will have to be. Perhaps what you need is auto-wikis run by various groups that document different 'stripes' of the internet; you might call it 'plaiding' the internet (they can overlap.)

In this scotch system, you take a point on the plaid, and you search for what stripes it falls into, and then ask them what they think of the sites in question.

Therefore the circle-jerk effect is impossible; nobody has exclusive domain, since there are at least two dimensions - vertical and horizontal (in reality far more) so it might look more like this:

A blogger blog on politics, computer science and poetry:

Stripes:
1. it is a blog
2. it is on politics
3. it is reactionary
4. it is on poetry
5. it is on modern poetry
6. it has essays
7. it is on programming
8. it is about science fiction
9. it speculates about the future
...

etc.

Feudle, having now determined the tartan of the site, can simply ask the interlocking groups what values to assign this blog - if I search for 'reactionary programming' it may know to ask politics, reactionary politics, computer science, and perhaps science fiction (or rather to check the indexed values for these guilds.)

No guild has guild 'control' because nobody belongs to one guild - I mean, consider how many different blogranking services there are. Ranking just needs to be a generalized, trusted service with a large enough number of players so that 1. it is difficult to systematically game 2. it is difficult to lock out or disenfranchise players who do not lock themselves out.

All in all, possible.

But the problem is, can't google just put this under the hood and we might not know the difference?

Thoughts?

March 17, 2010 at 7:28 AM  
Anonymous Pals said...

River Cocytus,

Good analysis, until you say “Ranking just needs to be a generalized, trusted service with a large enough number of players so that 1. it is difficult to systematically game 2. it is difficult to lock out or disenfranchise players who do not lock themselves out.”

You’re missing the point. The point is that ranking itself should also not be generalized and centralized, but dispersed and accountable to everyone. Everyone should fee free to start their own ranking service. And everyone should be free to ignore them all or use whichever they like. Only the best will survive—only by continuing to be good. The point is that at any level of content provision where you introduced centralized, unaccountable and coercive mechanisms, you will get crap.

Pol Pot would want the central planners to write everything. A modern statist would probably want the central planner not to write everything, but control what gets written and published. You want the central planner to not write or control writing, but to control the ranking of the writing. I go one further. Nowhere on this spectrum should anyone have a monopoly or anything be centralized. We do not need something “trusted”; if we could “trust” anyone then just let them write the whole internet.

This is similar to the argument for anarcho-capitalism, in a sense. A minarchist wants the state to enforce the elimination of coercion from human interaction. The anarcho-capitalist wants to eliminate coercion by taking away “legitimate” coercion and the ability of the state to enforce anything.

In both cases, liberty is the mother of order.

March 17, 2010 at 11:08 AM  
Anonymous Pals said...

TGGP,

For what it’s worth, your diagnosis of me is exactly contrary to the truth. I was actually your run-of-the-mill left of center Keynesian-leaning retard who thought all the interesting questions in economics were resolved in the 1950’s when god invented computers and economics become a mathematical science and that all that is left is for the good Economist Scientists to apply this science to designing the world economy properly. I then took the red pill, read Austrians, and after realizing that all of mainstream econ is garbage, became a libertarian on economic matters. I then started applying this same analysis to other aspects of life and became a full-on libertarian. I then applied it to politics and went completely batshit anarcho-capitalist.

Since you claim knowledge of both Austrian and mainstream econ, I’m curious as to what exactly you find worthwhile in mainstream economics. I believe it’s interesting to observe from a sociology-of-science viewpoint, to see how so many could be duped with such transparent bullshit for so long. And it might be important to understand the “national discussion” taking place among the Krugmanite retards. But other than that, I struggle to find anything in mainstream econ in the last 70 years that is worth knowing. All that is valid and logical is not original (pretty much all of the “free market” economists just take classical economics and apply some useless pointless math) and anything that is original is blatantly invalid bullshit (all the statist, Keynesian garbage that is responsible for ruining the entire fucking planet.) Game theory is an exception, but then again, that was started by Austrians and appears to me to be an extension of Mises’s ideas on human action; it introduces purposeful acting man to another purposeful acting man.

March 17, 2010 at 11:11 AM  
Blogger River Cocytus said...

Pals -

You misread. I never said centralized.

I did say generalized, meaning that there would be some level of standard for API's (for a search engine to plug into) and ways of ranking (so the results can be made sense of) for the striping/guilds to use. They would have to do this to some extent, or the job of figuring out how each group is rating things would prove to be the same task google is currently doing.

Of all of my errors it would be nice if you critiqued one I made!

March 17, 2010 at 1:42 PM  
Anonymous Web Designing Karachi said...

It’s quite appreciable that such information is being shared through a huge network. Keep it up.

March 21, 2010 at 9:59 PM  
Blogger Publius said...

This is bad writing. Painfully bad. 360 words of content dragged out to 3,600. Do you ever get to the point?

March 24, 2010 at 7:23 PM  

Post a Comment

Subscribe to Post Comments [Atom]

<< Home