dekstop: weblog

Visit my Flickr photo stream for more abstract eye candy.

I'm having a series of conversations with Falko Schmid about current interfaces to structured information, and about using existing web services to build all-encompassing information structure and search facilities.

Our conversation started with the realization of a problem: apparently nobody can build a single system that integrates all information. There have been many attempts to accomplish that (Google and Wikipedia being the most prominent and successful) -- but if the last years have shown us anything it's that there is no way to define a stable content structure that contains everything. There are just too many reasons why attempting this will fail. Even Wikipedia is only good for so many things -- you don't use it to find the best price for a book.

Why all-encompassing content structures always fail

• This would involve too much work. There simply is too much stuff out there.
• There is no stable structure. The value of Information changes, and is too dependent on context.
• There is no common interface. Different information wants to be treated differently. The power behind Flickr and Wikipedia and Amazon is that they don't want to be all things to everything; instead they specialize on certain information and specific methods to apply structure.

There simply is no way to build a single structure of everything.

And: there should be no way to build a single structure of everything. Nobody should own this. There is too much value in such a (hypothetical) information integration system that we should rely on a single service provider to own it.

But what is exciting about the current state of the web is that there are more and more services popping up that specialize on specific types of information, and which do that very well.

So during our conversation we quickly found that it would be pretty cool to find a way to integrate all of these services. Currently this is not possible at all: the interfaces used for each of these services are simply too diverse, to specialized. There is no standard API that bridges e.g. information on Flickr with information on Amazon, although depending on what you want to achieve it would be very interesting to do so.

Instead of building one all-encompassing structure of everything we should attempt something else: we should find a way to integrate each of the existing specialized structures. Content structure as a distributed system (small pieces loosely joined).

Think of how exciting this could be: there is no central system, but instead a simple interface that is understood by all kinds of services and clients. An interface that allows navigating different pools of information, that allows chaining of searches, combining different kinds of information.

We already have stable methods to read specialized information pools. Web services have APIs. And when there is information that can't be accessed via an API we can simply scrape it.

But what we need is a common interface, one that is simple enough so that people will actually implement it, but which is flexible enough so that we can use it to extract useful information.

What should such an interface look like? How do we describe connections between very different services with very different kinds of information? Is there a common denominator in all services that allows to build meaningful connections between them?

We think that there is a connection between all these services that makes such a bridging service feasible.

The current state of information structure on the web builds on one premise: people are gateways. They are both sources of and pointers to information. Google recognizes this in PageRank, where the source of information qualifies its value. And virtually any interesting web service today is social software, where a core aspect of the service is the connection between people and other people, and between people and content (e.g. their interests).

This is the idea behind blogs, tags, friend lists and many other kinds of information pools: you can now find interesting content by finding interesting people. And that is what a web service bridge should accomplish: not to automatically determine correlation between existing data, but to use existing connections between sets of information. And the most prominent of those connections are the people that create or consume information.

The idea we developed is that some kind of "global" content integration can be attempted with rather simple means. The web is in a state where this is feasible, and actually not at all complicated. And we believe that it boils down to this assumption that can be found in any interesting service: interesting people are gateways to interesting information.

"Interesting" is the operative word of the last sentence: the value of information is subjective and context-dependend. In order to find information that is interesting to you, it helps to find the right people and ask the right questions. And this is where combining web services can help a great deal.

It would be interesting to describe a common interface which allows to find links between data in unconnected services. One then could build proxies for each existing web service that serve as adapters to the nonstandard interfaces. (If the service providers don't do it themselves, everybody can build proxies that serve as adapters between the service's specialized interface and the common interface.)

I realize that all of this sounds a little vague, but bear with me. Coming up: some more specific scenarios for which this would become useful, and a first attempt to describe such a bridging interface.

---