Still pretty busy right now, so still no new posts for now... be patient.

Download

• freq08.mp3 (2.8 MB, 02:27 mins)

I'm closing the comments down for now, as I'm quite busy with other stuff and simply don't want to handle the new wave of comment spam that started a couple of weeks ago. You can still reach me via the contact form though.

I'll open the comments again at some point down the line, I'm quite proud of the average comment quality on this site, and I wouldn't want to miss your invaluable feedback.

But since I wrote the "Spotlight Helps Fight Comment Spam!" article I've received more spam comments than usual, and each day a couple of them aren't caught by the filter and force me to manually delete them. (Coincidence? I'm looking forward to whoising the spammer's domain names when I find some time, I'm expecing to see some hints towards an organized effort.)

Otherwise life treats me very well, I've had my final verbal exam and got my diploma, and am now looking forward to some major changes in my life. I'll post more details later; for subtle hints you could start watching my Flickr account.

Just so I won't forget, and because it's useful to know.

require 'RMagick'

# generate rgb image buffer
pixeldata = []
14.times do |y|
	100.times do |x|
		pixeldata << 1.0 - (x/100.0 * y/14.0)
		pixeldata << (x/100.0 + y/14.0) / 2
		pixeldata << (x/100.0)
	end
end

# convert to PNG
image = Magick::Image.constitute(100, 14, 'RGB', pixeldata)
rawdata = image.to_blob() { self.format = 'PNG' }

# display
puts "Content-Type: image/png"
puts
puts rawdata

...or simply write to file.

Result:

Any hints on more lightweight PNG writers? ImageMagick can be a pain to install. There are some neat tricks for image-builders on why's sparklines for minimalists page, but jzp's PNG code shown there is for a very specific image and visualization format, and I don't understand PNG enough to adapt it.

Update 2006-12-09: Olle Jonsson sends a link to png, a lightweight pure-Ruby PNG library by Ryan Davis. gem install png...

I just found that StrategicBoard offers their keyword search results in an OPML format that contains feed URLs -- which can be a remarkable tool in a number of situations. Such a service is an essential prerequisite for feed grazing, i.e. subscribing to blogs temporarily while they're writing about stuff that you're interested in.

(Feed grazing typically is part of automated processes, e.g. of simple news recommendation engines or topic exploration algorithms. Some desktop aggregators seem to support it already, unfortunately NetNewsWire isn't one of them.)

Here's the URL for such a search query:

http://www.strategicboard.com/index.php?fr=o&s=your_keyword

...and the result is an OPML of feed URLs.

Unfortunately the service has a major flaw. When there are no search results for your query, the reply will be an HTML error message:

<br><br><b class="gsr">&nbsp; No relevant posts were found on your_keyword&nbsp;</b>

...which makes it a nuisance for developers that want to consume this OPML feed, and will irritate most OPML parsers. Plus it conflicts with their Content-type header, which sends a value of application/xml. Why they don't simply return an empty list I don't know.

On a semi-related note, I whish the company's crawlers would stop sending a referrer URL with their "Strategic Board Bot" HTTP requests. My logfiles are full with their referrer spam. (Even more irritatingly, some of their requests contain the user agent string as referrer URL.)

Update 2006-07-10 -- Dudu Mimran of StrategicBoard writes in the comments: OPML bug will be fixed, bot referrer URL fixed since yesterday. Nice.

I'm using a combination of fairly primitive methods to cope with blog spam. As this blog doesn't get too much comments the amount of manual work is relatively limited; main line of defense is an old-fashioned and relatively short blacklist. I'm notified of incoming comments, and in the rare event that a spam comment gets through I'll inspect it for new keywords.

For a couple of months now it's become apparent that specific posts seem to attract more spam than others.

I just thought that it may be great to have a statistic of this phenomenon -- so that I could make an informed decision wether it makes sense to close comments for particular old articles.

Unfortunately the tools I use create no such statistics. I have a log of successfully filtered keywords, but unfortunately this log doesn't map blocked attempts to individual articles.

But I still have all comment notification emails of spam that got through the filter -- and these are a relatively accessible source of the required information:

• Comment notifications use a distinct mail subject, and mention the article URL in the email body.
• If I delete comment spam I will then also delete the respective email notification.
• I never empty my email trash folder.

... which means I still have all notifications for successful spam comments, stored away in a separate folder from 'legit' comment notifications. If I combined that with Spotlight and a short Ruby script, creating a 'successfully spammed articles' ranking is a piece of cake.

Taming Spotlight

The mdls command line tool lists all Spotlight metadata for a specific file, so if you point it to an email file (~/Library/Mail/.../Messages/*.emlx) you can get a list of the available metadata for filtering email messages. I was only interested in the mail subject, kMDItemTitle; but there are also kMDItemAuthors, kMDItemAuthorEmailAddresses, kMDItemContentCreationDate, and others.

The mdfind command line tool lets you then search the Spotlight database. I'm using a query like this:

mdfind -onlyin ~/Library/Mail/folder_name/ "kMDItemTitle == '*New Comment*'"

This results in a list of files:

/Users/.../Messages/11579.emlx
/Users/.../Messages/5077.emlx
/Users/.../Messages/5171.emlx
/Users/.../Messages/5218.emlx
/Users/.../Messages/5279.emlx
/Users/.../Messages/5280.emlx
/Users/.../Messages/5282.emlx
/Users/.../Messages/11550.emlx
/Users/.../Messages/11551.emlx
...etc

And Then Count

The .emlx file format is afaik proprietary, but when you open it in a text editor you'll see that it's a plaintext format and includes, among other things, the raw email message with full headers and body text. In my case I was only interested in extracting an article URL from each of those emails, so I didn't even bother to look at the file format but simply used a regular expression against the complete file content.

Then count each captured string in a hash, and finally print the ranking:

Searching... found 94 emails.
Parsing.........................................................
..................................... done.
Ranking:
11 times /weblog/2006/01/flip4mac_has_a_strange_eula/
9 times /weblog/2006/03/added_article_feeds_with_comments/
7 times /weblog/2005/08/screw_objectivism/
7 times /weblog/2005/12/mailfeedorg_has_launched/
7 times /weblog/2005/12/osx_10_4_3_phoning_home/
5 times /weblog/2006/06/notice_the_information_bar/
5 times /weblog/2006/03/jabber_server_space_starts_boiling/
4 times /weblog/2005/11/new_comment_spam/
4 times /weblog/2006/03/yarv_benchmark_with_rexml/
4 times /weblog/2005/11/team_ramrod/
4 times /weblog/2005/08/a_first_look_at_pandora/
3 times /weblog/2005/08/securely_connect_web_services/
3 times /weblog/2005/12/contact_form_spam_bots/
2 times /weblog/2005/12/qre_recommendations_from_your_database/
2 times /weblog/2005/08/small_feed_changes/
2 times /weblog/2006/01/gmail_html_view_as_default_view/
2 times /weblog/2005/07/ballmer_creeps_me_out/
1 times /weblog/2005/10/rhinola_javascript_for_the_server/
1 times /weblog/2006/06/midnightbot/
1 times /weblog/2005/12/printable_schedule_for_22c3/
1 times /weblog/2006/03/feed_readers_a_commodity/
1 times /weblog/2006/01/visualization_of_numeric_data/
1 times /weblog/2005/09/marc_mcdonald/
1 times /weblog/2005/11/100_dollar_laptop/
1 times /weblog/2005/11/type_managers_replacing_file_managers/
1 times /weblog/2005/07/backup_hell/
1 times /weblog/2006/02/delicious_404/
1 times /weblog/2005/10/webservice_auth_finally_coming/
1 times /weblog/2005/12/rails_inflector_in_ruby_scripts/
1 times /weblog/2005/11/22c3_schedule/

I was a little surprised to see the result. The ranking of total spam attacks, as opposed to successful attacks as shown here, probably looks a little different. But to get such a ranking I might have to parse Apache logfiles and correlate timestamps with the blacklist log -- much less fun.

Download

• comment_spam_stats.rb.txt, Ruby script

Related Articles

• Contact Form Spam Bots Ahead
• New Wave of Comment Spam: PageRank Indirection
• Critique and Countercritique

Via the De:Bug Blog I just came upon the "Mission Statement" of Jimmy Wales' new site, Campaigns Wikia:

Blog and wiki authors are now inventing a new era of media, and it is my belief that this new media is going to invent a new era of politics. If broadcast media brought us broadcast politics, then participatory media will bring us participatory politics. [...]

This website, Campaigns Wikia, has the goal of bringing together people from diverse political perspectives who may not share much else, but who share the idea that they would rather see democratic politics be about engaging with the serious ideas of intelligent opponents, about activating and motivating ordinary people to get involved and really care about politics beyond the television soundbites.

There are more and more people who 'get it', and I feel that we may be reaching a tipping point where another digital revolution reaches the masses and changes our societies.

The idea of huge, participatory, integrative communities. The idea of collaborative organizational structures on a very large scale.

From a slightly different angle, here's what I wrote two weeks ago:

For a while now I've been thinking about how the representative systems of our democracies are more and more obsolete, and should be replaced by systems that integrate people who care, as opposed to those who are simply getting paid to do the job.

My theory is that we are now living with systems that are too complex to manage for small groups of people. Systems where it simply isn't enough to have a couple of dozen people do the error checking, but where you need thousands.

And I believe, I know, I have experienced that the new 'digital' organizational structures we have evolved during the last decade may finally help us cope with this complexity.

It WILL Happen. Period.

I've been having a lot of discussions about this lately, both with friends and with total strangers. It's a really interesting topic to talk about with people you don't know. One such discussion about social filters and participatory systems, on a train from Heidelberg to Berlin, was one of the most memorable meetings with strangers I ever had.

There are many, many who instantly 'get' it, sometimes after a couple of minutes of explanations and examples. It's pretty easy to make the average person understand why a participatory community such as Wikipedia or (my generic example) the Firefox community may well be a precursor to future political systems.

Then there's a distinct second group of people who argue fiercely against what they perceive as 'mob rule'; people who can't fathom the thought of open, participatory systems that actually work. Typically these are people from Academia or political institutions, both of which are groups that are deeply characterized by their thresholds of entry. (I'm not joking here. That's anecdotal evidence, true, but happened too often to be a coincidence.)

And these are the people wee need to have discussions with. Because they're the gateways to making changes on a large scale, to changing the system from within. If we can't persuade them we at least must reduce their fear of these systems. We must demonstrate that what we're proposing is not Anarchy.

Because the alternative would be what the music industry is undergoing right now, with their version of the Conflict with Digital Values: the change is happening anyway, and the existing structures suffer a slow and violent death. And while that may be a fun thing to watch happen to the pop culture industry, I don't think we can afford to let that happen to our political systems.

Updated to revision 2.

Oh boy. Not only is is a great time to be alive -- it's a great time to be a programmer.

One year ago I wouldn't have thought that knowing the tools of bug tracking software or version systems or mailing lists or forum software would have relevance outside the small virtual communities that use them.

Now I know that these are powerful tools of political expression.

I've been bookmarking sites on del.icio.us with the tag "participatorydemocracy".

I just returned from a concert at WM Lounge, one of these places in Berlin that suddenly cropped up during the last couple of weeks, where you can watch the games and listen to bands play during intermissions. WM Lounge is a beautiful location in the center of Prenzlauer Berg, an old bath house with pillars and arches and flaking paint.

As there was no game scheduled for today they simply had four bands scheduled to appear, three of which actually played. I went to see Team Blender, who I got to know because they're friends with a friend of mine, and I think they're excellent.

Before I left my appartment I had received an E-Mail that was the beginning of a short but important conversation, and so I took my laptop with me, just in case I found an open wifi network. (Happens a lot less often in Berlin as you might think.)

I arrived early, so I had some time before the show started. Naturally I got out my laptop -- and what do you know, they had an open network just for the event. And I got a mail with great news, so the evening got off to a good start.

As I went home later that evening I found that I had to take the tram, because U-Bahns don't drive on week nights. The particular line I rode, M10, ends with a drive through Warschauer Straße, a fairly important part of Eastern Berlin. Well-known for being both one of the semi-official centers for "alternative" culture, and a mekka for students. It's still relatively cheap to live in this area, though that's quickly changing as more and more places get renovated.

During the ride home I had opened my laptop to write down some notes. And on a whim I opened KisMac to check out if Warschauer Straße had open wifi networks.

Turns out the street has quite a number of networks (see screenshot above), but most of them closed. That's Berlin for you. ;)

This is an updated version of the first MidnightBot article where I visualize the times of day that people post on their blogs -- and it might not be the last iteration either.

There were a number of reasons why I wasn't satisfied with the original graphs, mainly caused by the limitations of the software used (a simple Ruby sparklines library). Senorpako then suggested Processing, and that turned out to be a much better tool indeed. I'm not too fond of the Processing IDE, but you can just as well use Eclipse or any other editor instead.

The first thing I found after a number of iterations: the graphs I originally published were not only hard to read, they were sometimes even misleading, to the point where some of my conclusions derived from the graphs were flat-out wrong!

But before I go into details, first the new graphs.

Update 2006-07-05 -- text now includes links to the source data of three featured graphs: DrunkenBlog, My Boring-Ass Life, and 23C3 Wiki.

Update 2006-07-06 -- small layout edits. Sorry about that, I'll stop now.

Showroom

Site	History	24h	7d	Occupation
De:Bug Blog				Journalist and DJ
dekstop weblog				Student
DrunkenBlog				(Developer?)
eigenclass.org				(Developer?)
inessential.com				Developer
ranchero.com				Developer
My Boring-Ass Life				Movie director
The Lunatic Fringe				(Evangelist?)
villainous.biz				Artist
wortwechsel.biz				Designer/Developer
23C3 Wiki				Unwashed masses
History = vertical axis denotes time of day for each article, plotted in sequential order. 24h = posting frequency over the time of day, from 0:00 to 23:00. 7d = posting frequency over the day of week, from Monday to Sunday. Occupation = author's job description (mostly a guess).

*doh*

I really learned some things by comparing the old and current versions of the graphs.

Continuous lines were a really bad choice for displaying histograms. They ignore the separation between data points, and introduce intermediate values that don't actually exist. As a result the graphs become much harder to read correctly -- and indeed, most of my faulty conclusions can be explained by this.

	24h	7d
Old
New
DrunkenBlog, old and new. (data)

For example I suggested that DrunkenBlog only posts Fridays, while the day with the most articles was actually Saturday. This is a tricky mistake to spot, because you can't really read it from the old graph itself without measuring pixels, and you won't see it if you look at the source data, which is only a list of numeric dates. I only found out about my mistake after I compared the old sparkline with the new one (which clearly separates individual days).

New: History

While looking at the 24h graphs I became interested in some of their characteristics. Why does LunaticFringe have two peaks? Why do DrunkenBlog, Villainous.biz and others seem so evenly distributed over the day? As I thought about explanations I found that an important piece of context was missing in the original graphs: is the author's posting behavior changing over time?

So I added History plots, which are meant as companions to the 24h graphs. These plots display the time of day of each article in order of their release sequence, ignoring the actual dates. (I'm not yet sure if the concept behind these plots is easily understood, so please give feedback if you have better ideas.)

It turned out that these plots don't give a simple answer to the unexplained phenomena, but I think they add an interesting new dimension to the 24h graphs. Plus they give a sense of the size of each underlying data set, a piece information which previously used up a whole table column.

I find it interesting that although most History plots seem chaotic, each does have a distinct form.

And New Phenomena

There are two History plots that are dominated by distinctly non-chaotic parts: My Boring-Ass Life, and 23C3 Wiki.

My Boring-Ass Life

The History plot of My Boring-Ass Life starts off unusually organized, but about halfway through changes behavior and breaks into the same chaotic movements that can be seen on all other sites:

History	24h	7d
Kevin Smith breaks character. (data)

While double-checking the initial straight line with the source data I found that the Boring-Ass Life really did release earlier articles at recurring times of day, one article per day (most around either 11:00 or 15:00, which you can also see as spikes in the 24h histogram).

This might point towards a rigid publishing strategy -- after all, this site is a PR medium. It's not inconceivable that Mr. Smith writes a number of articles in advance, but only releases new articles at specific times every day. A look at his site may clear up why this behavior changed. (Something to do with the release of Clerks 2? Or the preceding press tour?)

23C3 Wiki (Recent Changes Feed)

The History plot of the 23C3 Wiki's 'Recent Changes' feed does seem even more peculiar. From about entry 20 up to close to the end, the curve rises and re-starts four times in what seems like a continuous line:

History	24h	7d
Edit war? (data)

But that's easily explained by adding some context. These 50 or so 'continuous' data points correspond to about 4 days of high editing activity after the Wiki was announced to a larger public on May 29th.

Note how the effect of the announcement gradually slows down: the line segments get steeper and steeper as the time span between individual Wiki edits increases.

In short, compared to the other sites the 23C3 Wiki's 24h and 7d graphs are utterly unrepresentative. But its History plot demonstrates an important characteristic of social systems: how a single event results in a series of reactions, until interest finally dies down.

Douglas Coupland - JPod. Page 190 of 449, as seen through the lense of my alien surveillance equipment. (view all)

OK the last two entries were kind of a downer, so before the rest of my remaining readers jump the sinking ship I thought I should resume with something a little more consumer-friendly.

I just finished Douglas Coupland's "JPod". As almost every other reviewer will tell you it's like an updated version of his early 90ies "Microserfs", which means it won't win huge literature prizes, but instead is light entertainment with a lot of insider jokes, jargon, an pop culture references. The novel often works best in its side notes, say when Coupland manages to illustrate or extrapolate complex social phenomena within a couple of half-finished sentences, often in a brilliantly funny manner.

Induced by summer-related laziness I thought I should simply reproduce my handwritten notes (hand... what?!) and call it a review. Let others write coherent sentences. At these temperatures nobody will actually read long texts anyway, so I could just as well bake some dense word pie and call it a day.

Super, iPhoto just crashed. Don't send a report, restart.

ISBN: 1-59691-233-2

The novel's plot is intermixed with fragments of spam, which you don't read even in print form (e.g. page 27).

Characters seem to believe that you can change the nature of certain things by giving them a different shape. One protagonist chooses "John Doe" as his new legal name, likes M&M's because they're popular in consumer statistics (p31).

Instead of lists, people are now writing 'profiles': semi-structured texts that attempt to capture the nature of themselves or another person, thing or concept. Ethan/"me": "Even thinking of [Karaoke] in this neutral detached listing format causes me to worry that I might somehow even unintentionally lure karaoke into my life." (p34) Then: "Mark, all I did was collect the data and present it." Mark: "No. I want to improve my profile now. I demand to be more than just a cartoon character." (p35) (Note that "improving the profile" just means finding better stories. The shape and meaning of these new 'lists' are highly influenced by the concept of Internet user profiles, even if they never become part of an actual social software system. People are the social software.)

Compare: "Neither of you are emotionally blanks", you're "special and unique" and "I'm going to draw up a standardized list" to prove it (p28) vs. "Mark, I think you're obsessing on this whole individuality thing. Revel in your averageness the way John Doe does." (p35), both from the same speaker. Protagonists seem overly concerned with choosing adequate means of self-expression (and with how these could be interpreted by others), to a degree where shapes become interchangeable. It's good as long as it captures attention. In that way people become gimmicks, as much as their things already are. Edible office equipment, heroin as a means of emotional self-improvement, wearing Chinese refugee clothing (because it's there) and then switching back to The Gap before stepping on a plane to China (because otherwise the Chinese might stare), etc.

And Consumption

Common modus operandi: watch it a bit, then poke it to see what else it can do. And: skip around to get a quick overview of the situation. Cf. Ethan, while listening to Mark: "Can we speed this up?" (p36).

Ethan: "The problem is, after a week of intense googling, we've started to burn out on knowing the answer to everything. God must feel that way all the time. I think people in the year 2020 are going to be nostalgic for the sensation of feeling clueless." (p248)

John Doe, after Mark ate his snack: "I don't want to eat your stapler, and I don't want another Handi-Snacks. That's not my point. You took it without asking, and then you act like private property is meaningless." (p250)

And the climactic dialogue between Ethan and Chinese mini-drug czar and refugee smuggler Kam:

Kam had an Ikea desk set up in the corner, and he paused a game of AmmuNation. "Ammunation!", I said. "All right! What do you think of it?"

"It's the best. It allows me to park my evil in one place so I can be a better person in the real world."

"That's thoughtful of you."

As may have become apparent, this time I didn't have the feeling Coupland describes a species from another planet; to the contrary he simply extrapolated our digital lives and added some accidental homicides. It's still a fun book though.

Now direct your attention elsewhere, I hear there's a hilarious new video on YouTube of a tech service guy accidentally dropping a TV set on a kitten. Oooh, look, a disabled pony. Click.

About the Book

I first heard of East Timor by watching "Manufacturing Consent", a documentary on Noam Chomsky. In this film Chomsky calls East Timor the great untold war story of the 20th century. He states that the atrocities committed by foreign nations in East Timor are comparable to Pol Pot in Cambodia. And he notes that, in contrast to Pol Pot, nobody seemed to notice.

(The film, btw, I now consider mandatory viewing for every member of a society that has TV, newspapers, and/or corporations. Much time is spent on Chomsky's view of how public opinion is shaped by political and commercial processes. It's great introductory material to the Modern State of Truth. School children should watch it. If you're a teacher: bring it in the classroom, please. See below for download links.)

The island Timor, sitting somewhere between Indonesia and Australia (Google Maps, Google Earth). East Timor is the darker part of the island, with the capital Dili marked by a white star. The lighter, western part of the island is now part of Indonesia.

An Attempt to Recapitulate

To me the story seems to go like this (please send in corrections):

Southeast Asian island Timor has been occupied by Western governments for centuries. According to the Chomsky documentary, Timor hosted one of the last surviving ancient civilisations in that part of the world, living a self-sustained life mostly secluded from modern societies.

The area around Timor is also known for its oil reserves. In the 1970ies Brits find major new oil reserves, in 1975 Indonesia starts a war under political pretenses, and things take a really wrong turn. US President Ford assures he would refrain from intervening. Even Australia starts salivating. It is blatantly obvious that the war is fueled by skewed politics and corporate greed, but the world doesn't seem to care. Timor is occupied, thousands die.

In 2002, after 30 years of struggle, East Timor regains political independence. A government is instated in a manner that seems a little opaque regarding the involvement of former occupants. The country's new military is a wild mixture of former resistance militia and pro-Indonesian soldiers. The country is slowly recovering from a state of shock, from a feeling of betrayal, but oil interests are still not cleared.

A couple of years later, in early 2006, things fall apart again. The former occupants feel like they should "help out" and clean the new mess, not admitting in public that the situation is a direct result of their previous actions.

And this is where we are right now. A big political mess, and a society on the verge of civil war.

Hm. Where have I heard this story before?

Watch, Listen, Think, then Write

This may sound like a Unicef commercial to you. It's a very simplistic summary, and I've deliberately chosen to be partisan in my descriptions. And the cute indigene on a boat as pictured above borders on cheap emotional manipulation.

But I was quite touched by the story. It's not that simple to get me interested in contemporary politics (I tuned out of local news years ago), but there are things that I really care about. I think Timor is a great example of the Modern Tale, of the power of deception in the hands of a selected circle. Of nepotism at the highest levels.

A new situation is unfolding right now in Timor, but although the media are a little more interested than 10-20 years ago it's still hard to find out what is actually happening. Compared to a decade ago the informal news circles are working quite well, but there are a lot of opposing positions and inconsistencies.

The Wikipedia talk pages of the relevant articles can serve as nice illustrations of the general confusion, of misinformation, and skewed sources. Cf. "East Timor talk page ", "2006 East Timor crisis talk page".

An analysis on Global Voices attempts to clear up some misinformation and demonstrates that there still are people who are trying to see through the mud -- but notice that all of the cited sources are translated from Portuguese.

That's what Chomsky is talking about in his commentary on modern media: It's hard work to stay informed. You need to be educated, you need allies you can trust. You need to watch many, many sources, and then compare. It's a fucking maze.

And on the other side it's hard to get a message across that disagrees with the common consensus. Nobody listens anymore when your argument takes more than 20 seconds.

Recommended Reading

• Wikipedia: 2006 East Timor crisis
• TailRank search: "Timor" (feed). Probably the best persistent news resource, covers a wide spectrum of sources.
• tumbleweed in timor lorosae, private blog of a Unicef volunteer

Mandatory Movie Download

• MiniNova search: "Manufacturing Consent"
• PirateBay search: "Manufacturing Consent"
• The film is also available on ChomskyTorrents.org, but you can't link to search results. Otherwise this link would work: "Manufacturing Consent"