Using Junk Mail Filtering techniques to Spot Common Threads
So a couple weeks ago, the good folks at O'Reilly MacDevCenter posted a concise article detailing how the Mail.app junk mail filter works. The article, The Fight Against Spam, Part 2 is somewhat light in details, but is a GREAT overview. If you've ever wondered about how junk mail filters work on your Mac, you should probably click over and check it out.I'll spare you the gorey details, but one aspect of Mail.app's junk mail filters is that it attempts to find clusters of messages in a "phase space." The space is a multi-dimensional place where you have a dimension for each word in the English language. Messages that have the similar words, will wind up being closer to each other in this imaginary space. Mail.app looks for clusters of email messages surrounding known SPAM messages, and guesses that since they have a lot of the same words, they're probably SPAM as well. (this is a gross oversimplification, of course...)
So I was just thinking... A lot of my online experience is in mailing lists and news groups. I scan these things visually, using subject headings as a key. Mailing list or news group threads tend to eventually wander off from the original topic, but the subject line frequently stays the same. Wouldn't it be nice to have a system like the junk mail filter to scan messages sent to a mailing list and cluster similar messages into a "thread."
Even more fun would be to use OpenGL to create a 3d representation of phase space clusters, possibly with links between clusters. The links could be labled with concepts or word groups that are the same (or similar) in different clusters...
Just an idea...
