Q's Squishy Thing of the Day

Just saw this today. Perhaps nobody else will find it interesting, but I do.

Essential WoW terminology in other languages

Q's Squishy Thing of the Day

Meet ShadyURL. It encodes URLs in things that look like spam, porn, malware, and cults/racism. Some of the things I've linked to various people in the last day:

http://5z8.info/PIN-phisher_a1x6x_nakedgrandmas.jpg
http://5z8.info/double-your-wang_z2f4t_nazi
http://5z8.info/worm.exe_i8k3z_stalin-will-rise-again
http://5z8.info/killallimmigrants_x2l9n_protocols-of-the-elders-of-zion.doc (best URL so far)
http://5z8.info/OPI423098BVMPEMBUDSGND423098889708807909788079_k3j7r_dogporn

And, just for fun, here's the Copyright Alliance blog: http://5z8.info/pirate-anything_w2x9e_racist-raps

And this blog: http://5z8.info/whitepower_o3q2x_inject_worm, which is awesome, because I'm actually 3/4 German

And yes, I have been made aware that I'm late to the party on this one.

& Content Filtering

Following a brief scuffle with Patrick Ross in the comments of the Copyright Alliance blog, I thought the topic I briefly discussed deserved elaboration: content filtering. That is, the analysis and identification of illicit content packets as they pass through a router, based on hashing (not so effective) or fingerprinting (more effective), usually followed by dropping the packet.

I'll discuss the full range of content filtering, though one thing at a time.

First of all, the specific type I referred to previously: filtering at the internet service provider level. Imagine an ISP that is being legally pressured to do something about file-sharing by copyright industry representatives, a situation hundreds of ISPs around the world find themselves in at this very moment.

Now, Mr. Sales Rep, from another company that makes deep packet inspection hardware, offers to sell a product to Mr. Business Suit from the ISP, that will solve the file-sharing problem, as many sales reps are currently doing. This company's tests show that the product is 95% effective at identifying and blocking traffic containing illicit copyrighted material, and has a low rate of false-positives. Naturally, Mr. Business Suit at the ISP looks at this product, and sees an amazing solution to all of their problems. Mr. Engineer at the same ISP looks at the product and sees a million-dollar paperweight, and tells Mr. Sales Rep to get out of his office. Nevertheless, convinced by Mr. Sales Rep, Mr. Business Suit purchases the product, and has Mr. Engineer install it.

Now the product has been installed, and everybody watches eagerly, as Mr. Engineer turns the new product on. Immediately the product begins logging transfers of copyrighted content by the thousands, and successfully blocks them. Yet Mr. Engineer looks at his network statistics and sees that not only is the product having 0 effect on the amount of internet traffic, there is still just as much illicit content being successfully uploaded by users of the ISP.

What could possibly have gone wrong? And why were the appraisals of the product so drastically different between Mr. Business Suit and Mr. Engineer, to begin with? Is the product defective? Did Mr. Sales Rep lie?

Well, not exactly.

What happened is that the ISP's users adapted effortlessly to the new piece of filtering hardware. While it's certainly viable, if implemented competently, to detect and block things like copyrighted content, this is only possible if you have access to the data being transmitted. The universal Achilles heel of such identification algorithms is encryption.

Modern file-sharing software supports end-to-end encryption - the same kind used to secure credit card transactions online: the uploader encrypts the data, the downloader decrypts it, and nothing in the middle can access the data in between the two, because nothing else has the encryption key. This "nothing" includes that million-dollar product our ISP just bought.

Now, this encryption is not a technology that needs to be developed, nor does it need to be downloaded and installed by the user. It's already there. If a user is able to share files through a P2P application, the encryption code is already in that P2P application; it needs only to be enabled by a user clicking a check box. And, of course, you can be certain that it will be turned on by default in future versions of said software if content filtering by ISPs becomes common.

In other words, each of those "blocked" uploads the product registers is merely the first of two attempts. A blocked upload is merely an upload that will succeed seconds later, after the user clicks the box to enable encryption (though if content filtering is widely deployed, users won't even need to do that). Thus, to make a long story short, while you have successfully prevented file-sharers from uploading unencrypted illicit content, you haven't actually prevented a single copyright infringement.

This is a theoretical problem, not an implementation issue. As such, there is no basis to hope that this is a limitation that will ever be overcome in the future.

But look on the bright side: Mr. Sales Rep got a nice commission off that million bucks the ISP paid his company, and as he technically never lied, the ISP has no legal recourse to argue fraudulent advertising.

However, the fact that ISP-level filtering is a technological dead-end should not be taken to mean that all filtering technology is useless. As stated, filtering technology can be effective, given that it has access to the data. Of course, the hackers of the world will continually work to find new ways to evade such filtering algorithms, but it should still be possible to successfully filter enough to justify the cost of the filtering hardware/software.

One example where this works to a satisfactory degree, both in theory and in practice, is YouTube. Because YouTube actually processes and decodes the content uploaded to it, it's impossible for it to not have access to the data - it couldn't function otherwise. As such, it always has access to the full, unencrypted content uploaded, at which point filtering of that content is possible, and in fact is already being performed.

Dumb file storage sites - sites like RapidShare - which store data without any regard to what type of data it is, fall somewhere in the middle. As they do not require access to the data itself, encryption is entirely possible, and would indeed be capable of evading any filtering of uploaded content done on the part of the site. However, use of this type of encryption would be much more of an inconvenience than is the case with encryption in P2P programs; in this case, encryption must be done manually, by the user, through a completely separate program (almost anything that can make ZIP files can encrypt them, for instance), and the encryption key must be distributed through other channels, such as forums that link to the encrypted file. As such, while filtering at the level of such sites will certainly not prevent such encrypted transmission of content (nor probably even a majority of total transmission), it's possible that filtering systems might reduce sharing of illicit content by some sufficiently valuable fraction by means of sheer annoyance.

Random Fact of the Day

You can't install SilverLight on a Pentium 2. It has to be something that supports SSE.

Random Fact of the Day

Watching CSI: Miami I learned something new: that receiving a liver transplant* can change your blood type to that of the donor. I'd never heard anything about that before, so I had to do a Google search before believing it. Apparently it's actually happened; once. It seems that stem cells in the liver managed to detach and land in the bone marrow of the recipient, and take over the marrow.

This makes me wonder. If I understand the biology involved correctly, this would only be possible if the blood type changed in a particular direction. In any other direction you'd get the same effect you'd get from getting an incompatible blood transfusion. Thus my question: might this have happened before, but been misclassified as simple organ rejection?

*In theory this is probably possible with most transplants, given that it involves stem cells being transported with the organ.

Random Fact of the Day

In .NET, String.GetHashCode computes the hash code every time the function is called. This is slightly surprising given that Strings are immutable, and thus could easily compute the hash code on construct or on the first call to GetHashCode (although both of those options do have associated overhead).

Someone I know believes that Java is the same way.

& Moral Panics

One topic that came up rather suddenly in IRC is the topic of the irrationality of humans when a person feels wronged. The particular topic in chat was that, I'm told, you should never, ever touch leaked materials, such as the Windows source or COFEE, because this tends to send companies (especially the one that produced said thing) into moral panics and refuse to ever hire you.

Think about this for a moment; a little bit of rational thought concludes that this is highly irrational behavior, reminiscent of the Pointy-Haired Boss (is there a Dilbert strip on this topic, I wonder?). If you were Microsoft, for instance, and you were looking to hire a programmer for the Windows team (although this could also apply to other parts as well), the #1 most desirable candidate for you is the one who has played extensively with the leaked Windows source, all other things being equal. Not only would categorically refusing to hire such a person result in no benefit, but it would materially harm you as a company, by refusing the candidate most beneficial to you. This is a case where moral outrage contradicts reason, and acting on that outrage results in self-destructive behavior that does more harm than good; or, as the saying goes, cutting off your nose to spite your face.

An alternate form of this is observed extensively in the copyright industries, who have a long history of various licensing and technology blunders with a detrimental effect to their own sales in the name of fighting piracy (and the goal of fighting piracy is, you know, to increase sales). In this case the moral outrage is provoked by a fixation on the amount of piracy; this is a fundamentally flawed measurement. The entire purpose of business is to maximize profit, and that is concerned (usually) solely with sales: reducing piracy (if you can even manage that) is of no benefit if doing so does not produce a net increase in sales at the same time; whatever the exact number of pirated copies may be is entirely irrelevant. And if you haven't managed to boost sales in the process, you're all the worse off because you're already out the money you spent trying to fight piracy.

(for those wondering, yes, the term "moral panics" is from Patry)

The Story of ThunderGraft

For those who aren't familiar with it, ThunderGraft was possibly the coolest of the three modding tools I released (the other two being MoPaQ 2000 and MPQDraft). Diablo, Diablo II, Starcraft, and Warcraft II: BNE all use PCM WAV audio (22khz 16-bit if I recall correctly) for their sound effects and music. Diablo used raw PCM WAV audio, while Starcraft introduced (and the other two used) a special "WAV compression", which compressed the audio in lossy ADPCM form, resulting in a compression ratio between 3:1 and 4:1 (though at the cost of audio quality); WAV compression, however, was implemented transparently in the MPQ API - the games saw what they read and wrote to the MPQs as simply PCM, and that's all the audio streaming API was capable of playing (in other words, the audio streaming functions were identical in all four games).

ThunderGraft was a utility that added the ability for all four of these games to play MP3s and Ogg Vorbis, in addition to standard PCM WAVs; even better, it did this in a version-independent way* (the very same ThunderGraft binary worked for all versions of all of those games). Naturally this was huge, especially in the days of dial-up (which was when the modding community first became big). You could use modern audio compression formats that yielded significantly smaller file sizes and higher audio quality compared to WAV compression. Yet ThunderGraft is all but dead now. Why is that?

MPQDraft and ThunderGraft have a few things in common. They're both very simple, clean, version/program independent, and highly effective. The reason is also the same: they both rely on very clean exploits of design to do their thing, without getting into messy exploits of implementation that depend on the precise binary patched. MPQDraft exploited the priority system in the MPQ API; ThunderGraft exploited the fact that the audio streaming API was encapsulated in functions supplied in Storm.dll.

The fact that the functions were entirely contained in Storm meant that I could cleanly capture calls to them and redirect them. For cleanliness, I opted to simply replace the streaming API in its entirety. Anything less would have been version-dependent, as it would have required all sorts of messy code modifications inside Storm's internal functions.

This meant that I needed my own decoding and streaming code to replace Storm's. At the time I just coincidentally happened to have such a thing handy. Specifically, a friend of mine by the moniker Dark_Brood was writing a game engine called Aegis, and happened to have audio decoding/streaming code handy for me to plug into ThunderGraft. He sent me a static library of the code, and in it went. Easy.

Where things took a turn for the worst was when he wrote the next iteration of his game engine. This involved rewriting a whole bunch of stuff, and integrated the various systems in the engine much more tightly (e.g. added garbage collection and other global things). This meant that it was no longer possible to simply extract the audio portion of the engine.

While in theory I could have just continued to use the old version, there was a big problem: I never had the code for the original version, nor did he save a copy after rewriting the code. This is a big problem because static libraries are compiler- and version-dependent. Now the only way I could even compile ThunderGraft was on the very same version the original was compiled on: Visual C++ 6, which is some 11 years old, now, and I haven't even had it installed for many years.

Thus, the only way to resurrect ThunderGraft would be to replace the decoding and streaming system entirely, and thus far I simply haven't managed to muster the effort. After open-sourcing MPQDraft I wanted to do the same with ThunderGraft, but was unable to readily do so for the same reason.

*There is one thing that's version-dependent in ThunderGraft, as there's simply no theoretical way to do it in a version-independent way: importing of non-WAV audio files into maps with StarEdit (the Starcraft map editor). Special support for this was required for several reasons: 1. StarEdit verifies that things imported are WAVs and refuses anything else, 2. it needed to know how long the audio file was in order for triggers to work right. The audio decoding and streaming in ThunderGraft, in contrast, was truly game- and version-independent.

& the Real World

Martial arts in a skirt, eh? I thought that kind of thing only happened in anime.

(found via http://www.darkroastedblend.com/2007/01/anti-us-north-korean-posters.html)

Random Fact of the Day

I just learned that some lizards (e.g. Gila Monsters, Komodo Dragons) are venomous.

MRI scans of a preserved skull showed the presence of two poison glands in the lower jaw. They extracted one of these glands from the head of a terminally ill specimen in the Singapore Zoological Gardens, and found that it secreted a venom containing several different toxic proteins. The known functions of these proteins include inhibition of blood clotting, lowering of blood pressure, muscle paralysis, and the induction of hypothermia, leading to shock and loss of consciousness in envenomated prey.

I'd always heard what had been previously believed - that they merely had toxic bacteria in their mouths that was transferred in bites.

Basic Word Order

Obviously, English typically has has the word order subject-verb[-(direct) object] (or SVO, for short). This can be altered in specific situations due to WH-movement (e.g. OSV in "What would you like?") and other structures, but SVO is the normal word order.

However, this isn't the only possible word order. Basic combination math tells us that there are six possible orders: SVO, SOV (e.g. Japanese, Korean, Latin, and Proto-Indo-European), VSO (e.g. Hebrew, Trique, and Caia), VOS, OSV, OVS; these are, however, not all equally likely.

Two rules have been developed which govern the prominence of different word orders. First, there's a tendency for the subject to precede the (direct) object. This is known as subject salience. There have been various theories on the exact reason for this; the general idea is that it's more natural for the subject to precede the object because they subject is typically the source of an action, and thus precedes the object in both cause and effect and chronological order.

The other is that there is a tendency for the object to sit next to the verb (on either side). Linguistics thus far has developed the notion that the object and the verb logically form a structure called the predicate, which stands apart from the subject (i.e. a clause is typically defined as a subject + a predicate). I haven't investigated the full depth of why this has been decided, so I couldn't really give more detail than that (though it's noteworthy that this idea is consistent with my hypothesis that language began as commands - the command forms the predicate, and the subject was added in later).

Thus, we have four classes of word order: those that meet both conditions, those that meet one or the other, and those that meet neither. In agreement with theory, SVO and SOV are by far the most common among languages, at 42% and 45%, respectively. On the distant second tier is VSO, which places the subject between verb and object, occurring with 9% frequency. On the again distant third tier are VOS, and OVS, each placing the object before the subject; these occur with 3% and 1% frequency, respectively. At the bottom is OSV, which violates both rules, and was not seen in any language in this survey of 402 languages.

Clearly the subject salience rule dominates in significance, as orders where object precedes subject are very rare (3% or less); although it's also true that languages where the object is not next to the verb are uncommon (9% or less).

Random Late-Night Thought

For a while I've been aware of a particular piece of linguistic evidence - namely, that cross-linguistically it is common for the imperative (command) form of verbs to be shorter than other forms, seemingly lacking inflectional affixes applied to other conjugations. For example, in Old English, the verb 'creopan' (infinitive form) is 'creope' for first person singular, 'criepth' for third person singular, but 'creop' for imperative singular. This led me to hypothesize that language might have begun as commands, and later evolved to support more general types of expressions by the addition of affixes or extra words.

Think about the significance of this for a moment. One of the most frequent differences between normal sentences and imperative sentences is that the imperative usually does not have a (stated) subject, while, depending on the language, general sentences may require subjects.

Now, one of the big mysteries of linguistics is how we came to have such radically different language systems as accusative, ergative, and topic-comment*. Yet if commands were all that was originally spoken, this provides us with a trivial answer: initially, there was only a direct object and no subject, thus that language would predate the differentiation of the three types.

As the language evolved further, eventually there would be the need to add in a subject; how exactly this was handled would then determine which of the three paths was taken. Accusative languages would place the subject in a separate case (nominative) from the direct object (accusative case). Ergative languages would classify the subject based on whether its role is the agent (ergative case) or patient (absolutive case). Finally, topic-comment languages would place the subject (the topic) completely apart from the rest of the sentence (the comment).

*Since I don't think I've talked too much about topic-comment structure, I'll briefly explain here. In topic-comment languages, a topic is stated for a sentence or set of sentences, then a number of comments are made regarding that topic. Japanese, Korean, and Chinese are like this, among others, although it's also possible to use a periphrastic form in languages like English (e.g. "As for the movie [the topic], we'll meet at 2 [the comment]").

Of particular relevance, one thing the topic can be used for is the subject of the sentence, e.g. "As for him, he'll be coming later" (though true topic-comment languages usually wouldn't duplicate the subject as English does - it would be more like "As for him, will come later"); this is frequently done in Japanese and Korean, for example. Of course, the comment may have a different subject than the topic, so topic-comment languages may also be accusative or ergative (e.g. Japanese is accusative). Here I am hypothesizing that initially the subject was represented exclusively as the topic, then further evolution allowed the subject to be within the comment itself (although whether this is true is relatively unimportant to the theory that languages began as commands).

Random Linguistic Fact of the Day

Technically English (and I think all Germanic languages) doesn't have a future tense. The future is rendered as a mood in English, using a modal (mood auxiliary) verb, in the same class as (and mutually exclusive with) "can", "may", "must", "would", etc. The past and present tenses, on the other hand, are true tenses, and both in the indicative mood (no modal verb or the "do" dummy modal verb).

The logical basis for this distinction has to do with the concept of realis. Essentially that means what it looks like: realis moods have to do with 'real' things - things which are considered certain to have already happened; while irrealis moods are not certain for one reason or another. There's a general tendency in language to regard the future as inherantly uncertain, and thus place it in an irrealis mood.

Whether this is a peculiarity of Germanic languages or is universal among Indo-European languages is unclear. In Latin there is a future tense for the imperative mood (commands - an irealis mood) as well as indicative (events that are certain - a realis mood), but not for subjunctive or supine, two other irealis moods.

For trivia value: Caia does not have tense; aspect and mood are used to imply tense, and if tense must be made absolutely certain, it can be indicated with adverbs. It has three basic moods (more complex moods are specified with helper verbs or particles): indicative, potential, and hypothetical. As in English, the indicative is used for events considered certain, and is used primarily for past and present tense. Potential mood indicates that an event is possible, but not certain; it is used for the future, among other things (although the preferred method of referring to the future is to reduce it to a certain, indicative present expression such as "I intend to go" or "I want to go", which is more precise). The hypothetical refers to events that are known to be false (hence talking about a hypothetical, counter-factual "what if" situation).

Random Linguistic Fact of the Day

Ever wonder why it's fairly common to create compound nouns from phrases (e.g. bird-watching, card-carrying), but in all these cases the object comes before the verb participle? Based on English word order it should be watching-bird, etc., yet it never is.

This is probably due to the fact that word order in Proto-Indo-European was very different than the word order used in English and most other Indo-European languages today. In particular, instead of the subject-verb-object order typically used today, PIE (along with more recent ones, like Latin) preferred the subject-object-verb word order. So you might say things like "Avem [bird] spectabam [I watched]" in Latin, which is exactly the order seen in the compounds.