The prolific Nils has a few things to say about MMO death penalties. At the risk of trivializing all that work, there's one idea I want to focus on (which Nils himself pullquotes) - the concept that an ideal penalty is an effective deterrent without actually harming the player or the game experience in the long run. It's a great idea, but one that I think is tough to implement in practice.
I've argued in the past that basically all death penalties that anyone has implemented in an MMO, whether gold, exp loss, gear decay, or even permadeath, can be described as a loss of the time it will take to get back to the state you were at before your untimely demise (whether that means killing a few mobs or re-rolling from level 1). In principle, developers could tack on a monetary component - for example, branding all future characters on the account with the scarlet noob label, so that the player would have to pay for a new account to escape their reputation - but I suspect that this option would not be tolerated by the overwhelming majority of the market.
An Example in ROM
Of the MMO's I play, the harshest death penalty is probably in Runes of Magic, where you suffer 5% of your next level in exp debt. A third of this amount is forgiven if you loot your tombstone, which does not expire, even if you log off and don't come back for months, so there's no irresponsible pressure to stay logged in to collect it at that moment. Assuming that you did not die due to a bug or a quest spawning an elite mob on top of you with no warning, the system is fair, and only requires the player to continue doing something that they enjoy doing anyway (obtaining exp). The problem is that the system is telling me that I should play the game in an overly cautious manner that ultimately makes the game less fun.
At the moment, my level 52 druid is looking at about 9 million exp for her next level, which means a bit over 300K per death in debt, assuming that I recover my tombstone before dying again. A typical daily quest is going to award somewhere from 40-70K exp, so I'm looking at a full day's worth of daily quests to pay off the debt. (Another option is to join a guild with a library in its castle, which forgives about 100K/hour of AFK time at my level, if you have something else to do in another window or offline while you wait.) The message is clear - stick to easy stuff with zero chance of failure or spend hours of time paying for your ambition.
As I've said repeatedly since I've revisited ROM, there are times when no risk, low rewards gameplay is vaguely amusing. The real fun, though, is pushing the envelope to see what exactly I can pull off. In general, at 52 I can beat level 47 elites and I can't beat level 48 elites, but there's no way to be sure unless I try. If I do try and I'm wrong, I'm out 300K exp and faced with a choice of whether I want to risk doubling my losses. If I don't try until I've gained a few more levels, it's likely that the eventual victory will end up not being all that challenging.
The bottom line is that I do less of what I enjoy about the game because of the penalty structure, and I ultimately spend less time playing the game as a result.