For many, many years, I've been fascinated with the concept of insanity in RPGs. This is indubitably most iconically represented by Chaosium's Call of Cthulhu game, where you make Sanity checks, and failures cause you to gradually lose your sanity.
However, as an admitted nut for the idea of dark fantasy, which by definition is a genre-bender category sitting somewhere between fantasy and horror, I've been fascinated for years with the idea of adapting some form of this concept into my fantasy gaming. This would, of course, give it a much darker, more horror-like feel, but I'm OK with that. In fact, I find that situation highly desireable, at least as a "sometimes" alternative.
The d20 Call of Cthulhu book was the first such attempt in the d20 system (that I know of) to model this idea, and it just grafted the d% BRP version of Sanity made famous by the original Call of Cthulhu system into d20 without any changes at all. This made for a somewhat clunky and inelegant approach, but it works. In fact, it works sufficiently well that WotC later "opened" the content officially by reproducing those rules almost word for word in their Unearthed Arcana book. Since they were opened up and UA was effectively added to the SRD, you can get them online for free easily enough, here being my favorite place to get SRD rules.
But, I became gradually more and more dissatisfied with them over time because of the clunky and inelegant design. Plus, they're actually kinda long, and way too clinical. Someone (probably Sandy Peterson, or one of the other original designers of the Cthulhu game) got a little carried away with the psychological jargon and detail. A little 11-page pdf by Ronin Arts and written by Bruce Baugh used to be available that took care of the first problem; it "nativized" the rules. But, it did it as closely as possible to the original version. So, while I find that ruleset is still a better alternative than the one linked above, it's still too clinical. Plus, I can't find that it's for sale anymore; it probably bit the bullet back when the d20 license cratered. Of course, that doesn't bother me; I've already got a copy. But I can't exactly recommend rules that aren't available, unless I'm willing to cut and paste them somewhere under the auspices of the OGL. Besides, there's a better alternative still.
I was poking through my copy of the d20 Freeport Companion (still available slightly renamed as the 3rd Era Freeport Companion from Green Ronin directly, or from RPGNow.) I needed to for my big class list post from a few days ago. After I looked over the classes, I turned around and looked at some of the rest of the rules, and I forgot that there's a condensed and simplified version of the madness rules in here. They take up exactly four pages, at least half of which is charts and a medium sized illustration. And they actually do a rather remarkable job of being faithful to the original Cthulhu rules, except without being clunky, cumbersome, overly jargonistic and technical or otherwise difficult to use in any way whatsoever.
I can't say that I'd be interested in using these rules every time I play, but sometimes I really like the idea of mechanically quantifying the horror of the game by watching the PCs struggle to maintain their sanity as well as their lives because of the choices and risks that they take. And when I do, I've got a go-to option here, completely compatible with and native to my system of choice, yet which honors and feels like the most iconic version of sanity rules ever published for the RPG medium.
NOTE: Actually, after posting this it occured to me that the d20 Call of Cthulhu rules were not the first attempt to model insanity in the d20 system that I'm aware of; the Wheel of Time game did it first. And very simply; it was just a half-page sidebar, if I remember correctly. That doesn't change my recommendation of Green Ronin's 3rd Era Freeport Companion for d20-compliant insanity rules, though. For one thing, it's a lot easier to find Green Ronin's book than the old and long out of print Wheel of Time book. For another, the Wheel rules were almost too simplistic. They had the opposite problem of the Cthulhu rules; in their effort to be playable, they were too bland. Plus, there wasn't any provision for ever really recovering from insanity. And I like the fact that Rob Schwalb's rules in Freeport actually feel faithful to the Cthulhu rules, even though they do so in a condensed and simplified fashion rather than in a technical fashion.
Basically, they're similar rules, made native to d20, but without the Psychology 101 lecture thrown in.