"Mr. Joshua, why in the world would you want to use the rules of D&D for horror?" I can already hear the cries (actually, I can; I started a discussion about this on one of my gaming messageboards, and right off the bat, the idea of "why don't you use an existing horror game?" came up.)
The reason is: everyone has D&D. Everyone who games, that is. The rules are right there, easy to find and nobody needs to relearn them. And even if you don't, hey, online SRD. They're free. Plus, D&D has tons of horror influences. Vampires, zombies, demons, Cthulhu-esque monstrosities; the game is rife with this kinda stuff.
What D&D doesn't have is a paradigm appropriate for horror. In D&D, you're expected to charge heroically into battle against the horrors of the night and smite them or whatever. In a horror genre game, you do that, you're dead or worse. Every time.
I think a few other small mechanical changes can help encourage players to keep in mind the paradigm that, even though we're using D&D rules here, we're actually playing Call of Cthulhu.
1) Level isn't necessarily something that players scale like a ladder. I can pick a level that has the PC power and capability that I think is appropriate for the game, and peg it there. All players create third level characters. You don't ever level up. Ever.
2) That doesn't mean that you can't have character advancement. You still get experience points. You can spend 1,000 to gain a skill rank (up to the maximum skill rank at 3rd level, which for class skills is 6 ranks) and you can spend 5,000 XP to gain a feat.
3) Forget CR. Forget wealth by level. Forget dungeoncrawling for profit and thrills. They are completely inappropriate for this genre. For that matter, be very cautious about combat. A primary theme of horror is that you're in over your head before you start. A straight up combat, unless it's with regular guys, cultists, or whatever, is one that you can expect to lose. You've got to be more clever than that.
4) 3rd level means any magic using classes have 2nd level spells. At best. If, for some bizarre reason, you really need higher level spells than that, they'll have to be converted into incantations and cast outside of combat. And expect to pay a hefty price for them while you're at it. Magic is unknown and scary, not a utilitarian tool for PCs to wield with impunity.
5) Still casting around for a potential fear/madness mechanic. I've actually got lots of options; I'm trying to figure out which one I like best.
Anyway, any comments? Other than that the image I picked for this post is only, at best, tangentially related to the topic at all by virtue of being a Google Image Search result for horror?