One of the 3.5 books that you'd think would have been right up my alley, but which I'm actually just now reading for the first time, is Heroes of Horror. While much of the advice is about what I expected (i.e., I hardly need it) there are a handful of interesting tidbits of stuff here and there. For instance, check out the following campaign model, "Death is Only the Beginning" followed by the soul-locked trait, which it references.
DEATH IS ONLY THE BEGINNING
Nonhumanoid monsters are extremely rare in this campaign, and each is likely unique. Perhaps they are the results of the gods’ anger or of curses or foul magical experimentation. Whatever the case, each one is practically impossible to kill; every monster is considered soul-locked (see page 47). Only those with the skills and fortitude to investigate each monster and determine what methods can be used to permanently eliminate it can hope to rid the region of the beasts that terrorize it.
The Soul-Locked Creature
If you really want to hammer home the notion that a particular creature is almost unkillable—or the fact that violence is not a winning solution—consider giving some of your monsters the soul-locked trait. This is something like a template, since it takes a preexisting monster and modifies it. It makes only a single change to the creature, however: namely, it cannot be killed under normal circumstances!
Soul-locking functions similarly to a ghost’s rejuvenation ability. Whenever the creature dies, it makes a DC 16 level check (d20 + HD). If it succeeds, the creature returns to life—or undeath, or animation—4d20 days after being slain (or twice that if the body is completely destroyed, such as by disintegrate or immersion in lava). Each soul-locked creature has one specific way in which it can be dispatched permanently. In some cases, this might be a particular type of weapon, or even a specific weapon. In most instances, the creature can be permanently defeated only through indirect means. If the monster is the manifestation of a familial curse, it can be banished forever only by making amends for the sin that called the curse down on the family in the first place. If it’s a demonic entity, the heroes must find the ancient symbols that drew it to the Material World and destroy them. If the beast is a creature of taint, the characters might have to cleanse an entire area of taint or lure the creature to a sacred purifying spring before it will stay dead.
Even though being soul-locked makes a creature almost impossible to permanently destroy, it does not increase the monster’s Challenge Rating, since it is no harder to defeat the monster in any given encounter. You should give the PCs extra XP as a story award—perhaps an additional 25% over and above the XP for defeating it in that particular encounter—when they finally best it permanently.This is very much up my alley, although it's not necessarily horror themed. The Percy Jackson books basically use the same model, after all.
I'll repost text from a post I made many years ago now which summarizes my take on monsters, because the two approaches complement each other a great deal. Anyway, the reposted text starts below:
A problem that I often see in roleplaying games, and frankly, in D&D in particular, is that monsters completely fail to be scary. Rather, they are viewed as tactical gamepieces and challenges. Arguably, this is what Gygax and Arneson wanted all along, but it is not something that I want, and I think the idea of reducing monsters to a statline that has tactical implications is to make them not monstrous at all.
Granted, it doesn't have to be like this. I've managed to engender some of the same kind of dramatic tension in D&D games (occasionally) as you get while reading a scary book or watching a scary movie... but frankly, not very often. Part of it is the paradigm and attitude that the players bring to the table; if they're playing D&D, then there's an assumption that they'll be facing challenges that they should be able to overcome if they're smart and tactically sound. It's just the tone and nature of the game, or at least it's often expected to be so.
I like monsters to be scary. I like players to really question whether or not they want to fight these monsters. But I admit that my success, what of it there has been anyway, in accomplishing this is something that I've done more intuitively rather than rationally, and I'm not 100% certain that I fully understand how to pull this off. So for this post, I'm going to noodle around some ideas and see where they go.
1) Monsters should be set pieces. The idea of going through a "dungeon" and fighting monster after monster has really diluted the concept. In any good story of supernatural horror, the monsters are never routine. Be careful about showing off too many monsters. Granted, D&D has truckloads of monsters. But don't assume that all of them are hanging around waiting to be discovered. Think of how the ancient Greeks did it. There were not "medusas"; there was one Medusa. There were not hydras, there was one Lyrnean hydra, etc. While you don't need to go to this extreme, keep in mind that monsters are much more monstrous if they are extraordinary. No matter how monstrous the description or the statline makes them appear, they never will be if they're routine. If you're not fighting monsters all the time, what are you having the PCs spend their time doing otherwise? Bad guys! Thugs, cultists, criminals, spies, and the like. Don't underestimate the value of contrasting monsters to larger hordes of plain old bad guys. Besides, bad guys are fun to fight for the most part on their own too. And also don't underestimate the challenge of dangerous animals. In this era of high powered hunting rifles, enclosed off-road vehicles, and completely tamed terrain, we forget exactly how dangerous it would be to come across a herd of wild elephants who feel that their calves are threatened out in the middle of nowhere with nothing to defend yourself with except a few sharp pieces of metal that you need to hold close to yourself to use. Or how dangerous a pack of wolves could be to a lonely traveler, or a pride of hungry lions.
2) Closely related to that, monsters should be unknown. There's few things more prosaic than the GM of a game casually announcing what his monsters are, when their properties, strengths and weaknesses are well known to all the players. Does this mean that you should only use unfamiliar and unusual monsters that you make up yourself or find in obscure third party sources? Of course not, but you should take steps to, again, keep your monsters from feeling routine. Think about possibly making them difficult to identify for a time. The PCs don't actually see them well until they're well into the thick of it, but they see the effects of their attacks on NPCs or something like that. Mix up your descriptions so that the PCs can't easily match them to a monster that they know. Statistics and descriptions can be decoupled and rearranged. One of my most memorable encounters was with a handful of hellhounds that I simply described very differently—I used some artwork from Paizo of Lovecraftian hounds of Tindalos to represent them, gave them a chittering Predator-like growl, and had their fiery breath transform into a vomit of tiny, toothy little demons that crawled all over their victims. Consider giving some of your monsters surprise abilities. A zombie that has a poisonous bite, or something. It doesn't have to be a big deal, just enough to keep the PCs guessing and unsure of what exactly it is that they're up against.
3) And closely related to that, make your monsters horrific. Granted, another of my most memorable scary monster encounters was with a little blighter that due to some relatively weak stats and some extraordinary good rolls by the PCs, ended up going down like a chump in the first or second round of combat, but that was the exception not the rule. He was creepy because of all the other stuff I had him do, but by and large a monster that the PCs aren't sure they can beat is one that's more likely to scare them than one that's only going to "reduce their resources by XY%" or something inane like that. In fact, ignoring the advice of the Challenge Ratings system completely (which is a good idea for a lot of reasons, only one of which I'll get into here and now), you can pit the PCs against foes which they literally can't defeat in straight-up combat. Monsters shouldn't often be creatures that cause heroic PCs to shout, "Huzzah!" and charge at to engage in melee; they should be monstrous. There should be a lot of doubt about how to deal with them, if they're up to the challenge, and what exactly can be done to get around the many strengths that monsters might have. Rather, you can have the PCs need to research specific dirty tricks or weaknesses that allow them to have a chance against the monster; without which they'd be committing certain suicide. A demon that can be banished back to its home dimension only via a desperately hurried ritual is scarier than one that can be banished back to its home dimension after the PCs just jump in and attack it.
4) And that gets a bit into my last tip; foreshadow your monsters. There's nothing worse than having a monster just pop out of nowhere, get defeated and then promptly forgotten. The chump monster that I mentioned above? He was mostly memorable because of the excellent (and extraordinary; I haven't quite had this level of success many other times, sadly) foreshadowing. While the PCs were attempting to sleep at a crowded and sleazy dockside inn, they were woken up when a young woman in the room next to them screamed. Barging into her room, they found that her eyes had been scooped out of her head. There was no sign of any attacker, just the sobbing victim's almost nonsensical cry that the last thing she saw was a hideous face over her shoulder in the mirror. A few more clues, a bit more foreshadowing... and then a half-glimpsed movement in their own mirror, and the players were keyed up, tense, and on edge. When it turned out that their invisible assailant was actually killed rather easily, it didn't diminish the feeling of dread and creepiness that the encounter had managed to elicit. And this foreshadowing harkens back to my point #1; if monsters are set pieces, then you can craft an entire "adventure" around finding and defeating one, which means that you should have plenty of opportunities to foreshadow, to drop in unsettling or horrific clues or feelings, and generally ratchet up the tension on your way towards the final confrontation.