For my first foray into Dungeoncraft, I need to depart substantially from Ray's articles. He starts off the series with a discussion of whether or not you should GM at all, and if so, what books you need to acquire (from the 2nd edition D&D line.) I'm going to assume that the first question has already been answered, and I'm also going to assume that you may be playing any system out there. My sample game will use D&D 3.5 as a baseline, but this exercise could be done with any game system that exists.
Rather, I'm going to talk a bit about another interesting phenomena in my system of choice, namely that there is so much d20 compatible material in print these days, much of it by "third party" publishers unrelated to Wizards of the Coast, that deciding what rules to use, at least for an experienced GM, is a big decision that isn't necessarily as simple as turning to the Player's Handbook. Of course, you could do very well with just the Player's Handbook, so why would I necessarily even want to look beyond that?
I think there are essentially two reasons to do so: variety and flavor. With so much variety available in print, lots of experienced players want to try more exotic options anymore. In print we have tons of books that detail custom races, alternate core classes, and of course, there's literally hundreds, if not thousands, of prestige classes in print. Of course, you have to be careful with what you allow here, as well; too much variety ends up with a mish-mash setting that you might have difficulty managing, or turning into a setting that's coherent or makes any sense at all.
Flavor's a more interesting item to discuss, of course. Looking through my collection of d20 books, I can see all kinds of flavor exuding from the various types of books I have. d20 Call of Cthulhu for example, is a dark setting where fear plays an important role in the campaign (both because of the Sanity mechanic, and because characters are typically much weaker and more vulnerable to the hazards of the setting than your typical D&D characters.) My Iron Kingdoms material has a gritty, early-Industrial edge to it, and lots of mechanics support that flavor. Rokugan likewise has a very specific pseudo-Japanese fantasy flavor to it; including a wide variety of alternate classes and races. That's just naming a few options off my bookshelf, but you get the point. Flavor is something that needs to be the first decision you make, which in turn will drive what rules you will allow in the game. Although this decision is probably best made in tandem with the decisions we'll talk about in our next installment, for now I'll assume that you can make a flavor-based decision based entirely independent of any other decisions about your campaign setting, and go from there.
Another thing to keep in mind if your flavor choices take you fairly far afield from the standard Player's Handbook style of rules, is that it could be difficult to keep what rules you are allowing straight and coherent. If I use three Player's Handbook classes, four Rokugan classes, alternate races from some other three sources, and prestige classes from who knows how many others, you might be spending a lot of time flipping through multiple books during play, which isn't really an ideal situation.
But here, if you don't mind spending a few hours going through the exercise, the Open Game License can come to your rescue. Compiling all the alternate classes and races into a single document that you can give to your players takes away the need for extraneous flipping and page turning, as well as making it very clear what is being allowed. By using all open content, you can also make this document completely legal under the OGL. Of course, preparing this initial document is a fairly tedious and lengthy task. For this reason, as few house rules as possible is certainly an easier, if not necessarily better, option.
Of course, there's also the fact that any given player will only need a small subset of the houserules available to him. If he's playing an alternate class, he just needs the rules for that class, not all the rules for all the classes that you use. So, in play, the problem is rarely as exacerbated as it initially seems like it could be.
How do I suggest going about it? First of all, I suggest you sit down for just a minute and decide what from the Player's Handbook you want to change and why. Take a real hard look at it; don't change just for the sake of being different, change it only if you really want a different feel than the standard game provides. Change only those things that directly impact that, and leave the rest alone.
Now let's look at an example; my prototype campaign setting that I'll develop using this methodology, the Dark•Heritage setting. First, I need to decide on a flavor, a tone or a feel for the campaign setting. In my case, that's an easy decision, as I have a clear favorite tone when it comes to fantasy, and anyone who's read this blog even a little bit will probably have picked up on that. I'll go for a low magic, low fantasy feel; a darker, grittier type of fantasy, somewhat like Glen Cook's Black Company novels, or George R.R. Martin's work, or China Miéville's books. In fact, paraphrasing Miéville, I want to create a setting that is almost "the anti-Tolkien" in some ways; although I really like Tolkien and Tolkien-inspired fantasy settings, they are also extremely prevalent, and I want something different this time around. I also want to include a "conspiracy theory" element to the campaign; ancient peoples that have meddled in the lives and fates of the people of the setting for generation upon generation, for their own dark purposes. Finally, I also want a sooty, "almost Industrial" feel, to further differentiate the setting from the standard Tolkienesque setting. So, with this desired flavor, what changes do I have to make?
Looking at the races chapter, I readily see that the Tolkien influence is perhaps strongest in this regard. In fact, the only race that I find works for the flavor I want is human. So, out with the others, and in with a collection of new ones. In this regard, I have lots of information I can borrow from other settings or books, but creating races is fairly interesting and easy work anyway. Mechanically, there's little that needs to be done. Some racial ability modifiers allows for some flavor and variety to create options your players can choose from. Each race should have a small selection of special abilities as well. There's a few caveats to this process, however. First, all ability score modifiers should be in increments of 2. Because of the way d20 works, you need to change the ability by 2 to consistently get a change of 1 on modifier for that ability. Also, keep in mind the way you will play the game. In most games, the three physical attributes (STR, DEX and CON) are somewhat more useful than the other attributes (INT, WIS, CHA) but if you intend to run a game in which social interaction and intrigue are as important as combat, than you make the judgment call on what an appropriate trade-off in abilities is (for most standard races, a +2 in one ability is countered by a -2 in another ability.) Besides the ability modifiers, each race should have a few small flavor mechanics as well. For example, humans get an extra feat at first level, and an extra base skill point every level (which is multiplied by 4 at first level, as always.) While nice, these aren't supremely powerful abilities, and unless you want to get into effective character levels, you should stick to balancing your races against the human. For my setting, I want to focus more on humanity, but the other races that I allow are "evolved" or changed humans, really. Hellspawn (tieflings), jann (fire genasi), vucari (shifters) and possibly one or two others sit alongside regular humans of multiple cultures for Dark•Heritage.
Looking through my collection of classes, I can fairly easily find a few classes that I can simply use as is with very little modification. So I add the classes from the Expanded Psionics Handbook, the Complete series, and a few others from the Freeport book, the Midnight book and a handful of others. I end up with a pretty nice list of classes, although I have to admit up front that several of them tread on the conceptual toes of some of the others a bit. I'm not worried about that, because I just see them as subtle variations on the same idea that could exist between individuals who bring slightly different training, techniques or traditions to their profession. I'll not worry about prestige classes yet, because there's too many options for me to summarize at this point, but if characters decide they want to start looking into available prestige classes, I'll sort through my material and allow a few to choose from, based on what their specific character goals are. Adding the E6 alternate character advancement top hat to my system, I greatly reduce the effect of runaway high level characters and runaway magic as the game goes on. Ideally, I might consider removing the spellcasting classes altogether, but that's a more drastic option that I'll keep on the backburner for the time being.
To give me just a bit more flexibility with magic (as well as to allow for the possibility of powerful Ten Who Were Taken level sorcerers), I'll allow Incantations as a house rule, which can allow me to introduce any higher level spell I need, or create my own, although most will come with steep personal costs to learn and to cast.
And finally, if I want a pre-Industrial, or early Industrial feel, I might want to include some primitive firearms. Taking the rules for these from Green Ronin's Freeport book, I find firearms that do lots of damage, but can only be fired once, or at most twice, per combat, and come with potentially disastrous misfire chances as well. The misfire rules are characterful, but ultimately a bit fiddly; most likely I'll ignore them completely.
With these fairly extensive house-rules, I'm ready to move onward. Because I already know that not all of my players own all of the books that contain rules I want to use, I'll want to provide an OGL-compliant document that has all my custom races, borrowed classes, magic and equipment rules in order to create their characters. Although this will take a little bit of time, and is a fairly tedious task, once it's done I can move on to more fun things again. You may not want to include nearly as many house rules for a variety of reasons; 1) you don't own as many books as I do to pick and choose from, 2) the flavor you desire isn't as different from standard D&D as mine is, or 3) you don't want to put as much work into it as I do. All of these are clearly valid reasons, and I would likely be in the same boat for many other settings I'd develop, but in this case, as I'm specifically looking for something different as well as demonstrate the flexibility of the d20 system, I'm willing to step quite a bit out of the pale for a unique experience that hopefully plays quite differently from regular D&D and is a memorable experience for my players. In the meantime, most of these houserules are already provided here on this blog; click on the house rules label, ignore all the posts about Anti-d20, and you'll be good to go. They can also be found on my campaign wiki here. Because of ongoing tinkering, the two sources... the wiki and the blog, might have very minor differences, and a few rules only appear on one site or the other but not both. Clearly, this isn't an ideal situation to be running the game, but since I'm not actually running it currently, I'm OK with leaving it in a somewhat disorganized state for the time being; I'll organize it better when I'm actually ready to use it.
When I come back next time, we'll be at step 2, in which the basics of the setting are established. You'll see at that point, that what I did there and what I did here really dovetail together. While that does make for some frontloading of the work required to develop a campaign setting, you'll also find that I'm a strong believer in only putting as much work into it as you need to, to avoid burning out before you get done. Not only that, the next stages are more fun!