The AD&D game essentially "out-competed"—or was heavily pushed by TSR, as the successor to the OD&D game. To a great degree the market agreed, and while there was a a dichotomous split in the D&D brand all the way through the end of TSR's corporate life, AD&D was clearly the face of the company, and the "other" D&D line was one that often lacked in direction, consistency and focus. This lack of focus and direction means that it's often treated as if it were a series; in reality, that's an artifact of chronology rather than of intent.
Even at the Acaeum they split D&D products into three big chunks; OD&D, D&D and AD&D. This split is not entirely justified. Some of the D&D line was specifically meant to follow in the footsteps of OD&D, and should be seen as part of a series with that, in terms of intent, presentation, and actual content. Some of it represents what Gary Gygax himself said, in the famous Sorcerer's Scroll column in The Dragon June 1979, or elsewhere. Sometimes EGG wasn't himself consistent.
The Holmes book, sometimes called BD&D (because it was the Dungeons & Dragons Basic Set, ergo BD&D—Basic D&D) was deliberately meant to be an introduction to AD&D for those who would otherwise be overwhelmed by it. (It says so, right there on page 6.) It was also meant to be a clean-up of what OD&D had become, and it was meant for a less sophisticated, and potentially younger audience. Because of this, it's a little hard to see this as belonging to any series at all; it really is a weird kind of stand-alone product that sits oddly and uncomfortably between OD&D and AD&D. Some reasonable speculation that's impossible to corroborate at this point suggests that it was really meant to be a clean-up and reorganization of OD&D, and that the "gateway to AD&D" aspect was mostly grafted on after the first draft was prepared, leading to a game that does the former much better than it does the latter.
The Moldvay Basic (paired with the Zeb Cook Expert to form the B/X version of the game) replaced the Holmes Basic, and was a much more substantial revision to the rules themselves Eschewing the idea of being a gateway into AD&D, it instead stood out as a line in the sand against the tournament trend of AD&D; it deliberately was a modification and update of the tone, feel and concepts of D&D, without much direct reference to AD&D at all, or to the Holmes book either one. It's no surprise that this is my favorite version of D&D to come out of this entire era; it's the vibe and feel of OD&D with better rules. (It needed to be revised one more time, honestly, with even better rules. One could argue that maybe Basic Fantasy Role-Playing Game is this edition.) Because when paired with the Expert set, the Basic set allowed for play into a relatively high level, this is in every sense a complete game that stands apart from the direction in which AD&D was drifting. It's a mistake to think that, even though EGG said so numerous times in print, they were mechanically really all that different, though. Most people I knew didn't really understand (or necessarily care) about the differences between the various versions of D&D, and their games in actual practices were chimerical in nature, freely borrowing elements from any as it suited their tastes, preferences and needs. This was also partially due to the fact that AD&D failed pretty spectacularly in its stated goal of eliminating diversity and rules questions; largely because it was so poorly written and organized that it actually probably created more confusion rather than less, and players would often default to a D&D paradigm if the AD&D one wasn't clear or readily to be found, or too much work to implement.
The B/X series was supposed to have had a third manual, a Companion, that would have extended the levels even further, and added more material to accommodate that higher level play. This ended up being a ghost product that never appeared, but it would have been pretty cool to have seen B/X evolve into BXC. Of course, I'm personally unsure of the necessity—or even the advisability—of higher level play than B/X already accommodates, but the B/X game was headed towards a kind of Conan-esque approach, rather than the AD&D save the world from bad gods approach, or the BECMI approach of apotheosis. So although I still probably wouldn't have used it much, it'd have been nice to have, and it would have also completely and totally enshrined the D&D approach (as opposed to the AD&D approach) as a completely viable alternative.
One could say that BECMI, the Mentzer boxed sets that replaced the Moldvay/Cook/Marsh B/X approach was that, but I think it in many respects, clearly was not. Although it did replace the B/X with it's own B and E, and added to them with C,M and I, the tone had completely changed. Whereas the B/X sets had a sword & sorcery feel, with dangerous characters, and were marketed towards adults (said so right on the cover) with the Mentzer sets, they were deliberately "dumbing down" D&D for a younger audience, no doubt part of their plan to continue to make AD&D the real flagship product.
Although details of it aren't public, of course, I've often suspected that due to the acrimonious partings between Gygax and Arneson and later between TSR and Gygax himself, included a lot of provisions that had the company producing products that it didn't really want to produce. I think the entire strategy of the split between D&D and AD&D was meant to keep D&D in print because they legally had to, while also maximizing sales of AD&D, which was oversold as a "totally different game", and therefore one in which Arneson was owed no royalties. Although I don't know what primary source she's quoting from, when you go to buy the Basic set as pdf, you get this historical context: "Mentzer claimed that the main reason behind this new edition of Basic D&D was that previous versions 'were not 'revised', merely 'reorganized.'' He clearly wasn't talking about the mechanics, which demonstrably had been revised in Moldvay's version of Basic D&D, but instead how the game and its rules were structured. Mentzer's version of Basic D&D thus made some large changes to how the game was taught and presented. Menzter's first two goals for the new Basic D&D were to make the game approachable by beginners and to make it learnable from the rules. Mentzer's Basic Set is thus laid out almost as a tutorial, with new rules and concepts being introduced to players very carefully; the rules about GMing are then introduced only after all of the basic player concepts have been discussed."
I'm not very personally familiar with the Mentzer rules, but what I hear of it was that it was more a difference in presentation than in system. It was deliberately dumbed down, as I said earlier, almost to the point of condescension. It was deliberately aimed at a younger audience. The tone changed from one more informed by the literature to one that resembled "Extruded Fantasy Product." It did have the Product Finishing department, so the garage-band feel of the earlier books was somewhat replaced with a more slick, professional-looking product (for better and for worse.) And it did add some complications such as skills and weapon mastery. But the rules changed relatively little from B/X to BECMI—at least if you stop after the first two boxes.
The Rules Compendium, or RC, was basically another repackaging of the BECMI boxed sets into a single book. Although it made a few minor changes, dropped a few rules, and picked up a few that came from sources other than the BECMI boxes themselves, the rules are highly compatible. In fact, the compatibility between B/X, BECMI and the RC are generally assumed to be among the highest for any versions of D&D; what changed following B/X was the tone, the implicit setting to some degree, and the "feel" of the game. This is, of course, less important than some make it out to be, because the feel of the actual game as played is much more informed by the players and GM than by the rules, especially if the rules are essentially the same.
It does strike me as a bit unusual that by the time the C, M, and I were added the BECMI (and were later reorganized as RC) you've actually got a fairly complicated game, comparable in many ways to AD&D—although with some major differences, of course. It also strikes me as unusual to think that by this time, you've basically turned D&D itself into a "fantasy heartbreaker", which seems to be a complete reversion of expectation. But since AD&D became the flagship and D&D itself became the often neglected game, it lost its developmental focus, it lost its clear ties to what was later called OD&D, and it became, in a way, a pale shadow of what AD&D was. Sadly, if the direction of the B/X game had been maintained; if the Companion had come out, and if the game had remained on that plane for some time, we would have really seen something unusual and potentially very, very interesting.
Quite honestly, I'd have been playing that game instead of AD&D back in the 80s. I like the spirit and feel of OD&D the best of all of the versions of D&D, but I don't particularly like the rules. The Holmes set reorganized this stuff so that it was more comprehensible, but then strangely cut a huge chunk of the material out so that you had to graduate to another rule-set to keep playing (the game itself touted AD&D, which was on the verge of being released, but since it hadn't actually been released yet, and wouldn't be completely released for a couple of years, it probably shunted some people into the OD&D game instead. B/X rejected the spirit of AD&D and went back to the original D&D paradigm, and updated and revised the rules. Still not as much as I'd like, but as I've said many times, this was the closest D&D got to getting it right in terms of what I wanted from the game, at least.
Later, with BECMI and RC, D&D lost that spirit and feel (even as it retained, mostly, the same mechanics) and eventually became essentially a poor imitation of AD&D itself. Given that that's my perspective, it's not hard to see why I became disillusioned with D&D altogether and eventually wandered away. Third Edition, where the AD&D and D&D brands were finally brought back together, enchanted me for a time until after many, many hours of play it became obvious that it really was more about the feel and tone of AD&D than it was D&D and I wandered away again. I still had plenty of fun with that system, and given the gaming group that I have now, I probably will again, but it's never going to be a system that I recommend anymore. If anything, my tastes have evolved into being even more free-wheeling, rules light, GM interpretation than even OD&D did, although built on a more stable, robust chassis of mechanics that are consistent and easy to use (curiously, a really stripped down version of 3e, which is what m20 essentially is.) It took a long time for me to arrive where I am, and there were both changes in me and my own tastes as well as substantial changes in the game itself to get me here.
UPDATE: Another guy posted somewhere, long ago, and I found it and read it and wonder how much it's true, that as BECMI matured, it started getting all kinds of rules; rules for splitting pirate loot, rules for running a merchant marine, rules for everything! If this is true (maybe I should buy the RC pdf; it's on sale for only $10—then I'll know first-hand) then D&D evolved near the end of its life-cycle into one of the ultimate simulationist systems of its age. This truly does bring D&D and AD&D together in spirit—and leaves the original paradigm completely unrepresented. No wonder the OSR thrived so much when it first started! It wasn't so much about nostalgia as I always thought, but also catered to a lot of pent-up demand for a system that was specifically designed to work best with a specific play style—which also happened to be the original playstyle.