There are a number of factors that have caused the decline of these games. Technology in the form of online roleplaying games (Blizzard's World of Warcraft) being a critical obstacle, and the advent of the more portable, more casually convened collectible card games (Wizard's of the Coasts' Magic the Gathering) drove money into the fantasy market, but also swallowed up more than its share of the money.
But there is another interesting aspect to the problems with pencil-and-paper role playing games that mirrors some of the problems of our culture.
The Table Top Roleplaying Game Hobby is Contracting
Ryan Dancy, EN World Forum, December 2011 (Hat tip: MR).
We began to view the market not as a series of product pyramids...but instead as a series of human webs that overlapped and interconnected. Where those webs were strong, the products flourished.... The limiting factor was human brains within which theses games could interconnect.
The more segmented those brains became, the weaker the overall social network was. Every new game system, and every new variant to those systems, subdivided that network further, making it weaker. Between 1993 and 1999, the social network of the TRPG players had become seriously frayed. Even if you just looked at the network of Dungeons & Dragons players you could see this effect: People self-segmented into groups playing Basic D&D, 1st Edition, 2nd Edition, and within 2nd Edition into various Campaign Settings that had become their own game variants. The effect on the market was that it became increasingly hard to make and sell something that had enough players in common that it would earn back its costs of development and production.
We looked around the industry and saw the same problem at virtually every company that had become successful: White Wolf had 5 World of Darkness games which were all slightly different, surrounded by a more diffuse constellation of games somewhat related to the Storyteller system but designed to be mutually incompatible. FASA had 4 games, none of which shared anything in common. Palladium & Steve Jackson Games both had “house systems” that they tried to use across their entire product lines, but they had ended up with the “Campaign Setting” issue that was bedeviling TSR; the variant rules at the edges of their games were creating independent game networks despite the shared DNA of the core. And we knew that inside every one of those companies they were seeing the same financial information we were seeing: Each new release was selling fewer and fewer copies, and in response, the companies were increasing the pace of releases trying to sustain planned revenues by volume of titles, not by volume of units. And it was killing everyone.
…My opinion is that the hobby gaming industry is going to transform into a very small niche business. It will cater primarily to an aging group of players who have made TRPGs their lifetime hobbies. As those players age, they’ll need less and less support in the form of commercially produced products. They will instead seek out community support tools to help them remain in touch with their hobby even as the social network they’re directly connected to becomes ever more frayed.
One of the huge attractions of the role playing game was the enactment of shared but personal fantasies with like minded people. But there is a commitment in time that our society works against. While it is true that the computer games, and card games don't offer the camaraderie of the table top games, they offer a reasonable facsimile, and there involvement involves much less preparatory commitment. Combined with the self-fragmenting nature of the hobby (over specialization), the group-based activity was bound to suffer.
Our culture is very much similar. Offered an enormous amount of choices, any popular group activity is likely to feel the splintering effect. The easier the activity is to initiate the more likely it will sustain itself, or possibly even expand. If the activity can either be performed on, or initiated through electronic media will have a huge advantage,
On the plus side for table top role playing games, at their esence they are not much more complicated than sitting around the fire telling stories. You don't even need a table. To the extent that idea has been born - to have a group share in the creation of a story - there is no reason that it cannot be reborn endlesssly in simpler non-commerical form. The fireside tale is not a common practise today's western world, but to the extent that there are people around, it is not likely to ever completely die off.