5E D&D’s Playtests – Which Was Better?

Before 2024’s version of 5E Dungeons & Dragons, there was the One D&D playtest. And before 2014, there was the D&D Next playtest. These were two very different playtests aligned with different goals and producing very different results. Now that time has passed, let’s look back and see what we can learn from comparing them.

But First, a Thanks for SRD, Sage Advice, and Errata

The D&D team has been awesome lately, providing us with the promised SRD 5.2 so that creators can more easily create third party products for 5E 2024 (I would love an RTF version of the SRD, please). In addition to an excellent landing page with guidance, we also have a Creator FAQ. And, we now have Sage Advice and Errata, including all of the old errata files in D&D Beyond and PDF format! We even have an updated version of the Basic Rules, which is fantastic (I would also like to have a PDF and RTF version of the Basic Rules). Huge thanks to Wizards and the D&D team for this. Fantastic!

Now, on to the subject of playtests!

Different Goals

2024’s playtest was called One D&D. Why? Initially, to reflect their goal of creating the final version of D&D: a version of 5E updated to stand the test of time, requiring maybe only minor tweaks thereafter. Later, the team changed its mind on what One D&D signified as a term, and then soon dropped the term altogether. Very little was done to communicate goals externally, and the emphasis seemed to change over time.

2014’s D&D Next was all about moving forward, leaving the immediate past behind. It was distancing itself from 4E, declaring itself different. Interestingly, designers also expected D&D Next to be the final edition, but for different reasons. D&D’s limited financial success during 3E and 4E meant management saw D&D primarily as fuel for licensed deals, so the designers were largely left alone to create the version of D&D they wanted. The team worried this was the final version ever.

In a future blog I’ll examine this history more deeply, looking at just how different these goals were. For now, let’s note that the goals were very different, with different levels of managerial oversight.

Length and Number of Packets

The One D&D Playtest for 5E 2024 began in late August, 2022. A total of 10 playtest pdfs were released, with the last one landing late December, 2023. Each pdf had a narrow focus (some just a few classes) and could be downloaded within a relatively narrow window, followed by a week or so when a survey could be filled out.

The D&D Next playtests for 5E 2014 started in May 2012 and ended in September 2013, but closed playtesting continued until July 2014. Thus, 2014 had a far longer playtesting period. Where One D&D had 10 pdfs, the playtest program for D&D Next had 16 public packets. Each packet had multiple pdfs, providing a fully playable version of the rules. In addition to public packets, there was closed alpha testing. This brought the total number of playtest packets to almost 50! D&D Next had a lot more playtesting in sheer number of packets released.

2014’s D&D Next also had a significantly different approach. Even the very first packet provided a completely playable version of the rules. The packets often changed significantly, reflecting major design adjustments. Playtesters could see the rules evolve as a result of feedback. For example, in the very first packet, characters lacked skills. Later packets tested the concept of Expertise Dice, and later there was the eventual concept of a proficiency bonus and Expertise dice were tested as something only a few classes might use. Blogs on the old D&D web site discussed design theory, helping fans understand what was being tested and how each packet changed.

One D&D’s Limited Audience and Hurried Approach

The vast majority of playtesting for 2024 was based on people who downloaded a packet at home, read it, and decided whether to fill out a survey. It was very rare to hear of anyone who actually played with the rules, both because of the short survey window and because we never had monsters or revised DMG rules with which to run an accurate playtest.

2024’s One D&D initially wanted to have some public playtest opportunities. The intention had been to offer playtesting at the Virtual Weekends online games (a great program, check them out if you haven’t), but Wizards never managed to provide something suitable for playtesting or that met the Virtual Weekend timelines. Conventions sometimes had staff discussing One D&D, but it could not actually be played there. (2024 conventions offered the 50^th anniversary adventures, but by then playtesting was done.)

Playtesting for One D&D seemed to be wishful thinking that kept running out of time. Even towards the end of One D&D, key staff said in videos that we would get to playtest new encounter rules, new monsters, and more. We never did. Moreover, 2024 never provided even a single playable encounter or monster for playtesting. (As an aside, it’s interesting to see MCDM and Critical Role / Darrington Press both used multiple playable playtest packets during their playtests of the Draw Steel and Daggerheart RPGs.)

D&D Next Reaches Diverse Audiences

While the overall number of playtesters was higher for One D&D, D&D Next worked harder to reach diverse players. 2014’s design team had a goal to appeal equally to previous and new players. To achieve that, it sought different kinds of players.

Learning from play was a key for the designers. In December 2011, even before 5E was public knowledge, WotC invited members of the media, bloggers, and organized play leaders (including little old me!) to its offices to hear about the upcoming playtest… and to actually play the early version. Bruce Cordell ran my table!

Playtest packets provided adventures, and these and additional adventures were offered at key conventions to reach that crowd. This included running Caves of Chaos at Winter Fantasy in January 2012 with the very first packet. D&D designers were on hand, observing tables and taking notes. We also had a playtest packet containing the Reclaiming Blingdenstone adventure, featured at Gen Con and PAX West that summer. The Isle of Dread was available September 2012. Winter Fantasy convention saw the classic G1 Steading of the Hill Giant Chief in 2013 and the Isle of Woe (a 3E organized play scenario) in early 2014. Confrontation at Candlekeep, which I was proud to help write with Shawn Merwin and Greg Bilsland, was at Gen Con and PAX 2013 and served as a preview for 5E.

D&D Next also had alpha closed playtests using adventures that were being written. The Daggerford adventures, later sold in Ghosts of Dragonspear Castle at conventions, were first part of special playtest packets. Same with early versions of Murder in Baldur’s Gate, Confrontation at Candlekeep, Legacy of the Crystal Shard, Scourge of the Sword Coast, Mud Sorcerer’s Tomb, and other adventures. Pause to let it sink in – this is an incredible amount of playtesting of complete rules, with feedback based on actual play as the game evolved. And it wasn’t just at homes and conventions…

Wizards also reached out to gaming stores. The D&D Encounters program was a huge success at hundreds of gaming stores across the US. If you have spent time at gaming stores, you likely know that the crowd is usually very different than the crowd at conventions and also tends to be completely different from the folks who are active online. Each store’s community is often its own bubble. Tapping into the D&D Encounters program allowed Wizards to reach these isolated audiences.

Playtesting at stores started with Against the Cult of Chaos (Feb 6 to April 3, 2013), allowing play with either 4E or D&D Next (some later adventures even supported Pathfinder, so as to draw in PF fans so they might hear about what Next was doing). Playtesting continued through D&D Encounters with Storm Over Neverwinter, Search for the Diamond Staff, the special Vault of the Dracolich multi-table event adventure, Murder in Baldur’s Gate, Legacy of the Crystal Shard, Scourge of the Sword Coast, and Dead in Thay. A total of 7 in-store seasons, each spanning weeks, plus several one-time store adventures, such as Vault or the preview adventure for Legacy of the Crystal Shard. This brought tons of playtest reports from actual play to Wizards from very diverse audiences. The approach also sold lots of players on 5E.

Surveys and Feedback

Each One D&D pdf contained rules text with a particular focus. The initial pdf provided rules for character creation, including new races (later called species) and backgrounds.

WotC said that the first One D&D survey was filled out by 39,000 users. We weren’t given subsequent numbers, perhaps indicating that the first was the biggest. (You can see my review of the first two playtest packets and surveys here.) I suspect interest declined, because the surveys were often frustrating to fill out, with unclear questions and insufficient ways to provide feedback. It was hard to know what we were being asked. Say a power is cool but problematic in play, taking up too much time. Should we say we love it as a player, rate it low because it is bad for play, rate it average after considering both factors? Wizards seemed to notice this. For one playtest packet they discussed in a video how the survey had said one thing, but online discourse another, so they adjusted for online discourse. (How did the team know which of the two was more accurate, or that either was?)

Survey Ratings were converted to 1-100%, and the team said anything above 70% indicated the community supported it. Over 80% indicated the current form was very close to what would please fans, while 60-69% could be corrected through redesign. Below 60% would likely be scrapped. For this first pdf, fans did not care for the Ardling species of celestial bipedal animals, so they were scrapped. The first revision of the dragonborn wasn’t well received either and was modified.

Surveys for One D&D felt like grading on pass/fail. “The designers want X change, will you approve it?” This is very different than asking, “Hey, here is an idea we have. Which parts of it do you like, if any? If it needs to change, what should change?” I think this hot/cold on/off approach to playtesting, and basing it on players only reading the material, led to inferior results.

The One D&D playtest material was at times surprising. The team floated changes to critical hits, including that spells would no longer have crits. The 5E community never had an issue with critical hits… why was this being considered? It received low ratings and vanished. In other cases, we weren’t asked about changes (such as whether we liked Downtime). 2024’s final form is full of quirky and imperfect rules that don’t quite solve problems, including Stealth and Hiding, Ability Scores tied to Backgrounds, Bastions, Crafting, Inspiration, and more.

Playtesting didn’t suitably present alternatives for players to consider and weigh in on. And because playtesting often presented and rated items disassociated from the whole, it was only in the published version of 2024 that most fans would make an actual character and see the full problematic nature of how Origins associated Ability Scores and Feats with Backgrounds.

For 2014’s D&D Next, each fully playable packet was also followed by a survey. The D&D Next surveys placed a high value on figuring out what different types of gamers wanted. Conversations then continued on the official D&D forums (sadly long gone) and there were closed forums for alpha playtesters with further discussions. Because there was so much actual play of cool adventures and new concepts, the quality was likely higher. Playtest packets evolved over time, proving that our feedback was shaped by the community. The process felt like a back-and-forth between designers and community.

Lessons for the Future

Looking back, the D&D Next playtest program seems far superior. The following is a list of approaches that I hope future versions of D&D, and other major RPGs, will consider:

• Communicate to the community clear goals for the edition.
• Tie goals closely to playtest packets. If the goal is a game equally pleasing to new and casual players, each packet and survey needs to test for that.
• Consider closed playtesting in addition to public playtesting so you can test with experienced players and DMs who can provide different and often deeper insights.
• Playtesting needs to be long enough to test all major aspects of the game’s core rules (rules, monsters, characters, encounters, gameplay).
• At least some packets and ideally many should provide fully playable playtests so playtesters and designers can see how the game is actually playing across all pillars of play.
• A game’s audience is diverse and compartmentalized. It is vital to reach beyond online crowds to conventions and stores, to reach both hardcore and casual players, to have many types of players actually play the game. This also doubles as marketing. Beware an echo chamber.
• Tie playtesting into a strong organized play program, ideally one reaching stores, homes, and conventions.
• Provide varied playtest adventure experiences, as these will test the game differently.
• Surveys need to frame questions carefully, noting the nature of the question and allowing for clear feedback.
• Surveys need to do more than vote down ideas, capturing the “why” and allowing for comparison. Fans may like what the current game provides. Surveys should capture that. How could One D&D have captured what fans wanted for Backgrounds and Ability Scores?

Taking a Breath

I strongly believe D&D Next had a superior approach to playtesting. I hope future versions of D&D learn from that. At the same time, no playtest process is perfect. All teams start by promising to spend as long as needed playtesting and then find there is no time left and the books have to go to print. I am familiar with playtesting at other companies across the years and I believe Wizards does a better job than many, if not most, companies. Running a playtest program is a hard and thankless job. Let me pause and thank both the 2014 and 2024 teams for what I am sure was tremendous work in the name of creating better D&D.

It’s also important to note that business concerns often cause playtesting and design to shift during an edition. More on this, next time.