Katherine Druckman talks to Angela Byron about Drupal 7 and managing a distributed open-source development team.
Angela Byron is one of the most respected contributors in the Open Source community. She has been recognized for her efforts by Google, receiving the 2008 Google-O'Reilly Open Source Award for Best Contributor. She's passionate about the Drupal Project and has worked tirelessly to ensure that the latest release of the popular Web platform and CMS is the best version yet. We talked to Angie about her role as co-maintainer of Drupal 7 and about what it takes to manage such an important, massive open-source project.
KD: You've had a very busy three years taking the lead as core maintainer on the newly released 7th version of Drupal. Tell us a little more about what being the Drupal 7 core maintainer means. What does it entail?
AB: Dries Buytaert is the Drupal Project's founder and project lead, also known as our Benevolent Dictator for Life. He holds the keys (as in commit access) to Drupal core. Each new release of Drupal, Dries also appoints one other person, called a co-maintainer, who is also given commit access and the responsibility to help set the vision and direction of the release alongside him.
Neil Drumm was the core co-maintainer in Drupal 5, and he prioritized usability improvements, including a graphical installer. Gábor Hojtsy was the core co-maintainer of Drupal 6, and he prioritized numerous internationalization improvements. In Drupal 7, I was selected. Things I prioritized included usability, quality assurance, “developer experience” (API consistency and other things that make Drupal more enjoyable to work with), and making Drupal more accessible to designers, themers and people with disabilities.
“On paper”, a core co-maintainer's responsibility is to review and commit patches submitted by the core developers. In practice, I found that 80% or so of my time was spent not doing that, but instead on more “human” endeavors: identifying people working in similar areas and encouraging them to work together, helping brainstorm architectural direction on certain patches, mediating heated arguments, helping new contributors get involved, promoting major initiatives and so on. So in some ways, it was a community management role, but with commit access thrown in.
KD: Which part of the three years has been the most intense? I would guess these past few months, but then I wonder what the pattern has been over the course of working on this version. How was it at the beginning versus the middle and end?
AB: Drupal's release cycle goes in three phases:
“code thaw” where we make our wildest dreams come true and add new features, break APIs and generally pursue world domination.
“code slush” where we focus on polishing the rough edges from code thaw, API consistency, UI cleanups and performance.
“code freeze” where the APIs get severely locked down, and bug fixes/stabilization is the name of the game.
The intensity varies, and generally whenever we have a major deadline (for example, a Drupalcon is coming up or a code freeze date is about to be declared), things heat up a lot. Most of our major features were introduced a week or so before one of those major deadlines.
One of the most challenging things was maintaining momentum after code thaw (aka the “fun” part) completed. People could no longer add their pet features; instead, the attention turned to “the slog” of bug fixes and incremental improvements. The number of core contributors died off significantly as they focused back on their contributed modules or other projects, and those who were left had to carry the pretty tremendous burden of taking us from some 150 critical issues down to zero so we could release.
KD: Drupal has a lot of core contributors. It's up to around 1,000 now, isn't it? Obviously a small number of those contribute the bulk of the code, but it's still an enormous group of people to coordinate. Can you tell us about that experience?
AB: Yeah, the count from the commit logs was a little less than 1,000, but that doesn't include people who reported bugs, tested patches and so on. That number includes a set of maybe 50–75 hard-core people who spend their lives in the Drupal core queue daily and then an enormously long tail of others who contribute only a handful of patches. And, bear in mind that every single one of those patches has to be reviewed and then committed by Dries or me.
It's both incredibly exciting and challenging to work in this environment. My favourite parts were forming battle plans and kicking around ideas on how to make Drupal 7 the most amazing release possible, helping new contributors submit their first patch, seeing the community consciousness about things like usability, design and accessibility gradually evolve and accept their importance. The core contributors are amazing, smart, dedicated and passionate folks who pour their hearts and souls into Drupal, and I'm honored to work with them every day on IRC and the issue queues.
However, this level of closeness also meant things also could get a bit personal sometimes. And when they did, it really, really hurt, because these folks are all my friends, and we have worked so closely together for years. There definitely were days when I needed to step away from the computer for a few hours and get some perspective, and even considered throwing in the towel altogether more than once.
On the whole though, the experience was absolutely amazing. I'm infinitely proud of the team and what we've managed to accomplish with Drupal 7.
KD: What can you tell us about the decision-making processes involved in developing and maintaining this sort of a project? How do you collectively decide on a course of action or the best approach to a problem? Programmers can be married to their ideologies. How do you deal with that?
AB: Drupal is very much a consensus-based community. Problems are identified, solutions proposed, code written and then discussed among at least two but possibly dozens or hundreds of people. No change, big or small, makes it into the upstream code unless it has been marked “reviewed & tested by the community”, which means at least one other person has looked at it and given it a nod of approval. This is a great “community engineering” strategy, because it ensures better quality code and encourages developers to be civil to each other so they can find reviewers for their patches.
There's typically very little contention around straightforward bug fixes. On some of the more esoteric or architecturally facing issues, however, lots of people start throwing around opinions on approaches, and sometimes heated ones.
In my role as core maintainer, the best thing I could do in those situations usually was simply monitor the discussion carefully, but keep mostly quiet except when things escalated to personal attacks. In almost all cases, if developers are given free reign to hash things out among themselves, they're able to come to a mutual resolution without any intervention. It's important that this happen most of the time, in order to build camaraderie and respect among the development team.
Occasionally, however, a stalemate was reached and intervention by a core maintainer was needed. In this situation, I normally try to take the folks aside in IRC and see if we can hash out their differences together. Oftentimes a heated discussion on the issue queue dragging on for days can be settled in minutes when both parties are brought together with a mediator. I try to summarize the opposing views and explain what is good about each perspective in an attempt to disarm some of the defensiveness that might be brought into such a meeting. Neither person's idea is “wrong”; they both have pros, but we need to come to the right decision that might be some mixture of both.
In cases where we weren't able to come to resolution, I'd post my best attempt at a mutually agreeable solution. This actually very rarely was the actual solution we went with, but it had a way of “resetting” the conversation to be around a new suggestion rather than the old ones, which usually helped everyone play nicely again—usually. On the whole, I just tried to remind people that we're all here to make Drupal better, even that stubborn git who can't see your point of view yet.
KD: What excites you most about Drupal 7? Do you think Drupal 7 improvements will increase Drupal adoption?
AB: Feature-wise, it's hard to pick—there were so many things we added to Drupal 7—but I'd say overall it comes down to the following things:
Image handling now is included out of the box without the need to download six or seven additional modules. This is huge for adoption, at least for people who want something other than text on their Web sites, who apparently exist. It also brought with it a host of improvements with our native file API, which has interesting implications for extending Drupal for use as a document management system.
The new database abstraction layer has a lot of people really excited. Both from its support of new features, such as transactions, and support for more database back ends, but also its new object-oriented syntax. Basically, everything that ever sucked about our database abstraction layer has been fixed in Drupal 7.
The new entity and field paradigm is an important shift in the Drupal Project. In the past, Drupal was very “content-oriented”, and lots of features were developed that would extend “nodes” or pieces of content in the system. This resulted in a lot of people trying to shoehorn things that weren't actually nodes (such as users, comments and so on) into nodes so they could benefit from these features. In the future, these same features will be developed as fields, which then can be used across any entity in the system—users, comments, content, taxonomy terms and more.
The automated testing framework and more than 30,000 tests we added to Drupal 7 has had profound effects on our community development process. We know instantly if any patch in the issue queue works or breaks existing tests. We are free to refactor subsystems knowing we didn't break any of the existing functionality, and the “test-driven development” mindset is slowly working its way into our development community's consciousness as a best practice.
Overall, I'm simply thrilled with the amount of diversity in the core development team now as compared to the last release. We have an accessibility team, a usability team, a markup and design team, and a documentation team dedicated to improving the core in ways that go beyond the code. My hope is that the leadership shown by early pioneers in these fields will open the floodgates for new contributors in the Drupal 8 process, and that Drupal 8 improves upon Drupal 7 in all of these areas and more.
KD: I have played with Drupal 7 quite a bit, but I can't say I've really put it to full use yet, and I keep hearing about how much more usable it is. How much do you think this will affect the Drupal learning curve?
AB: Usability was something that saw tremendous attention during the Drupal 7 release cycle, and we saw a radical transformation in the culture of the Drupal development community and how seriously this barrier of entry was treated.
Just after Drupal 6 was released in February 2008, Dries, myself and several other major contributors went to the University of Minnesota (yes, Minneapolis in February—that's how much we love Drupal) to perform our project's first formalized usability study. We were given a room with one-way glass, tools like eye-tracking software to tell where people were looking on the screen, and the University found several participants who had previous Web development experience with tools like WordPress, Movable Type and Dreamweaver but not with Drupal. In other words, people in our project's direct target audience.
The results were absolutely shocking and completely transformed the way I look at Drupal. We found that participants were completely lost as to whether they were on the front end or back end of their Web sites. They were unable to find major administrative sections in order to perform basic tasks. They were mystified by Drupal jargon, and on and on.
A usability team was formed who set about fixing many of the problems identified in testing, and they fixed a number of important and obvious problems in the existing UI. Additionally, Acquia funded Mark Boulton and Leisa Reichelt, a design and usability expert, to take a holistic view of the Drupal administrative experience and make more wide-sweeping changes, in a community-driven, collaborative usability experiment called D7UX. Both initiatives worked in tandem to provide Drupal 7 with a new administration theme; a set of common administration patterns, such as a toolbar and contextual links; and a task-based administrative information architecture. We haven't yet been able to test formally how the usability work we did in Drupal 7 improved the situation over the results we saw with Drupal 6, but preliminary feedback from the broader community has been pretty awesome.
This usability work doesn't fully address the dreaded “Drupal learning cliff”, per se. There still are an awful lot of things you need to know in order to be a successful Drupal site builder, like what modules you should use for what and what the heck weird words like “taxonomy” and “node” mean. However, Drupal 7 hopefully should require a lot less customization from site builders to put it in front of their clients and give them a better leg up on answering the question, “Great. I have a Drupal site installed...now what?”
KD: It's supposed to be more scalable too. Can you tell us how?
AB: A number of new features in Drupal 7 help with the situation where your site needs to accommodate huge blitzes of additional traffic, assuring all visitors of a speedy experience:
Master/slave replication support in the database abstraction layer—database writes are slower than database reads, and reads happen way more often. A master/slave setup allows you to separate reads from writes, so that the main database storing all the information isn't additionally overburdened with read requests, which can be routed instead to read-only slave databases.
Reverse-proxy support—a reverse proxy, such as Varnish or Pound, can greatly speed up Web site access by caching copies of pages and then intercepting requests to serve them, saving the Web server from having to handle page requests directly. Drupal's settings.php configuration file now contains directives to enable reverse-proxy support.
Support for content delivery networks (CDNs)—services exist for caching static files, such as images, CSS and JS, across multiple geographically distributed computers, which then can be served to visitors more quickly than a round-trip to the Web server where Drupal is stored. By invoking hook_file_url_alter(), modules can re-route requests from Drupal's files directory to services, such as Akamai or Amazon CloudFront.
There's a high-performance distribution of Drupal 6 core called Pressflow (pressflow.org) from which a lot of these scalability improvements originated.
KD: For our readers who are more comfortable with WordPress, Joomla or even platforms like Ruby on Rails and Django, what do they need to know about Drupal, and in particular Drupal 7, in order to have the best experience getting started?
AB: For folks coming from other CMSes, such as WordPress or Joomla, the biggest hurdle to getting started with Drupal is often the “LEGO block” approach Drupal takes to building sites with modules. It's common in other CMSes that if you want to add a photo gallery to your site, you simply search for a photo gallery extension and choose from a list of prebuilt all-in-one options.
In Drupal, however, the trend in modules is more toward small, generic, re-usable components that can be combined and mixed and matched in lots of various ways. There's not much in the way of off-the-shelf photo gallery modules for Drupal. Building a photo gallery in Drupal typically involves creating a content type to provide a data entry form for images, adding an image field in order to upload photos to the content type, creating a view of photo images, and so on. Although more elbow grease is required at the outset, the advantage is that the photo gallery you end up with can behave exactly as you want. And, the same module that provides an image field for photo galleries also can be re-used to provide album covers and user avatars, while the same module that provides a photo gallery view can be used to create event calendars, RSS feeds and other types of listing pages. This level of customization and re-usability is what attracts people to Drupal, but it definitely requires a tinkerer's mindset.
For folks coming from frameworks, such as Django or Ruby on Rails, the main thing to realize about Drupal is that it's more of a “framlication” than a pure framework. Drupal provides ample APIs for dealing with file handling, session management, internationalization and so on, and it also provides a “hook” system from which Drupal's base behavior can be extended. However, it also makes some base assumptions that what you're building is a Web-based application tracking things like users and content. The advantage of this is you don't need to recode a new login system every time you build a site on Drupal; this type of low-level functionality is provided for you in an extensible way. But, it does mean if you don't agree with some of the base assumptions Drupal makes, you'll need to spend a bit of effort developing a module to alter behavior you want to change. The best advice I probably could give to folks coming from more traditional Web frameworks is to take the time to explore what Drupal can do without writing a line of code, which is fairly substantial. Then, learn the extension mechanisms Drupal provides—hooks, the theme system, the localization layer and so on—to make customizations in a forward-portable way.
KD: You've worked on some major, large-scale Drupal projects, and Drupal was also selected as the platform for Whitehouse.gov. What about Drupal lends itself to those types of sites?
AB: I think Drupal hits a sweet spot in that it's free, open source and an extremely capable framework that's constantly evolving. It can be highly customized to particular use cases, and it has an ever-growing community with a lot of expertise. Many of the enterprise-level clients we work with move to Drupal from less-capable, proprietary CMSes that have thousands or even hundreds of thousands of dollars per year in licensing fees, with bugs they can't fix themselves because they're beholden to a vendor's schedule. So the idea of moving to something they can be trained on internally or hire outside expertise to implement quickly is very appealing.
