Vouga of Arabia

Last week was Eid al Adha (a.k.a. "Hajj break"), which for me meant only that there were less students around campus, and a lot of the facilities were shut down. I did get the opportunity, though, to take the bus into downtown Jeddah and do some shopping and sightseeing.

There are really only two places you can go outside of campus. The first is Thuwal, a fishing village right outside and visible from KAUST. The dining hall on campus is a typical cafeteria, serving generic international food;¹ restaurants in Thuwal offer quite a different experience. To begin with, the dining area is segregated into different section, neither visible to the other: one for men, and the other for families. (In practice, a family is any group of people containing at least one woman; even groups of obviously different races don't raise any eyebrows. At least around here -- I've heard that in the more conservative regions of the country, near Riyadh, "family" might be interpreted more literally.) Secondly, the customary way of eating is on the floor, on a carpet, with no utensils. Over the past few years, though, the amount of business coming in from KAUST has encouraged restaurants to slowly invest in western concessions: they have a few tables and chairs they keep in the back, and a few sets of silverware, that they bring out as needed. Thirdly, at a seafood restaurant I visited a few weeks ago, the main course wasn't ordered off of the menu: instead, you first visit a fish market in a building next door, and pick out the fresh fish you would like to eat. Only after you've selected your dinner, haggled a bit over price, and informed the staff how you'd like the fish cooked, do you get seated and order drinks and sides off of a menu.

The second possible destination is Jeddah, the largest city in the area (population ~3 million) and about 45 minutes away from KAUST.² A bus travels there and back once a day, which is lucky because I wouldn't want to drive even if I owned a car: lanes are strictly suggestions (and the shoulder of the road counts as a lane), driving over sidewalks and medians is the accepted way of taking shortcuts and U-turns, feral donkeys and camels wander onto the street, etc etc. Only in Boston have I seen crazier drivers (where, in addition to the ubiquitous double-parking, I've witnessed one clever guy's solution to one-way streets: drive at full speed in reverse.)

The bus makes three stops in Jeddah: the Mall of Arabia, Red Sea Mall, and downtown. I've never been to the first two (and I'm not particularly interested in doing so.) I was warned by a coworker: if you visit downtown Jeddah by yourself, take a map and a compass, and give yourself two hours to find your way back to the bus. I completely ignored him, of course, and got horribly lost. Jeddah's downtown is a vast labyrinth of twisty streets and alleys, with a scattering of larger shopping centers and hundreds of smaller stores selling incense, carpets, clothing (including abayas; I promised to pick one up for a friend), bars of gold, more different kinds of dates than I ever imagined existed, etc. The shopkeepers are all terrificly friendly. Haggling over everything is a little off-putting at first, but eventually you realize that you're arguing over what amounts to less than a buck or two.

The mosques at KAUST announce the five daily prayers, but unless you are unlucky enough to live directly under the minarets of the campus Grand Mosque the prayer times have no significant effect on daily life. Not so at all in Jeddah. When a prayer time begins, all customers are kicked out of stores and the stores close, for a not insubstantial amount of time. According to coworkers, one trick is to order take-out right before the prayer begins, and eat while waiting for it to end; of course, this strategy still requires knowing when the next prayer will begin, which changes daily. Another unique aspect of shopping in Jeddah is that stores tend to be clumped into groups offering identical wares: one block will be entirely devoted to carpets, the next to paint, etc. At first this arrangement didn't make any business sense to me: if you're opening a carpet store, why place yourself right next to your competitors? But it was explained to me that this system has established itself as a Nash equilibrium of sorts: customers looking for a carpet know to look (only) on the carpet store block, and a carpet vendor trying to set up shop elsewhere would only starve himself of customers.

1. the one exception, of course, is the lack of pork: "bacon" here always means small bits of beef. [↩]
2. Mecca and Medina are also nearby, but strictly forbidden to non-Muslisms. [↩]

Lessons Learned Volunteering on an Internet Project

Zelda Classic started out, in the dark forgotten days of DOS, as a clone of the very first Nintendo Zelda game, painstakingly reverse-engineered by a pair of high school students in Ohio. What set it apart from the numerous other ROMs and pirated copies of the game on the Internet was the "quest editor" bundled with it: using this simple graphical editor, fans of the game with no particular experience programming could quickly and easily create their very own game, with completely new levels and graphics, but the same Zelda look and feel.

Over time the game grew in popularity and functionality: the original creators moved on to other projects, but new programmers picked up where they left off and ported it to Windows, modernized some of the user interface, and added many, many new features to the quest editor. It became possible to create games with polish rivaling the official Zelda games on the Super Nintendo and Gameboy.

Zelda Classic was at its apogee around the time I first became involved with the project's development seven years ago; it had been featured on TechTV's show The Screen Savers, and in EGM magazine, and as a result more users than ever were playing the old fan-made "quests" and creating new ones. Zelda Classic was downloaded over a million times, and several hundred fan-made quests were submitted and "published" on the official site.

But whereas the fan community was thriving, development had stagnated. The last stable release was years old; newer betas had occasionally been publicly released, but these betas were notoriously buggy and widely avoided. Under the theory that fresh blood would hurry things along,¹ a call went out for new developers. Devoting time to a project one good cease-and-desist letter away from being permanently shut down wasn't going to make me rich or famous, but it looked like a fun way to hone my coding skills. I signed on.

I wasn't Zelda Classic's original programmer, nor was I ever the lead programmer during the time I actively contributed to development. But I left a significant mark on the project and am proud my accomplishments: the C-like scripting language I added to the quest editor is one of its most ambitious and complex features; I led the team in number of bugs fixed; and I became webmaster of the code repository, official project site, and associated community forums.² Most importantly, I experienced first-hand the challenges of working on and leading a large, popular, volunteer software project. Below are the three most important lessons I learned during this experience.

 1. The allure of Cool Things is irresistible.

When a programmers volunteers to work on a project, she does so because she is full of ideas about Cool Things that need to be added to the project, as soon as possible. This feeling is completely understandable: everyone who's used a tool for long enough accumulates a long list of ways the tool could be improved. Fixing bugs is boring, time-consuming, and thankless. Adding a new feature, on the other hand -- especially a Cool, flashy, long-requested one -- is fun, brings glory, and satisfies the basic human need to press hands into new sidewalks, to leave a mark that you can later point to and say, "I was there, I contributed to that!" If the new feature doesn't quite work right at first, or doesn't play well with the Cool Things that others added yesterday, well, there's plenty of time to fix that tomorrow, when/if the natives complain enough.

The problem only gets worse as times goes on, due to combinatorial explosion. In the original Legend of Zelda, for instance, there are bodies of water blocking the player's path, that can only be crossed in one of two ways: by using a stepladder, or riding a raft. Code is needed to make sure nothing breaks if the player tries to use the ladder while already riding the raft, and vice versa. Easy.

Now we add a new way to cross water: flippers that give you the ability to swim. There are four ways adding flippers might break the code: what happens if the player tries to use the raft or ladder while already swimming through the water? What if he tries to dive off of raft or ladder while wearing flippers? All of these possibilities must be taken into account.

Now we add a new item that lets the player jump short distances. The player can use this item to jump across narrow streams. He can also jump into lakes -- if he has the flippers, he'll dive in and start to swim; otherwise, he'll drown. Can the player jump into or across water if he's already standing on the ladder or raft? We have to write code to deal with that. What if the player tries to use the ladder or raft while in midair over water? Clearly that shouldn't be allowed. And wait -- what if he tries to jump while already in the water, swimming with the flippers?

Every new feature added exponentially increases the number of possible interactions, and thus the number of bugs, since some of these interactions are inevitably missed. Debugging starts to feel like a Sisyphean task, as the system becomes complex enough that a "fix" for one bug introduces a new bug somewhere else.

We tried several strategies to keep testing and debugging manageable: we recruited as many users as we could to perform beta testing; we decreased the time between releases of betas, to tighten the testing and feedback loop; we urged voluntary "feature freezes" where only changes to the code that fixed bugs, and added no new features, were allowed. These measures helped, but weren't enough: without managers or deadlines we simply didn't have the self-discipline to 1) carefully consider the long-term consequences of the changes we introduced; 2) adequately test our changes; or 3) force ourselves to slow down and ship something stable before moving on to the next Cool Thing.

There are several measures open-source projects have taken to solve this problem: mandatory review of every change by a second developer; more central leadership with one person ultimately responsible for deciding which features to include and reject; separate branches for experimental new features still being worked on, and for well-tested new features that will definitely be included in the next release; etc. I woudn't start any new project with as large a scope as Zelda Classic without thinking very seriously about how to reign in feature creep, keep developers on target, and ensure development sticks to a coherent long-term plan.

2. The user community is your most valuable resource.

I've been very impressed by the quality of the custom quests created by users with Zelda Classic -- as I mentioned before, many of these are larger, more polished, and more fun than even the official Nintendo games, and I can't imagine how many hours of time must have gone into crafting them. Equally impressive, though, was the amount of effort the community was willing to contribute to the project.³ Us programmers had barely enough time and resources to do the programming... from beta testing, to marketing, to helping new users work the program, to designing and maintaining the web site, many tasks crucial to the smooth running of the project were handled by fans of Zelda Classic who volunteered their talents, unsolicited.

The corollary is that the user community is an Internet project's most valuable resource. The most ominous sign I see for the future of Zelda Classic is that this community has slowly dwindled over the last few years. The lack of a recent new version of the program is partly to blame, but there were several other preventable factors at play. Firstly, the official forums were repeatedly hacked and infected with malware, leading many users to abandon the forums and the project altogether. Secondly, as Zelda Classic grew in popularity, the community divided into two camps: friends of the original programmers, early adopters, and other users who had been around from the very beginning; and "newbies" who had only recently heard of Zelda Classic and decided to check it out. The former became vested contributors -- people who, by virtue of their having been around the longest, and having contributed most to the project, felt they were entitled to greater privileges, and fewer rules, than the new arrivals. The former resented the later for being on average younger and less mature, and asking many basic questions; the latter resented the former for being arrogant, rude, and cliquish. This friction grew until the community split, with most of the newer users migrating to their own, more welcoming community. The code itself didn't fork -- it couldn't, being closed-source -- but the code was hardly the project's most precious resource -- the users were.

Incidentally, vested contributors are a significant problem facing many collaborative efforts on the Internet -- Wikipedia and Stackoverflow and two such projects that spring to mind. For both of these projects I predict overindulgence of vested contributors as the most likely point of failure.⁴

3. You can't take back moves.

As you release new versions of software, there are two approaches you can take to deal with user data (in the case of Zelda Classic, user-made "quests") created with old versions of the program:

• The Apple approach: You don't worry about backwards-compatibility. Each version of your software makes a clean break from previous ones. This is the case for different version of Apple's operating system: if you're lucky, the program you wrote for one big cat continues to work on the next one, but there's no guarantee it will do so. This approach is great for the programmers of Mac OS, as it gives them the freedom to start completely from scratch, learn from their mistakes, and redesign or ditch completely those features of the operating system that didn't work as well in practice as they'd hoped. It's not so great for the application programmers: the burden of making sure the application continues to work over time has been passed down to them.
• The Microsoft approach: You try to keep your software as backwards-compatible as possible. Programs written for Windows XP or Vista (and, for sufficiently popular programs, even Windows 95 or DOS) continue to work on Windows 7. This situation is ideal for application developers: they write a program once, and Microsoft, not them, has to worry about it continuing to work in the future. It's also great for users of the program, since the program continues to work even if its developers go out of business, get hit by a bus, etc. But maintaining backwards-compatibility forces Microsoft to be very conservative about what it changes from version of Windows to another, and forces Microsoft to bloat its software with components that serve no purpose but tricking old programs into thinking they're still running on e.g. Windows XP, so that they continue to work correctly.

Zelda Classic chose the Microsoft approach long before I joined the project. It's the approach that's friendliest to end-users, and so is a natural choice for a free program entirely reliant on its user community for survival. But it's a choice that has cost us dearly in terms of program complexity.

Suppose you're using Zelda Classic to make your own quest, and you decide you want to make the player lose his sword midway through the game (he gets robbed by bandits, say.) You look through Zelda Classic's features, but see no way of taking away an item that you've already given to the player. You ask on the Internet if anyone has any suggestions -- and another user tells you that, although there is no official way of taking away the sword, there's a bug in the current version of Zelda Classic where if you try to give the player a second sword while he's holding his sword, a shield, and is facing left, the player loses both swords instead. "Perfect!," you think. You've found a creative way of working around the constraints imposed on you, and you can put that bandit scene into your game after all.

Years later, someone reports that giving players a second sword isn't working correctly in some cases. We investigate, find the bug, and fix it -- but now all kinds of old quests aren't working anymore; bandits are robbing players but the sword doesn't disappear. And we promised old quests would continue to work flawlessly in the future! "Oh come on," we might grumble. "What were you thinking?!? You're obviously exploiting undocumented, buggy behavior. How could you possibly think it's a good idea to rely on this bug?" But there's no telling how many other people also discovered and exploited the bug, and we don't want all of their quests to stop working, so we grit our teeth and modify our code: if a quest was created before a certain date, use the buggy sword addition code. Otherwise, use the new, correct sword addition code.

This situation happens all of the time: a sufficiently large group of users is infinitely ingenious and no more immune to the danger of Cool Things than the developers. If features can be combined in unintended ways to create a neat effect, they will find the combination and use it. They will find undocumented features used for internal testing. They will find bugs and exploit them.  And you will only find out after you make a seemingly-innocent change to the code and discover it breaks a bunch of old quests.

Another consequence of backward-compatibility is that every new feature you add becomes an indefinite commitment. If the feature is poorly thought out, it will hamstring you. Over time Zelda Classic has accumulated many such missteps -- options and settings that almost nobody ever uses, but that interact in complicated ways with the rest of the program, and are endless sources of bugs. For more experimental features, we tried telling users, "this new feature is strictly beta. We might take it out at any time; please don't create quests that rely on it." We even incorporated a popup window with that same message into the program itself. Users would rely on the feature anyway, with the best intentions: at first they just wanted to play around with it and test it. But as they invested more and more time into works in progress depending on the feature, they became increasingly reluctant to give it up.

Striking a balance between backwards-compatibility and freedom to revisit past mistakes is a challenge even Microsoft is struggling to solve, and there are no easy answers. But there are several things Zelda Classic could have done differently to partially tame the problem. Most importantly, and relating back to the first lesson, we could have been more conservative about which features we expose to the public. We could also have compromised a bit on backwards compatibility; if I were to start over from scratch, I would wrap Zelda Classic in a meta-program that detected what version a quest was created with, and downloaded that version of the quest player if needed, instead of trying to make the most recent version of the quest player compatible with all old quests.

1. This is a classic blunder, almost as well-known as "never get involved in a land war in Asia" and "never go against a Sicilian when death is on the line." [↩]
2. A role I still hold, more by default than any abundance of spare time, or love of web work. [↩]
3. Perhaps I shouldn't be too surprised, since I too was "the community" before signing on as a programmer... [↩]
4. Though Wikipedia has quite a few other problems. [↩]