Seibeclass="underline" And how can programmers learn to better recognize where they are on that curve?
Norvig: I think you set the right environment, where it’s results-oriented. And I think people can train themselves. You want to optimize, but left to yourself you optimize your own sense of comfort and that’s different from what you really should be optimizing—some people would say return on investment for the company, others would say satisfaction of your customers. You have to think how much is it going to benefit the customer if I go from 95 percent to 100 percent on this feature vs. working on these ten other features that are at 0 percent.
At Google, I think it’s easy because we have this “launch early and often” philosophy. And because of the way the company is, for a number of reasons: one, most of our products we don’t charge any money for so it’s easy to say, well, go ahead and ship it; how much could they complain? The other one is we’re not stamping CDs and putting them in a box so if there’s something that’s not complete today or even if it has a bug, it’s not a disaster. Most of the software is on our servers so we can fix it tomorrow and everybody gets the update instantly. We don’t have this nightmare of installing updates. So it makes it easier for us to say, “We’re just going to launch things and get some feedback from the users and fix the stuff that needs to be fixed and don’t worry about the other stuff.”
Seibeclass="underline" If you’re working on the design of a big system what are the tools you use—do you sit down with a pad of graph paper or a UML drawing tool?
Norvig: I never liked any of these UML-type tools. I always thought, “If you can’t do it in the language itself that’s a weakness of the language.” I think a lot of what you’re doing, you’re dealing at a higher level. At Google a lot of what we do is figuring out how to break things up and parallelize them. We’re going to necessarily run this on multiple machines but we’ve got so many users and, for many applications, so much data; how’s that going to work? So we’re thinking more at the level of machines and racks of machines rather than at the level of functions and interactions. Once you get that straightened out, then you can start diving into individual functions and methods.
Seibeclass="underline" And so that level of description is just prose?
Norvig: Yeah, mostly. Sometimes people draw pictures. They’ll say, “We’ll have this server here that will be serving these kinds of requests and then it’s connected to this server and then we’ll use these various tools for storage and big distributed hash tables and other types of things. We’ll choose these three tools off the shelf and then we’ll argue about whether we have to build a new one; which of these existing ones works or do we need something else?”
Seibeclass="underline" And how do you evaluate that kind of design?
Norvig: You show it to the people who’ve done it before. They say, “Oh, it looks like you’ll need a cache here—it’s going to be too slow but there should be a lot of repeat requests, so if you install a cache of this size here that should help a lot.” You have a design review where people look over it and say whether they think it makes sense and then you start building it and testing it.
Seibeclass="underline" And you guys have formal design reviews? You worked at NASA where they had a very formal design review.
Norvig: Nothing formal like NASA. The stakes are lower for us because, as I say, it’s easy for us to have a failure and recover from that. At NASA usually the first failure is fatal so they were much more careful. We don’t worry about that much. It’s more of a consultation, I think, rather than a review.
There are people who officially read design documents and comment on them. You go through that and get your design approved. But it’s still much
Norvig: I think it was something that had always been done and so people accept it. Well, I shouldn’t say that completely. Some people it takes a while to get used to it. One of the typical failure cases is a new hire comes in and they’re not used to doing this kind of thing so they just start an experimental branch and they have all their code in there and you keep on telling them, “Gee, you don’t have any check-ins yet.” And they say, “Yeah, yeah, yeah, I’m just cleaning it up—I’ll check it in tomorrow.” And then another week goes by and another week goes by and eventually they have this one gigantic check-in. And then it’s a problem that too much time has gone by, it’s hard to evaluate it all at once, and some of the things they’re comparing against have changed out from underneath them. Then they see what a headache it is and they learn not to do that.
Seibeclass="underline" So that’s on the coding side. Are there skills that the reviewers develop?
Norvig: There certainly are people who are known for being better reviewers than others. There’s a trade-off of when you submit a review—do you try to get somebody who will give you a lot of good feedback or do you try to get somebody who will just say “OK” as quickly as possible?
Seibeclass="underline" So what makes the better reviewers better?
Norvig: Well, that they catch more things. Some of it is the trivial stuff of you indented the wrong number of spaces or whatever but some of it is, “I think this design would be cleaner if you moved this from here over to there.” So some people will do more of that and others won’t bother.
Seibeclass="underline" Sort of related to that, does every good programmer turn into a good architect when they grow up? Or are there some people who are brilliant coders but only at a certain level and they should never be allowed to do bigger designs?
Norvig: I think different people have different skills. One of our best search people is by no means our best programmer, in terms of how the code looks. But if you say, “Here’s this new factor that we have—you know, how many times people click on this page after they’ve done such and such—how do we fold that into our search results?” He’ll say, “Oh, on line 427 there’s this variable alpha and you should take this new factor and raise it to the second power and multiply it by 1.5 and add it to alpha.” Then you experiment for a couple months trying different things and you find out he was right except it should have been 1.3 instead of 1.5.
Seibeclass="underline" So that suggests he just has this very good mental model of how the software works.
Norvig: He understands the code perfectly. Other people can write code better but he understands all the implications of what goes where.
Seibeclass="underline" Do you think those are related? It often seems that people who write the worst spaghetti code are the ones who can hold the most in their head—that’s the only way they could possibly write code like that.
Norvig: Yeah, I think that may be.
Seibeclass="underline" So the reviews here are less formal here than at NASA. What are the other differences between the “engineering” and “hacker” ethos, in the best sense of both those words?
Norvig: One big difference is organizational structure and how software is accepted. Google was founded as a software company, and they went out and hired a CEO who has a PhD in computer science from Berkeley, hired a VP of Sales who has a computer engineering background—it’s throughout the whole company. At NASA they’re rocket scientists! They aren’t software guys. They say, “Software is this necessary evil. Straight line code, I can sort of understand; if it’s got a loop in it, that’s kind of iffy. Then if there’s a branch statement inside the loop, ooooh, that’s getting away from what I can solve with a differential equation in control theory.” So they’re distrustful.