Knuth: When I wrote TeX originally in 1977 and ’78, of course I didn’t have literate programming but I did have structured programming. I wrote it in a big notebook in longhand, in pencil.
Six months later, after I had gone through the whole project, I started typing into the computer. And did the debugging in March of ’78 while I had started writing the program in October of ’77. The code for that is in the Stanford archives—it’s all in pencil—and of course I would come back and change a subroutine as I learned what it should be.
This was a first-generation system, so lots of different architectures were possible and had to be discarded until I’d lived with it for a while and knew what was there. And it was a chicken-and-egg problem—you couldn’t typeset until you had fonts but then you couldn’t have fonts until you could typeset.
But structured programming gave me the idea of invariants and knowing how to make black boxes that I could understand. So I had the confidence that the code would work when I finally would debug it. I felt that I would be saving a lot of time if I waited six months before testing anything. I had enough confidence that the code was approximately right.
Seibeclass="underline" And the time savings would be because you wouldn’t spend time building scaffolding and stubs to test incomplete code?
Knuth: Right.
Seibeclass="underline" Other than the fact that they’re so nicely typeset now, do you think your books would be very different if you hadn’t spent ten years writing TeX?
Knuth: Good question. The experience of using structured programming in a not purely academic way—in other words, I’m not just thinking about invariants in toy programs, but in real programs—probably had a fair influence on how I’m describing algorithms in the new stuff I write now. Or if it hasn’t, it should.
I wouldn’t have known about caching and trends in the way computers change and things like that if I was just going on in the same mold of going from the literature to writing my books. While I was writing Volumes I, II, and III, I wasn’t writing programs like TeX that that would be more typical of a large programming practice. They would be toy programs. So it gave me more of a perspective on numbers and quantity.
It’s really amazing, though, when you’re writing a book, the influences that will make you choose different words. It’s mysterious how it gets in there. That’s the most important influence of writing TeX—that it gave me a different kind of a mental take on things so that my sentences come out different. They’ll be a little bit less hedgy. There is a tone that comes out in the whole thing, of confidence or something.
Seibeclass="underline" Do you think you were a dramatically better programmer when you finished TeX than when you started?
Knuth: Well, yes, because of literate programming.
Seibeclass="underline" So you had better tools, but had you actually improved your skills?
Knuth: I learned a terrific amount while I was doing it. One of the things I learned was how much software occupies the brain. It was a much more difficult task than I expected. I couldn’t teach classes full-time and write software full-time. I could teach classes full-time and write a book full-time but software required so much attention to detail. It filled that much of my brain to the exclusion of other stuff. So it gave me a special admiration for people who do large software projects—I would never have guessed it without having been faced with that myself.
Seibeclass="underline" So programming is harder than writing books, and somewhere I read something where you said that it’s impossible to estimate how long it will take to write books. Does that then mean that it’s even harder to estimate how long programming will take?
Knuth: Yeah, right. That’s a very good corollary.
This year I’ve written probably three major programs which are pushing one hundred pages of code—literate code, with 8.5x11 pages. Two of them are related to each other, so it’s more like two and a half major programs. And about 150 small programs. Probably more than I did the previous year. So I programmed galore this year on small programs but also, a couple of them were things that took a month or more to do.
Seibeclass="underline" And did you expect them to take a month?
Knuth: Well, I expected one of them to take a month. I knew that it wasn’t going to be easy but I didn’t know how much richness there was going to be, so I added more features as I got to using them. I think it is always going to be true that a person who manages programmers should not expect it to be predictable.
Seibeclass="underline" In addition to writing The Art of Computer Programming and TeX, you’re also the inventor of—and advocate for—literate programming, a way of writing code so it can be more easily read by people. And you wrote WEB and CWEB, tools that implement literate programming languages based on Pascal and C.
Knuth: So you say advocate—it’s sort of my shtick to say that this is good. But I also am the kind of guy that’s uncomfortable preaching or trying to convert someone. I think programming is a lot like religion; people have their beliefs. Some people like to force their beliefs on others. Others say, you know, here’s what I think; I can’t prove that this is the best thing, but it sure works for me. Then you hope that other people will try it and come to the same conclusion. But I don’t like going out and telling people what they ought to believe.
Seibeclass="underline" Well, maybe you can explain why you like it so much and how it differs from illiterate programming.
Knuth: The first rule of writing is to understand your audience—the better you know your reader the better you can write, of course. The second rule, for technical writing, is say everything twice in complementary ways so that the person who’s reading it has a chance to put the ideas into his or her brain in ways that reinforce each other.
So in technical writing usually there’s redundancy. Things are said both formally and informally. Or you give a definition and then you say, “Therefore, such and such is true,” which you can only understand if you’ve understood the definition.
Or you’ll say, “We define a equals the such-and-such to be the set of all leading elements.” So this informal term, the set of all leading elements, is complemented by the mathematical description of how we constructed the set a.
So literate programming is based on this idea that the best way to communicate is to say things both informally and formally that are related. And it just provides a natural framework for switching between the natural language, English, and the formal language, C or Lisp or whatever is your formal language, and putting this together. So that, to me, has to be a win for documentation.
Now, the other thing is, as I write the program, I don’t have to present it in the form that the compiler wants to see it. I present it in the form that I think is easiest for a reader to understand.
You can write your code bottom-up and make subroutines that give you bigger and bigger things and your confidence builds because now you can do more things. Other people write top-down; they start out and say, “Well, I have this problem to solve, so first I’ll do this and then I’ll do this.”
When I write a literate program I can choose between these as I like. And almost always the way my final program comes out is in the order in which I actually thought of the things myself. So I’ll start out and I’ll say, “I have this problem to solve, so first I’m going to have to solve this and then I’m going to have to solve that.”
But then I say, “Now let’s start building some tools bottom-up.” We have the goal in mind but we build a few bottom-up tools and then we’ll go back and do a little top-down. But in what order we do this is, first I write about what I thought about the first day I had to work on this problem. And then, the next chapter would be the thing I decided to tackle next.