I’d written a bunch of code and there was some newsgroup where I posted that I was looking for a job and, oh, by the way, here’s a bunch of code. Peter Norvig saw it and scheduled an interview. My girlfriend at the time had moved out here to go to UC Berkeley, so I followed her out.
Seibeclass="underline" Norvig was at Berkeley then?
Zawinski: Yeah. That was a very strange job. They had a whole bunch of grad students who’d been doing research on natural language understanding; they were basically linguists who did some programming. So they wanted someone to take these bits and pieces of code they’d left behind and integrate them into one thing that actually worked.
That was incredibly difficult because I didn’t have the background to understand what in the world they were doing. So this would happen a lot: I’d be looking at something; I’d be completely stuck. I have no idea what this means, where do I go from here, what do I have to read to understand this. So I’d ask Peter. He’d be nice about it—he’d say, “It totally makes sense that you don’t understand that yet. I’ll sit down and explain it to you Tuesday.” So now I’ve got nothing to do. So I spent a lot of time working on windows system stuff and poking around with screen savers and just the kind of UI stuff that I’d been doing for fun before.
After six or eight months of that it just felt like, wow, I’m really just wasting my time. I’m not doing anything for them, and I just felt like I was on vacation. There have been times when I was working really a lot when I’d look back at that and I’m like, “Why did you quit the vacation job? What is wrong with you? They were paying you to write screen savers!”
So I ended up going to work for Lucid, which was one of the two remaining Lisp-environment developers. The thing that really made me decide to leave was just this feeling that I wasn’t accomplishing anything. And I was surrounded by people who weren’t programmers. I’m still friends with some of them; they’re good folks, but they were linguists. They were much more interested in abstract things than solving problems. I wanted to be doing something that I could point to and say, “Look, I made this neat thing.”
Seibeclass="underline" Your work at Lucid eventually gave rise to XEmacs, but when you went there originally were you working on Lisp stuff?
Zawinski: Yeah, one of the first projects I worked on was—I can’t even remember what the machine was, but it was this 16-processor parallel computer and we had this variant of Lucid Common Lisp with some control structures that would let you fork things out to different processors.
I worked a little bit on the back end of that to make the overhead of spawning a thread lower so you could do something like a parallel implementation of Fibonacci that wasn’t just completely swamped by the overhead of creating a new stack group for each thread. I really enjoyed that. It was the first time I’d gotten to use a fairly bizarre machine like that.
Before that I was bringing up Lisp on new machines. Which means basically someone’s already written the compiler back end for the new architecture and then they’ve compiled the bootstrap piece of code. So I’ve got this file full of binary, supposedly executable code for this other machine and now I’ve got to decipher their loader format so that I can write a little C program that will load that in, make the page executable, and jump to it. Then, hopefully, you get a Lisp prompt and at that point you can start loading things in by hand.
Which for every architecture was bizarre, because it’s never documented right. So it’s a matter of compiling a C program and then looking at it byte by byte—byte-editing it in Emacs. Let’s see what happens if I change this to a zero; does it stop running?
Seibeclass="underline" When you say it wasn’t documented right, was it that it wasn’t documented correctly, or it wasn’t documented at all?
Zawinski: It was usually documented and it was usually wrong. Or maybe it was just three revisions behind—who knows? But at some point you tweak a bit and then it would no longer believe this was an executable file and you had to figure out what was going on there.
Seibeclass="underline" So that’s something that comes up all the time, from the lowest-level systems programming to high-level APIs, where things just don’t work the way you expect or the way they are documented. How do you deal with that?
Zawinski: Well, you just come to expect it. The sooner you realize that your map is wrong, the sooner you’ll be able to figure out where it went wrong. In my case, I’m trying to produce an executable file. Well, I know the C compiler will produce one. Take the good one and start converting it into the bad one until it stops working. That’s primary tool of reverse engineering.
The hardest bug I’ve ever fixed, I think, was probably during that period at Lucid. I’d gotten to the point where it’s running the executable and it’s trying to bootstrap Lisp and it gets 500 instructions in and crashes. So there I am leaning on the S key, stepping through trying to figure out where it crashes. And it seems to be crashing at a different place each time. And it doesn’t make any sense. I’m reading the assembly output of this architecture I only barely understand. Finally I realize, “Oh my god, it’s doing something different when I step; maybe it’s timing-based.” Eventually I figure out that what’s going on is this is one of the early machines that did speculative execution. It would execute both sides of the branch. And GDB would always take the branch if you single-stepped past a branch instruction. There was a bug in GDB!
Seibeclass="underline" Nice.
Zawinski: Right. So then that takes me down into, “Oh my god; now I’m trying to debug GDB, which I’ve never looked at before.” The way to get around that is you’re coming up to a branch instruction and you stop before the branch, set a break point on both sides, and continue. So that was how I proved that really was what was going on. Spent like a week trying to fix GDB; couldn’t figure it out. I assume a register was getting stomped somewhere, so it always thought there was a positive value in the branch check or something like that.
So I changed the step-by-instruction command to recognize when it was coming up on a branch instruction and just say, “No, don’t do that.” Then I can just lean on the S key and it would eventually stop and I’d set the break point by hand and continue. When you’re debugging something and then you figure out that not only is the map wrong but the tools are broken—that’s a good time.
Working on Lisp systems was especially weird because GDB was completely nonfunctional on Lisp code because it doesn’t have any debug info—it’s written by a compiler GDB has never heard of. I think on some platforms it laid out the stack frames in a way GDB didn’t understand. So GDB was pretty much an assembly stepper at that point. So you wanted to get out of the GDB world just as quickly as you could.
Seibeclass="underline" And then you’d have a Lisp debugger and you’d be all set.
Zawinski: Right, yeah.
Seibeclass="underline" So somewhere in there Lucid switched directions and said, “We’re going to make a C++ IDE”.
Zawinski: That had been begun before I started working there—it was in progress. And people started shifting over from the Lisp side to the Energize side, which is what the development environment was called. It was a really good product but it was two or three years too early. Nobody, at least on the Unix side, had any idea they wanted it yet. Everyone uses them now but we had to spend a lot of time explaining to people why this was better than vi and GCC. Anyway, I’d done a bit of Emacs stuff. I guess by that point I’d already rewritten the Emacs byte compiler because—why did I do that? Right, I’d written this Rolodex phone/address-book thing.
Seibeclass="underline" Big Brother Database?