Where's Waldo?

Many thanks to Jonathan Zittrain for joining us this last week to talk about Internet governance. JZ is always thought-provoking, entertaining, and leaves you thinking about things you haven’t thought about before. I feel lucky to count him as a friend and colleague.

Talking about the Internet Engineering Task Force (IETF) is always fun and interesting. The IETF is governance as done by the geeks; it doesn’t really exist (at least legally), it has participants rather than members, and the members (even when they work for and are supported by a company or government) are expect to represent themselves, not their employers. It is a technocracy, running on the principles of loose consensus and running code. In many ways, it is just a somewhat larger version of the groups of graduate students who got together when their advisors told them to write the code for the original ARPAnet. But it is hard to argue with the power of what they produced, even if you can’t understand how they could have done it.

The other aspect of the IETF that tends to confuse people steeped in governance is its enforcement mechanism. Passing laws or creating standards isn’t much good if there is no way to encourage or force others to follow those laws or standards. After all, passing a law doesn’t do much good if you don’t have police to enforce the law.

But here the IETF is different, as well. It has no enforcement power. If you don’t implement a standard that the IETF publishes as an RFC, no one will challenge you. There are no fines to pay, and no one goes to jail. Nothing happens.

Except, of course, that you can’t communicate with any of the computers that do implement the IETF standard. Nothing says that a computer has to speak TCP/IP, and nothing happens if the computer doesn’t. Including getting the traffic from the other computers that do implement the standard.

In fact, there are lots of IETF RFCs that haven’t been implemented. There is even a group of them that are famous (well, famous in the IETF community) for being April Fools jokes. Some of my favorites are RFC 3092, an etymology of the term “foo”; and the standard for electricity over IP (RFC 3251). Not all RFCs are taken seriously, even those that are meant to be by the proposers.

But the core RFCs define the interoperability of the Internet, and as such they become self-enforcing. You don’t have to follow them, but if you don’t you are shut out of the community. And if you want to replace them, you need to get others to not only agree to the replacement, but get them to do so simultaneously with everyone else. Which is pretty much impossible. So the best approach is to simply go along with what everyone else is doing, and follow the standard.

This is much of the reason that groups like the ITU or various parts of the United Nations, that would dearly love to have control over the Internet, can’t quite figure out how to take that control. They might declare that they own the standards (they in fact have). They can insist that everyone change to use their standard (they have done this, as well). But they can’t make it worth anyone’s while to make the change, so they have no enforcement mechanism.

It’s enough to make a bureaucrat cry. Which is enough to make the geeks smile, and continue…

I’m late in posting again this week, but this time I have a reason. Our discussion last time (and many thanks to David Eaves for visiting and leading the session) was about the interaction of the Internet and government. By coincidence, I had agreed to go to Washington, D.C. on Friday of last week to give a talk to interested Senate staffers about cyber security. So I thought I’d wait until after the trip to see if going to the seat of the federal government would trigger any thoughts.

The trip was fascinating– I had been asked to give a talk that was the conclusion of a month-long series of talks and panel sessions, organized by the Sargent at Arms for the Senate, on cyber security and social media. The Sargent at Arms is, essentially, the head administrator of the Senate, running all of the departments and groups that allow the Senate to do its work. My audience was made up of members of these administrative units, along with staff members for the Senators themselves. There were about a dozen people in the room, but the talk was also broadcast on the Senate’s own version of CNN, both within the Senate office buildings (there are many) and to the field offices of the Senators.

The room where I gave my talk was one of the (many) Senate hearing rooms. It was impressive, even by Harvard standards. Beautifully painted ceiling (with a zodiac motif), high ceilings and huge windows, lots of wood and carvings, and a raised area with a table and chair for the Senators (blocked off so no one would enter the space). After the talk I got a great tour of the Capitol itself, one-on-one with a staff member of the computer security group, which let me go all kinds of places that are generally not open to the public. The size of the place, the scale of the rooms, and the history recalled were all pretty awe inspiring and a bit overwhelming.

The only places I could compare it to that I have visited are the Coliseum in Rome, the Doge’s palace in Venice, and St. Peter’s in the Vatican. All monuments to their empires, all built at the height of that empire’s power.

But as I was feeling the awe (and pride) caused by seeing the Capitol, I couldn’t help but think of the places I knew out in Silicon Valley, in the New York tech scene, or in the tech companies around Boston. None of them were as beautiful and impressive as what I was seeing. But there was also a sense of ponderousness, of self-satisfaction, and of slow moving deliberation in the halls of the Senate that contrasted sharply with the feeling of excitement, experimentation, and energy that I remember from the tech companies.

All of which makes me wonder about the effect and interaction between the world of government and the world of technology. We talked some in the seminar about how technology can improve the delivery of services by government, but often that is just government adopting technology that has been used in the rest of the world like reasonably designed web sites and APIs that allow access to information in a way that enables others to write useful applications. This may be new and different in the world of government, but has been the norm in the rest of the world for a decade or more.

David’s stated worry was that government could use technology to impose a surveillance state and become far more controlling than anything thought of by Orwell. We have seen some things (like the Snowden revelations) that might back this view up, but so far I haven’t seen evidence that the government agencies can more quickly enough or competently enough to really carry this off. Nor do I think that the government  believes that it has to…the environment in which those running the government, like the Senate, are designed to make them feel that they are already masters of the world. Why would they need to do something different?

I have a very different worry– that the tech companies will move so fast in comparison to what happens in government that they make the Senate and the rest of government pretty much irrelevant, at least on a day-to-day basis. Yes, we will need the government bodies that deal with defense and foreign affairs to continue dealing with those subjects, but our everyday lives will be molded and controlled by corporations that move so fast and change so rapidly that the government bodies that are supposed to regulate those companies, protect us from their abuse, and insure that they don’t overstep the bounds of law and ethics are simply left behind. It has taken a year for the federal government to even start investigating what role tech companies like Google, Facebook, and Twitter played in the last election. Imagine how quickly a tech company would go out of business if it took a year to react to something like that?

I’m generally an optimist, and I don’t think that tech companies (or any other kinds of companies, for that matter) are actively evil. But they are currently pretty much unanswerable to the public, and this is beginning to worry that same public. We need to find some way of addressing these issues, but it won’t be by slowing down tech so that it matches the pace of government. The time scales are too different, and the incentives are too far out of alignment. We need a new approach to these problems, one that combines speed with responsibility. Our technologists need to think beyond the technology to the effects of that technology, and our legislators and regulators need to learn to understand both where technology is and where it is going. I don’t see an easy answer, but this is a problem we need to solve.

Part of our reading for the discussion of the Internet of Things this last week was a selection from the National Academies study Embedded Everywhere. I was part of the study group, which published the book back in 2001 (after a couple years of the study). It’s interesting to look back at something like this– we got so much of the big picture right that I’m surprised, especially since we got almost all of the details wrong.

What we got right was how ubiquitous networked sensors were going to be. We saw them being used to help with traffic congestion, agriculture, climate science, seismology, and a host of other things. All of which is being done.

What we got wrong was how all this would happen. We talked about “smart dust,” a notion that the miniaturized sensors, with low-power networking and long-lasting batteries, would be sprinkled around the environment so that the data they collected could be fed back to banks of servers. There was even a project at UC Berkeley that produced what they called motes that were seen to be a first step along this path. Somewhere in my office I think I still have one or two of these hanging around. But it turned out that these were never as small as we had hoped. Batteries didn’t get better as fast as we thought they might. And calibration of large numbers of sensors turns out to be an amazing difficult problem to solve unless you have some way to get a human to deal with the individual sensors.

Instead, all this happened through cell phones. There were cell phones back when we wrote the study, but they were used as, well, phones. They didn’t have GPS embedded. They didn’t include accelerometers, or other forms of sensing. The networks they connected to were cell networks, optimized for voice and pretty bad (and limited) when transferring data. They were interesting and useful devices, but they weren’t the kinds of sensor platforms we were envisioning.

Like the early users of the ARPAnet and the Internet, we didn’t see what Moore’s Law and the acceleration of network technology was going to do to our little phones. Within a short number of years Apple introduced the iPhone, which was really a small networked computer with enough sensors to make things interesting. Restrictions on the accuracy of civilian GPS were lifted just before the study was published, but we had no idea the size of the impact that would have. As sensors, cameras, and microphones became smaller and better, they got pushed into the phones. The networks for the cell phone system got better and better, both in bandwidth and reliability. Calibration ceased to be a problem, since all of the sensors were paired with a person who could help with the calibration. Soon all of the data we had hypothesized being sent to clusters of servers was being gathered and sent. Just by a different set of technologies than we had been able to imagine.

The connection of people to the networked sensors caused other good things to happen, as well. People could get involved in the projects that we originally thought would be machine-only. There were community government projects to allow people to report pot-holes (automatically, based on bumps that their cars encountered), damaged sidewalks (where pedestrians could take a picture of the sidewalk, with a GPS tag, and send it to the local government), and the monitoring of air pollution or how well transit systems were keeping to their published schedules (which have given way to real-time schedules that tell you where the busses or trains are, not where they are expected to be).

It’s another reminder to all of us technologists. We tend to think of what the technology can do on its own. But the most valuable uses of the technology pairs the technology with people who use it, sometimes in unexpected ways. We think of how we can use machines to replace us, rather than how we will use machines to enhance what we do. But the valuable uses of technology are in enhancing the human enterprise, and that’s how we end up using the technology, even when it wasn’t designed for that. A lesson that we need to keep in mind, since we seem to constantly forget it.

End-to-end Arguments in System Design is one of my favorite papers in the computer science universe. It is well written, and it clearly states a design principle that was followed in the creation of the network we now know as the Internet. It gives some nice examples. What more could you want?

Well, I’d like more papers like that. In the computer science/software engineering/programming community, we don’t often talk about what makes a good design, or why we made the design decisions that we make. Oh, there are lots of book and articles that tell you how you ought to go about doing design. Maybe you should use pattern languages, or you should adopt an agile methodology, or you should do everything in UML diagrams. I’ve even seen combinations, saying that you should use an agile pattern language (no doubt producing UML diagrams). All of these can be useful as mechanisms for communication of a design, but I don’t find any of them actually talking about what makes a good design good, or how to go about doing such a design.

Writing about good design is far rarer. There is Butler Lampson’s classic Hints for Computer System Design. This is a great paper, but it is also pretty old (although not, surprisingly, outdated). There are a couple of books that tried to talk about design at different levels (Beautiful Code for programming, and Beautiful Architecture for collections of programs), but the results in both are mixed (full disclaimer; I wrote a chapter in the second and am not at all sure that it was much of a success). I’ve always like Paul Graham’s Hackers and Painters (the blog rather than the book), but it is more a contemplation on the art and craft of programming rather than on design. I’ve tried to talk about how to get good design, but it is a very slippery topic. Even Fred Brooks has written a book on the subject, which I like but is also, in the end, somewhat less than satisfying.

One of the reasons, I believe, for the lack of literature on the subject is that it is so hard to say anything that doesn’t seem to be either trite or just wrong. We all can agree that simplicity is a good thing, but what makes a design simple? Breaking a system into modules that make sense is trite; saying what it is for a module to make sense is difficult. You can recognize a good design when you see one, but explaining what a good design will be before the fact– well, I’ve never seen that done all that well.

After thinking about this for some time, I’ve come to the conclusion that good design is more art than science, more craft and taste than knowledge and process. As Brooks says, often the best that one can say is that good design comes from good designers. Good designers are often trained by having apprenticed with other good designers, but that’s about as far as you can go with the explanation. Even that training (which is really an apprenticeship) may not be all that conscious. I know of companies that set up mentoring programs in the hopes of getting good design training, but my apprenticeship (and I remember it well) was much more like that in Kill Bill, Vol. II– I spent a lot of time doing the computing equivalent of carrying water, and often felt bruised and battered. But I learned.

This is another way in which system design is like art– the way you learn to be an artist is to try, hear criticism from a  more experienced artist, and try again. Like a painting, the design of a system is never finished– there are always things you would like to change, decisions you would like to revisit. Sometimes you decide that it is good enough, and you ship. But you know you could always do better.

Our discussion in our Freshman Seminar this last week concerned how technology in general and the Internet in particular could be used to help governments (at all levels) be more responsive and deliver services better. We were fortunate to have David Eaves join us; he has been an advocate for open data and using technology in government for some time, so getting advice from someone who has been there is really helpful.

What I keep thinking about, after the discussion, is the number of large technical projects within various parts of government that fail. There are the really visible failures like healthcare.gov (on its initial rollout) or the attempts to implement a case management system for the FBI or attempts to replace the air traffic control system (which has failed multiple times). Depending on the statistics you want to cite, 72% of government IT projects were in trouble in 2009, or that 70% of government IT projects fail, or that only 6.4% are successful. David claimed that things are not that different outside of the government, and you can certainly find studies that agree with this. In fact, reading these studies, it is surprising that anything ever works at all.

My problem with all these studies is that they fly in the face of my own experience. I spent about 30 years developing software in industry. I was on lots of projects. There were some that weren’t successful in the market for one reason or another. There were a couple that were stopped when the company I was working for got bought and the new owners decided that those projects weren’t the sorts of things they were interested in. But I was never on one that failed because we just couldn’t get the system to work.

David did distinguish between projects that were done in “technology” companies verses those done by everyone else, and I certainly worked in technology companies. But over the past 6 years I’ve been working in one part or another of Harvard Information Technology. Harvard is hardly a technology company (don’t get me started…), but in that time we have successfully rolled out a new course management system, a new student information system, re-vamped the Identity and Access Management system, moved most of email from local servers to the cloud, and done a ton of other projects as well. Not all of them have done exactly what everyone hoped they would do, but that have all pretty much worked. None had to be abandoned, or re-written from scratch, or got deployed and then turned into a disaster.

So what is the difference between the facts and figures that we see about project failure and my own experience? Maybe I have some sort of magic about me, so that projects I join or observe are somehow saved from the fate of all of these others. That would be really useful, but I don’t think it is the right explanation. I think I’m good, but I don’t think I’m magic.

I’m more inclined to think that the difference has to do with what the managers of the projects care about. In most of the government projects I’ve heard about, and in lots of the non-governmental projects that have failed, managers have been more concerned about how things get done than anything else. That is, the worry is what kind of process gets followed. Is the project being run using the waterfall model (which was first discussed in a paper saying that it was the wrong way to manage a software project) or various forms of agile development (which is a new cult), or some other method? These are approaches that managers really hope will make the development of software predictable, manageable, and most importantly, independent of the people who are on the project. All of these models try to make the developers interchangeable parts who just do their job in the same way. Doesn’t matter who is on the project, as long as the right process is followed.

This is in contrast to what I saw through my career, and what I see in companies that might be thought of as “tech” companies now. In these projects, the worry was all about who was on the project. There was a time I gave talks about what I called the Magnificent Seven approach to software projects. The process was straightforward: hire a small group of experienced professionals, let them deal with the problem as they saw fit, and if you found a kid who could catch fish barehanded ask him or her along. This was hardly an idea that I came up with by myself; you can see it in The Mythical Man Month and other things written by Fred Brooks.

A process-based approach seems a lot more egalitarian, and in some ways a lot more fair. It means that you never have to tell someone that they aren’t good enough to do the job. It is good for the company, because you don’t have to pay outrageous salaries to engineers who are probably a pain in the tail to manage because they think (often rightly) that the company need them more than they need the job (since, if they really are that good, they can find another job easily). So I certainly understand why managers, and large bureaucracies like various levels of government, want to focus on process rather than individual talent.

But then you have to explain the difference in success rates. If focusing on process gives a success rate somewhere between 40% and 5%, and focusing on talent does a lot better (I don’t have numbers, but my anecdotal experience would put the success rate of really high performance teams at the 85%+ range), then maybe making quality distinctions isn’t a bad idea. I’m not sure how you get the various levels of government to accept this, but I think if we are going to have governments that are dependent on good technology, we need to figure out a way.

A recent graduate with whom I did considerable work recently wrote and asked me the following question:

What are your thoughts about how academia versus industry contribute to the expansion of the world’s knowledge? My thoughts from speaking to people from both sides are that it seems that in academia, knowledge is much more open to sharing, but progress is much slower (because of multiple things: the need to apply for grant money, the multiple roles that professors must juggle, the difficulty of the problems being tackled, the lack of readily accessible consumer data, etc.), and that in industry, there are probably tens or hundreds of companies trying to do the same thing (like spam detection, for example) but who do not want to share their knowledge, but that the competition and that the money incentive makes development faster. Do you think that working in one versus the other is more effective or efficient if the furthering of knowledge is the ultimate goal? This is one of the big questions I’m thinking about as I’m trying to figure out where I best see myself in 10 years.

I’ve been thinking about this problem, and some variants, for a considerable period of time, so the response was fairly length. On the off chance that others might be interested, I post it here:

And this is why I love working with students— they always ask the simple questions :-).

There may have been a time when the answer to this was actually simple. The idea was that academics worked on long-term, highly speculative research topics, generally funded by the government, and published papers for all the world to see. Then industry would pick up some of these ideas, figure out the commercial application, apply the pure research, and take the product that resulted from applying that research to market. Academics worked on the long-term (5-10 year) problems, and industry worked on the short term (.5-2 years). A simple and rational pipeline, with each group knowing what it was doing and how it interacted with the other. If you wanted to work on the high-risk, high-impact (but likely to fail) stuff, you became an academic. If you wanted to have immediate impact on the real world, with some better guarantee of success, you worked in industry.

This is a nice picture, but if it has ever actually been true in the computer/software sector, it was before my time. As long as I’ve been watching and taking part, real research was done all over a combination of academia, industrial research labs (Bell Labs, BBN, Xerox PARC, Digital’s SRC, Sun Labs, MSR) and people in product groups at various companies, both large and small. I remember a time when it seemed that everyone in the industry worked in Research and Development; the difference between being an academic and working for a company was whether you wrote the paper before or after releasing the product.

But there are some changes that have occurred over the past 10 or more years that have made the picture even more muddled than it was, and thus make the question harder to answer.

One of these changes is that the amount of research funding going to academics in the CS field has been going down, and the risk profile of the funding agencies has been going down with it. There was a time when NSF and DARPA would give out lots of money, and be willing to take chances on things (like the internet) that didn’t have any obvious payback and might not have worked at all (there were plenty of research projects that didn’t). As the amount of money has decreased, the willingness for risk has decreased, as well— while there are some projects that seem pretty risky, for the most part the perception is that for a project to get funded, you need to show that it will succeed. This leads to people getting startup funding to do a year or so of research, applying to funding agencies for that work (or a small follow on to the work), and then using the funding to do something new that they can then ask for the next round of funding to support. Again, it isn’t always like this, but it is like this often enough that it can be problematic. By the way, it may not be that the actual amount of money has gone down (I believe that it has), but the amount hasn’t gone up in proportion to the number of academics who are trying to get it. So that makes things strange, as well.

At the same time, the number of industrial research labs seems to be decreasing, along with the funding available for such labs. Big places are willing to do some really adventurous stuff (look at Google X, or at least the rumors from Google X), but the work is not done in the open, may not be shared, and when it is often is covered by patents. Which is natural; these companies want a return on their investment. But it does limit the scope of the spread of the innovation or knowledge. In a lot of cases, companies are now willing to take more of a chance on research than academics, because they realize that the payoff to being the first to figure something out is so huge. So some of the really speculative, long-range work is being done in industry, but you hardly hear about it (think of the self-driving car).

And then there is a third trend. What seems to be happening more and more is that innovation and real research is being outsourced to startup companies. If you have an innovative idea, you start a company and work on the idea. Then if the idea works out, you either take over the world or (more likely) get bought by a larger company. This is a really attractive model for those who fund innovation; they have essentially out-sourced the problem to the VC community. The government doesn’t have to spend any money at all. The existing companies only have to buy when they know that the idea works. And the VCs are willing to put up the initial money, because the payback for the companies that get bought or get to go public is large enough to make the initial investment profitable. This certainly seems to be the way Oracle does research these days (they don’t even do much hiring; most of the people they add to the company come in through company acquisition). Ryan Adams recently had his little company bought by Yahoo, so sometimes the line between academic, startup, and established company can be even more blurred.

Of course, such outsourcing also means that the time depth of start-up company research is dictated by the patience of the VC community, which seems to be limited to a couple of years (at best). And the research better have a clear commercial application. 

All of this has to do only with how the initial research gets funded. The real question centers on how you could most effectively add to human knowledge. Which is a lot harder than just getting funding, because once you get funding, you then need some way to get people to recognize what you have done.

Academics do this by writing papers and giving talks, which sometimes works. But there are a lot of papers that get written and never really get read, a lot of talks that are heard and immediately forgotten. By the same token, there are lots of products that are really great pieces of technology that, for one reason or another, never get adopted. Before inventing Java, James Gosling invented NeWS, an elegant and fully functional window system. But NeWS went nowhere; instead the industry of the time adopted X-windows, which a lot of us thought was not technically as nice. Dick Gabriel and I have been arguing over why LISP lost out to C or Multics lost out to Unix for a long time, but whatever the reason was it was not purely technical. I remember being told by Ivan Sutherland, who has done more great technology than just about anyone I know, that all you can do as a technologist is make sure that the work is good; adoption is a question outside of your control. A hard lesson, but true.

After all of this evasion, let me try to answer the real question, which is what should you do if you want to push forward the boundaries of knowledge? And the answer will depend on how you want to live your life, not on which is more likely to push those boundaries successfully. As an academic, you have much more freedom, deciding on your own research agenda and working with students who you will get to advise and direct. In industry, you probably won’t have that sort of freedom for 5 to 10 years (and that’s if you enter with a Ph.D.); you will be part of someone else’s group for that time, working on their problems. But while in industry you will not have the worries over funding (my funding cycle when at Sun Labs was a couple of weeks), and the people you will be working with will have a higher level of skill and discipline than is generally found in students. But the problems you will work on will be constrained, to some extent, by the market and you may not be able to share all you know. The environment of a startup gives you some of the freedoms of an academic, but also brings the pressures of the market. 

And, of course, a final consideration is just what is meant by “furthering human knowledge.” One way of doing this is to come up with something that no one has ever thought of before, and getting everyone to understand it. This might be a new theorem, a new system, or a better way of doing things. Java, for all its flaws, certainly contributed to human knowledge in some way; the IP protocols did the same. But these sorts of contributions are few and far between. When they happen, it is a combination of insight, perspiration, and luck; no one knows how it really happens, but when it does it is pretty amazing.

But the other way to further human knowledge is to train the next generation in how to further knowledge. This can also be done in all of the contexts spoken about above— I mentored a lot of people when I was in industry, start-ups teach in their own kind of way, and being a professor is (supposedly) explicitly about that. As my career has moved along, I’ve grown to appreciate this way of furthering knowledge more and more; it may not be as romantic or immediately satisfying, but you end up playing the long game, knowing that your contributions will outlast you and not be subject to quite so many whims. Which is the other reason that I love working with students— it is part of my attempted contribution to the expansion of human knowledge.

A lot of Beatle’s songs have been running through my head this last little while, but Sgt. Pepper is getting most of the mental traffic. In part this is because of a recent personal anniversary (will you still need me?) but mostly because it was 20 years ago that Java(tm) was first made public. My, how time flies.

There was actually a bit of a debate on the JavaSoft (that was the part of Sun that was responsible for Java) mailing lists; the first public release of the alpha technology preview of Java went out March 27, 1995, while the first announcement of the language and environment was during the Sun Developers Conference held May 20-something in that year. To give a bit of context, that was the same Sun Developer’s Conference when the Memo of Understanding between Sun and Netscape that placed Java in the browser was announced. For those of you who don’t know what Netscape was, go look it up. At the time, Netscape was a much bigger deal than Java, but time has a way of changing those things.

Java had actually been wandering around Sun internally for a couple of years before that; during the 10 year anniversary I found that I had the electronic copy of the oldest known (then) spec for the language, which you can see here. First note that the language was not then called Java. It was originally named Oak, after the tree outside of James Gosling’s office. But the trademark folks said that name was already taken, so the commercial name was changed to Java. The rest, as they say, is history.

The Oak spec is recognizably Java, but is also different. The exception mechanism at the time did not require that exceptions be declared or caught; this was changed before the first official release. The language was a lot smaller (full description in under 30 pages), and the mass of libraries, frameworks, and imported features from other languages weren’t there. This is a snapshot of Java in its purer form, before it became important enough to be fought over by different companies, before the community process, and before all those who wanted it to be something else had their way with it.

I still like to program in Java (the Standard Edition; don’t get me started on the Enterprise Edition), since at its core it is still the simple, object-oriented, strongly typed language that it started out being. I do use generics, even though I hate the way they were implemented and argued against them (and still think they aren’t needed). I’m glad to see lambdas introduced, although I could have lived without them. And I do sometimes wish that we had had a few more years to develop the language before it became the global phenomenon it became. But it ruled much of my life in technology, and it was fun.

So happy birthday, Java. And thanks for all the good times. The bad ones weren’t your fault…

This semester I’ve had the good fortune to be involved in the pilot of the Digital Problem Solving Initiative, a program being started out of the law school and the Berkman Center to allow Harvard students (from all schools) to get some mentored experience in dealing with various sorts of digital solutions. I’ve been leading a group of law students, FAS graduate students, and college students looking at some of the issues around the research data coming in from the HarvardX courses that are running on the edX platform. It’s a great group, and we have a nice combination of law questions (just what does FERPA require, anyway?), policy matters (how do we keep privacy in these sorts of data sets) and programming problems (they really think we can use *this* data?).

The group is a lot of fun, and we’ve had a stream of visitors as well to liven things up. A couple of weeks ago, Doc Searles and his wife joined us. Doc is one of the authors of the Cluetrain Manifesto, and I mentioned that it was one of the three or four books that all programmers should read. After the session, a couple of the participants asked what the others were.

Which got me to thinking. Most of the people who talk about the three or four books that everyone should read only tell you one of the books at any one time, and if you take the union of all those times the total is in the 20s or 30s. So trying to come up with all of them at once is difficult, and takes some thinking. It’s much easier to come up with one (the “desert island book”), or 20. But four?

But here we go…

First on the list would be Tracey Kidder’s The Soul of a New Machine. This is a non-fiction report of the development of a 32-bit minicomputer by Data General in the 1980s. But the reason to read a book about a long-forgotten machine with what is now a laughable computing capability by a company that no longer exists is to see how little the technology industry changes. The personalities, problems, and solutions described in this book are as relevant and real now as they were then. There are the same personalities (the architect, the team lead, marketing, management), the same situations (dispair, joy, post-shipping depression), and the same unvarying truths (the way to get the impossible done is to give the task to an intern, but make sure you don’t tell the intern it is impossible). I re-read this book every couple of years, and it still rings true.

Second on the list is Fred Brooke’s The Mythical Man Month. Talking about a machine that predates the one described in Kidder’s book, this book contains most of what is true about software engineering. This is the book that first enunciated the principle that adding people to a late project makes it later, but there is much more here. This is where I learned how to estimate how long a project will take, how to size a team, and so much else. The problem with this book is it is such a classic the everyone reads it early in their career (or education) and then forgets it. It should be re-read early and often. These truths don’t change, and the combination of this book and Kidder’s will remind you that high technology is still mostly about people (or, as I’m sometimes known to say, the technical problems are easy compared to the problems of primate behavior).

Third on the list is the aforementioned Cluetrain Manifesto. It packs a lot of truth about the new network world in a small number of patges, and is necessary reading for those who deal with the new internet world. A lot of what is said in this work is now common knowledge, so I sometimes worry that those reading it now won’t understand how radical it was when it was first published (and on the Web, no less). But other parts of the book are still not clearly understood, and are good to read.

My fourth book would be Kernighan and Ritchie, The C Programming Language, more generally known by its initials, K and R. This is the only overtly technical book on the list, and given how few people live their lives in the C language anymore, may seem an odd choice. Programmers shouldn’t read this book to learn the C language (although you can’t read it and not learn the C language). Programmers should read this book to understand how computers work, and to see an example of great writing about a technical subject.

Any computer language is a model for computation. COBOL models computation as a filing cabinet, allowing you to take something out of the cabinet, do some simple things with it, and put it back. Java models computation as a set of interacting objects (unless you use J2EE, in which the model is, well, more like a filing cabinet). LISP models computation as a set of composable functions.

C models computation as a computer; more precisely C models computation as a PDP-11. But this isn’t a bad model to learn, as most of our current computers are, at heart, PDP-11s. Learning C lets you understand what the computer is doing; it is the most useful assembly language you can learn.

But the main reason for reading K and R is to be exposed to the most elegant exposition of a programming language (and programming) I know. The elegance of the writing is so pervasive that you don’t even notice it (which is true elegance), but everything just makes sense. As a model for how to explain the complex, there is no better example.

And those are my four. Unlike most books in the software section of your bookstore (or Amazon), I’m reasonably confident that they will be the four I would pick in ten years. They have all not only aged well, but become better with age. Something we can all aspire to do, both in our work and our selves.

A recent thread on an email list I subscribe to started with the question of why there are so many roles for technically oriented people at Google (engineer, lead, manager, product manager, production assistant, etc.) and then moved to the question of how decisions can be made in such an environment. All of this was around the more general topic of how to encourage innovation. I was asked to comment as someone who had some experience in the tech industry. But this is a hard problem, needing more space than a good email. It is also a topic that might be of interest to more than the subscribers of that list. So I decided to move the discussion over here.

Let me start by saying that, in my experience, the worst (and least innovative) technical organizations are those in which there is a single career path that starts with individual contributor and moves to various levels of manager. This might work for some functions (although I can’t think of an example right off hand), but is very bad for engineering. The only real advantage that this sort of organization has is that it is clear who gets to make decisions. The higher up on the organizational tree one is, the more authority one has. It is efficient, lines of authority are clear, and if you are trying to do something technically innovative you are doomed.

This kind of organization has the well-known drawback of making bad managers out of good engineers, but it also means that many of the most important decisions that are technical in nature are made by people who are spending their time on management. Having spent time as both a technical contributor and a manager, let me assure you that it isn’t just the skill sets that are different. The main difference is the way you spend your time in the different roles.

Engineers (and, by that term, I mean anyone who is building something, whether it is a web site, a new OS, or a service) need blocks of time to think about what they are doing. There is a lot of context that needs to be swapped in to the brain, and a lot of concentration that is required to delve into all of the corners of the implications of any decision. They shouldn’t spend much time in meetings, and the unit of time for their schedule should be no less than an hour (it’s even better if it is the day).

Managers, on the other hand, are interrupt driven. If you get a full hour of time to spend on some topic, it is a bonus. A really effective manager may occasionally disappear to do long-range strategy or planning, but once that plan is in place the decision process is to compare an action to the plan and then do the action if it fits and do something else if it doesn’t fit. Contemplation is not part of the job description.

Dual Track Systems

A much better organization is one in which there is a dual career path. One path takes a person into management. The other keeps a person in engineering. And each lets a person move up the organization, increasing scope of authority, pay, and prestige. The first company that I can remember having this sort of plan was Digital Equipment Corporation. I lived in the dual track at Sun for a long time. Most high tech companies have adopted some variation of this scheme, and many that you might not think of as high tech have also decided that this is a good idea. I’d love to see something like this within the IT organization at Harvard.

Most dual track system will have a number of points at which your career can branch. You start as an individual contributor, but at some point (sometimes early on) you can opt to go into line-management or become a technical lead. Line managers and technical leads share responsibility for a development group. The technical lead makes the technical decisions, while the manager makes business decisions, deals with budgets, does reviews, and the like. Clearly, these two people need to work closely together. The manager needs to be technical enough to understand what the technical lead is doing (if for no other reason than to explain the work to other managers) while the technical lead needs to understand the business case for the technology. But the creation of the business case is the job of the manager, while the creation of the technology is the job of the technical lead.

At the best companies, this dual track continues all the way up the organization. At the old Sun (and now at Microsoft, IBM, and lots of others) there was the job of Distinguished Engineer, which was the technical equivalent of a director or first-level vice president. The D.E.s (generally) had no reports, but designed and led the implementations of major parts of the technology. Becoming a D.E. was not something that just happened as a matter of course, just as becoming a director or a v.p. did not just happen. These engineers were often associated with a manager at the same level to oversee larger and larger parts of the company. At the very top of the company, the Chief Technical Officer was the technical representative on the executive team.

Decision Making

This did mean that the authority to make decisions was often unclear. While the distinction between a technical decision (made by the technical person) and a business decision (made by the manager) might seem clear, it often isn’t. The best groups had a team of leaders who could work well together, when those two didn’t work well together, it was not a pretty. When D.E.s decided that they could manage, bad things happened. When managers decided that they were sufficiently technical to make design decisions, worse things happened.

And then there are those companies that have decided that two decision makers is not enough. The thinking there is that while there is a clear distinction between the technical aspects of a project or product and the management of that project or product, there are other stake-holders that need representation. So the role of product manager is introduced to speak for the customer. VMware had such a system, which I believe is also used at Microsoft. At its best, this form of user-representation can provide valuable input into design. At its worst, it introduces the moral equivalent of the political officer in the Soviet military– someone who can second-guess any decision without having the responsibility to actually do the work.

From what I’ve heard, Google has introduced even more roles. At best, this will allow more points of view to be represented in the decision making around what is to be done. At its worst, this will make it harder to move quickly and will make everyone feel less responsible (since there were so many voices). It will be interesting to watch; I’ll express my own skepticism that it will work, but we will see.

Missing the Point

But at bottom, all of this talk about roles, and career paths, and tracks misses the most important point. It confuses management, organizational structure, and process with leadership, which is the thing that is really important.

All of the successful groups, products, and technology development I’ve ever seen or heard about were organized (intentionally or, more often, by chance) around a strong leader. Sometimes this was a manager, more often it was a technical person, and on occasion I’ve seen it be a marketing person or a product manager. But someone needs to be willing to insist that they know where everyone should be going, and have the force of both vision and personality to get others to sign on.

This is not something that you are going to get out of any definition of roles, or career track, or process. My own observation is that when a company tries to insure innovation through one (or more) of these things, it is probably time to look around for another gig. Innovation is the sort of thing that happens in all sorts of ways, almost none of which can be replicated or scaled. It is like good design– all of the best studies show that good design comes from good designers. Innovation comes from (and is driven by) innovative people, no matter what their title or role and no matter what process they use. This makes it impossible to manage, which in turn makes many managers nervous. But it is, I’m afraid, the truth…

The great thing about starting up a blog is that you get asked interesting questions. Even something as vanilla as my introductory post brought some good comments (and some help in setting up the Word Press theme; thanks!). Then along comes Ruby Bright and asks one of the big questions. As Ruby so beautify put it,

I’m not all that sure that I have the answer to this question; if I did I’d be rich and famous. People have written books about the problem (Crossing the Chasm is probably the most famous). I’ve thought about the problem a lot. And so have a lot of others, much smarter than I. When I worked at Sun Microsystems, this was a constant topic of conversation. It caused the company to move from workstations to servers (an innovation that worked), and to support Java (much more innovative, but maybe not so successful financially), but in the end we didn’t find another innovation that the company could support. I’m sure that this is a topic of conversation (all the time) at places like Google, and Facebook, and Microsoft. Doing something new is really hard, especially when what you are currently doing is working well. It is even harder when you are well known, because then everything you do is watched by everyone, and there is the worry that failure will dull (or even ruin) your reputation.

I remember seeing this risk-aversion develop in the Java organization. When Java first came out, the people working on the project thought of it as a new way of writing software, with the virtual machine, object-oriented language, and portability from machine to machine no matter what the operating system. There were all sorts of new things that we wanted to try. Innovation piled on innovation. It was really fun.

But then Java took off, and we had other companies as partners and the price of the company stock was determined by the latest news story on something that had gone wrong (or, less often, something that had gone right) with Java. Within a couple of months the question stopped being “what new can we do with this shiny thing?” and became “how do we keep from making any mistakes?” And a lot of innovation was stopped or never started.

Fortunately for those of us who want to innovate in the IT world of Harvard, we aren’t under quite the same set of constraints. Harvard is not, at core, an IT organization (or at least it doesn’t believe that it is, at core, an IT organization– this may be the subject of a future post). Since IT is a support for the core activities of teaching and research, we are not under quite the same microscope as we would be if Harvard were a hi-tech company.

And innovation, as I continually tell my colleagues in HUIT, is not an activity that can be planned, or managed, or controlled by some process. It is messy, and unpredictable, and requires leaps of faith. It is hard. But there are a couple of things that seem to help.

I’m a big fan of skunkworks projects– small groups of engineers that go off and build something out of sight of most everyone. This is a great way to build a prototype. You need to do it fast, and you need to understand that you might fail, and be willing to take the risk. But with the right people this can work amazingly well. The first version of what became the Common Object Request Broker Architecture (CORBA) was written by a group of 5 engineers in six months in a skunkworks (I had the great pleasure of leading that team). When we did the Jini work at Sun, the team was hidden from the rest of the organization by being put (organizationally) in Human Resources. Making such an investment is something I hope we soon have the freedom to do, at least at some scale. It may be that we involve students to keep the costs down, but we should try something.

I’m also a fan of importing good ideas from elsewhere, and then making them our own. This may be in the form of adopting an open-source solution that we then customize for Harvard. It may be in the form of taking work someone else has done here, and adapting it for a new purpose. To quote the sage Tom Lehrer, “brilliance borrows, genius steals.” We should be unashamed of using the work of others (always respecting IP laws, of course, and giving credit where it is due).

But most of all, we need people to be willing to take risks. Managing projects in any organization is like managing a financial portfolio– if you want major rewards, you need to allow some risk. You don’t want every investment to involve major risk, but you need to have some to balance the low-risk parts of what you do. It is best to manage risk with some vision of where you want to go; this is a vision of the future. I’m trying to encourage some risk taking in the organization, and there is plenty of support for trying this out. We will see if it will allow us to innovate. I think it will, if we let it.