Where's Waldo?

A recent thread on an email list I subscribe to started with the question of why there are so many roles for technically oriented people at Google (engineer, lead, manager, product manager, production assistant, etc.) and then moved to the question of how decisions can be made in such an environment. All of this was around the more general topic of how to encourage innovation. I was asked to comment as someone who had some experience in the tech industry. But this is a hard problem, needing more space than a good email. It is also a topic that might be of interest to more than the subscribers of that list. So I decided to move the discussion over here.

Let me start by saying that, in my experience, the worst (and least innovative) technical organizations are those in which there is a single career path that starts with individual contributor and moves to various levels of manager. This might work for some functions (although I can’t think of an example right off hand), but is very bad for engineering. The only real advantage that this sort of organization has is that it is clear who gets to make decisions. The higher up on the organizational tree one is, the more authority one has. It is efficient, lines of authority are clear, and if you are trying to do something technically innovative you are doomed.

This kind of organization has the well-known drawback of making bad managers out of good engineers, but it also means that many of the most important decisions that are technical in nature are made by people who are spending their time on management. Having spent time as both a technical contributor and a manager, let me assure you that it isn’t just the skill sets that are different. The main difference is the way you spend your time in the different roles.

Engineers (and, by that term, I mean anyone who is building something, whether it is a web site, a new OS, or a service) need blocks of time to think about what they are doing. There is a lot of context that needs to be swapped in to the brain, and a lot of concentration that is required to delve into all of the corners of the implications of any decision. They shouldn’t spend much time in meetings, and the unit of time for their schedule should be no less than an hour (it’s even better if it is the day).

Managers, on the other hand, are interrupt driven. If you get a full hour of time to spend on some topic, it is a bonus. A really effective manager may occasionally disappear to do long-range strategy or planning, but once that plan is in place the decision process is to compare an action to the plan and then do the action if it fits and do something else if it doesn’t fit. Contemplation is not part of the job description.

Dual Track Systems

A much better organization is one in which there is a dual career path. One path takes a person into management. The other keeps a person in engineering. And each lets a person move up the organization, increasing scope of authority, pay, and prestige. The first company that I can remember having this sort of plan was Digital Equipment Corporation. I lived in the dual track at Sun for a long time. Most high tech companies have adopted some variation of this scheme, and many that you might not think of as high tech have also decided that this is a good idea. I’d love to see something like this within the IT organization at Harvard.

Most dual track system will have a number of points at which your career can branch. You start as an individual contributor, but at some point (sometimes early on) you can opt to go into line-management or become a technical lead. Line managers and technical leads share responsibility for a development group. The technical lead makes the technical decisions, while the manager makes business decisions, deals with budgets, does reviews, and the like. Clearly, these two people need to work closely together. The manager needs to be technical enough to understand what the technical lead is doing (if for no other reason than to explain the work to other managers) while the technical lead needs to understand the business case for the technology. But the creation of the business case is the job of the manager, while the creation of the technology is the job of the technical lead.

At the best companies, this dual track continues all the way up the organization. At the old Sun (and now at Microsoft, IBM, and lots of others) there was the job of Distinguished Engineer, which was the technical equivalent of a director or first-level vice president. The D.E.s (generally) had no reports, but designed and led the implementations of major parts of the technology. Becoming a D.E. was not something that just happened as a matter of course, just as becoming a director or a v.p. did not just happen. These engineers were often associated with a manager at the same level to oversee larger and larger parts of the company. At the very top of the company, the Chief Technical Officer was the technical representative on the executive team.

Decision Making

This did mean that the authority to make decisions was often unclear. While the distinction between a technical decision (made by the technical person) and a business decision (made by the manager) might seem clear, it often isn’t. The best groups had a team of leaders who could work well together, when those two didn’t work well together, it was not a pretty. When D.E.s decided that they could manage, bad things happened. When managers decided that they were sufficiently technical to make design decisions, worse things happened.

And then there are those companies that have decided that two decision makers is not enough. The thinking there is that while there is a clear distinction between the technical aspects of a project or product and the management of that project or product, there are other stake-holders that need representation. So the role of product manager is introduced to speak for the customer. VMware had such a system, which I believe is also used at Microsoft. At its best, this form of user-representation can provide valuable input into design. At its worst, it introduces the moral equivalent of the political officer in the Soviet military– someone who can second-guess any decision without having the responsibility to actually do the work.

From what I’ve heard, Google has introduced even more roles. At best, this will allow more points of view to be represented in the decision making around what is to be done. At its worst, this will make it harder to move quickly and will make everyone feel less responsible (since there were so many voices). It will be interesting to watch; I’ll express my own skepticism that it will work, but we will see.

Missing the Point

But at bottom, all of this talk about roles, and career paths, and tracks misses the most important point. It confuses management, organizational structure, and process with leadership, which is the thing that is really important.

All of the successful groups, products, and technology development I’ve ever seen or heard about were organized (intentionally or, more often, by chance) around a strong leader. Sometimes this was a manager, more often it was a technical person, and on occasion I’ve seen it be a marketing person or a product manager. But someone needs to be willing to insist that they know where everyone should be going, and have the force of both vision and personality to get others to sign on.

This is not something that you are going to get out of any definition of roles, or career track, or process. My own observation is that when a company tries to insure innovation through one (or more) of these things, it is probably time to look around for another gig. Innovation is the sort of thing that happens in all sorts of ways, almost none of which can be replicated or scaled. It is like good design– all of the best studies show that good design comes from good designers. Innovation comes from (and is driven by) innovative people, no matter what their title or role and no matter what process they use. This makes it impossible to manage, which in turn makes many managers nervous. But it is, I’m afraid, the truth…

The great thing about starting up a blog is that you get asked interesting questions. Even something as vanilla as my introductory post brought some good comments (and some help in setting up the Word Press theme; thanks!). Then along comes Ruby Bright and asks one of the big questions. As Ruby so beautify put it,

I’m not all that sure that I have the answer to this question; if I did I’d be rich and famous. People have written books about the problem (Crossing the Chasm is probably the most famous). I’ve thought about the problem a lot. And so have a lot of others, much smarter than I. When I worked at Sun Microsystems, this was a constant topic of conversation. It caused the company to move from workstations to servers (an innovation that worked), and to support Java (much more innovative, but maybe not so successful financially), but in the end we didn’t find another innovation that the company could support. I’m sure that this is a topic of conversation (all the time) at places like Google, and Facebook, and Microsoft. Doing something new is really hard, especially when what you are currently doing is working well. It is even harder when you are well known, because then everything you do is watched by everyone, and there is the worry that failure will dull (or even ruin) your reputation.

I remember seeing this risk-aversion develop in the Java organization. When Java first came out, the people working on the project thought of it as a new way of writing software, with the virtual machine, object-oriented language, and portability from machine to machine no matter what the operating system. There were all sorts of new things that we wanted to try. Innovation piled on innovation. It was really fun.

But then Java took off, and we had other companies as partners and the price of the company stock was determined by the latest news story on something that had gone wrong (or, less often, something that had gone right) with Java. Within a couple of months the question stopped being “what new can we do with this shiny thing?” and became “how do we keep from making any mistakes?” And a lot of innovation was stopped or never started.

Fortunately for those of us who want to innovate in the IT world of Harvard, we aren’t under quite the same set of constraints. Harvard is not, at core, an IT organization (or at least it doesn’t believe that it is, at core, an IT organization– this may be the subject of a future post). Since IT is a support for the core activities of teaching and research, we are not under quite the same microscope as we would be if Harvard were a hi-tech company.

And innovation, as I continually tell my colleagues in HUIT, is not an activity that can be planned, or managed, or controlled by some process. It is messy, and unpredictable, and requires leaps of faith. It is hard. But there are a couple of things that seem to help.

I’m a big fan of skunkworks projects– small groups of engineers that go off and build something out of sight of most everyone. This is a great way to build a prototype. You need to do it fast, and you need to understand that you might fail, and be willing to take the risk. But with the right people this can work amazingly well. The first version of what became the Common Object Request Broker Architecture (CORBA) was written by a group of 5 engineers in six months in a skunkworks (I had the great pleasure of leading that team). When we did the Jini work at Sun, the team was hidden from the rest of the organization by being put (organizationally) in Human Resources. Making such an investment is something I hope we soon have the freedom to do, at least at some scale. It may be that we involve students to keep the costs down, but we should try something.

I’m also a fan of importing good ideas from elsewhere, and then making them our own. This may be in the form of adopting an open-source solution that we then customize for Harvard. It may be in the form of taking work someone else has done here, and adapting it for a new purpose. To quote the sage Tom Lehrer, “brilliance borrows, genius steals.” We should be unashamed of using the work of others (always respecting IP laws, of course, and giving credit where it is due).

But most of all, we need people to be willing to take risks. Managing projects in any organization is like managing a financial portfolio– if you want major rewards, you need to allow some risk. You don’t want every investment to involve major risk, but you need to have some to balance the low-risk parts of what you do. It is best to manage risk with some vision of where you want to go; this is a vision of the future. I’m trying to encourage some risk taking in the organization, and there is plenty of support for trying this out. We will see if it will allow us to innovate. I think it will, if we let it.