CQ2

I’ve recently moved into a new job at Novell, working on our strategy for worldwide services and planning for our next fiscal year is keeping me busy.  But I still, fortunately, deal with real clients and real problems too.  This one is classic: the client has several hundred old Unix and RHEL servers that they want to move to SLES.  Great!  We want to help.  So they negotiate the server deal and then want to know the cost to migrate.  How much is it going to cost, in total, to go from what they have today to what they want tomorrow?  They ask for estimates on a per-server basis; how many hours would it take to migrate a Solaris server to SLES?  Ten hours?  A thousand hours?  So they bring in the consultants, the dreaded consultants.  They’ve tried to avoid slowing down the deal but there’s no avoiding it now.

Well, you’ve done this before, they say, you’re grizzled veterans of the data center; is it two or ten hours for a server?  And the consultant — and I’ve been in this situation, believe me, it sucks — has to say, “Well, it depends.  It could be a thousand hours.”  Which is what everyone is expecting him to say because you can’t get a straight answer out of a consultant.  They’re always going to tell you “it depends.”  Right.

And even if there is all the time in the world, this particular answer needs to be in writing on the buyer’s desk by EOD today or the sales guy isn’t going to make his number for the quarter which means that he’s not going to make ‘club’ (his incentive travel event), which his wife is really looking forward to, so this damned consultant is not only not answering a simple, reasonable question from the beloved customer but they are also very directly making his wife mad at him, with the attendant consequences.

Perhaps you think I joke?  Or exaggerate?

Making matters worse, some nerd named Chad has downloaded OpenSUSE onto a machine in their testing lab and moved a couple of apps without incident (some directory changes, a few lines of code) and based on that experience has estimated that moving the three hundred servers will take approximately an hour each.  Seriously: we have clients who want us to tell them that moving unknown production workloads from one operating system to another will take less than two hours per server.

So the consultant sighs and starts to ask questions: What do the workloads on these servers actually do?  Online banking is different from warehouse management.  What platforms are they running?  (What version of J2EE?  What version of RHEL?  What version of Manugistics?)  Are they going to change anything else besides the operating system when they do this move?  Is the software custom or off-the-shelf?  What’s it written in?  If they say something like current Java apps running on a 2.6 kernel going to the same JVM on another distribution, that would be one thing.  If you are looking at non-ANSI C custom code on RHEL 3 on a complex multi-tiered app, that’s something else.  (Moving from the 2.4 kernel to the 2.6 kernel on any distribution is much harder than moving from one current distribution to another.)  What about storage, and backup, and disaster recovery?  Systems management?  There are a thousand more architectural details that you need to understand (one data center or many?  resource utilization?) but everyone is getting impatient with you and your endless questions.

Then you start getting into the enterprise-y aspects, which is where the real time and cost come in.  There’s a difference between Chad moving an app from one platform to another as a technical exercise and the actual time that it takes production applications to go from one to another.  What’s the testing regime?  I would expect that production code moving from one distribution to another would require real testing (stress/performance, UAT, etc.).  Would you include that in the estimate?  What about security?  Does the new OS have to go through a security audit at the company?  (Answer: yes, and it’s going to take a long time for the online banking app, believe me.)  Documentation?

This is all super-boring and bureaucratic and definitely not technical so the nerds aren’t interested and think it’s worthless and the sales guy is hearing his wife screaming at him and the buyer is saying, “Why is this so complicated?”

So, should we skip the backup part?

Really, the way to do this kind of thing is to do a quick assessment and figure out some kind of prioritization and rough sequencing, but that would require the client to spend time and money helping you to figure out how much to charge them and they are naturally leery of such a thing.  You desperately want to avoid getting locked into a fixed figure because you still have no real idea how complex the problem your being asked to solve is, but that is what the client and the others are asking for.

So you end up with a fudge; you commit to moving some edge servers and a cluster of supposedly simple apps and you sign up to do a security-approved core build and an assessment for the rest so that the project can get started and the customer can show progress to their boss and the sales guy can make his number.

Now you’re faced with months in the lab at the client site with Chad explaining to you how completely screwed up their environment is and how there’s no way that he’s going to give up his Solaris servers and anyway they’ve tried to do this themselves a bunch of times already and it never works because it’s not really a current release of Manugistics and they did some customization that they probably shouldn’t have…

Michael Nygard has his head in the computing clouds, suggesting that not only is cloud computing in our future, but that there’ll be many of them. He’s right.

Everyone who runs a large data center is today faced with the same set of interconnected environmental problems; space, power, and heating/cooling. And these are environmental not just in the sense of tree-hugging but also in a straightforward practical sense: there is no more space, there is no more power, there is too much heat and not enough cooling. These problems were the domain of junior people a few years ago, worrying about where, physically, to locate all the new Windows boxes. Then it was middle managers trying to sort out power and HVAC issues: “If we deploy a new phone system in our building we won’t have enough power to do any upgrades in the data center,” that sort of thing. Now environmental issues are front-and-center for senior IT management and if you’re a “red-shift” kind of company, for senior corporate leadership too.

You can cloak it if you want to in green terms but businesses are faced with real operational issues that they need to address regardless of their perspective on global warming or riverine dolphins.

Alongside these environmental issues, data centers are also facing a crisis of manageability. A large enterprise data center is a staggeringly complex thing, too complicated. Also, if the truth be told, most of them are not that well run; would you expect, for example, that an auto parts distributor would have great technology management skills? No, of course not, and the fact is that they probably wouldn’t want to spend the money to acquire that talent and technology even in they could; their differentiation, the competitive advantage of their business, lies elsewhere. So they have a complicated, and sub-optimized, technology infrastructure.

The answer to all of these problems — Monday edition — supposedly lies in virtualization. Novell gets brought into these conversations because inevitably data center managers have a roadmap that looks something like this:

(more…)

The question that I’ve always had about agile methods is: where does the project come from? Based on my limited knowledge (and I’m like a like a pagan at a theology convention here), the agile movement assumes a defined project or problem at the outset and then figures out where people should sit: by themselves, with a friend, or with a group. This is all fine to me; you take your Mountain Dew and sit wherever you want. But where is the problem coming from? Are you working on the right problem? How do you know?

I understand user stories and all that, but at that point you’ve already dedicated a team to working on the problem and so they go and — gasp! — actually talk to users. But how did they get tasked with that problem, the redesign of the inventory reorder system say, rather than some other problem, updating the contractor billing system, say? Do agile methods go upstream?

12 April 1865, Appomattomax courthouse.

Joshua Chamberlain was selected to receive the Confederate surrender. He describes the scene thus:

The momentous meaning of this occasion impressed me deeply. I resolved to mark it by some token of recognition, which could be no other than a salute of arms. Well aware of the responsibility assumed, and of the criticisms that would follow, as the sequel proved, nothing of that kind could move me in the least. The act could be defended, if needful, by the suggestion that such a salute was not to the cause for which the flag of the Confederacy stood, but to its going down before the flag of the Union. My main reason, however, was one for which I sought no authority nor asked forgiveness. Before us in proud humiliation stood the embodiment of manhood: men whom neither toils and sufferings, nor the fact of death, nor disaster, nor hopelessness could bend from their resolve; standing before us now, thin, worn, and famished, but erect, and with eyes looking level into ours, waking memories that bound us together as no other bond;–was not such manhood to be welcomed back into a Union so tested and assured?

Instructions had been given; and when the head of each division column comes opposite our group, our bugle sounds the signal and instantly our whole line from right to left, regiment by regiment in succession, gives the soldiers salutation, from the “order arms” to the old “carry”–the marching salute. Gordon at the head of the column, riding with heavy spirit and. downcast face, catches the sound of shifting arms, looks up, and, taking the meaning, wheels superbly, making with himself and his horse one uplifted figure, with profound salutation as he drops the point of his sword to the boot toe; then facing to his own command, gives word for his successive brigades to pass us with the same position of the manual,–honor answering honor. On our part not a sound of trumpet more, nor roll of drum; not a cheer, nor word nor whisper of vain-glorying, nor motion of man standing again at the order, but an awed stillness rather, and breath-holding, as if it were the passing of the dead!

I think you’d have to be more hard-hearted than me to not be moved by that scene. Chamberlain, hero of the second day of Gettysburg at Little Round Top with the 20th Maine and later president of Bowdoin College, continues:

What visions thronged as we looked into each others eyes! Here pass the men of Antietam, the Bloody Lane, the Sunken Road, the Cornfield, the Burnside-Bridge; the men whom Stonewall Jackson on the second night at Fredericksburg begged Lee to let him take and crush the two corps of the Army of the Potomac huddled in the streets in darkness and confusion; the men who swept away the Eleventh Corps at Chancellorsville; who left six thousand of their companions around the bases of Culps and Cemetery Hills at Gettysburg; these survivors of the terrible Wilderness, the Bloody-Angle at Spottsylvania, the slaughter pen of Cold Harbor, the whirlpool of Bethesda Church!

The whole passage is well worth reading, especially on the anniversary of that “chill gray morning.”

At Brainshare, Novell’s annual user conference in Salt Lake City, our CTO, Jeff Jaffe, announced a new technology vision, code-named “Project Fossa,” [pdf] intended to enable computing and collaborating with agility. The fossa is a cat-like mammal from Madagascar, sort of related to raccoons, weasels, and palm civits. (Fossas may be viverrids like civits or the falanouc, another Madagascar endemic; the taxonomy seems to be contested.) Fossas are supposed to be very agile, and if you have little kids you know them as the villains in the animated movie Madagascar. The project’s name is also a play on Free and Open Source Software (FOSS).

Here’s some press coverage including the priceless hed “Novell focuses future strategy around endangered mongoose” from the UK edition of ZDNet.

Amidst the continuing Microsoft acquisition saga, Yahoo is making some interesting strategic moves, principally towards more openness and what used to be called the Semantic Web. This is smart, I think, and makes a lot of sense. Will it be valuable? Is their timing right? Not sure; that’s what makes it a business.

They’ve opened up their search engine, via their Open Search Platform initiative and are now extending that to an ‘open search ecosystem‘ that builds on the data web. Details are still emerging, but it looks like Yahoo is going to use lightweight semantics to try to connect data silos, rather than the traditional, now heavyweight, view of the Semantic Web — what the cool kids now call Web 3G.

(more…)

Novell just announced that we are acquiring PlateSpin, which will greatly expand our virtual server management capabilities for the data center. I’m excited to welcome the Canadians. This is a good move for us, for them, and for our customers.

Andrew Lo, an MIT professor who just sold his company to Natixis Global Asset Management, has a fascinating paper on a little-noticed meltdown in the financial markets this past August. Written with his student Amir Khandani (credited as the lead author), the paper is not exactly academic and not exactly easy to read; it’s more like a combination of a whodunit detective story set in Greenwich, CT and a business school case study. The topic is three tumultuous days in the lives of hedge funds. The funds in question follow a quantitative ‘equity market neutral’ strategy.

The authors argue that the huge swings these long/short funds saw in the second week of August started with an essentially incidental event, probably the liquidation of one of funds due to unrelated events. They speculate that a big bank or hedge fund was forced to liquidate their equity market neutral portfolio, perhaps to reduce their risk due to overexposure in the mortgage markets, which have been in turmoil.

This sell-off triggered a cascade of unexpected events as the quantitative funds, firing on automatic, launched into an unchecked downward spiral.  These funds are, in normal times, engaged in a vicious arms race to develop better trading algorithms, faster technology platforms, better access to data, and newer strategies.  But the events of 6 August pushed them to their limits, and beyond.  I have heard that one of the funds had been evaluating SLERT, Novell’s low-latency Linux extensions, and rushed it into production in the August emergency.  I imagine that other IT infrastructure components were equally taxed.

Fund managers, up against margin requirements and with losses completely out of line with their forecasted risk exposure, were forced to liquidate their positions, which exacerbated the problem.  Unleveraged, Khandani and Lo estimate that a model long/short portfolio experienced a loss of 12 daily standard deviations.  Leveraged, of course, the losses were much greater, and this for a strategy that has low risk by design.

Another odd quality of the events of 6 August 2007 is that they were so large and so isolated; no one outside of the hedge fund quant universe noticed the explosion.  No other financial indicators budged.  The financial industry was dominated by concerns about low-quality mortage exposure, not long/short hedge funds.  Later in the week, the model strategy that Khandani and Lo constructed to re-enact the events of the week rebounded and made up most of the losses.  If the managers of the funds had enough liquidity, and courage, the week ended up looking pretty normal.

The paper’s conclusion is that hedge fund systemic risk seems to be greater than expected, at least in the quantitative long/short funds, many of which closely resemble each other.  With so many quants fishing in the same small pond, the rocking of one boat means that other boats may get swamped, or sink.

And every one of them words rang true
And glowed like burning coal

Ian Rogers from Yahoo Music, and formerly of Nullsoft, says it straight: check out his blog entry and slides from a talk he gave to “some friends in the music industry.” (Nice use of Flickr for slide presentation.)

Eight years ago he, and anyone else who was thoughtful about the issue, knew that legally selling music by the individual track in an unprotected MP3 format was the future of the music industry. But it’s taken eight years for someone, i.e., Amazon, to actually do it.

Eight years resonates with me, because I remember a small internally-focused conference we had on the future of what was then called e-business in late 1999 or early 2000, almost exactly eight years. One of the ‘tracks’ of the conference was on the future of the music industry and I remember being told by one of the presenters, who was with Sony Music if I recall correctly, that it was a ‘future that has already happened,’ in Peter Drucker’s felicitous phrase. He described more or less exactly what Rogers is saying, eight years later.

People knew.

I remember thinking that it was unclear to me exactly how the future of digital music was going to unfold — remember, there was no such thing as the iPod yet — and particularly if subscription services were going to win out over album or song purchases. But those were details; Rogers is absolutely right that this has been waiting to happen for *eight* years now. Incredible.

I especially like his point about the “context” of the music becoming increasingly important; he says that this is a limitation of iTunes, since it’s a music spreadsheet. Which is mean, but funny. And true.

There’s a rule (I might be making this up, so don’t tell your kids) that says that a financial derivatives market will grow to eventually become ten times the size of the underlying physical market, so that the oil futures market, for example, is bigger than oil contracts market by an order of magnitude.

I think (and I’m definitely making this up; tell your kids) that the same applies for interpretations of a source, so that the Bob Dylan discussion ‘market’ (here for example) should be at least ten times the size of the actual Dylan corpus. This is certainly true for religious texts; the Talmud, the derivatives market, is much larger than the Torah — the physical market, to torture the metaphor — as is the Buddhist commentarial tradition, which is enormous, largely untranslated, and little-read in the West to date.

The problem with the Dylan commentarial tradition, besides being dry and mostly lacking a sense of humor as per the requirements of the genre, is that it’s faced with a moving target; it’s as if they were still writing the New Testament.

But imagine, if you will, if you could embed the source texts along with your commentary, just like the religious authors do. That would put your music into context. And there are always going to be large contexts, communities, or commentarial traditions. So maybe the quote at the top is referring to Baudelaire, or the Bible, or to a 13th century poet. But I could help Bob sell some music if I could link to the source so that my dear readers could listen to the song as they read about it, or the future of music.

Back in April, Mozy announced a deal to provide backup services for all of GE’s employees — all 300,000 of them — which impressed me enough to start using Mozy’s free offering.

You download a small application, tell it the folders or filetypes you want to back up, with a 2GB limit, and it trickles the updates back to their servers during periods where you’re away from your machine. I’ve used it successfully with Windows and Mac. You agree to accept the occasional promotional email from Mozy-approved advertisers, but I don’t think I’ve gotten any. It’s a great service that simply solves the terrifying backup problem.

Now TechCrunch is reporting that EMC has acquired them for $76m, on just $1.9m in venture capital. Good move for EMC and great news for the Mozy guys, who had earlier rebuffed offers from Google.

I still would like to see an open source version that can use a variety of back-ends including S3; I think that would be an ideal solution. But for now, Mozy!