I used to think developing for the Internet would be no different than any other sort of product development: there would be some sort of software methodology used to write products, which would ship on a given timetable, with features reduced and a complete QA cycle done prior to releasing a product. The only difference here is what “product release” means: for an Internet company it means uploading your software to a server or bank of servers, while for product development it means pressing a CD master and sending it off for duplication.

In many ways I thought Internet development would be easier than Product development: you only have to make sure your software works on one server rather than thousands (or millions) of individual computers, each with their own quirks and oddities. And while various operating systems go out of their way to help isolate your product from other software running on the same system, various ‘hacks’ tend to burrow into the operating system, creating unpredictable features which may or may not break whatever API you are using.

What I didn’t appreciate is how different Internet development is to Product development.

At first I thought that somehow management where I work was screwed up: everyone was all into “agile” development (as opposed to “waterfall” where I worked before)–and their implementation of “agile” seemed to borrow all the worst elements without any of the benefits. (Waterfall is also a silly methodology, but at least it implied forcing everyone to a similar timetable so different teams didn’t stomp arbitrarily over other teams, as seems to be the case here.) But now I’m realizing that two features of the Internet make the risk/reward equation such a completely different animal that, aside from the fact that you write code and test it, Internet development and Product development share damned near nothing in common.

The first factor is the fact that you only push your software on one computer, a computer you own. This means that it is possible to push incremental improvements damned near daily.

Yeah, various software update panels (Symantec’s LiveUpdate, Apple’s Software Update panel, Microsoft’s Windows Updater) allow your company to push patches out to your customers. But at best this is a mechanism that can be used quarterly: because you are pushing software to potentially millions of computers, you really only have one chance to get it right, and that sort of testing takes time. And even then, you still run a risk of screwing it up, as Microsoft learned with it’s latest Vista patch.

But if your target system is just one server, a server that you own and control–well, the natural risk to pushing software out is significantly lowered. Sure, large Internet companies put arbitrary barriers to pushing out patches outside of the normal delivery cycle (where I now work, there is a committee which approves such things), because even where I work there is at most 8 server locations with two boxes each, it’s not the same as pushing software down LiveUpdate.

And that leads to the second difference, which puts incredible pressure on extremely short delivery cycles: software bugs are easier to fix and push out to production.

Because the cost of delivering a fix is extremely low (relative to a product update), the threshold for what constitutes an emergency requiring everyone to drop everything and push out a fix is lowered–and thus everything becomes an emergency. And, likewise, because everything is an emergency and fixing it in production is easier, there is pressure to (a) shorten the development cycle, and (b) less pressure to think through a problem to make sure the solution being implemented is the correct one.

In the systems I’ve worked on, the code quality is (from my perspective, with four and a half years creating products and another 9 years consulting) complete crap: there are places where deadlocks are just patched over, and fundamental flaws are ignored. One system I’ve done some work on desperately needs a complete rewrite, and other systems I’ve seen I’d never in a million years engineer that way.

And while it’s easy to chalk these things up to incompetence, I’ve realized that the economics of an Internet company forces these sorts of shortcuts: experiment, get something out the door, and move on–and if it breaks, take a couple of days and upload a fix. We don’t get six months or even six weeks to think through a problem–fix it in six days or else you’re too slow.

I don’t think I like working for an Internet company.

Sadly, what worries me more is that the attitudes people are learning at Internet companies are being brought over to Product companies–and rather than think through problems these new generation of software developers are slapping crap together because they’re more used to a two-week development cycle than a six month development cycle. And God help us if critical software systems (or, for that matter, consumer electronics) are developed on Internet time!