Category Archives: Commentary

Portable readers, players and video devices.

Posted on by
Reply

It’s clear now. The RIAA and the Vivendi Corporation (who backs NBC and NBC Universal) are friggin’ morons.

Really, though, it’s not their fault. Most broadcast networks, music companies and book printers have built their business model on the simple premise that they are, first and foremost, distribution companies. And until recently, distribution has been a very hard nut to crack: to distribute television you were required to win approval for a broadcast tower from the FCC, build a broadcast tower (which is damned expensive), and line up material to broadcast, which required both local producers of content and being part of a franchise for content to rebroadcast. Put together a line of advertisers (which requires a sales department), and you have yourself something which may cover one town. Wash, rinse and repeat a few thousand times with capital costs in the billions to cover a nation.

To distribute music or books required a nation-wide series of stores to sell the material at, and a line of companies which could press the music, and an army of technicians who could create the content–a barrier to entry which implies that you also need an army of managers who can make sure the content you sell makes the entire enterprise profitable.

Today, slap it up on YouTube or upload it to your MySpace account.

And here’s the real problem with these distributors. They’ve built their entire business model–their livelihoods–based on a whole bunch of assumptions: it’s nearly impossible (or at least very difficult) to time-shift broadcast material. You either watch it live, or you don’t watch it at all. You either buy a large disk from a local store, or you hear it live on the radio. You buy the bound material from a local store, or you read it at the library (at great inconvenience). And that’s it.

They didn’t want to be inconvenient: it’s just that their business model was built on the fact that there wasn’t a more convenient way to watch television, listen to music, or read a book.

What’s most interesting to me about television was that it’s great success came from the fact that television is a more convenient way to watch material than a movie theater, which was (prior to the invention of the television set) the only way you could watch news casts and movies and mini-plays. (And prior to the invention of the motion picture, the only way you could watch a mini-play was live, waiting for a troop of actors to come to town.)

So Vivendi’s television arm owes it’s existence to the fact that it is more convenient to watch television than to drive down to the off-Broadway district in your local community.

Here’s an irony for everyone involved. The Internet is more convenient.

It’s more convenient to download a television show and watch it than it is to watch it live on television: I can watch the show when I want to, and not when I’m supposed to. It’s more convenient to download an MP3 track and transfer it to whatever device I want–and have thousands of tracks on that player–than it is to lug around a whole stack of CDs (and hope I don’t scratch one). It’s more convenient to download my technical books onto a slim device like the Sony eReader or the Amazon Kindle than it is to carry around tens of pounds of books.

And the old players are all fighting this.

And worse: they’re fighting companies like Apple who see that the point of being able to download music or TV content is not one of digital distribution (or digital promotion)–it’s one of convenience. The Apple iTunes Music Store and the iPod and the iPhone were not purchased because Apple spread some strange pixie dust or Jobs distortion field into each device: they succeeded because they are pleasantly designed devices which are easy for the average person to use. They deliver on that convenience.

And NBC wants to route people to the Amazon Unbox system which is a pain in the ass to navigate and use–because they’re afraid of Apple. Because they don’t understand why Apple succeeded in music and think that whatever it is, it has to be stopped.

By failing to understand this, Vivendi has essentially declared war on convenience.

The reason why people download content from Bittorrent is not because they want to steal material. Often the cost of setting up a Bittorrent client to download content is higher than simply giving Apple a buck to dowload a song–though the people using Bittorrent have realized that setup and use is a cost that they’re trying to lower. No; they’re doing it because it is more convenient than the established distribution channels.

I’m excited about the Amazon Kindle because it’s convenient: I can download and buy my books anywhere. I can keep hundreds of books on me at any time. And what makes that nice is not because I’m a voracious reader of fiction–I want a way to store all of my technical books that is more convenient than devoting a whole bookshelf. I want a reference library which can be accessed easily without having to find a book on a whole row of bookshelves in the other room. I want to carry my collection of technical books with me without having to use a wheelbarrel to carry them all.

It was the promise of the Sony eReader–but that failed because Sony couldn’t make the deals with publishers that Amazon may be able to make. (Sony’s selection of technical books suck hard–though Sony has the advantage of being able to read PDF files, which means I can convert technically-minded web sites to something I can carry with me–though that advantage may be negated by the iPhone, which can simply got to those web sites.)

I’m going to take a wait and see attitude–but hopefully Amazon will succeed in doing something in books that Apple did in music–and the television producers are fighting like cats and dogs–getting all that content into a form factor that is more convenient than hundreds of pounds of dead tree matter.

Cheap Two Factor Authentication

Posted on by
3

Security authentication relies upon three factors: what you know, what you are, and what you have.

What you know: the canonical example of this is a password. It’s something you know: something you’ve memorized and, when asked, you can repeat it. This is a PIN number on an ATM card, or the answer to things like “your mother’s maiden name”.

What you are: this is some physical attribute about yourself. Your fingerprint, your eye color or the pattern of veins at the back of your eye–or the relative length of the fingers on your hand: these are all attributes which are things that you are.

What you have: this is a physical object that is in your possession. The perfect example of this is the key to your house: it is a physical object that allows you to get into your home.

The idea of two-factor authentication is simple: by having two of the three items above, you can then prove that you’re ‘you’ so you can get access to your account, your money, or your property. A perfect example of two-factor authentication is using your ATM card to withdraw money: you cannot withdraw money unless you can show you have something (the card) and you know something (the PIN number). The strength of two-factor authentication relies upon the relative strength of each factor of authentication: in a sense, the overall strength of two-factor authentication is the strength of the first times the strength of the second.

Which is why ATMs are secure even though passwords (a 4 to 6 digit number) are so bloody weak: even though the password itself is extremely weak, you also need to have something in order to withdraw money. Having something times knowing something is stronger than just knowing something by itself–even if the thing you know can be easily guessed.

This also illustrates the danger of some types of two-factor authentication: they can easily collapse into one-factor authentication (thus making it extremely easy to steal your money) through a simple act. In the case of an ATM card, two-factor authentication becomes one-factor authentication if you write your PIN number on your ATM card. Now anyone who has your ATM card can withdraw money–and they don’t have to know your PIN, they can just read it off your card.

Another example of two-factor which collapses into one-factor authentication would be a pre-printed card with a random number: you can memorize the random number on the card–essentially turning your ‘two-factor’ authentication into a memorized one-factor authentication. Not that this is bad: generally longer passwords are more secure than shorter passwords, and banks are missing out on a bet when they limit ATM passwords to 4 to 8 numbers. Even so, this really is no longer two-factor authentication–which is why there are devices out there (such as key fobs) which randomly generate a number on a synchronized clock: the number constantly changes in a seemingly random way, forcing you to have the device on your possession so you can enter the randomly generated number.

Banks have been required for the past year to come up with two-factor authentication for on-line accounts–and they have failed dismally at this, as illustrated here: Wish-It-Was Two-Factor, where banks essentially require you to pick essentially three passwords rather than one: a real password, a ‘question’ and an ‘answer’. It’s not really two-factor authentication: it’s simply a much longer (and harder to remember) password which frustrates people. And again, while having a longer password is generally more secure, it’s not two-factor authentication.

It struck me one cheap way that banks can create two-factor authentication by something you know and by something you have. It’s easy, really: when you open an account with the bank, they send you a piece of paper with a table of fifty random numbers or words or phrases, all numbered on the paper. So, for example, on that page you’d see:

1: 148191     2: 381912    3: 108910

and so forth.

When you’re asked to log in, the login dialog box then asks for three things: your username, your password, and the number that is in cell N on your access page.

By making this list long enough, you virtually have to have possession of the paper to guess the password. It becomes relatively easy for someone to then find the number, rather than to guess the answer to the question “what is your third-favorite existential author” or some other nonsense. Which makes it honest to goodness two-factor authentication–and the cost to administer this is no greater than the cost to print out an additional page to send to the user.

If you use the idea, you don’t have to credit me.

I think the wave of the future…

Posted on by
1

I’ve been giving some thought as to what I think the wave of the future will be, in part because if I were to come up with a good idea for a product, I’d contemplate leaving here in a heartbeat and go build whatever it is I think will be the wave of the future.

And I think I know what that is.

As an aside, if you’re going to build the next great product–look for the next Google or the next Apple, the best place to start is with “I want.” As in “I want a great camera with a built-in GPS so I know where I took my pictures.” Don’t build a product because you think someone else will buy it–build something that you want to buy yourself. Unless you’re completely weird, chances are, someone else will also want to buy what you want to buy.

And I know what I want.

I want easy to use consumer electronics. I want an answering machine that doesn’t require a friggin’ Ph.D. to use. I want a digital recorder that doesn’t make me feel stupid. I want a way to get e-mail without feeling dumber than a pile of rocks.

Actually, what I want, honestly speaking, is stuff which is designed around ease of use.

I think, by the way, that this has been Google’s secret sauce. Say you’re looking for widgets. Go to Yahoo! (disclaimer: I work at Yahoo) then go to Google, and tell me which is easier for your grandmother to use.

Yahoo!: a window full of crap; hard to identify which of a dozen boxes I should type into to find something.

Google: a single edit form. Type ‘widgets’ and go.

It’s Apple’s secret sauce: build easy to use (and I mean easy to use for your grandmother, not easy to use for your hacker friends who think “easy” is writing specialized JavaScript to a web site’s back door) software in beautiful looking hardware.

And that’s what I want as well: stuff around my house that my grandmother could use if she were still alive. Not stuff full of a thousand flashing lights which to a demented geek says “cool high tech” but to someone who is older, intimidates her back to her bedroom under the comforting warmth of a comforter while she wishes for the good ol’ days when things didn’t flicker, flash, and beep at her like demented little monsters with trapped fireflies for eyes.

Our components and gadgets are designed for people with attention deficit disorder, and it doesn’t have to be.

I think I’ve changed.

Posted on by
Reply

There was a time, oh, maybe 20 years ago or so, when I really wanted to write the next great thing. A new drafting tool, maybe, or a new software tool for managing software versions, or something… Partially I wanted to come up with something because it was intellectually challenging, and partially so I could make enough money to be comfortable.

Today, I still would love to sit down and come up with some software product to build; something which would challenge my programming skills. Maybe a spreadsheet in Java, or a drafting tool or something–something which would intellectually stimulate me. But I find myself wishing I had more and more time to build cabinets or boxes from scratch.

I think it has something to do with the fact that I find that I love the process of creating–and perhaps because there are very few problems that I see which challenge me that wouldn’t require me going back to school and getting a Ph.D.

And money wise, I’m comfortable. Not rich, which I would define as having enough money to allow me to maintain my standard of living without ever having to work again. But I’m definitely comfortable: I have most of what I want, and the things I don’t have–well, it’s either something that would require a major lifestyle change (such as a cabin in the woods), or it’s something that honestly would be more trouble than its worth.

So I find myself not really wanting the time to go off and start the next great Internet “thing”–because I’m comfortable. (Which, on a side note, explains why the next great thing generally is built by the young: they have less to lose and much more to win than someone who already has a wife, a nice car, and a nice house in Los Angeles.) Perhaps I will figure out the next cool thing and go off and start a new company–but because it’s an extension of my career path, not because I have this burning urge to ‘make my mark.’

Really, though, I want to just have a week or two where I can build a set of home-made cabinets and book cases. Something with flowers carved into the stiles, maybe.

Yesterday’s Design Patterns

Posted on by
Reply

I blog for a variety of reasons. To practice my writing and communications skills. To vent frustration in a safe place. For public intellectual exhibitionism.

Today I blog to remember something I had forgotten.

A design pattern, generally speaking, is a common pattern that we use to solve a problem. Unfortunately that observation has been reduced by most programmers I know into an exercise in name calling–and like the practice in high school of teaching history as a dry list of dates and event names, has a way of turning a wonderful observation as to how problems can be solved using common themes into mindless recital. (The next time someone tells me “why didn’t you use a fubar pattern here?” I’m going to smack them.)

Design patterns, however, are not just a dry list of names and UML class diagrams, just as history is not a dry list of dates and event names. And some design patterns from yesterday are nearly forgotten–at least I had nearly forgotten them–which as far as I’m concerned is the same thing.

All this came about because I was reading the paper Regular Expression Matching Can Be Simple And Fast. Because I’ve spent the past couple of months locked in meetings and other brain-scrambling exercises, I figured it would be good to exercise that most important muscle, the brain, on an interesting problem: writing my own regular expression matcher engine in Java, and do it without looking at the code in the paper. (Yes, I know: I can just use java.util.regex, or compile the code in the paper–but the point here was not to mindlessly go “oook! regular expression!” like a brainless monkey. The idea here was to engage in a mental exercise. Do you think Sudoku puzzles are pulled out of thin air without first computing a solution then working backwards to create the puzzle?)

The first design pattern comes about in the discussion of the Regular Expression paper; this is the State Machine. While I’ve linked the Wikipedia paper, the key thing to remember is that a state machine consists of a set of states S and a set of actions A, such that each state {s1} triggers action a1 and transitions to a new state s2 depending on the action’s outcome.

Typically states are represented as an integer, and your state machine code often looks like:

int state = 1;
boolean done = false;

while (!done) {
    switch (state) {
        case 1:
            state = doAction1();
            break;
        case 2:
            state = doAction2();
            break;
        ...
    }
}

What’s great about state machines is twofold: first, it helps break a computational problem into small chunks of code. This can be useful if you need to write code that fits into a small footprint (like an embedded processor you’d find in a remote control or a watch or calculator), and is especially useful in this context if you are emulating a multi-threaded process in an environment where you don’t have a multi-tasking operating system to rely upon. (I once built a print spool server that ran on a 6809 embedded processor without any operating system by building the entire network stack as a sequence of states in a state machine. I could then monitor up to 8 separate incoming network connections at the same time simply by having my program walk through 8 separate chunks of memory which held the state for each incoming connection and running the state machine in each: update state in slot 1, then update state in lot 2, then…, then update state in slot 8, then repeat with slot 1, etc.)

The second thing that is great about state machines is that it allows you to represent a dynamic program which is built from some input string (such as a language specification in BNF form or a regular expression parser); in this case, each state is not an integer in a switch statement, but an object in a linked list of objects. It’s great; rather than hard-code how your program works, you load the string, build the state machine, and your execution code simply works the state machine until it’s done.

The second design pattern that came about when creating the regular expression parser–and I’m kicking myself because it took me four hours to notice this last night (because of the brain-scrambling effects of managerial meetings) is the shunting yard algorithm, which is used to convert a statement in infix notation into a parse tree which can then be turned into a state machine or solved or turned into assembly language code.

Typically the shunting yard algorithm is used to solve algebraic equations in infix notation, so we can convert the string “34 + 4*5 – (-3 + 4) * 8” into a parse tree (shown here in LISP style notation) (- (+ 34 (* 4 5)) (* (+ (- 3) 4) 8)), which can eventually be solved: 

(- (+ 34 (* 4 5)) (* (+ (- 3) 4) 8))
=> (- (+ 34 20) (* (+ -3 4) 8))
=> (- 54 (* 1 8))
=> (- 54 8 )
=> 46

The nice thing about the shunting yard algorithm is that it works for any string which uses a series of prefix, postfix and infix operators: we could use it to solve, for example, -(1+2) + 3!, where the first minus sign is a negation operator, and the “!” is a factorial operator.

The algorithm itself is rather simple: you need an operator stack (a push/pop stack which stores the operators you’re using, such as ‘+’, ‘*’, etc), and an ‘eval’ stack, which can be as simple as a push/pop stack of numbers or a stack of parse trees or whatever intermediate representation you’re using to store the equation. Then it’s just a matter of reading the string as a series of tokens: each value is pushed onto the eval stack (popping prefix operators as needed after the push, updating the eval stack as needed), and each operator is pushed onto the stack, popping and evaluating operators that are of lower precedence. The details are in the Wikipedia article linked above.

What took me four hours to remember–and why I’m upset at myself today–is that this algorithm is not just good for solving math equations, but is useful for any sort of notation which is expressed as a series of prefix, infix and postfix operators.

Such as regular expressions.

When you get right down to it, a regular expression is a series of values (letters or characters you’re matching against), with a series of implicit and explicit operators.

‘Concatenation’ is an implicit operator here. Using # to represent the concatenation operator, it’s easy to see that /ABC/ (which matches the substring “ABC” in DABCDEF) can be represented as ‘A’ (i.e. “match letter ‘A'”) # ‘B’ # ‘C’.

The ‘|’ (or ‘or’ operator), which for the expression /ABC|DEF/ matches either “ABC” or “DEF”, is an explicit infix operator: the regular expression /ABC|DEF/ can be represented by the parse tree (in LISP notation): (| (# A B C) (# D E F)).

And the ‘*’ (match 0 or more), ‘+’ (match 1 or more) and ‘?’ (match 0 or 1) are all postfix operators with higher precedence than the concatenation operator: ABC* matches the string ABCCCCCCCC or AB and all in between.

The point is that all of this (including the ‘[…]’ character group, which is simply a single value object matching multiple letters, and the ‘(…)’, which operates in the same way as the parenthesis in an algebraic statement) can all be parsed using the shunting yard algorithm.

As an aside, what’s really spifilicious about the regular expression algorithm in the paper is the observation that a state doesn’t have to be a single integer value, but can actually be an array of values, with each value representing a specific action to carry out–and a single action can spawn two or more values representing two or more actions to carry out at the same time.

For example if you have a regular expression ‘a*a’, essentially meaning “match 0 or more ‘a’ characters, followed by a single ‘a’ character, and you’re matching against “aaa”, then is the second ‘a’ in that string part of the ‘a*’ (match 0 or more ‘a’ characters), or part of the terminal ‘a’ (match a single ‘a’) character? In the case of the algorithm outlined in the regular expression paper at the top of this article, the answer can be ‘both!’ In other words, you can maintain a state which is an array of vectors, and if we represent our regular expression ‘a*a’ with ‘a*’ as 1, ‘a’ as 2, and ‘done’ as 3, we could have our state go from {1} (read first ‘a’) to {1,2} (read second ‘a’) to {1,3,2} (read third ‘a’) to {1,3,3,2} (read end of string) to {3,3,3}: that is, we can simultaneously be doing multiple things at once.

This, by the way, is the point of the paper: by maintaining multiple “states” at once, or rather, by representing our current match state as a vector rather than backtracking (that is, rewinding the matcher back so we can try an alternate matching strategy), we can solve degenerate matching cases without having the program grind to a halt.

I’m blogging this so I can remember. And what I want to remember is this: that the shunting yard algorithm and the state machine are cornerstone design patterns nearly a half century old. For a regular expression, they both can be brought to bare on the problem: the shunting yard algorithm can be used to build a parse tree from a regular expression, and the parse tree can then be used to build an optimal state machine for matching strings against the regular expression.

And more importantly, they are two forgotten tools in a very old toolbox which are a hell of a lot more powerful than the ‘singleton design pattern’, which, to be honest, barely deserves to be named outside of the halls where computer scientists engage in taxonomy for the purpose of obtaining tenure.

Long Term Brain Scrambling

Posted on by
Reply

Here at Yahoo! (and you can tell I work for Yahoo! by adding the damned exclamation point–the company name is “Yahoo!”, not just “Yahoo”) I’m finding myself traveling about once a week to the mothership where I meet with many people to help coordinate and manage a large scale integration project.

And every evening I go home, sit in front of my computer thinking about hacking–but between taking care of an elder friend of mine and the events of the day, I find myself looking at code that should be trivially obvious–and seeing greek instead.

One thing that struck me about Paul Graham’s essay Holding a Program in One’s Head, he talks about the things that can cause a programmer to become inefficient. Basically, as a programmer you hold a program in your head–and there are events (meetings, destractions, etc) which scramble your brains, causing you to spend a half hour (at best) reloading your mind with the problem you’re trying to visualize.

I think it’s worse than that: I think all management functions are designed to scramble your (programming) brain.

And long term exposure may cause permanent brain damage.

Management is essentially incompatible with software development: it’s why the best ideas usually come from either small companies or small business units in large companies–business units which are essentially run like small startups.

So for the past few months I’ve been dealing with what are essentially management functions: communicating to other teams, coordinating actions, creating delivery dates and filing expense reports.

Hopefully I can get out of this loop before permanent damage sets in and I can no longer code my way out of a paper bag.

Update: A friend pointed me to the following site which can help exercise your programming brain: Project Euler, which provides a number of mathematical problems which can really only be solved by writing a short snippet of code. Some of the problems they have wind up being o(N!) or o(2N) if you take the trivial solution–causing the amount of time to solve the problem to explode, unless you also figure out a more efficient algorithm.

If I can’t clear my head this weekend, I may just spend a few hours solving problems from the Euler site.

Twenty Years (at least!) of integrated development environments

Posted on by
Reply

Back in 1987 or so, a small company put together an interesting development tool on the Macintosh called “LightspeedC”, which was later renamed “Think C” for legal reasons. What made this tool exciting is that rather than using a series of ‘makefiles’ and command-line tools for building your application, the development environment was integrated into two tools: a development environment which contained a text editor and a compiler, and a separate debugging tool which was automatically launched when you started debugging your code.

What made all this really exciting was that it was an “all-in-one” environment: while it was not as flexible as Unix ‘make’, if you played within the restrictions imposed by the IDE, you got one-button compilation, one keystroke compile and debug, and the ability to double-click on an error message and have the program automatically put the cursor at the point where the error occurred.

This was revolutionary and influential–though there were earlier IDEs, including an old tool called “Hippo C” on the Macintosh which boasted a similar (though less powerful) integrated environment for developing code.

The key to developing code is the “edit/compile/debug” cycle: the faster and more transparently you can go from ‘edit’ to ‘debug’, the faster you can make changes to your code, and the less code you feel like you have to write between testing your changes. If it takes less than half a second or so from changing some code to seeing if that code works, you can damned near debug code at the same time you write it.

And an integrated development environment allows this to happen: with the old method of a separate editor (vi, say), you would make your code changes, escape out of vi, go to the command line and type ‘ant’ or ‘make’ or whatever, wait for the code to compile–write down the line number (or open up a separate window) where the error occurred, open up vi, change the mistake, exit vi, run make again, start up the debugger tool (assuming you even have one!), run your program–it’s a mess.

In Eclipse, the compile error is shown as you type–so you fix your errors as you make them. And to debug, you press the ‘debug’ button.

I note this because for some strange reason I cannot figure out, even though we have the most powerful development tools on the planet today–tools which even make the goodness of ‘Think C’ look like primitive flint knives and crudely crafted clubs, for some fucking reason I cannot fathom, the code I’m working on is still built using command-line tools and editors first invented in the 1970’s!

There is nothing about our project that cannot be handled in Eclipse or NetBeans or the JetBrains IDEs–yet co-workers who are younger than ‘vi’ insist upon using build methods that are older than they.

Next think you’ll know, they’ll insist upon giving up their cars and commuting to work by horse-drawn wagons.

And now for something completely different.

Posted on by
Reply

When I started this blog, I wanted to have a place where I could put my technical rantings. If you knew me in person, you’d realize that I’m rather passionate (in a “meet my little friend” beat people to death with a baseball bat and bury the remains in the back yard under the flowers sort of way) about quality software development.

This is a completely different rant.

I have a friend, an elder gentleman who was in the process of having all of his savings squirreled away by an unscrupulous fellow who served as his “power of attorney” and was going to ship my friend off to a hell-hole in some God-forsaken part of the country to die.

And so I stepped up to help.

Not to go into the details, but we’re slowly unraveling the problems he had in his life, and we’re in the process of putting his life back together. My wife is going to help clean up his apartment; we’ll get him a new television set and a new bed, get everything organized, just the way he wants it–so when he goes back home, everything is just perfect.

Bottom line–and I cannot possibly feel more strongly about this: If you are unwilling to be an advocate for someone who finds themselves in trouble, then you have absolutely no business asking for an advocate when you find yourself in trouble.

You mean people don’t do this?

Posted on by
Reply

Paul Graham has, as usual, an excellent article on the art of software development: Holding a Program in One’s Head

I have only one question. In his article he writes:

You can magnify the effect of a powerful language by using a style called bottom-up programming, where you write programs in multiple layers, the lower ones acting as programming languages for those above. If you do this right, you only have to keep the topmost layer in your head.

You mean people don’t do this?

In my mind this is the first Design Pattern; the most important one from which all other Design Patterns derive from and pay homage to. It was the most important thing I learned in college when studying networking models. It was the key to my understanding when I started writing software for the Macintosh under System 1. (Yes, I’m that old.) It helped me crack object oriented design. This design pattern: that you develop software in layers, which each layer supporting the one on top of it–this pattern is so ingrained in my mind that I cannot think of writing software any other way.

You can see a beautiful example of this approach in the TCP/IP networking stack: each layer of that stack from the hardware to the application layer is designed to do a very simple and predictable job, and to support the layer on top. The magic of sending a packet of data reliably across the world in an error-prone environment works because of a layer of components, where each component is designed to exactly one thing and do that one thing well–and each component relies upon the services provided by the service below to do that thing well.

My first parser was built this way: I wrote a layer of software that did nothing but tokenization, on top of a layer that did nothing but identify comments and stripped them out of the input stream, on top of a layer that did nothing but read bytes and track the current line number. And those layers sit on top of an operating system call that itself is built in layers: from a layer that does nothing but read blocks from the disk and spools them out in chunks at the request of the application to a layer which does nothing but communicate with the hard disk via a simple protocol, to a layer in the hard disk responsible for stepping the read head motor and interpret the magnetic pulses on the platter.

And on top of this tokenization layer was another layer which did nothing but read in and interpret the tokens representing the top level constructs of the language, which then called subroutines which did nothing but handle the language constructs for small pieces of the language.

I wrote (and have on my web site) an ASN.1 BER parser; it does nothing but handle reading and writing BER encoded byte streams. A man (whose name escapes me now because I’m at work–me culpa) has been in contact with me about writing an ASN.1 parser which uses my BER parser to handle tokenization; I’ve advised him to write code that handles the parsing, validation and interpretation of an ASN.1 specification. And in some cases I’ve said that essentially the BER parser layer isn’t the right place to handle certain features in an ASN.1 parser, such as validation of the input data stream.

See, part of the art of programming is also understanding what belongs in a layer, and what belongs in a new layer. Just as much as the art of programming is about understanding what pieces are part of the same functionality–and should therefore be part of the same layer.

At its root, each layer should be simple, self-contained, and do exactly one thing and do that one thing well. So if the code you’re working on doesn’t do this, then get up from your computer, go for a walk, and think about how to break it up into the proper layers.

Like perpetual motion machines and anti-gravity…

Posted on by
Reply

After reading this article on Slashdot: Hiring Programmers and The High Cost of Low Quality, and simultaneously attending an internal meeting where they discussed an internal reorganization and the promise to “execute” on a new project using “agile development methodologies”, I just had an epiphany.

Software development methodologies, such as “Agile” and “XP” and “Scrum” and the like are all attempts to build a perpetual motion machines: they are all attempts to reverse the laws of physics discovered fourty years ago, especially the observation that the productivity difference between an excellent engineer and a poor engineer can be more than a factor of 25. Each of them promise management that they can overcome the barriers in finding good programming talent by imposing some sort of order. And each of them ultimately either fail–or at best manage to allow a company to limp along.