A new location for my blog.

So it appears there have been a series of automated attacks against the WordPress site I was maintaining at LunarPages, which has caused the admins there to try to tamp down my site. Which is fine. But their attempts have been a little bit, erm, heavy handed, breaking some essential features of the site.

So I’ve moved the blog component to a new location and have set up autoredirect.

I’m working through the content to make sure things were updated correctly, but hopefully things should be sorted out shortly.

A good example of “solve the problem you have, not the one you don’t have.”

How the Trendiest Grilled Cheese Venture Got Burnt

Forget Mars colonies and AI. Kaplan declared he had “developed a set of technology that allows us to make the perfect grilled cheese.” The innovation was as meaningful as it was miraculous: the sandwich had “that nostalgic thing,” Kaplan explained. Grilled cheese sandwiches were the fast food equivalent of Proust’s madeleines, priming them for disruption.

Read the rest of the article.

I feel like this is a perfect example of premature optimization, or solving the problems you think you have, and not working on the problems you really have.

And regardless of if this applies to software or to grilled cheese sandwiches, the base of the problem is this: you’re working on the things you want to work on, rather than the things you need to work on. This often leads you astray.

The surest sign that you’re working on the wrong problem comes from when other people start copying certain features:

Chains like Starbucks copied many features that originally distinguished The Melt, such as its digital loyalty program. Though The Melt may not have behaved enough like a restaurant, other restaurants have begun behaving like tech companies, eroding the startup edge that had initially given The Melt an advantage.


“[T]echnology was the promise, and it also may have been the Achilles’ heel,” said a former Melt employee, who declined to be named. “That’s where the arrogance was: We’ve got all this money, we’ve had success in our individual careers in the past, so we can’t get it wrong.”

The ex staffer told me the experience had been humbling. Given the chance to do it over, “we should have been spending a lot more time on the food, the customer experience, the management, and the operations.”

Read: we worked on the problems we wanted to solve, rather than the problems that would have made us successful.

“I think if you’re looking for the angle of, like, what went wrong, I would say that nothing went wrong,” Kaplan told me when we last spoke. “But what we did learn is that the quality of the food is the most important reason why someone comes to a restaurant.”

Don’t work on the problems you want to work on. Work on the problems you have. Sometimes this requires slowing down and realistically looking at the problems in front of you–and prioritizing those problems.

After all, an electronic loyalty program which automatically rewards you discounted food doesn’t matter if your food is crap.


At what point did my life stop being about implementing increasingly clever algorithms which do more and more interesting things, and started being more about hammering on an API framework to behave in ways not originally envisioned by the designer of that framework, to mesh with the whims of designers who cannot be bothered to read the fucking user interface guidelines?

Just sayin’

A follow up from yesterday’s post: further observations.

Securing Driverless Cars from Hackers Is Hard. Ask The Ex-Uber Guy Who Protects Them

Now, Miller has moved on, and he’s ready to broadcast a message to the automotive industry: Securing autonomous cars from digital attacks is a very difficult problem. It’s time to get serious about solving it.

Eye roll.

Look, securing cars or so-called IoT devices is not hard.

But it requires three things–which in today’s Agile-driven cowboy-hacking development culture, is fucking damned near impossible:

  1. Planning: Security is not something you bolt on. It’s not an add-on feature like adding a new button to a web app which carries you to a disclaimer page. Security is something that has to be planned for from the bottom-up–and in today’s broken development culture, “planning” is just something we don’t do well.
  2. Understanding: You need to understand the core tools available to you in the encryption toolkit and use them correctly. You need to understand that every security system for authentication, including tools like single-sign on, access control authorization, and securely storing information are based on the three encryption patterns I covered yesterday: one-way hashing, symmetric-key encryption, and asymmetric-key encryption. You need to understand the limits of these patterns, and understand that no matter how you wrap them in some architecture, the architecture only works because it either relies on the features of each of these patterns (and does it either well, or poorly), or it relies on “security through obscurity.”And you have to remember that “security through obscurity” in today’s world is simply not enough.
  3. Proper testing: You need to have a QA team who specializes in testing security. You need a team who specializes in penetration testing: in trying to figure out how to break the existing system. You need to verify your security architecture before you build everything else on top of that architecture–otherwise you may find yourself re-engineering major chunks of your code in response to security lapses. You need to review your architecture with your security team–because eventually hackers will also understand your architecture, and hackers will have a hell of a lot longer to try to break into your system than your QA team will have. (You may spend six months developing the software for the on-line systems in your car–but your car will be out there for 10 years.)

Now if you’re the type of developer who doesn’t understand why salting passwords is important–and why, while you may not want that salt string to get public, it doesn’t matter as much as having a fundamentally insecure system that uses obscurity to hide access–then you really need to hire someone who does. That person needs to be a system architect: someone who is in charge of designing and specifying the complete architecture of your system. (Security cannot be championed by Project Managers; he has no ability to specify how things work. And a senior developer doesn’t have the authority to force fundamental architectural changes over the objections of management.)

And if you think simply switching on SSL will do the trick–please step the fuck away from your IDE.

Because while SSL may help prevent eavesdroppers from sniffing traffic between two devices he doesn’t control (though not always; Symantec’s Endpoint Security and other similar AV scanners have the ability to crack the contents of HTTPS packets)–I fucking guarantee you a hacker will download your application and can sniff the contents of the packets from his desktop or mobile device so he can reverse engineer your API.

And if your idea of security is to put the car’s VIN as a parameter in the HTTPS GET and POST requests to the server–then you are a fucking moron.

Security really isn’t hard. But it requires thought, dedication and a willingness to properly test–all things we no longer do in a development environment.

Frankly if I were to put together a team which was creating secure software for a bank or a car company or an Internet device, I’d hire a technical writer (to help keep all the documentation straight), and I’d keep at least a 1:1 ratio of developers to QA. I’d also hire a separate security QA team dedicated to penetration testing: people who know how to do a security audit, who know what “fuzzing” is, who can also test the security of our physical systems. I’d draft a set of policies and procedures for our customer support, our back-end deployment team, the guys who maintain the software–and those policies and procedures would cover everything from access to the source kit to physical security of the servers to the length of passwords used internally. And let my security QA team audit that as well.

And I’d use a hybrid waterfall/agile process with a very long (at least 6 month) release cycle except for emergency hot-fixes.

Contrast to a typical development team today handling web or mobile development: we skimp on QA hiring, I haven’t seen a technical writer embedded with a development team in decades. We also tend to use a short (2 week) agile development cycle where long-term planning and architectural design take a back burner to adding the latest feature–which may be fine if you’re creating a web site but sucks if you need to think deeply about anything.

So really, security isn’t hard.

It just takes a fundamental change in development culture.

On password security.

From an AT&T Developer Program e-mail: How secure is your e-mail password?

If you want to be more careful with your e-mail on your mobile device, make sure you have turned on SSL. And you should never give your phone to others because turning SSL off is easy to do.

I have a few notes.

  • SSL is not a panacea. SSL does help slow down hackers. But not by much.

There are a variety of tools which can crack open SSL requests–and this has been true for more than a decade, though some commentators are only just now figuring this out. Symantec’s Endpoint Protection firewall, for example, scans for computer viruses in all HTTP transactions–including “protected” HTTPS requests. It does this by generating a trusted SSL certificate within the firewall which uses one of Symantec’s cert keys, signed using the web site URL of the web site you are accessing. Thus, the firewall can reach out, make a request against a web site, decrypt the traffic, scan it, re-encrypt it using the internally generated SSL key, and send the result to your computer.

There are a number of desktop tools which do the same thing, nominally designed to help developers understand the messages being sent by their desktop or mobile devices. I personally own Charles, a web debugging proxy application, which does more or less the same thing. (For debugging physical hardware you can set up your computer as a proxy for your mobile devices.)

Given this, you can pretty much assume any hacker worth his salt can easily peak into the structure of the requests and responses you make against a web site, reverse engineer your protocol, and make requests against your server–SSL or no.

And even if a hacker doesn’t have access to a certificate against which to generate signed SSL certs on the fly, a hacker can certainly insert a new root certificate into a computer he has access to, and sniff the traffic from that computer using the same techniques Symantec does with their Endpoint Security system. Bad if you run an IT shop full of computers.

  • There are techniques for encrypting a password sent to a remote computer to solve this problem. Various solutions for encrypting a password have been around for a while. Too bad no-one uses them.

One technique used to authenticate without sending a password in plain text is APOP protocol, first described in the e-mail POP3 protocol. Similar ideas (from more than a quarter century ago) have been floating around for at least as long. And the idea is this: when authenticating, instead of simply sending a command like:

{ "req": "login", "username": "woody@alumni.caltech.edu", "password": "Pa55w0rd" }

You’d do three things:

  1. Request a unique token. This can be a timestamp or a UUID generated by the server. The server stores the token returned associated with the current session.
  2. Use the token to generate a hashed response. For example, calculate SHA1(token + salt + password), where salt is an agreed upon constant common between the client and server. Both the client and server generate this result.
  3. Send the generated hash response to the server. The server then compares the hash with the client’s sent result, and authenticates if the hashed password matches.

In today’s world, doing this on a Javascript server is pretty trivial. And a hacker sniffing your traffic will see the following:

  "req": "login", 
  "username": "woody@alumni.caltech.edu", 
  "password": "9d6786ab1b8ded2e8ccb2079efafa18612c8c7b8f1888f3d25ee2c7528c164e9" 

This has a number of nice properties.

  • It is immune from replay attacks. A common technique for hackers to use is replay attacks; they record the packets going back and forth to a server, and replay the packets in order to gain access to the server. By obtaining a unique token from the server and using it to encrypt the password, you cannot replay the login packet to gain access to the user’s account.
  • It is hard to reverse engineer the password by sniffing packets. The next time the user logs in, the packet that gets sent for logging in looks like:
  "req": "login", 
  "username": "woody@alumni.caltech.edu", 
  "password": "5d12af8a0031f83c3e32c5513ecbabfec37fbef48b2b002ece20c7abab8e0e80" 

There is no pattern here that is easily discerned because the password being sent is a SHA-256 hash of a UUID + a token (can you guess the token?) + the password we used above.

There are a number of other techniques, but most of them are variants of the above. Some variants encode the password first (for example, the password is first hashed with a salt so that the password is never stored in plain text).

Side note: There really are only three encryption techniques that are used in encryption, which underlies all security. Most security techniques basically are an arrangement of each of these encryption techniques, each of which have their own strengths and weaknesses:

  • Hashing which converts some data into a hashed value. Hashing is a one-way technique; you can hash a value easily, but it is nearly impossible to unhash a value once hashed. (At best you can search for a “collision”; another string which generates the same hash. It’s one reason why you always salt your hashes–because most techniques used for finding a “collision” are much harder to execute if the input string must fit some specific criteria, such as containing a common well-known prefix.)

    (Examples include MD5, SHA-1, SHA-256 and bcrypt. Please don’t use the first two. Try to use the last.)

  • Symmetric-key encryption/decription where a string is encrypted and decrypted using the same encryption key. Symmetric-key encryption requires the common key to be shared between the code encrypting something and the code decrypting something–making the encryption key a weakness.

    (Examples include Blowfish and AES.)

  • Asymmetric-key encryption/decryption where a different key is used to decrypt a string from encrypting the string. Asymmetric-key encryption has the nice property that, if one key is kept private and another is made public, we can do all sorts of interesting things. For example, if the encryption key is made public and the decryption key private, then we can create a system where anyone can send a receiver a message–but once encrypted only the receiver with the decryption key can read the message. (This is how PGP e-mail encryption works.)

    Likewise, if the key used to encrypt is made private but the decryption key public, then we can create a mechanism for “signing” files: a file is properly signed if, once the encrypted message is decoded, the encrypted message matches some expected value–such as the checksum of the file being sent. This is how message signing is done for things like SSL certificates on the web.

  • There is a fourth technique that is used to help two clients agree upon a common key, and that is Diffie-Hellman and related algorithms, which permit the sharing of a shared secret without the possibility of an eavesdropper discovering the shared secret. It helps solve the problem of a client and server sharing a shared secret for symmetric key encryption of a message–and that is at the core of how TLS encrypts messages. (Add in public/private key signing to validate a certificate, and you have SSL and TLS in a nutshell.)

All encryption techniques are basically built on these three tools (and network protocols may use the fourth).

Each of these tools have their own strengths and weaknesses. For example, asymmetric key encryption assumes at least one of the two keys are kept secret (either the encoding key or decoding key, depending on how the key is used), which means if you use asymmetric key encryption for sending secret messages to multiple receivers, either the message must be encrypted multiple times or the secret key must be shared amongst multiple receivers.

(It is this fact that made me raise an eyebrow when someone creates a messaging app which they claim is completely secure end-to-end. Because if the system permits a single message sent from a client to a server to be re-sent to multiple receiving clients, it means either the server actually has enough information to decrypt a message (and re-encrypt the message to multiple receivers), or it means the server has the ability to share the encryption secret amongst multiple receivers–exposing the server to all the information necessary to decrypt traffic.)

Let’s get back to the topic at hand.

  • Encrypt passwords on your database.

When securing passwords on your web server or server application, the other thing you need to worry about is properly securing passwords in your database. Assume a hacker will eventually read the contents of your database tables–which means all the password security in the world is pointless if you store passwords on your server in plain text.

This is a common mistake, and you can tell if a web developer made this mistake when, as part of password recovery, they simply e-mail you the password. (As an aside, nothing pisses me off more than a web site that demands your password be strong–demanding you use an upper-case letter, lower-case letter, punctuation mark and number–then they e-mail your password in plain text when you ask to recover your password. Fuck you for making me jump through hoops when you can’t be bothered to do the same on your web site.)

There are a number of ways we can secure passwords against spying eyes on a database server. At its core, looking at the list of encryption techniques above, it should be clear we can either hash passwords (again, using a salt of some form), or we can use symmetric-key encryption. (Asymmetric-key encryption certainly is possible, but if your server is the same thing encoding password and decoding passwords, going through the additional effort for asymmetric-key encryption seems silly. Under some circumstances, however, it may make complete sense–such as if you implement a separate sign-on server to handle a large web site.)

A big advantage of using hashing is that it is one-way. This is the technique used by nearly all secure web sites and is also behind the encryption of passwords in Linux and on the Mac. (The principle difference in various techniques involve the hash involved: secure systems use Bcrypt, which has the property of being computationally expensive–and thus very slow to calculate. Of course in today’s world where we can buy the equivalent of a room full of Cray X-MP supercomputers for less than 50 dollars, computational complexity is a shallow panacea: it’s not hard to imagine a dedicated hacker wiring up multiple dedicated processors to accelerate the calculation of a Bcrypt hash to look for collisions.

Ultimately security is not about guaranteeing that your system won’t be penetrated. It’s about increasing the cost of penetration. It’s a series of best practices–backed with knowledge as to how encryption and hashing work–which allow you to slow down a hacker.

Think of computer security very similar to the physical security of a building.

The point is not to make it impossible to break into a building; that is simply impossible. Someone equipped with enough time, enough equipment and the willingness to use force can break into a building.

The point is twofold.

  1. Make it hard. If it is harder for someone to break into a building than the value of the stuff inside that building, then they may not bother.

    You see this with houses verses warehouses verses banks: the more valuable the contents, the harder it is to break into that building.

  2. Try to capture information about who broke in. It’s why security cameras are employed around secure buildings. It may be impossible to prevent someone from robbing a bank–but if you can catch the robber, you can then punish him for the crime.

    (This is why computer logging is an important aspect of security: so you know who tried to break into your system.)

Sure, SSL is an important element of security.

But if you’re not thinking through your entire security stack, from the protocol used to send a password to the way you encrypt passwords in your database and way you physically isolate the database from the network, simply turning on SSL is just fooling yourself into thinking things are safe.

It’s the equivalent of locking the front door while leaving all the windows open.

Interview questions.

Programmers Are Confessing Their Coding Sins To Protest a Broken Job Interview Process

A number of programmers have taken it Twitter to bring it to everyone’s, but particularly recruiter’s, attention about the grueling interview process in their field that relies heavily on technical questions. David Heinemeier Hansson, a well-known programmer and the creator of the popular Ruby on Rails coding framework, started it when he tweeted, “Hello, my name is David. I would fail to write bubble sort on a whiteboard. I look code up on the internet all the time. I don’t do riddles.” Another coder added, “Hello, my name is Tim. I’m a lead at Google with over 30 years coding experience and I need to look up how to get length of a python string.” Another coder chimed in, “Hello my name is Mike, I’m a GDE and lead at NY Times, I don’t know what np complete means. Should I?”

When I interview people I do rely on technical questions.

But I use them as a form of behavioral interviewing.

Meaning I don’t give a damn if you can solve the problem. I really don’t. What I care about is how you act when I give the interview.

Further, I limit my technical questions to two phases–and I do this only after breaking the ice and talking about you and your previous jobs. (The “why did you leave your last job, why did you go to the college you did, how is your dog doing?” sorts of questions.)

The first phase of technical questions: I’ll ask very basic questions about the platform I’m interviewing you for. If you’re a Java programmer I’ll ask about different collection classes. If you’re interviewing to do iOS development, I’ll ask what a view controller is. Questions which, if you are actually actively writing code, you should know the answer because you deal with those things on a daily (or weekly) basis.

Now I don’t care if you stumble on the questions. I expect you to stumble. It’s a God-damned interview, and no-one is comfortable during an interview. You want a job, I’m waving money in front of your face–of course you’re going to choke when I ask you what a UINavigationController is. It’s a stupid question, and I must have a hidden reason for asking a stupid-simple question, right?

But these questions are designed to screen out those who are pretending to be an iOS developer, or pretending to be a Java programmer, or pretending to be an Android developer. There are a lot of people out there who claim 5 years of Java programming who haven’t actually written one line of code in Java.

The second phase of technical questions: I will ask you to write code on the whiteboard. I have two standard questions I ask (one which requires you to write code with a single loop, another with two loops), and I’ll happily walk you through the exercise if necessary.

But even with the second phase, what I’m looking for is your behavior. Do you remember to close your curly braces? Do you know what the modulus operator (‘%’) is? Do you know how to write a for loop? And if not, do you ask? Do you think or do you panic? Do you try to correct me? Do you push back?

I don’t care if you can write code on a white board. I really don’t.

But I do care how we interact. I’m not watching if your code is correct–though bonus points if it is. I’m watching to see how you approach the problem.

Sometimes someone blows through my questions quickly, and at that point I start digging out the harder questions: what is the P vs NP problem. How do you calculate prime factors? Can you write a quick sort from memory?

But if we’re on those questions, I guarantee you three things:

First, you’ve got the job if you want it.

Second, I’m trying to impress you on how smart the other people you will be working with–because I’ve noticed many programmers like the idea of working with smart people.

And three, I’m trying to kill time because I’ve already decided I want to hire you, but I still have a few minutes to fill.

I’ve gone through a lot of interviews, and I’ve noticed most interviewers ask technical questions–but have absolutely no fucking clue how to ask them or what to look for when someone tries to solve them. They’re just looking for the correct answer, not for someone who seems smart and motivated. And that often leads them to making poor hiring decisions–overlooking those who would make great additions to their team, hiring people who may not actually work well with the existing group culture.

But remember: you’re hiring a human being; someone you will work with, someone you will chat with over coffee, someone you will use as a sounding board when you have a problem you’re struggling with.

You’re not hiring a fucking search engine.

Yet another Slashdot comment.

Where someone asks my thoughts about maintaining comments or code documentation:

Comments, like many things in software development, are really a matter of personal style–though they do serve the purpose of making it clear what your code is supposed to do, so the guy who maintains it later can have an understanding of what is supposed to be going on.

So I tend to lean towards a block comment per non-trivial (10 lines or more) function or method–though sometimes the block comment may be at most a sentence. (I don’t think the name of the method or function is sufficient commentary enough simply because cultural differences may make a non-native English speaker unable to translate your personal naming conventions.) I also tend to lean towards describing how complex sections of code work if they appear “non-trivial” to me–meaning if I’m doing more than just calling a handful of other methods in sequence or looping through a bunch of stuff which was clearly noted elsewhere.

I also tend to write code like I write English: a single whitespace used to separate two blocks of code, and moving functionality that is associated into their own blocks (where shifting functionality does not impair the app) is preferable. I also tend to block chunks of methods in a class which are related with each other; the Apple C/C++/Objective C compiler provides the #pragma mark directive to help provide a one line comment to help identify the functionality of those groups of methods.

If I’m doing something really non-trivial–for example, implementing a non-trivial algorithm out of a book (such as a triangulation algorithm for converting a polygon into a triangle) I will start the block comment with a proper bibliographic citation to the book, the edition of the book, the chapter, page number, and (if the book provides) algorithm number of the algorithm I’ve implemented. (I won’t copy the book; the reference should be sufficient.) If the algorithm is available on a web page I’ll cite the web page. And at times when I’ve developed my own algorithm, I will often write it up in a blog post detailing the algorithm and refer to that in my code. (When done as a contractor I’ll look for their internal wiki or, if necessary, check in a PDF to the source kit.)

Back when I started doing this a long time ago, technical writers were sometimes used to help document APIs and the project requirements of the project. The reason for having a separate technical writer was because by having a person embedded in a project who is not intimately familiar with the project, they know the right questions to ask to help document the project to an outsider, being an outsider themselves. Developers cannot do this. Developers, being too close to the code they work on, will often run the code down the “happy path” not thinking of the edge cases to run, and they won’t describe things they think is obvious (like “of course you must call API endpoint 416 before calling API endpoint 397; the output of 416 must be transformed via the function blazsplat before calling 397, and how do you get the input to blazsplat?). A really good technical writer could also help to distill the “gestalt” of the thing, helping to find the “why” which then makes code or project specifications incredibly clear. (You call endpoint 416 because it’s the thing that generates the authentication tokens.)

Back when I was required to train for CISSP certification (I never took the test), one section discussed the three states of knowledge–later used by Donald Rumsfeld in a rather famous quote he was lambasted for. There are things we know we know. For example, I know how to code in C. There are things we know we don’t know. For example, I know I don’t know how to code in SNOBOL. But then there are also things we don’t know we don’t know: this is why, for example, why often speakers never get any takers when he asks for questions: because his audience doesn’t know what questions they should ask. They don’t know what they don’t know.

I think there is a fourth valid state here–and we see the computer industry absolutely riddled with them: the things we don’t know we know. These are the things we’ve learned, but then we think somehow “they’re obvious” or “everyone knows them.” They’re the things we don’t question; the things we don’t think need to be taught, because we don’t know they need to be taught. And the problem is, developers–having invented the code they need to describe–don’t know that they need to describe it to people who haven’t spent weeks or months or years studying the problem and working towards a solution.

I know I suffer from it, I know others who suffer from it. It’s why I think we need people checking our work and helping us to document our interfaces. And it’s why any answer I give for how to comment code will be insufficient: because of the knowledge problem that I don’t know what I should be telling you if you want to understand my code.

And it’s why I think we often don’t comment enough: because we don’t know what we know, so we don’t know what we need to tell other people.

Took, what? 20 years for someone else to say this?

58% of high-performance employees say they need more quiet work spaces.

Our open industrial spaces are frustrating our best people and likely impacting our end products.

I mean, this would be obvious to anyone who had read The Mythical Man-Month and considered the consequences of some of that book’s observations. But apparently we’re too smart to make such observations…