Nice Idea

Snagged from Daring Fireball: Adobe edits the development cycle

Probably the most effective thing we did was institute per-engineer bug limits: if any engineer’s bug count passes 20, they have to stop working on features and fix bugs instead. The basic idea is that we keep the bug count low as we go so that we can send out usable versions to alpha testers earlier in the cycle and we don’t have the bugalanch at the end.

The goal is to always have the product in a state where we could say “pencils down. You have x weeks to fix the remaining bugs and ship it”. The milestones are primarily measurement points, with the acceptance metric being quality instead of something like “all features in” or “UI Frozen”.

There are two things I’ve seen or heard of which I believe is a really nice idea, software management wise. This is one of them.

The other is the idea of the weekly team meeting. I know most people hate team meetings–and they can become obnoxious if you have a large team. But I’ve been on projects where we never held meetings–and on one project I couldn’t have picked out half of my teammates in a police lineup if I had a gun pointed at my head.

On the other hand, I think team meetings should be limited to one hour a week. Stand-ups (where everyone is supposed to meet for 15 minutes, like the huddle before a game play in football) strike me as a completely useless exercise that only makes managers feel good that they’re “doing something”, and team meetings that last more than an hour or meet more than once a week are also equally a waste of time.

Simple Rule Of Thumb

Lots and lots of column space has been devoted to the proper techniques for doing software management and software engineering.

I have one. It’s a simple one–but strangly one that I haven’t seen honored on any project I’ve worked on in a large colaborative environment, much to the detrement of the team. That rule of thumb is this:

If you are using an IDE, and you cannot simply open up the project file for the project you are working on and press “debug”, then your build process and project file is broken.

Now I’ve heard all sorts of objections to this rule of thumb on the projects I’ve worked on. “Oh, my project contains a bunch of DLLs which require special code in order to be hooked into the OS.” (Uh, DllRegisterServer, anyone? And the Visual C++ debugger will automatically run DllRegisterServer for you when you hit the debug button.) “Oh, my project is scattered across a whole bunch of different projects.” Then your build process is broken, since every modern IDE I’ve encountered can handle multiple projects. “Oh, I’m building something specialized like a device driver.” Okay, in that case I can see your point: the IDE tools may not be quite up to snuff. And one project I’m working on hooks the OS at a very low level–but really, how many people are working on low-level back-door OS hooking code? And shouldn’t that be segregated in such a way so that the majority if your code can operate even without the hooks?

By and large the biggest problems I’ve encountered which prevent a newbie from opening up a project and pressing the “debug” button is because the other developers just don’t care enough to make the build process simple. They cannot be bothered to look up how the IDE they’re using works; they cannot be bothered to read the documentation that describes how COM DLLs are loaded–and so they create these humongous piles of junk trying to solve a problem that was already solved with a more elegant–and IDE friendly–technique.

Fundamentally, if you cannot just open the IDE and hit “debug”, your project is broken. And if you cannot figure out how to make it so you can just hit the “debug” button, perhaps you need to spend some time figuring out why, rather than creating all sorts of weird workarounds because you’re too lazy to RTF (IDE) M.

Q&A

Q: All the cool kids are using Java 5.0 or later. Why are you using Java 1.4?
A: Because v1.4 runs on MacOS X v10.3, which means if you want your Java code to run on 10.3, you need to use Java 1.4.2. And because I’m a bit of a simplicity person: while generics, the new for(;;) loop syntax, and auto-boxing is cool (and nice and useful and all that), they don’t really matter to me all that much.

One of the reasons why such compiler-time tools are useful to many programmers (as I discover time and time to my chagrin where I work) is that many are too lazy to actually test their damned code by at least stepping through the newly written stuff before checking it in. So having auto-boxing and typesafe enums allows more stuff to be found at compile time because some programmers don’t bother to check it at run-time is a good thing: it just takes one idiot programmer who doesn’t check his work to ruin your whole day.

Me; I single-step through my code as I write it. In my opinion, if your project requires more steps to debug than opening up the project file in your IDE and pressing the “debug” button, the build process for your project is hopelessly broken and a defect should be filed immediately. So unless it means there is something subtle going on (like in the CIMMail project, natch, which causes one transaction in a thousand to screw up), or unless you misunderstood the specifications, your code will be relatively solid and reliable the first time around.

WebProxy posted.

A final fix (turns out ThinkGeek.com sends an HTTP/1.1 response with a “Content-Type” header but no length, so I had to change the reader code to take that into account), and the proxy server code now appears to work with every web site I visit.

So it’s now posted.

This is a Java server program which listens to incoming connection and proxies (forwards) the HTTP requests to another server. This basically implements the proxy protocol, though this is a bit of an “itch scratch” on my part–so I cannot guarantee the code will work for you, even though it seems to work for me. It uses the common tools library that is also posted here–“common tools” being a nice phrase for “jar file which catches all of the common crap that doesn’t have a good home elsewhere”, and helped me exercise the bugs on my HTTP client and server code.

Of course if you want something robust and industrial strength, may I recommend looking at the Apache foundation web site instead? This stuff is small potatoes, but as I said, it does appear to work for me, so have at it!

Here is the web site with some information.

D’oh!

So I uploaded the common tools package to the web site last night–and realized that I had no version information in the jar file or the documentation files. How do you know if you have the latest thing?

Quick tweak to the build.xml file for all uploaded projects to add a build number to the jar and the javadocs, uploaded everything to the server–and all is well in the world.

What I did, by the way, was to use <propertyfile> in ant to manage a build properties file, then use the resulting properties to generate the correct manifest and copyright information. So now the copyright information contains the version number corresponding to the build number use to build the docs–which generally is the same as the build number for the jar files. (But not always; some of my code checks to make sure the jar file of other projects are up to date, which triggers a build number increment on that other project.)

The latest additions I made to the common files include code for an HTTP server framework–so I could build my proxy, a simple and stupid ‘debug’ IO Stream class which allows you to hook an OutputStream to an InputStream or OutputStream chain so you can see what’s going through the pipe, as well as a watchdog timer class which has the nice property that you can tell an object in the timer to reset itself.

Nothing earth-shattering here, and all are things you probably can find better examples of elsewhere. But what the heck; it entertains me, which is really what all this is about. Right? If someone else accidently finds all this stuff useful, then so much the better.

Really odd proxy bug. Fixed. (Yay!)

I sorted out a really interesting proxy bug, which means that my proxy code now apears to work correctly with WordPress posting.

What was really happening is this: in my test harness I’ve turned off the connection shutdown timer code, so the connection stays alive indefinitely. But on the WordPress side, as I was typing in a long post, it would shut its side of the connection down.

Now I have a call in my outbound connection code which was verifying the connection was open by calling the java.net.Socket calls ‘isClosed’, ‘isConnected’, ‘isInputShutdown’ and ‘isOutputShutdown’ routines to see if the connection was still open. If the connection had been closed, this was supposed to bring the connection down, discard the socket, and reconnect it fresh.

And apparently, even though all of these calls were indicating the connection was open, the moment I sent data out, I was getting an ‘end of connection’ error on reading the input stream from the socket.

So I kludged it: if I get an end of connection when I expect a response, close down the socket, re-open it, and try the connection again–but only once. If it fails a second time, bail.

This appears to do the trick–I managed to post this through the firewall code, where yesterday I was failing.

That means I can clean this up, post it to the ‘net (probably on Monday) and set up my proxy so I can get my e-mail from work… 🙂

The Internet is held together by duct tape and bailing wire!

So I finally figured out what was wrong with my proxy code.

The World Wide Web is held together by duct tape and bailing wire! I hadn’t realized that HTTP/1.1 requests aren’t uniform across the Internet–or even on the same damned server. A request can ask for a /1.1 request and get a /1.0 response–and I wasn’t handling /1.0 responses well. Further, some servers go so far as to just close the damned connection rather than return a 404 error. (*sigh*) So after working through all of the niggly issues (thanks to Safari, which provides helpful diagnostic tools to allow me to find the things failing to load) and a ton of debugging statements, and I’ve finally arrived at a proxy server that appears to work.

(In fact, I posted this message through my proxy engine.)

The proxy isn’t the most efficient thing in the world: for example, my connection caching code is serializing requests that are being made in parallel to the same web server. (Thus, even though Safari happily opens up five separate threads to process requests faster, I catch them and funnel them through one connection.) So the net effect is a connection which is pretty slow.

And of course I’m not caching or doing any of the nice things proxy servers are supposed to do.

But it appears to work!

I’ll clean it up and post it later this weekend.

Update, two seconds later: I went to hit ‘publish’, and nothing. WTF? Okay, back to the drawing board. Apparently I’m mangling the POST message being made when I tried to submit this post.

WTF?

So I’m thinking of switching jobs because I hate the commute. Of course the best time to think about switching jobs is when you don’t need to nor really have to–in my case I’d be perfectly happy staying put for another 5 years, but honestly the drive is killing me. So I’m “exploring my options.”

One “exploration” involved talking to a head-hunter who then sent me to a site to take a stupid on-line test.

I get it: it’s a great screening mechanism to filter out the bozos who claim years of Java or SQL or C++ but who wouldn’t know the difference between a class and a struct or a select verses an update–and lord knows I’ve interviewed enough people where I work who claimed expert knowledge but who couldn’t write a function call to save his life.

But ohmygod that was the worst test-taking experience I’ve ever had. Perhaps I’m just old: it’s the first test I’ve ever taken on-line. But I like being able to go back and check my answers, and I like being able to run through the test out of order so I can spend extra time on the hard questions. The whole “you have three minutes and if you don’t hit the OK button we’re taking you to the next question” combined with the “hit the back button at your own dooooooom!!!” thing drove me bats. And that says nothing about staring at tables full of tiny tiny little text and a SQL statement and the question “how many values will this return? 8? 9? 11? 13?”

I have a headache.

Progress on the HTML Proxy

I managed to work out the bug in my previous post. Turns out that it’s a bug in my software and a bug in whatever is handling the e-mail redirect at LunarPages. My bug: I was sending the ‘Host:’ HTML header twice. LunarPages? They were grabbing the list of host values, and concatenating them in order to produce the redirect location. So what was happening was in my host header I was writing

Host:www.chaosinmotion.com
Host: www.chaosinmotion.com:80

and apparently they were grabbing the value–presumably as part of some sort of JSP processing system, which I guess was returning the value

"www.chaosinmotion.com:80, www.chaosinmotion.com"

and they were just slapping a resource identifier at the end (“/forwardaddress”) and an “http://&#8221; on the front and sending it out.

I assert it’s a bug in their software because the RFC says that the Host: field of an HTTP request can be a host name optionally followed by a colon and a port number. Their software is broken because when they attempt to redirect people to their on-line e-mail reader application (which is on port 2095), if you set the Host string to

Host: www.chaosinmotion.com:80

you get redirected to http://www.chaosinmotion.com:80:2095

There are some other issues I’m ironing out. But it appears my proxy redirector is working on about 95% of the web sites I’ve tried it with. Of course it’s failing on the one web site I want–the on-line e-mail application I want to connect to from work–but at least I’m making progress.

Oddities while building an HTTP proxy.

I’m building an HTTP proxy. Because I can.

I’ve modified the HTTP classes within the common library to allow me to build an HTTP server; it’s just a matter of taking incoming requests and forwarding them, spitting back the response.

Or so I thought.

One web site, hosted at LunarPages, seems to be sending a forward request (302) in reply to a request I’m making–but for some reason the Location filed is being munged. Instead of setting the URL to something that looks like:

Location: http://www.chaosinmotion.com/forwardaddress.html

I seem to be getting something like:

Location: http://www.chaosinmotion.com:80, www.chaosinmotion.com/forwardaddress.html

What the hell?

Once I sort it out I’ll post it–because where I work they have the firewall from hell, and I want a proxy server running somewhere so I can get my e-mail.