“I failed to appreciate either of those facts until I’d actually done a lot of it myself. Once I appreciated these things, I understood that we have imperfect software BECAUSE the people who make it are dedicated to writing quality software for end users, not in spite of that fact.”
All I can say is that such excuses don’t make software suckage feel any better. Software tends to attract a lot of idealists, because software has the potential to provide large amounts of value for small initial investment. But, as Eric points out, writing software that is actually usable, reliable, and valuable involves all sorts of tradeoffs that affect the cost/benefit equation.
I’ve been involved with many software projects over my career, including IT, horizontal, vertical, and shrinkwrap. And although most of them shipped, and were considered “successful”, I have always been bothered by the disjoint between what I feel “should” be the cost/benefit and the actual cost/benefit results. We can invent excuses all day long, but at the end of the day I can never convince myself that these are anything other than excuses. I believe that there is tremendous waste in the process; far more waste in software than in other industries. This is why I don’t get too excited when people debate about whether Americans work too many hours. Especially in the software industry, the amount of overhead is so huge it’s a wonder we get paid at all. The idealist in me says that if people would slow down, spend more time thinking, engage in proper planning, and stop making excuses for the constant fire drills, we’d cut the number of hours worked in half. The extra hours we work are, IMO, evidence of the software industry’s high margins and tolerance for waste than evidence of any necessity.
To be sure, I’m not saying that poor planning and “fire drills” are the only source of the massive waste that I’m talking about. In times past when I led development teams, I took great pride in making sure that things were scheduled in a way that nobody was ever required to work overtime. But doing this is as much a process of padding the schedule for waste, scheduling in appropriately small discrete tasks, and so on. It certainly didn’t mean that there was no waste. While Eric may be right that developers tend to have good intentions, we have all sorts of well-known biases which lead us to have skewed priorities. Developers like to design for extensibility. Developers like to optimize the crap out of a code loop and ignore the database. And the list goes on.
We also have a tendency to consider ourselves to be better than we are. The developer who is optimizing the crap out of his code loop will look with scorn on the developer who doesn’t, while himself being looked down upon by the developer who has more experience to inform his scalability priorities. And once we have completely mastered a particular aspect of the process, we get promoted or bored.
And even when you have teams full of people who have tons of experience with development, project management, and so on, marketing (as in Marketing 101, not just advertisement) tends to be the high-order bit, and in today’s world that leads to lots of “strategy tax” tradeoffs and randomization.
In any case, this wasn’t meant to be a comprehensive list of the sources of waste in the software industry. Suffice it to say that I believe there are huge amounts of wasted human energy and precious little legitimate excuse other than the fact that software generates large enough return on investment that we tolerate the waste. The waste is systemic, IMO, and not something that’s going to be fixed with “XP”, CMM, development tools, or any other quick fix. For what its’s worth, I think that open-source is no panacea, and in fact is one of the biggest black-holes sucking away human talent needlessly these days. How many man-hours have been spent building a clone of the 30 year-old Unix operating system? There are many better areas for us to be applying talent. And I don’t mean to diminish the professionalism of Microsoft developers. The product teams here are some of the most well-tuned machines I have ever seen, but “best” is not the same as “perfect” or even “as good as possible”.
Who knows? It’s a tough problem, and IMO doesn’t get any easier to stomach with experience.