Engineering Ardor

The Wrong Tool for Every Job

jeffrey — Sat, 05 May 2007 23:51:52 +0000

I spent several of my last few days on my last job tidying up design documentation, javadoc, and header comments before leaving. Some parts of the header comment template were useless. How many hours was it worth to have me go through and label each file with its file name, author, and creation date? I doubt that in the year 3007, the historians of that time will be browsing millenium-old source code header comments to gain insight into our life and times. And if I’m wrong, they’ll probably do what we do now and check the repository change history. As it happens, a good CM system will keep track of who checked something in and when. Most astonishing of all, it’s quite easy to figure out the file name by… I dunno… looking at the name of the file on the file system? Believe me, I’m all for sensibly applied engineering rigor, but sometimes the wrong tool for the job just ought to be put back in the tool box.

There He Was… Gone

jeffrey — Sat, 05 May 2007 12:33:36 +0000

I’ve switched jobs again. Actually, I went back to my previous employer. That in and of itself is of no use to you, so I’ll skip over it. There are, however, some lessons from my now ex-employer that are worth learning if you want to retain your employees, or I suppose, if you want to drive them away in droves.

Integrity Matters

My ex-boss had a bad habit of lying to potential customers. (I only say potential because we didn’t actually have paying customers. I only say lying because his statements were untrue.) Need something to manage all your data? Our product does that. Need something to predict the effect of your manufacturing shop changes? Our product does that. Need something to translate between eight file formats? Our product does that. Need something that will train transportation security workers? Our product does that. Not gettin’ any? Neither is our product.

Point is, he then expected his employees would trust and have confidence in him. Even though the quarterly bonus program he kept promising for nine months never got off the ground. Even though the upcoming business projects and prospects he described never materialized. Even though he instructed me to falsify a time sheet on a federally funded project. For the record, folks, there’s only one acceptable answer to that last one: “Boss, Kansas gets too cold in winter to end up in Leavenworth.” Suffice to say, my time sheet remained unchanged, accurate, and legal; our conversation has since been reclassified as a misunderstanding.

If you don’t act with integrity all the time, where does it begin and end? Do you start acting honorably when you leave the house in the morning? On the drive in to work? When you clock in? Only after your second coffee? When talking to your own people? Existing customers? Potential customers? When you preface your words with “Simon says?” If you don’t display good character and integrity at all times, why should I trust that you’ll act uprightly when dealing with me?

Responsibility, Accountability, and Authority

Authority is the delightful creamy center sandwiched between the dry chocolate wafers of responsibility and accountability. Consider:

If you have accountability and authority, but no responsibility for something, you get Not My Job, syndrome. I see this all the time in engineering organizations. The design engineers are held accountable to some degree if the product is terrible and they have authority to make design changes, but it’s “the ilities” job to make sure the system is reliable, safe, and of sufficient quality. It’s testing’s job to find the bugs. It’s usability’s job to change the neon orange eighties era airline seat color scheme to something pleasant.

If you have responsibility and authority without any accountability… well… look at most of the United States federal government for good examples of this.

If you have responsibility and accountability with no authority, you have the job I just left. It was my responsibility to make sure we ended up with a high quality product that delighted our user, and it was my behind on the line if that goal wasn’t met. However, I also had to comply with all the detailed design decisions of our VP of engineering, ensuring I had no authority to design a successful product. Unfortunately, his decisions bring us to the next point.

One Size Fits None

“To a database person, every nail looks like a thumb. Or something like that.” - Jamie Zawinski

I am told that by quoting jwz, I will make myself appear to be one of the cool kids. Or something like that. Undeniably, the man has a point. We had a data storage infrastructure that had to be used for all data storage needs of the software: application data, cached calculations, user preferences, everything. Worse yet, not only was the data storage mechanism one size fits none, but it came with a preexisting data structure, which the VP of Engineering was dead set against changing or expanding in any way. Need a concrete example? We were working on software that needed to control access to certain data using access control lists and permissions. The One True Data Structure had no such construct. The VP of Engineering reasoned that because permissions were initially assigned during the process of creating an account, we could create an instance of our workflow engine and store the permissions on an event thrown by the workflow engine during account creation. To retrieve, we merely needed to instantiate the workflow engine and sift through all the events it had thrown to find the permissions for a particular user.

A Blunt Butter Knife Is Not A Scalpel

We had few tools, but we made up for it by using them poorly. Even with CVS, it shouldn’t have taken six different checkouts from two or three branches to get the code for one of our projects. With all the free alternatives available, we could’ve had a more sophisticated bug tracking tool than a pad of legal paper. And the clay tablets, why oh why did we leave them in the oven so long. They came out brittle and cracked.

What I Want

Keeping me happy, and using Proof By Arrogance I will claim other engineers as well, is not so difficult.

Give me smart people with integrity with whom to work.
Give me an interesting problem to work on. (Note, some people will be very happy to work on a boring problem with a pleasing enough work environment and paycheck.)
Give me the accountability, authority, and responsibility to work on that problem.
Give me a decent environment and tool set to use, or even better the authority to select my own.
Get out of my way so I can get things done.

Why My Broken Air Conditioner Poses Challenges for Reputation Systems

jeffrey — Sun, 25 Mar 2007 21:39:19 +0000

Over this past summer, our air conditioner failed. It wasn’t a quiet failure of the compressor. The outflow line for the condensation water blocked up and sent water down over and through the oil furnace below. In looking at the state of the HVAC system, particularly the oil furnace, it became apparent that they had been poorly maintained for a lengthy period of time, despite our service contract with our oil company. Getting recommendations from neighbors and friends is one way to help avoid this kind of abuse of trust. Many sites on Teh Intarwebs use reputation systems as well, but most are oriented around transactions with short durations. Many industries, including health care, HVAC maintenance, accounting, and investment work over considerably longer periods.

The failure mode of the existing air conditioner was pretty interesting. Our air conditioner was (and our new one is) a split system. The unit outside is the condenser, which compresses refrigerant turning it into a liquid. This process generates heat, which is dissipated in part by a fan blowing air over the condenser coils. The liquid refrigerant flows through piping into the basement.

Inside the basement is another box, called the air handler unit. The air handler unit blows air throughout the house. As is not unusual, our air handler unit sits above and inline with our oil furnace, a fact relevant to the failure we suffered. As the refrigerant enters the air handling unit, it passes through a valve that releases the liquid into a much lower pressure environment. The liquid boils to a vapor, reducing its temperature to the refrigerant’s boiling point. Remember, this stuff isn’t water, so the boiling point is actually well below the temperature of a comfortably cooled room. The cold refrigerant vapor flows through coils in the air handler unit, chilling the air that passes over the coils. As the warm, humid air is chilled, it loses some of its moisture. Condensation collects on the coils in the air handler unit, drips into a pan, and drains away from the unit in a drain pipe.

That’s how it’s supposed to work. In our case, the drain pipe plugged up and the condensation water backed up until it was pouring down over and through the oil furnace onto the basement floor. Great. It turns out the drain pipe went through the basement wall below ground level. We never found anywhere outside where it surfaced to discharge. I suppose it’s not surprising it got plugged up, being a pipe to nowhere.

In diagnosing this problem, we also took a good hard look at the state of our oil furnace. Despite religiously keeping a maintenance appointment with the oil company every six months, our furnace was in a deplorable state of cleanliness and repair. Our new HVAC technician asked how many years it had been since we had our system cleaned and maintained. In his words, “They were cleaning this thing every six months? What — did they just come down here and bang wrenches for a couple minutes and leave again?” The flue was corroded to the point of developing holes. The flue, clean-outs, and furnace itself were thick with soot and particulate debris. Every tunable item was set precisely to minimize the efficiency of the furnace, contributing to a high oil burn rate. And the line to the furnace was leaking oil.

I am somewhat to blame for this. I wasn’t around to keep an eye on what they were and weren’t doing. Of course, I’m not sure I would’ve known the difference. I definitely should’ve recognized the conflict of interest in asking my heating oil vendor to maintain the efficiency of my furnace. As I’m covered by their service plan, avoiding necessary repairs saves them money. Any cleaning they don’t do degrades the efficiency of the furnace, increasing my consumption of their product. Any fine-tuning they do to ensure a high burn rate again means I buy more of their product.

Perhaps I should get my forehead tattooed: “I was one of the ones born every minute.” One might also suggest that my new HVAC technician was exaggerating his criticism to ensure that he’d get future work from me and possibly sell me a new furnace and air conditioner. I’d like to think that by then I was wary enough to do sufficient fact checking to verify his claims.

The world is too complex to detect this kind of grift in every area of life. It’s quite possible that the oil company’s owner is tracking the money he’s made from cheating me on a desktop secured by paying a consultant hundreds of dollars to do no more than install a free (as in beer) firewall and anti-virus product. Learn enough about how your car works to avoid being taken at the auto shop, and you may neglect checking whether your prescription drug was approved shortly before a senior official left the FDA for a highly paid position at the drug’s manufacturer. There just aren’t enough hours in the day.

Common sense does help. Most people don’t fear they’ll be answering emails from deposed Nigerian royalty any time soon, but it’s not so easy to detect an abuse of trust by a domain expert hired precisely because of their specialized technical expertise. Asking lots of questions helps, as the answers should be self-consistent at a minimum. An optometrist once recommended I consider laser correction for my vision.

Me: “I’m not convinced yet. I’d like to see some evidence of what happens over the long haul — no point in getting perfect vision now just to find out I’m guaranteed to spend my old age completely blind.”

Doctor: “Oh, long term studies have been done. It’s completely safe.”

Me: “Oh really? Just where did they find seventy year olds who’d had this surgery done in their thirties, when the current technique is less than ten years old? Was the time machine they used completely safe, too?”

I’m still wearing glasses.

It’s hit or miss, though, depending on whether you ask the right questions and catch any inconsistencies. A lot of professionals don’t like being second guessed in every step of their work. Would you enjoy it if your boss stood over your shoulders questioning you about every variable name and subroutine you used? Even those that don’t mind may be talented in their craft but poor teachers. Worst of all, it still doesn’t scale well enough — still not enough hours in the day.

I can Google someone and their company, but that can be a time consuming process as well, as I have to wade through dozens of random forums and blog posts trying to put together a coherent picture. Also, there tends to be a publishing bias towards positive information. Reading diatribes and flames, you might not think it, but there are still many people who won’t put names to a bad experience for fear of retaliation or being sued. (You don’t see my oil company’s name in this post, do you? Well, until the end of this contract season, they’re still responsible for delivering the oil that keeps my home heated. Maybe freezing wintery days aren’t the best time to accuse them of something somewhere between negligence and grift. I wonder, does that make me sensibly cautious or a coward?) Lastly, for some traditional brick and mortar businesses, there might not be enough information to find. I was recently looking for a lawyer and an accountant, and there just wasn’t much to be found online about either firm I selected.

The world continues to sprout new services of a specialized and technical nature. Reputation and referrals will become more important in choosing with whom to do business. Reputation systems are already quite common on Teh Intarwebs. Ebay’s rating system for buyers and sellers may be the best known, but almost every recent community web site has some feature for separating the contributing members from the trolls. I found my new HVAC technician through an online service that features a rating and feedback system for HVAC, plumbing, electrical, roofing, and remodeling contractors.

The system had its flaws, though. It relies on short-term feedback. This is good, as the user’s memory of the transaction is still fresh. It’s also bad, as quality of work may not be apparent for months or years. New house foundations may look great but crack a decade later. One might be satisfied with a family dentist for a long time, then visit another dentist and find out that years of neglect require a root canal and bridgework to correct. There’s a joke that roofers don’t mind offering 30 year guarantees on 20 year roofs because they’ll be out of business in 10 years anyway.

Existing reputation systems don’t seem to be built for this kind of long haul with changes in opinion. They do best with small-scope, short-duration transactions, such as individual posts on a message board, sales at an auction site, or visits to a restaurant. They focus on collecting a wide breadth of data by trying to maximize the number of users who provide feedback. There’s little follow up over time. Of course, there’s a nuisance threshold here too. No one is going to fill out a survey a year for every home repair, product purchase, and professional service they use. It’d just be overwhelming, especially as in most cases there wouldn’t be a significant change of opinion.

So what would I do differently for a reputation and rating site for trade contractors and professional services? At a minimum, I think such a site would need to:

Address the problems existing reputation systems face, such as encourage a good user response rate by keeping the rating process simple and flexible. I shouldn’t have to fill out a three page survey. Perhaps an overall one to five rating would be sufficient with comments describing the rationale behind the rating. Spam, trolls, and other sources of bad data will have to be filtered to keep the site useful.
Pick an empty niche. No site would easily span every industry. There are already sites handling feedback for online stores. Weakly represented are trades and professional services, like HVAC, electrical, general contracting, architects, lawyers, accountants, and doctors. Start focused and branch out once the system is running smoothly.
Avoid any actual or appearance of collusion with the vendors to be rated. The service I used to find my new HVAC technician is paid by the contractors to provide leads to people like me. I hope they run their reputation system honorably, but there’s an economic incentive for their contractors to get high ratings so people like me will want to hire them.
Follow up later. Whether a fixed system or by random sampling, request that a set of users update their opinion. Give them a convenient one-click link to indicate no change in their views. Give them an equally convenient one-click link to come back and explain that one year after installation the roof had to be completely redone or that their five-star, huge-refund tax return didn’t hold up to an audit.
Provide tools to make sense of the data. It’s no longer a matter of knowing that someone’s average rating is high or low. It’d be necessary to see changes over time. If 4500 out of 5000 total ratings are high initially, but 90 out of 100 total ratings are terrible a year later, the overall average for all time may still be high, but there’s a consistent downturn a year after service. Help users see how people’s impressions change. A good choice might be some graphic showing the trend over time, made as an image map to drill into details like individual comments.
Handling the passage of time also means making the user aware of significant changes to the company as well. Perhaps a company provides terrible service until new management turns them around and they improve, or vice versa. Perhaps a contractor dissolves his old company and forms a new one to shed the history of complaints and lawsuits under the old name.

If I’m just ignorant and there’s a site already doing this, let me know. I’d like to have a look at it. Or if you’re currently working on such a project, drop me a line — I’d be interested in how you’re solving this problem.

Another Toe Tagged Project

jeffrey — Wed, 01 Nov 2006 23:42:58 +0000

I was recently asked to go take a look at a web site project. Ordinarily, my answer to requests for a critique like this is, “Are you willing to pay my ‘I don’t want to do this,’ rate?” This person, however, I owed. Big. I almost asked, “Would you be willing to take an internal organ instead? I have a spare kidney!” I don’t think she would’ve taken it. She wanted the web site reviewed. And so I began my foray into a text book case of all that can go wrong in web development.

This person I was helping out — let’s call her the Customer, because she was — had retained the services of her university’s IT department to develop a web site for her organization. She provided what she believed was a very complete set of requirements for exactly what needed to be built, yet the project was well over deadline and didn’t look right to her. She wanted me to provide a sanity check. Was this the best that could be done with the web? Were her expectations too high? Was this a fundamental limit of the technology? Was there something she could have done to prevent this or could do to rectify it?

Customer, Requirements, and a Chorus of Angels

It turned out this was the simplest kind of site imaginable. No database backing. No server side scripts. No site search function. No Java widgets. No Javascript. Absolutely nothing but good ol’ fashioned static content. She provided a Word document with every bit of the content to be put on the page: text, images, sidebars, navigation, and a first cut at styling and layout. I say first cut because she wasn’t given any guidance on the width to which she should lay out - and not for lack of asking - or formatting conventions like leaving link text blue and underlined. (Yes there’s irony here given that the style sheet for this site messes with the link style — I just haven’t gotten around to correcting that yet.) She wasn’t rigidly inflexible about the requirements either, and actively solicited their input where they felt something wouldn’t display well or adhere to the usability conventions of web sites. Print documents she knows; the Intarwebs she doesn’t.

I know it sounds like I’m describing the Platonic Ideal of a Customer - she had her requirements in order and left room for flexibility of implementation. I promise I am not exaggerating. And yes, I believe the cloud cover over my house did break to allow a single beam of golden radiance to illuminate my screen when I read her specification — okay, that part I’m exaggerating. But it’s true nonetheless that this project did not suffer from an indecisive Customer with poorly conceived notions for a grandiose ball of scripts and database hacks.

The Site, Or Shall We Call It “Ground Zero”

Then I looked at the results. I expected to spend a half hour poking through and giving her an overview. Soon I had a notebook out frantically scrawling notes. I think I wrote “Here Be Dragons,” a handful of times, and I remember using enough exclamation points to populate a fanboi forum post. But these were not light-hearted exclamation points; they were laden with foreboding, a lightning strike arcing down to the point at the bottom.

The gripe list is summarized below. Note that this isn’t meant to be a set of guidelines universally applicable to all web sites (although some probably are) — they’re meaningful in the context of the particular look, feel, and set of information she was trying to portray.

There was a logo on the top left of the page suggesting a way to navigate back to the home page, but it wasn’t linked.
There was a bread crumb trail (list of links followed to get to the present page), but not all the text links in the trail were actually links, meaning you couldn’t reliably navigate using the trail.
There was some extraneous formatting, borders and boxes around paragraphs in the side columns, that didn’t really look intentional. I guess you had to be there and see it, but it looked off
Across several pages and within a single page, several different color and formatting conventions were followed for href links, making it impossible to consistently and easily identify links. And none of these styles were the usual blue with underline. Moreover, some non-link text was formatted in ways that suggested links that could be followed.
Some of the foreground and background color choices made the text difficult to read
Many of the images had been stretched or compressed without regard to their aspect ratio. Given that many of these were photographs of people, it gave the whole site a fun house mirror effect.
URLs for other sites were listed on some of the pages without being linked to those other sites.
Some of the link names and titles of pages didn’t match their contents. For example, a link to a statement from a person went to a page with a matching title but contents that read more like a third person biography. The link to that person’s biography went to their statement about the site and its purpose.
Some of the links pointed to missing anchors within pages.
The navigation was confusing in places. On a product list page, you might click on “Order Now” only to be taken to a product description page with its own “Order Now” button. Recall above that I said this side had no scripting. This is still true, clicking the second “Order Now” button downloads a PDF order form.
Several pages present on the server file system had no links from the anywhere on the live site. These were intended to be part of the site and omitted accidentally.
Overall, there was a general lack of consistency to the site. Content that is in boxes on one page floats without a border around it on the next. Text that’s styled like the links on one page aren’t links on another. All in all, it gave the impression of being shoddy workmanship.

All in all, this is pretty amateurish stuff. Any competent developer would catch this stuff with a quick once over of the site. Not only did the developer not catch it, but when pointed out on a page as an example of a problem with the whole site, he fixed only that page and not everywhere else the problem occurred.

Rewarding the Guilty and Blaming the Innocent

Customer’s point of contact for this project was, well, I’ll call him Irving Thomasso Guy. IT Guy was the directly assigned labor on the project. He built the site. From her first description of him, I thought him well-meaning but incompetent. I reasoned that since he had no talent with graphic design, layout, or site usability, he must be a pretty good database and scripting developer.

This seemed plausible, as IT Guy was assigned to the project at the first meeting where it was discussed. No one took the time to analyze the kind of site being built to recognize that good aesthetics were all that mattered in the complete absence of any actual coding work. There was no project manager of any kind for the project; where a good project manager spending even 5% time overseeing this trivial job would’ve quickly picked up that the wrong person had been assigned. In fact, Customer described to me that another developer clearly had the right skills and ideas, because in walking past IT Guy’s desk, he would make suggestions that resulted in significant improvements to the site. Too bad he was just passing by and not given the job.

It became apparent, however, that the problem ran deeper. IT Guy had a sign over his desk reading, “Clock In, Clock Out.” Apparently these are the watchwords of his work ethic. On several occasions he committed to several days of development work knowing he’d be out of the office for training the rest of the week. At one point, he told Customer that if he had designed the site, he might take pride in its development, but as it was her design, he was just there to mechanically do what she told him to get the thing online. I think I actually take more pride in folding laundry than this guy does in his… you know… career.

Bottom Line, and Digging Ever Downward

Bottom line, this was a failed project. Months overdue. Customer provided a list of changes required before the site would meet its initial requirements, and it was returned with fewer than half of them made. I guess IT Guy thought she wasn’t going to check, or something. If her organization’s IT department had been operating as an independent contractor or firm, prompt payment would be the least of their worries. A court date might well be in their future.

This is something that frustrates me about IT departments; the desire to not make waves in one’s own organization tends to shield them from the results of incompetence. I’m not saying that all large organization IT departments are necessarily incompetent, just that it seems easier for them to get away with it.

Ultimately, I suggested Customer cut her losses. The site is static HTML with some CSS for formatting. There’s no real scripting work to speak of, so there’d be little learning curve for someone else to come up to speed on the site. She knows exactly what changes need to be made and where. I suggested she get a CD with the files on it from IT Guy and take it elsewhere. Of course, he lied to her again and said that wasn’t possible. I hope I don’t end up having to fly out there…

Application Service Providers and Beef Bouillon Flavored Jello

jeffrey — Tue, 12 Sep 2006 12:57:35 +0000

My workplace shares our street with a goodly amount of vacant class A office space. Apparently this space was left vacant by an Application Service Provider that went from being the largest technology employer in the county to bankruptcy court in record time. A few years ago, we were told ASPs were the way of the future - the desktop was dead, and a brave new era of applications served from the net was revolutionizing the way we worked and played. Now most of the revolutionaries are dead and buried, and the few remaining ASPs look like they’re trying to sell to large enterprise customers rather than you, me, and everyone. To me, the interesting question is, “Why?”

It’s because application service providers were trying to serve beef bouillon flavored jello.

More accurately, gelatin dessert, as Jell-O is actually one particular brand of gelatin dessert made by Kraft Foods, Inc. However, the brand is so successful that too few people now realize that translucent, wiggling, fruit-flavored desserts have any other name. And if I write it “jello”, I’m pretty sure I’m not infringing on anyone’s trademark.

Now I’m on the hook to explain what application service providers have to do with unappetizing meat-flavored gelatin desserts. When I was in college, a group of us banded together to stave off starvation. With teams of four or five cooking for the whole group of forty, each of us only had to cook one meal a week to get hot meals all week. Even better, ingredients are cheaper in bulk. Every term, one fool would get himself elected as steward. The steward’s job was to pick out the recipes and order the food.

One day, my team showed up and found neither recipes nor ingredients. A tasmanian devil whirlwind spun down the hall past the kitchen. From somewhere inside it, we heard our steward’s voice. He’d been late ordering the food, so there were no recipes and no ingredients but whatever was left around from last week.

“I’m sorry guys. Just improvise something.” The expertly thrown butcher knife passed through the empty air where he had been, harmlessly lodging in the corridor wall outside the kitchen.

We took stock. We had all manner of oils, vinegars, spices, and basic pantry goods. So far, so good. Main ingredient fodder? We had several heads of lettuce, a couple pounds of butter, and several pounds of popcorn kernels. Lovely.

“Popcorn and salad!” Joseph cried out triumphantly. We sprang into action like the smoothly coordinated team that we weren’t. Patrick and I were assigned to improvise a dessert. The only thing we found even resembling a dessert ingredient was a box of Knox unflavored gelatin. By this time, we were tired, angry, and knew the meal was going to flop anyway. Our eyes came to rest on some beef bouillon cubes. Hot water, beef bouillon cubes for flavor, and gelatin to turn it into a dessert treat. Perfect! We ended up added some red food coloring to give it a bright, cheery, cherry-flavored look.

After a round of death threats for the woefully inadequate salad and popcorn dinner, we took the gelatin dessert trays upstairs and set them down.

“Jello!” People were rising from their seats as soon as we set it down. “What flavor is it?” someone asked. I looked down at the trays.

“Red!” I said with great honesty and conviction.

“Ooh, cherry.”

I barely heard the reply; the din of chairs flying back from the table and hitting the wall was too loud. I kid you not, some people were climbing over the tables to get at our beef bouillon masterpiece. No flatware. No silverware. Just a frenzy of hunger. I cried out for civility, for someone to go get bowls and spoons, but there was just a dissonant growling in response.

Eric was the first to reach the tray. He scooped up a generous mouthful in one hand and tossed it back, chewing. Everything slowed down in that nifty bullet time effect, as I watched his eyes widen and the expression on his face sour from delightful anticipation to horrible realization. As he whirled on his feet and starting clawing his way over the advancing horde, trying to reach the kitchen sink, I could hear him mumbling “Oh shit! Oh shit! OH SHIT!” Colin took one bite, looked down at the remainder in his hand, looked up at me, and I ducked just in time as his helping of dessert sailed harmlessly overhead. Word quickly spread through the ranks that this concoction was beyond foul. I took cover behind a couch as someone - I don’t know who - yelled, “We’re gonna put the rest of this stuff in yer bed!”

My partners in crime and I fell into (actually dove into, intentionally) the same problem I see with application service providers. We were providing our customers with something they really didn’t want and had no incentive to accept. It solved our problem: we needed something called dessert with no dessert ingredients. But it didn’t solve their problem of being hungry after an inadequate meal.

The big advantage of the Application Service Provider business model is that the software vendor gets a steady, predictable source of income rather than having to worry about how many users will buy an upgrade and how much “maintenance” they can get away with charging on a license (as though software needs to be cleaned and oiled periodically). But this doesn’t solve any problems the customer has. In fact, it removes flexibility the user has with traditional desktop apps. If cash is tight, I can always choose not to upgrade some software this year. Microsoft Word hasn’t had a new feature I really needed in over a decade, so I don’t need to upgrade it until I get so far out of date I can’t exchange files with other businesses. With an ASP, I’d be paying a steady stream of cash whether I needed the periodic updates or not, whether I was flush with work or scrambling to stay in business. Good for them, yes, but as a customer, what’s in it for me?

ASPs did claim other benefits. Products would be higher quality because the ASP only had to develop for their own servers, a known and controlled environment. Except that the client environment was still every bit as varied at the customers’ sites. Help desk costs for their customers were supposed to be lower. Except that ASPs didn’t really eliminate the support burden. It just changed the problems users had from “Why has Word been crashing ever since I installed my BouillonSaver screen saver?” to “Why has [ASP application] been broken ever since I upgraded the Gellatinizer 3000 personal firewall?”

Users care about getting their tasks done, and they care about how much they have to spend to do it. Features, usability, quality, price - these things matter. Users, on the whole, couldn’t care less whether the software runs as a desktop app or over “Teh Intarweb.” And ASPs just didn’t offer any substantial improvement in the feature set, quality, or usability of software. They just offered a payment model more convenient for them and less so for customers. They were solving a non-problem of their customers’.

Any time I think of a company trying to make money solving a non-problem, I think of our beef bouillon jello. We used only the finest ingredients, paid careful attention to the cooking, and produced what was probably the finest batch of beef bouillon jello ever made. We had the best possible solution for people craving a wiggly, jiggly, salty, beefy dessert - which, as it turns out, isn’t something our customers wanted.

If you cannot concisely and concretely describe both a typical user of your product and how your product will solve a problem that user has, you better learn to duck and cover, because you’re serving beef bouillon jello.

[Note, now that it's about a year later -- if I can ever carve out some time to post again, I could get a decent article about why web applications are doing so well. The technology's finally there so it is less trouble to build, deploy, and support a web application, and web apps are offering things you just can't get from your own desktop, like interaction and collaboration with the rest of the known universe.]

Test-wide Setup and Tear Down of JUnit Tests

jeffrey — Thu, 17 Aug 2006 23:17:47 +0000

JUnit 4.0 introduces the @BeforeClass annotation, providing a way to run a class-wide setup method before any of the tests in that class are run. The analogous @AfterClass annotation allows for tear down of those fixtures. I work on a project that hasn’t yet upgraded to JUnit 4.0, and although less widely publicized, it is possible to accomplish the same thing with 3.8.x.

Think before doing this. Tests should be independent of one another. If your tests share fixtures, side effects from one test could affect the results of others. Sometimes, however, a test fixture is just too expensive to set up and tear down with every test. In my case, I needed to set up a database and import a substantial quantity of data. Doing this for every test would’ve made the suite prohibitively slow.

The TestSetup class makes test-wide setup and tear down possible in JUnit 3.8.x. Consider the following class:

	package com.burningvoid.jexample;

	import junit.framework.Test;
	import junit.framework.TestCase;
	import junit.framework.TestSuite;
	import junit.textui.TestRunner;

	public class ExampleTestCase extends TestCase {

		public void setUp() {
			System.out.println("Per Test Case Setup");
		}

		public void tearDown() {
			System.out.println("Per Test Case Teardown");
		}

		public void testA() {
			System.out.println("Test A");
		}

		public void testB() {
			System.out.println("Test B");
		}

		public static Test suite() {
			return new TestSuite(ExampleTestCase.class);
		}

		public static void main(String[] args) {
			TestRunner.run(suite());
		}
	}

If I run this class, the setUp and tearDown methods run once before each test, producing output of:

.Per Test Case Setup
Test A
Per Test Case Teardown
.Per Test Case Setup
Test B
Per Test Case Teardown

For those accustomed to IDE support for JUnit, the periods in the output are due to the text JUnit test runner. It prints a period for each test.

Ok, now change the suite() method to the following:

	public static Test suite() {
		TestSuite ts = new TestSuite();
		ts.addTestSuite(ExampleTestCase.class);
		return new ExampleTestSetup(ts);
	}

This requires the addition of another class.

	package com.burningvoid.jexample;

	import junit.extensions.TestSetup;
	import junit.framework.Test;

	public class ExampleTestSetup extends TestSetup {

		public ExampleTestSetup(Test test) {
			super(test);
		}

		public void setUp() {
			System.out.println("Test-wide setup");
		}

		public void tearDown() {
			System.out.println("Test-wide teardown");
		}
	}

The TestSetup decorator’s setUp and tearDown methods are called before and after all the tests in the suite, resulting in the following output.

Test-wide setup
.Per Test Case Setup
Test A
Per Test Case Teardown
.Per Test Case Setup
Test B
Per Test Case Teardown
Test-wide teardown

If you need test setup and tear down that covers more than one class, simply create a top level test suite that collects all of your test classes into one decorated suite, such as:

	public static Test suite() {
		TestSuite ts = new TestSuite();
		ts.addTestSuite(ExampleTestCase.class);
		ts.addTestSuite(SomeOtherTestCase.class);
		ts.addTestSuite(YetAnotherTestCase.class);
		return new ExampleTestSetup(ts);
	}

As far as I know, the @BeforeClass and @AfterClass mechanisms aren’t quite as flexible, in that they only apply to a single class, and thus can’t be applied to an entire suite. I say this not having used JUnit 4.0 yet, though, so please correct me if I’m incorrect about this.

A Plea for Configuration Management Sanity

jeffrey — Sat, 12 Aug 2006 22:21:07 +0000

I’ve taken configuration management and revision control for granted. I first lost work because I wasn’t using a repository system early in life, and I resolved that such a thing would not happen again. During my senior year in college, I was working on a project with a randomly assigned classmate. When we started, I explained I’d set up a CVS repository for the code and gave him the directory for CVSROOT. Next time we met, he told me he didn’t have any trouble finding the code, but asked why the file names had commas and a couple random letters at the end of them. In retrospect, it was wrong of me to laugh, but I honestly thought he was making a joke with a deadpan delivery. When I walk onto a project, I can make an educated guess about how it’s going to go based on how much work it takes to get up and running with the SCM and build system.

Getting to know a project is akin to meeting a person. Whether consciously or not, I form a first impression based on a handful of rules of thumb accumulated over the years. One rule of thumb is that it should take no more than a day to have a functioning copy of the source code, build environment, and automated test setup. If it takes longer - hunting through branches for the right version of the code, tracking down library versions, remembering which tags actually need to be checked out, or wrestling with a tool chain that was never standardized - then I know I have a lot of remedial work ahead of me.

I’m not including time to hassle IT folks to get an account created or setting up test environments with device hardware in the loop. That’s a separate issue for another blog post. This is just getting the productivity loop going: check out code, write tests, write code, build, test, declare victory.

There’s no magic to a successful CM system. The right things to do are written up in a thousand FAQs, user manuals, and technical books. Here are a couple that have been on my mind of late.

First, standardize your tool chain. Decide what editor, build system, compiler, linker, and testing tools are used on the project. Keep installers for those programs around, unless they’re included in the operating system on your development systems. If you’re really zealous about this, you could even standardize the operating system and revision you use on your development platforms.

Programmers are individuals; doubtless someone will protest that they absolutely must use their favorite editor and command line interface to the revision control system or it will completely sap their productivity and make small children cry. You have two good options. Either refuse to budge, or tell them they’re welcome to substitute as long as no one else on the project can tell the difference. For example, if you standardize on Eclipse, a skilled emacs user will be able to edit code without anyone knowing the difference. In my book, that’s okay. However, if someone decides to replace the Makefiles with Ant build.xml files, then everyone on the project will see the difference: either everyone will have to convert to using Ant, the two build systems will have to be maintained in parallel, or at the very least the files for the redundant build systems will have to be stored in the repository. Those are very noticeable changes.

Next, put the libraries upon which your program depends under revision control. There used to be good reason not to do this. SCM systems would get bogged down handling large binary files. With modern server hardware and SCM software, however, this just isn’t a concern for most projects. The advantage is that someone checking the code can check out the libraries as well and be assured that everything necessary to build the project is there. If you still can’t do this for some reason, keep a standard set of library files outside your SCM system, and keep a tight reign on them. Basically, if you don’t have the tool support you, you’ve decided to do your own manual configuration control of the libraries.

It should also be easy to figure out how to check out the source code. By that, I mean that it should be clear what branches and tags are applicable for a given development task. There are lots of guides on how to do this, and in recent days, they haven’t even conflicted that much. Whatever repository layout you choose, just be sure that it’s easy to find where new code is being developed, old releases are being maintained, and particular side projects are being carried out.

I have to take a moment to rant against branches. Everyone’s been very excited about lightweight branches, lately. We’ll have a branch for the released code, one for each maintenance release, one personal sandbox for each programmer, another sandbox for each different task the programmer is working on, another branch for demos to visitors. It’s the branch-making drinking game: “Every time someone says we need to be more agile, create a new branch in the repository!” I’ll admit, the ability for recent SCM tools to support lightweight branching and merging is pretty interesting. What seems to be missing is the cognitive support to developers to navigate the branches that they create. The tragic result is a repository full of branches, where no one is sure what branch is used for what purpose. My best suggestion is to use branches - they are helpful - but use them only as necessary and with a clear naming scheme, so that there’s no confusion as to what goes where.

When populating the revision control system, don’t forget about test artifacts. If your product needs startup or configuration data, check it in. If the tests need test data, check it in. I shouldn’t have to go hunting through the tape archive of a programmer who got canned three years ago, trying to find his test data to determine whether his code (which has been sitting in the repository untouched for those three years) ever actually worked right according to his own unit tests. These unit tests, as you may have guessed, should also be checked into the repository.

Bottom line, everything necessary to build the software ought to go into the repository, and it should go in organized such that it’s ready to build when pulled back out. Now that this rant is aside, perhaps we can get back to working on solving the real problems.

Skating at Work

jeffrey — Sun, 16 Jul 2006 02:03:05 +0000

Years ago, a few friends introduced me to a new use of the word “skating” to mean shirking one’s work responsibilities. One is skating when at work, seemingly productive, and deliberately accomplishing little or nothing. Note the difference from other forms of work avoidance, such as absenteeism. One of the best executed examples of skating was a guy helping clean a building. He spent an entire week with a bucket of soapy water and a sponge sitting and looking out a window. When someone came by, he wiped the sponge over the window a few times. When left alone, he went back to looking out the window.

I just left a job where I worked in a department of about fifty people. Some of them were the best with whom I’ve ever worked. Others were a questionable use of perfectly good space, light, and breathable air. Among the latter, I observed repeated and common threads that begin to form a pattern language of skating. I catalog a couple of them here not to encourage their use, but to suggest ways to detect and counter them.

Please keep in mind that the goal is to drive toward getting the work done, not to assign blame. I don’t care who shot John; I just want to make sure we don’t shoot him again.

Name: Can You Spare a Dime

Context: You are asked to perform a task that you want to avoid. You are a manager with responsibility for a budget.

Problem: You want to block the task from being performed.

Solution: Explain that you have insufficient resources to do the job. Not having enough people or material resources are good choices. Putting a high price tag on an undesirable initiative is a good way to kill it before it gets started. Worst case, even if it doesn’t work, the allocation of excess funds provides some consolation.

Remedy: Working around this objection depends a great deal on the local political situation. Management must be convinced that the cost estimate is inflated. I usually try to get details on the assumptions underlying the cost estimate; that often exposes the truth.

Name: Fireman and Arsonist

Context: You want to avoid the intimidating prospect of day to day work according to a plan.

Problem: A predictable, measured, and planned pace of work makes it easier to evaluate progress and quality of work. Work must therefore be made to inherently chaotic, requiring heroic fire fighting efforts.

Solution: Ensure that there’s always a crisis. If the workplace becomes calm, pick an issue and create a sense of emergency surrounding it. It takes a good sense of rhythm to get everyone jumping through rings of fire regularly. The reward is that workers trying to keep hold of a chaotic job are often lauded for their efforts with little examination of the quality of their work. An expert at this pattern is seen as a hero for having exceptional crisis management skills. The risk in this pattern is that the effort to fight the fires may be too great to qualify as shirking responsibility.

Remedy: This is a difficult pattern to overcome, particularly if the fires are genuine. I try to help the fireman/arsonist to start looking past the next fire drill and planning ahead. Unexpected changes and critical issues are a part of any engineering job. The goal is to make as much of the work relaxed and routine as possible. The trick is reigning in the arsonist so that he or she doesn’t continue to create trouble. If necessary, I’d have regular, informal discussions with the arsonist about relative importance of their assigned tasks.

Name: Houdini’s Escape

Context: You work off-site from your manager, such as by telecommuting.

Problem: You’ve been so successful with the other patterns in this list that you’ve now done very little for a long time. You wish to avoid this becoming apparent.

Solution: Disappear. Stop answering the phone. Don’t reply to emails. Occasionally send a short email to explaining the work is almost done or on its way. You might get lucky; some tasks really do become non-issues after a particular deadline or milestone has passed. In most cases, however, this probably won’t work forever. At best it forestalls the inevitable.

Remedy: Overcoming this pattern can be a time sink. I find one of the best tools is close contact. Next time I see this pattern, I’ll probably try a quick daily stand up meeting with the Houdini. It may also help to break tasks into small chunks of work and track status to a fine resolution. Early detection of trouble is essential. Catching the beginning of the escape routine makes it easier to nudge my wayward worker back on course.

Name: Nobody Knows the Trouble I’ve Seen

Context: You are asked to perform a task that you want to avoid.

Problem: You need to reverse the decision to perform the task.

Solution: Explain that the proposed task is not feasible. It could be that the process or methodology doesn’t apply to your situation. It could be because there are more important issues to address. It could be because the effort itself would be detrimental rather than beneficial. But whatever you do, make it clear that having to do as asked would add considerably to the already unreasonable sacrifices you are making.

Remedy: This one is pretty easy to overcome. I explain that the decision has been made and the task must be carried out. I find I have to pay extra attention to following up, however, as this worker can be expected to drag his or her feet.

Name: Twenty Questions

Context: You’re attending a meeting during which you’re expected to report status on a task, and you haven’t made significant progress.

Problem: You don’t want to get caught having not done anything.

Solution: Go on the offensive. Instead of providing status, immediately start asking questions about the task, the method to accomplish it, points of contact for information, and your teammates work as it touches on yours. By asking questions, you make yourself seem motivated and active, covering the fact that you really haven’t done anything. I once had the privilege of witnessing a master at this: she even managed to get her work assigned to others as action items from the meeting.

Remedy: I have two options to handle this: endurance or steering. If going the endurance route, I let my worker wear himself or herself out asking every question imaginable, then calmly repeat the inquiry about what’s been done and what hasn’t. With enough endurance, eventually the slacker will tire and admit to a real picture of what’s been done. Of course, if I had enough time to stand around wearing someone down, maybe it wouldn’t matter that the work isn’t getting done.

The second option, and my preference, is steering. I simply steer conversation back to the point. I explain that questions need to be saved for later to keep the status meeting focused. It’s important not to allow other team members to get drawn into lengthy discussion; questions for coworkers should be taken offline. Some people are pretty slippery, so this may have the feel of nailing jelly to a tree. I try not push too far in a public forum after confirming that there’s a problem. I can’t think of a circumstance where there’s something to be gained by publicly embarrassing someone. It’s better to take up the topic later in a private setting.

Name: Waiting for Godot

Context: You have a task to complete, and it requires inputs from others, peer or supervisory review, or coordination among departments.

Problem: You don’t want to do the work.

Solution: Work diligently until you reach a stopping point that requires action from someone else. Saunter over to the water cooler while you wait or web surf at your desk. If the task is particularly important, then it’s vital that you not do any other work while waiting, as waiting for this first task is the most important thing you can be doing. If someone asks for status, explain what you’re waiting for and from whom. This subtly suggests that you are blameless for the lack of progress. Definitely do not mention that you haven’t followed up in three weeks or are getting push back that you don’t know how to overcome.

Remedy: This pattern can be mitigated with a very simple line of questioning. I keep asking for more information. If a task is held up by the need for data, I ask what data is needed, what needs to be done to get it, who’s working on it, the schedule to which they’re working, and the contingency if the deadline passes. My goal is to force the layout of a very clear plan and timetable to accomplish the work. If this conversation instead ends up exposing the lack of such a concrete plan, I help put one in place. (Remember, the goal here is never to trip someone up to justify blaming them. The goal is always to drive toward getting the work done.)

Name: White House Press Secretary

Context: You are asked a question to which you ought to know the answer.

Problem: You don’t know the answer, or it would be against your best interest to reveal it.

Solution: Pick some other question for which you do have an answer that doesn’t harm your interests. Provide that answer instead. If the question is asked again, explain your irrelevant answer in greater length and detail as though the problem is with the listener’s comprehension skills. An expert at this technique will exercise sufficiently poor communication skills that his or her supervisor will have difficulty telling if the answer really addressed the question or not.

Remedy: I do not know of an elegant way around this problem: best to go straight through it. I explain that the answer I’m getting isn’t sufficient, then ask the question again. I do consider, though, that I may be the source of the problem and try to ask my question more clearly.

Job Listings and Past Experience

jeffrey — Sun, 09 Jul 2006 17:32:00 +0000

I started this blog now because I believe I’ll have more free time to devote to hobbies. I’ve just switched jobs. As I looked at my options, I read many listings for software engineers. The emphasis these listings placed on experience in a particular programming language, tool, or framework was disappointing.

Requiring that a programmer must have experience in one particular language or tool is like insisting that a waiter must have experience waiting tables at a seafood restaurant. Who cares if his experience was at a steakhouse instead? The essential skills, knowledge of how restaurants operate, and ability to deal with difficult customers are the same. If he’s thrown off because the plates have steamed crustacean on them instead of grilled cow, he’s no use to me anyway.

I’ve heard the counter-argument. “But steak and lobster have such different centers of gravity, so the plates balance differently. And it’ll cost us a lot to teach him the particular gotchas of this kind of food, like not letting the lobsters catch his fingers with their claws!” The idea here is that every language and tool has unique quirks, and that a developer must have experience with those quirks or they’ll get mired in a productivity sapping tar pit of small mistakes. There may be some truth to this argument in the first few days on the job, but not over the medium and long term.

I’m not dismissing the value of experience as a whole. The right kind of knowledge and experience are valuable. More experience working as a developer is better than less. I’m now a far better developer than I was five years ago. I’ll be better still in another decade. Domain knowledge and experience, too, are more than surface deep. Picking up a thorough understanding of a problem domain takes time, and in the meantime the less experienced programmer may do some amount of well-intentioned harm. Don’t expect someone with a background in developing UML modeling tools to produce a good design for a real-time, embedded signal processing system. Those distinctions make for better hiring criteria than years using a language or tool.

When hiring, I look for developers with strong skills in areas that contribute to delivering software. A great developer should:

Know how to produce good, clean code.
Understand requirements specification, design, and testing.
Have solid experience with the things that keep projects running smoothly throughout the life cycle, to include development process, configuration management, peer review, and work estimation.
Have superior soft skills such as drive and motivation, verbal and written communication ability, the capacity to work with a team, and leadership capability.
Domain knowledge applicable to the project.

See what’s missing from that list? Specific languages. Specific development tools. Specific APIs. Specific processes. It doesn’t matter whether the candidate’s experience is in C or Perl, NetBeans or Eclipse, CVS or Subversion, PSP or XP. To a good developer, those are just variations on a theme. And yet, these variations get top billing in job listings: 8 years experience in C; not C++, because we don’t understand the similarity. Ideal candidate must fog a mirror held under nostrils and supply own blank stare.

Insisting a candidate have experience with a particular language or tool set assumes that it’s a difficult to translate development knowledge from one language syntax to another. For a short time there was a fad - forgive me, trend - toward languages with very English-like syntax. Those languages never took hold, in part because they solved the wrong problem. The syntax was never the hard part; the right kind of thinking was and is.

A few old college friends of mine say they’ve solved this problem by lying shamelessly about the buzzwords they know. Once they land the job, they hit the bookstore and pick up a few reference books. By the time they walk in for their first day on the job, they know everything they need to know about the language and tools du jour. I can’t bring myself to do this. As for you, gentle reader, this is between you, the little you with the pitch fork on your left shoulder, and the little you with the halo on your right shoulder.

On Blogging

jeffrey — Sun, 09 Jul 2006 17:20:01 +0000

When I first became familiar with putting up personal web pages, I thought “I’ll never do that.” Such is the power of the word “never” that shortly thereafter I had a few small pages. The “Infrequently Asked Questions” title stuck to those pages because it seemed doubtful that anyone else would get any use out of them. I haven’t updated them in years, but they remain online because I still get an email every few weeks thanking me for them.

When I first became familiar with blogging, I thought “I’ll never do that.” Such is the power of the word “never” that… here we are.

There are a number of thought provoking and useful blogs online. There are also legions of bloggers writing diaries without the good taste to hide them in their sock drawers. That latter category doesn’t hold much appeal for me. I’m going to impose a few ground rules here that, were I king of the forest, I would write into the law of the land.

I’ll only update this blog when I think I can add something useful to others.
I’ll only add entries when I think I’ve written them without spraining the English language too badly.
If that means the blog is infrequently updated or abandoned altogether, so be it.