Over this past summer, our air conditioner failed. It wasn’t a quiet failure of the compressor. The outflow line for the condensation water blocked up and sent water down over and through the oil furnace below. In looking at the state of the HVAC system, particularly the oil furnace, it became apparent that they had been poorly maintained for a lengthy period of time, despite our service contract with our oil company. Getting recommendations from neighbors and friends is one way to help avoid this kind of abuse of trust. Many sites on Teh Intarwebs use reputation systems as well, but most are oriented around transactions with short durations. Many industries, including health care, HVAC maintenance, accounting, and investment work over considerably longer periods.

The failure mode of the existing air conditioner was pretty interesting. Our air conditioner was (and our new one is) a split system. The unit outside is the condenser, which compresses refrigerant turning it into a liquid. This process generates heat, which is dissipated in part by a fan blowing air over the condenser coils. The liquid refrigerant flows through piping into the basement.

Inside the basement is another box, called the air handler unit. The air handler unit blows air throughout the house. As is not unusual, our air handler unit sits above and inline with our oil furnace, a fact relevant to the failure we suffered. As the refrigerant enters the air handling unit, it passes through a valve that releases the liquid into a much lower pressure environment. The liquid boils to a vapor, reducing its temperature to the refrigerant’s boiling point. Remember, this stuff isn’t water, so the boiling point is actually well below the temperature of a comfortably cooled room. The cold refrigerant vapor flows through coils in the air handler unit, chilling the air that passes over the coils. As the warm, humid air is chilled, it loses some of its moisture. Condensation collects on the coils in the air handler unit, drips into a pan, and drains away from the unit in a drain pipe.

That’s how it’s supposed to work. In our case, the drain pipe plugged up and the condensation water backed up until it was pouring down over and through the oil furnace onto the basement floor. Great. It turns out the drain pipe went through the basement wall below ground level. We never found anywhere outside where it surfaced to discharge. I suppose it’s not surprising it got plugged up, being a pipe to nowhere.

In diagnosing this problem, we also took a good hard look at the state of our oil furnace. Despite religiously keeping a maintenance appointment with the oil company every six months, our furnace was in a deplorable state of cleanliness and repair. Our new HVAC technician asked how many years it had been since we had our system cleaned and maintained. In his words, “They were cleaning this thing every six months? What — did they just come down here and bang wrenches for a couple minutes and leave again?” The flue was corroded to the point of developing holes. The flue, clean-outs, and furnace itself were thick with soot and particulate debris. Every tunable item was set precisely to minimize the efficiency of the furnace, contributing to a high oil burn rate. And the line to the furnace was leaking oil.

I am somewhat to blame for this. I wasn’t around to keep an eye on what they were and weren’t doing. Of course, I’m not sure I would’ve known the difference. I definitely should’ve recognized the conflict of interest in asking my heating oil vendor to maintain the efficiency of my furnace. As I’m covered by their service plan, avoiding necessary repairs saves them money. Any cleaning they don’t do degrades the efficiency of the furnace, increasing my consumption of their product. Any fine-tuning they do to ensure a high burn rate again means I buy more of their product.

Perhaps I should get my forehead tattooed: “I was one of the ones born every minute.” One might also suggest that my new HVAC technician was exaggerating his criticism to ensure that he’d get future work from me and possibly sell me a new furnace and air conditioner. I’d like to think that by then I was wary enough to do sufficient fact checking to verify his claims.

The world is too complex to detect this kind of grift in every area of life. It’s quite possible that the oil company’s owner is tracking the money he’s made from cheating me on a desktop secured by paying a consultant hundreds of dollars to do no more than install a free (as in beer) firewall and anti-virus product. Learn enough about how your car works to avoid being taken at the auto shop, and you may neglect checking whether your prescription drug was approved shortly before a senior official left the FDA for a highly paid position at the drug’s manufacturer. There just aren’t enough hours in the day.

Common sense does help. Most people don’t fear they’ll be answering emails from deposed Nigerian royalty any time soon, but it’s not so easy to detect an abuse of trust by a domain expert hired precisely because of their specialized technical expertise. Asking lots of questions helps, as the answers should be self-consistent at a minimum. An optometrist once recommended I consider laser correction for my vision.

Me: “I’m not convinced yet. I’d like to see some evidence of what happens over the long haul — no point in getting perfect vision now just to find out I’m guaranteed to spend my old age completely blind.”

Doctor: “Oh, long term studies have been done. It’s completely safe.”

Me: “Oh really? Just where did they find seventy year olds who’d had this surgery done in their thirties, when the current technique is less than ten years old? Was the time machine they used completely safe, too?”

I’m still wearing glasses.

It’s hit or miss, though, depending on whether you ask the right questions and catch any inconsistencies. A lot of professionals don’t like being second guessed in every step of their work. Would you enjoy it if your boss stood over your shoulders questioning you about every variable name and subroutine you used? Even those that don’t mind may be talented in their craft but poor teachers. Worst of all, it still doesn’t scale well enough — still not enough hours in the day.

I can Google someone and their company, but that can be a time consuming process as well, as I have to wade through dozens of random forums and blog posts trying to put together a coherent picture. Also, there tends to be a publishing bias towards positive information. Reading diatribes and flames, you might not think it, but there are still many people who won’t put names to a bad experience for fear of retaliation or being sued. (You don’t see my oil company’s name in this post, do you? Well, until the end of this contract season, they’re still responsible for delivering the oil that keeps my home heated. Maybe freezing wintery days aren’t the best time to accuse them of something somewhere between negligence and grift. I wonder, does that make me sensibly cautious or a coward?) Lastly, for some traditional brick and mortar businesses, there might not be enough information to find. I was recently looking for a lawyer and an accountant, and there just wasn’t much to be found online about either firm I selected.

The world continues to sprout new services of a specialized and technical nature. Reputation and referrals will become more important in choosing with whom to do business. Reputation systems are already quite common on Teh Intarwebs. Ebay’s rating system for buyers and sellers may be the best known, but almost every recent community web site has some feature for separating the contributing members from the trolls. I found my new HVAC technician through an online service that features a rating and feedback system for HVAC, plumbing, electrical, roofing, and remodeling contractors.

The system had its flaws, though. It relies on short-term feedback. This is good, as the user’s memory of the transaction is still fresh. It’s also bad, as quality of work may not be apparent for months or years. New house foundations may look great but crack a decade later. One might be satisfied with a family dentist for a long time, then visit another dentist and find out that years of neglect require a root canal and bridgework to correct. There’s a joke that roofers don’t mind offering 30 year guarantees on 20 year roofs because they’ll be out of business in 10 years anyway.

Existing reputation systems don’t seem to be built for this kind of long haul with changes in opinion. They do best with small-scope, short-duration transactions, such as individual posts on a message board, sales at an auction site, or visits to a restaurant. They focus on collecting a wide breadth of data by trying to maximize the number of users who provide feedback. There’s little follow up over time. Of course, there’s a nuisance threshold here too. No one is going to fill out a survey a year for every home repair, product purchase, and professional service they use. It’d just be overwhelming, especially as in most cases there wouldn’t be a significant change of opinion.

So what would I do differently for a reputation and rating site for trade contractors and professional services? At a minimum, I think such a site would need to:

• Address the problems existing reputation systems face, such as encourage a good user response rate by keeping the rating process simple and flexible. I shouldn’t have to fill out a three page survey. Perhaps an overall one to five rating would be sufficient with comments describing the rationale behind the rating. Spam, trolls, and other sources of bad data will have to be filtered to keep the site useful.
• Pick an empty niche. No site would easily span every industry. There are already sites handling feedback for online stores. Weakly represented are trades and professional services, like HVAC, electrical, general contracting, architects, lawyers, accountants, and doctors. Start focused and branch out once the system is running smoothly.
• Avoid any actual or appearance of collusion with the vendors to be rated. The service I used to find my new HVAC technician is paid by the contractors to provide leads to people like me. I hope they run their reputation system honorably, but there’s an economic incentive for their contractors to get high ratings so people like me will want to hire them.
• Follow up later. Whether a fixed system or by random sampling, request that a set of users update their opinion. Give them a convenient one-click link to indicate no change in their views. Give them an equally convenient one-click link to come back and explain that one year after installation the roof had to be completely redone or that their five-star, huge-refund tax return didn’t hold up to an audit.
• Provide tools to make sense of the data. It’s no longer a matter of knowing that someone’s average rating is high or low. It’d be necessary to see changes over time. If 4500 out of 5000 total ratings are high initially, but 90 out of 100 total ratings are terrible a year later, the overall average for all time may still be high, but there’s a consistent downturn a year after service. Help users see how people’s impressions change. A good choice might be some graphic showing the trend over time, made as an image map to drill into details like individual comments.
• Handling the passage of time also means making the user aware of significant changes to the company as well. Perhaps a company provides terrible service until new management turns them around and they improve, or vice versa. Perhaps a contractor dissolves his old company and forms a new one to shed the history of complaints and lawsuits under the old name.

If I’m just ignorant and there’s a site already doing this, let me know. I’d like to have a look at it. Or if you’re currently working on such a project, drop me a line — I’d be interested in how you’re solving this problem.

This entry was posted on Sunday, March 25th, 2007 at 5:39 pm and is filed under Crafting Software. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.