Taxing my badometer

Posted by Ben Simo

I've used great software. I've used some horrible software. I've used lots of software with annoying bugs. I regularly spend a great deal of time working around bugs in software that I use on a daily basis. However, I think I have recently found the worst yet. It is not the worst because it fails to work. It is not the worst because it is missing features I want. It is the worst because it appears to work on the surface yet generates bad data that has a direct impact on my stress level, my finances, and my interaction with the IRS.

Dealing with the IRS -- or even thinking about it -- is already stressful enough. The tax code is complicated enough that it is easy to accidentally report the wrong income or deductions. If it were easy, we'd see accountants lining up at soup kitchens instead of raking in the cash helping us common folks with our tax returns.

For most of the past 10 years, I have elected to use commercial tax preparation software to help me complete and file my tax returns. I've used some tax software that was good, and some that was bad. I've even tried some freebie and online options. The software I used this year is nothing short of a software atrocity.

This software was swarming with bugs -- although most were not obvious. I suspect that most users didn't even notice.

After installing the software, I dutifully entered more financial data than I care to track. I tried a couple what-if scenarios in the process: such as a last minute IRA contribution. I also entered some estimated values before entering the real data so that I could determine if I should take one option or another. I also entered some data for deductions that I later removed after learning (due to reading the IRS publications) that they not apply to my situation.

I did not follow the interview process from start to finish. I went back and forth in this process entering data, revising data once I clarified the rules or gathered up all my evidence for income and deductions. After several hours, I thought I had a complete return. I had entered and verified all my data in the software's "interview" interface.

Then the automated error check failed. It would not let me continue until I fixed data in a form. This was a form that I had accidentally selected somewhere in the process that contained no data. I then tried to delete the form by unchecking a box that made the options that went the form available. The software then informed me that I could not delete the form there. I went to the portion of the interview where it told me to delete the form and there was no delete option. I then discovered that I could delete the form from the program's "forms" list. However, I later found out that going back to any point in the interview recreated the form. There was no option in the software to permanently get rid of my accidental selection.

I discovered that there was no way to print my return for review until I got to the last step of the eFile process. This required that I go back in the process and tell it that I wanted to file on paper. I printed the return and was surprised to see that the data on the return did not match the current data in the interview interface. Some values that I had entered and then deleted or changed were in the printed return. My return was inaccurate. I was glad that I went through the tedious process of getting to the print feature. If I had eFiled without first printing, I would have submitted a return that did not match the data I entered -- and the data shown to me on the screen.

I then spent several hours deleting and reentering data until the printed forms finally matched the data I entered.

In the midst of my review, I noticed that the software had forced me to take an option that resulted in a lower federal tax bill. However, selecting this option instead of another equally valid option significantly increased my state tax bill. The software forced me to have an overall higher tax bill because it only considered the federal return when deciding which option was best for me. After about an hour, I realized that I could remove a deduction in order to force the software to take the alternate option.

Once everything was in order and the displayed and printed data matched, I selected the eFile option to submit my return. I entered bank account information to pay my taxes online. I then discovered that the eFile service I paid for did not allow me to submit electronic payment to the state. My returns were submitted. My federal payment was sent electronically. However, my state payment still had to be mailed. I could have filed my state taxes and paid online using the state's web site as I had done in previous years. Instead I thought it would be easier to use the integrated system. I wish that the software would have told me that I could not electronically submit my payment before it took my money for the eFile service.

And to top it all off, the software did not tell me how to send in my payment by mail. I spent nearly an hour going through the system's help and the vendor's web site seeking guidance. I finally found an option for an online chat with customer service. It then took customer service a half hour to figure out how and where I should send my payment.

I asked if I could have a refund due to the buggy software and the eFile service that was not as good as the state's free eFile service. I was directed to call them by phone to get a refund because the could not give refunds via the online chat. I called the number and waited on hold of over a half hour before I was disconnected. I called back and got a recording stating that the customer support office was closed.

The bad software was topped off by bad customer service. I was angry. My badometer was pegged.

I then did a little exploratory testing in the application. I discovered additional bugs that created incorrect tax returns. I noticed that the software said it would calculate things for me and then ask me for the value without giving any assistance. I found places where the instructions in the application did not match the IRS publications. I even found a way that I could get it to calculate a refund of any value on the state return without changing the income or deductions.

I understand that tax software companies have a very small window in which to do their development and testing. I am amazed that this software was released. The poor quality -- and the agravation it caused -- have ensured that I will not be a returning customer next year.

I suspect that the software passed all the test cases created by its makers. I suspect that it even passed multiple automated tests. This just goes to show that passing all the preconceived scripted tests does not make a quality product. I do not know what methods this company uses for testing their software, but I suspect that it is primarily scripted testing.

I believe that exploratory testing (and model-based automated testing) would have been more likely to find the bugs I encountered than scripted testing.

As we test software, we need to consider the infinite possibility of data and work flow variations. We can't test all the variations, but we can vary what we test. It is easy for individual testers to select similar test options when they think they are selecting data randomly. Seek out variety. Use random data generation tools to provide more variety.

How would you test tax software? How would you ensure that bugs like the ones I encountered don't aggravate customers?