Last weekend, I attended the 1^st FROSST (Front Range Open Space Software Testing Conference) in Thornton, Colorado. Thanks to Ben and Heidi for putting on such a great open space conference. I attended the Metrics, Open Source, Performance, Automation, and Agile sessions. Below are a few comments about each:

Metrics

This was an interesting topic with consensus for only one outcome of the metrics. Provide Management with enough valuable information about the software to make a reasonable judgment on whether the software is ready for prime time.  The metrics could be in the form of raw data, graphs, or just plain old testers speak. Some items to be considered were bugs, percent of software covered (or not tested at all), the number of tests executed / failed, and traceability to requirements.

One new idea was discussed for providing a visual indication of bug severity vs the likelihood of occurrence. On a white board, put a sticky for each bug in the iteration. If all the bugs end up in the top right quadrant, you may have an issue. This will give you a good visual indication on whether there are issues with the release.  

 Open Source

This was probably my favorite session since I am pro Open Source. There was concern from some that using OSS would take longer to be productive (specifically for test automation). The overall consensus after discussion was it didn't really matter whether the tool was commercial or OSS, the real investment was the people using the tool. This led to a discussion on the real cost. Using an OSS tool would save a significant amount of money if dozens of licenses were required.

The framework developed for automation is extremely important whether you use commercial or OSS.

 If your organization does go with an OSS solution, look at the tools language and the support you will have on the development team, from associates, and the OSS community. Also, give back to the OSS community. You don't have to provide code, but product feedback and reporting bugs always helps. You want to make sure that the community is very active and responds in a timely manner to your issues.

Specific tools discussed commercial and OSS were QTP, Robot, Watir, Selenium, and Eclipse Test & Performance Tools Platform (TPTP).

Agile

I won't lecture about the Agile methodology since we are an Agile Team. One of the developers in attendance and his team has mastered one significant concern with Scrum. "Don't ever write bugs during the iteration, instead write failed tests. Nobody is done with the iteration until all failed tests are passing". I like it.

Performance

We were privileged to have Scott Barber in attendance. Scott is the Chief Technologist of PerfTestPlus, Executive Director of the Association for Software Testing, Co-Founder of the Workshop on Performance and Reliability, and co-author of the Microsoft patterns & practices book Performance Testing Guidance for Web Applications. Scott provided some very simple strategies for eliminating 80% of software performance issues. I would encourage you to visit Scott's website.

Automation

Probably the most important advice provided in this session was testing your business rules at the API level instead of the GUI. GUIs are always changing and the amount of effort to maintain them is unbearable for small organizations. If GUIs have to be automated, use small building blocks and reusable blocks of functionality. Again, your framework is the most important part of any automation strategy.  Two good reads are:

 

Grow Your Own Test Harness Naturally, by Kevin Lawrence http://www.developertesting.com/archives/month200508/20050824-Grow-your-harness.html

 

Test Automation Snake Oil, by James Bach http://www.satisfice.com/articles/test_automation_snake_oil.pdf

 

Happy Testing,

 

Jeff Peters
Software Quality Assurance Manager

jpeters@xaware.com

http://www.xaware.org/>