Test Executives - part 3

I started writing about test executive software last month, and then a couple weeks ago I wrote about off the shelf software.  Now I want to write about my experiences with "rolling your own" test executive.

I've worked with homegrown test executives at two different companies.  At the first company the test executive evolved out of a couple of different programs for testing different features of the same product.  In the second company, I worked with a test executive that had been written several years before I started. I'll address each one in a separate paragraph.

The executive I wrote myself was somewhat rudimentary.  For the products we were manufacturing, there were six distinct tests you could perform.  Each of those tests had from about 3 to 20 different numerical parameters that could be adjusted.  This just screamed out for scripting, so that's what I did.  The end result had the configurability I needed, and it was often used by test technicians, but it was missing several other features of an off the shelf system test executive:

• It was only usable within the Labview framework I had written.  For example, it couldn't interface with any modules written in C++  without a lot of extra coding.
• There was no programmatic logic in the scripting.  The scripts consisted entirely of what tests to perform under what conditions - no if-then or optional looping allowed.
• Some reporting tools existed, but mostly in simple formats (saving CSV files or bitmaps of graphs).
• It was written with testing a specific product, and had to be overhauled to test some other type of product.

The other test executive was written in .NET and made use of Measurement Studio for certain graphical presentations.  The scripting tools had quite a bit of programmatic control, and the reporting tools were more extensive.  But there were different problems.

• Because it had been designed as a reconfigurable tool, it was horribly complicated to use.  We didn't let technicians near it without a lot of training.
• It was only usable within the .NET framework and didn't play well with other modules.
• It was written with testing a specific product, and had to be overhauled to test some other type of product.

That's a brief synopsis of the two in-house test executives.  So, what is the point I'm trying to make?  I'm not really saying that you should run out and buy a copy of Test Stand.  It's certainly a nice piece of software, but it's expensive and may be overkill for what you need.  I guess the lesson I've learned is that if you get to the point were you want to develop something for yourself to reuse over and over, learn from how the OTS software does it.  Specifically:

1. Don't make it overly complicated
2. Make it generic enough to use for different tasks
3. Use programmable logic in the scripting
4. Have plenty of reporting tools

Test Executives - part 2

A couple of weeks ago I started writing about test executive software.  I've decided that this topic is another three-parter, so in this 2nd post I'll write about off the shelf (OTS) software.  The third post will cover in-house test executives.

I mentioned last time that I have my own definition of a test executive software, but I never wrote that definition.  Well, for me the software is defined by what I expect it to do.  The three things I expect at a minimum are:

1. Configurable testing.  I need the capability to switch up the order in which specific parameters are measured and under what conditions they are measured.  This is typically done with scripting or configurable sequence files.
2. A solid, usable GUI.  A test executive is often used by a test technician instead of an engineer, so you need software that doesn't require a lot of care and feeding.
3. Easy to see what's happening.  This is somewhat related to #2 above but more specific.  I like graphs, reports, and charts.

The OTS platforms I have experience with are TestStand and ATEasy.  I wrote about my experiences with TestStand a year ago (a three parter, starting here and continuing with this and that).  To be honest, I'm a lot more nebulous on ATEasy and my knowledge of it consists of three data points:

• I sat through a long seminar on it once.
• I worked with a guy who had used it before and liked it.
• I evaluated it for about a week at a previous company.

Given that limited knowledge, I think it would serve as a cheap, limited alternative to TestStand.


On a final note, NI published a checklist for evaluating test executives.  While the list is a little self-serving and obviously slanted towards NI software, it's worth consideration as a starting point.

Happy SIX years

Six long years ago today I wrote my first post on this blog.  Since then I've written close to 200 posts on various topics.  Based on some of the feedback I've gotten, at least a few of them have been helpful.  So here's to six more.