Test Executives - part 3

I started writing about test executive software last month, and then a couple weeks ago I wrote about off the shelf software.  Now I want to write about my experiences with "rolling your own" test executive.

I've worked with homegrown test executives at two different companies.  At the first company the test executive evolved out of a couple of different programs for testing different features of the same product.  In the second company, I worked with a test executive that had been written several years before I started. I'll address each one in a separate paragraph.

The executive I wrote myself was somewhat rudimentary.  For the products we were manufacturing, there were six distinct tests you could perform.  Each of those tests had from about 3 to 20 different numerical parameters that could be adjusted.  This just screamed out for scripting, so that's what I did.  The end result had the configurability I needed, and it was often used by test technicians, but it was missing several other features of an off the shelf system test executive:

• It was only usable within the Labview framework I had written.  For example, it couldn't interface with any modules written in C++  without a lot of extra coding.
• There was no programmatic logic in the scripting.  The scripts consisted entirely of what tests to perform under what conditions - no if-then or optional looping allowed.
• Some reporting tools existed, but mostly in simple formats (saving CSV files or bitmaps of graphs).
• It was written with testing a specific product, and had to be overhauled to test some other type of product.

The other test executive was written in .NET and made use of Measurement Studio for certain graphical presentations.  The scripting tools had quite a bit of programmatic control, and the reporting tools were more extensive.  But there were different problems.

• Because it had been designed as a reconfigurable tool, it was horribly complicated to use.  We didn't let technicians near it without a lot of training.
• It was only usable within the .NET framework and didn't play well with other modules.
• It was written with testing a specific product, and had to be overhauled to test some other type of product.

That's a brief synopsis of the two in-house test executives.  So, what is the point I'm trying to make?  I'm not really saying that you should run out and buy a copy of Test Stand.  It's certainly a nice piece of software, but it's expensive and may be overkill for what you need.  I guess the lesson I've learned is that if you get to the point were you want to develop something for yourself to reuse over and over, learn from how the OTS software does it.  Specifically:

1. Don't make it overly complicated
2. Make it generic enough to use for different tasks
3. Use programmable logic in the scripting
4. Have plenty of reporting tools

Test Executives - part 2

A couple of weeks ago I started writing about test executive software.  I've decided that this topic is another three-parter, so in this 2nd post I'll write about off the shelf (OTS) software.  The third post will cover in-house test executives.

I mentioned last time that I have my own definition of a test executive software, but I never wrote that definition.  Well, for me the software is defined by what I expect it to do.  The three things I expect at a minimum are:

1. Configurable testing.  I need the capability to switch up the order in which specific parameters are measured and under what conditions they are measured.  This is typically done with scripting or configurable sequence files.
2. A solid, usable GUI.  A test executive is often used by a test technician instead of an engineer, so you need software that doesn't require a lot of care and feeding.
3. Easy to see what's happening.  This is somewhat related to #2 above but more specific.  I like graphs, reports, and charts.

The OTS platforms I have experience with are TestStand and ATEasy.  I wrote about my experiences with TestStand a year ago (a three parter, starting here and continuing with this and that).  To be honest, I'm a lot more nebulous on ATEasy and my knowledge of it consists of three data points:

• I sat through a long seminar on it once.
• I worked with a guy who had used it before and liked it.
• I evaluated it for about a week at a previous company.

Given that limited knowledge, I think it would serve as a cheap, limited alternative to TestStand.


On a final note, NI published a checklist for evaluating test executives.  While the list is a little self-serving and obviously slanted towards NI software, it's worth consideration as a starting point.

Happy SIX years

Six long years ago today I wrote my first post on this blog.  Since then I've written close to 200 posts on various topics.  Based on some of the feedback I've gotten, at least a few of them have been helpful.  So here's to six more.

Stumper questions - vindicated

Hmmm, I wrote this post last week about trick questions in interviews.  I decided I wanted a little support for it.  Synchronicity struck: THIS was published in the NYT just last week.

I particularly agreed with the statement, "On the hiring side, we found that brainteasers are a complete waste of time."

Interviewing - stumper questions

In the process of digging through the list of partially-completed posts, I found a link on EDN that talked about technical questions you can ask prospective employees.  Rereading that little tidbit made me think about my stint phone screening college graduates for HP in the late 90s.  The company had a long list of pre-vetted questions that I could ask these poor souls.  The questions ranged from basic EE problems ("describe how you would implement a low pass filter") to programming issues ("what is a linked list?") to twisty questions (i.e. - the infamous "water in a locked room with a dead body").

Anyway, the point I wanted to make was that in today's world it seems somewhat silly to pepper a candidate with questions like this.  I mean, if you want to check their credentials then go take a look at their LinkedIn page.  Heck, just google them.  What's more important, in the long run, is what kind of person are you hiring and how will he/she fit into the group?

Test Executives - part 1

For many years now I been a little obsessed with the subject of test executives.  I'll explain why in a minute, but first let me try to define what I think of as a test executive.

Wikipedia has a specific description of test executives, but my definition is more specific to my experiences.

So why do I want to write about this topic?  Ten years ago I was working for a small division of a Fortune 500 company.  We had just bought a complicated test/manufacturing tool developed by a consulting firm.  This firm had written a script driver that they used for most of their hardware projects.

That was an epiphany moment.  I had written a script driver for a specific test instrument when I worked at HP back in the late 90s, but I'd never really thought about writing one that I could reuse across projects.  Nor had I considered buying one off the shelf.  In the ten years since that moment I've worked at four other companies that developed four separate solutions.  So over the next couple posts I will write about different test executives - commercial as well as "roll-your-own" - and compare and contrast my experiences with each.

Historian Software

A week ago I sat in on a presentation for a type of software I had never had dealings with before: historian software.  It was GE's version of the software, called Proficy.  Before the demo I did a quick check on what the software was, and it really just sounded like a database with a nice GUI.  

But after the presentation I have to admit that it's really a lot more than that.  I'm not sure if I'll ever use anything like this, since it seems geared towards enterprise-level applications and I gravitate to startup companies.  But it's still neat. 

It's always cool to learn about something new in this field.  Maybe I'll check out NI's historian software, NI Citadel.