Archive for March, 2009

Software Verification vs. Validation

Thursday, March 26th, 2009

For some reason it just really bugs me that these two terms are incorrectly interchanged so frequently.

Part of the problem is that the document General Principles of Software Validation; Final Guidance for Industry and FDA Staff (2002) does not do a good job of differentiating actual verification and validation activities. They just call everything validation.

The recent MD&DI article Building Quality into Medical Device Software provides a pretty good overview of the these regulatory requirements, but is a another case in point.  The article talks about "software validation" at every step just like the FDA document.

Another similar article on this subject is Software Validation: Turning Concepts into Business Benefits.  It is also confused. e.g. (my highlight):

... software validation involves the execution of tests designed to cover each of the specific system requirements.

No, testing specific requirements is a verification activity! It's no wonder most people are confused.

These definitions, Difference between Verification and Validation, are better as they highlight the sequencing of activities:

Verification takes place before validation, and not vice versa. Verification evaluates documents, plans, code, requirements, and specifications. Validation, on the other hand, evaluates the product itself.

From here (warning PDF):
verification vs. validation

Validation activities (usability testing, user feedback, etc.) are much harder to define, execute, and document properly than most verification testing.

Here are the golden rules:

Verificationwas the product built right?

Validation: was the right product built?

I guess I should get over it...

UPDATE (5/12/09):  Good definitions from here: Diagnosing Medical Device Software Defects Using Static Analysis:

Verification and validation are terms that are often used in software. However, it is important to understand the difference between these two distinct but complementary activities. Software verification provides objective evidence that the design outputs of a particular phase of the software development life cycle meet all of the specified requirements for that phase by checking for consistency, completeness, and correctness of the software and its supporting documentation. Validation, on the other hand, is the confirmation by examination and provision of objective evidence that software specifications conform to user needs and intended uses, and that the particular requirements implemented through software can be consistently fulfilled.

UPDATE (8/6/09):  The importance of proper V&V can not be overstated. The FDA is watching: FDA still enforcing regulations for validation of enterprise software.

UPDATE (2/11/10): I just noticed that the guidance document link on the FDA site was changed and fixed it. When I reviewed the document I found that even though it was "issued" in Jan. 2002 it had been recently updated (11/6/09). The later sections (4, 5, and 6) still use the term validation generically, but the updated document does distinguish between verification and validation:

3.1.2 Verification and Validation

The Quality System regulation is harmonized with ISO 8402:1994, which treats "verification" and "validation" as separate and distinct terms. On the other hand, many software engineering journal articles and textbooks use the terms "verification" and "validation" interchangeably, or in some cases refer to software "verification, validation, and testing (VV&T)" as if it is a single concept, with no distinction among the three terms.

Software verification provides objective evidence that the design outputs of a particular phase of the software development life cycle meet all of the specified requirements for that phase. Software verification looks for consistency, completeness, and correctness of the software and its supporting documentation, as it is being developed, and provides support for a subsequent conclusion that software is validated. Software testing is one of many verification activities intended to confirm that software development output meets its input requirements. Other verification activities include various static and dynamic analyses, code and document inspections, walkthroughs, and other techniques.

Software validation is a part of the design validation for a finished device, but is not separately defined in the Quality System regulation. For purposes of this guidance, FDA considers software validation to be "confirmation by examination and provision of objective evidence that software specifications conform to user needs and intended uses, and that the particular requirements implemented through software can be consistently fulfilled." In practice, software validation activities may occur both during, as well as at the end of the software development life cycle to ensure that all requirements have been fulfilled.

All Atwitter

Sunday, March 22nd, 2009

I'm finally all atwitter.  Better late than never. Use the link on the lower right or click here:

Follow me on Twitter

It seems like it will be a good place for quick thoughts, links, and discussion.

Contradictory Observations and Electronic Medical Records

Tuesday, March 3rd, 2009

Martin Fowler has an interesting discussion in his ContradictoryObservations post.  This little slice of medically related software design insight is particularly relevant because it highlights (at least for me) the complexity of the use of electronic medical records and their interoperability.

In a broader sense I suppose it also shows some of the underlying difficulties that face the Obama administration's new EMR adoption push.  But I'm not going there.

The concepts of observationsrejection, and evidence are good, but they're just the tip of the iceberg:

rejected and evidence

Even after you've modeled the data interactions, how do you effectively communicate these concepts to the user?  Or to another EMR that doesn't know about your model or how it's used?

Martin's view is that:

Most of the time, of course, we don't use complicated schemes like this. We mostly program in a world that we assume is consistent.

Unfortunately, many of the issues facing electronic medical records do require complex solutions. And even when the world is consistent, how you implement a solution may be (actually, will probably be) very different than how I implement it.  Either way, interoperability will always be a challenge.

We're going to need lots of good software design tools to solve these problems.

Kindle 2

Sunday, March 1st, 2009

Kindle 2I'm a big reader. When my family told me that they ordered a Kindle for me I was pretty excited. Well, that was over two months ago.  We were notified a couple of weeks ago they would be shipping a Kindle 2. It finally arrived!

The Kindle 2 hardware has a modern sleek look and feels great in your hands. At 10.2 ounces it's lighter than most of the books I read.

Here are the things I like:

  • Electronic-ink: The displayed text looks like a real book and can be read anywhere you'd normally be reading.  You can also change the font size to your liking. Because there's no back light, it's easy on the eyes.
  • Whispernet: The Amazon broadband 3G network is subscription-free when you buy the device. This gives you access to the "Kindle Store" as well as the rest of the internet (beware: see Browser notes below). Being able to download purchased books and free previews is the ultimate in instant gratification. There are a number of free e-book sites (e.g. Feedbooks)  that can also be accessed directly from the Kindle.
  • Search, bookmarks, and annotations:  At first you might think that not being able to "flip" through the pages of a book and spot interesting content would be a negative. That may be true for some people. In the long term though, being able to easily find long forgotten content along with your own bookmarks and thoughts is a significant value-added.  I see these features as a real game changer.  Here's a simple example: How many times have you purchased a book that you already own? With an e-reader, it won't happen again.
  • Built-in Dictionary: I'm always looking up words. With the Kindle, you just point at the word and the definition appears at the bottom of the page. Sweet!
  • Personal files: Amazon provides an e-mail based service for converting common file formats (e.g. PDF, DOC, etc.) in to e-book files. This is very handy for keeping reference and personal material on the device.

Here are the things that I've found annoying:

  • Electronic-ink: Static pages are great, but all page turns have a noticeable "blink". Apparently the screen must be blanked with all black before updated text is drawn. You do get used to it though.  The other major issue are the display problems with menus and cursor movement. The pop-up menu is sometimes not completely displayed. Also, you can end up with multiple pointers on the screen if you move the cursor to quickly.
  • 5-way Pointer and Page buttons: The 5-way pointer takes some getting used to. I accidentally purchased a book when I tried to move the pointer up but inadvertently pressed select. Fortunately, I would have bought the book anyway. The page buttons (Next, Prev, and Home) seem kind of tight -- you have to snap down pretty hard to activate them. They look nice, but feel clunky.
  • Keyboard: It works, but the keys are very small and hard to use.
  • Web Browser: The "experimental" Basic Web browser is completely unusable! It doesn't render anything properly and is impossible to navigate. On the Experimental page it says that it "Works best with web sites that are mostly text." And what sites would those be? None!

The text-to-speech feature seems the work -- the voice isn't very natural though and would get on my nerves quickly. But that's OK because I don't plan on using this feature.

IMHO, the benefits of electronic book reading and content management far outweigh the annoyances of the Kindle. You're not going to be browsing the Web or typing e-mails with this device.  The Kindle is for reading books! If you expect more than that you should consider purchasing a different device.

UPDATE (3/2/09): Here's a thorough review of reading newspapers on the Kindle 2: Reading the New York Times on Kindle 2.