Agile development in a FDA regulated setting

I ran across an interesting Agile v. FDA discussion the other day. For those that are not familiar with what a FDA regulated product means, I'll give a brief overview.

In order to market and sell a medical device in this country -- OUS (outside US) medical device regulations are different -- you must have FDA approval. Most of the time, this involves a FDA Premarket Notification 510(k) submission. Your company must be registered with the FDA and is subject to periodic on-site visits by FDA inspectors to audit your quality system records. There are two important points here:

  1. Getting approval: In order to sell your device you not only have to prove safety and effectiveness, but you also have to demonstrate that the design and development of the device -- including the software -- follow Quality System Regulations. The Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices details these requirements.
  2. Keeping approval: After you receive 510(k) approval the FDA can pull your device off the market (the dreaded "recall") at any time due to complaints or unsatisfactory audit results.

What this means is that your on-going software development process must adhere to a well defined quality system process. As noted in the discussion, the FDA guidance does not dictate that a particular process must be used. The quality system process itself is designed by you -- actually, your entire company -- and simply needs to ensure that you are designing and building a quality product. The difficult part is that you have to be able to prove to the outside world that you have actually followed that process.

I've spent just about my entire career developing software for devices that were under FDA regulatory control. In the old days (pre ~1996) the FDA did not have a good concept of the software development process as part of it's quality system regulations and inspectors did not usually scrutinize software design and development. Nowadays, the FDA has a much clearer understanding of the software development process. It's that understanding that is the one of the central issues with respect to adopting an Agile development process for software in medical devices.

Let's first look at the FDA Good Manufacturing Practice (GMP - Quality System Regulation) requirements. It's Subpart C--Design Control (§ 820.30) that's of primary interest. Here's the outline:

  1. General
  2. Design and development planning
  3. Design input (specifications)
  4. Design output (coding)
  5. Design review
  6. Design verification (Was the product built right?)
  7. Design validation (Was the right product built?)
  8. Design transfer
  9. Design changes
  10. Design history file

The critical issue for the software development process is that each of the items 3-7 require a formal review and approval step. This is the reason why most companies that develop medical device software have chosen to use a quality system process that follows the Waterfall model for their development.

Waterfall

This sequential approach is a natural fit for allowing you to review and document each step in the process.

Now let's look at the Agile process. One of the better descriptions of the Agile development process is 'The New Methodology' by Martin Fowler. There a number of flavors of Agile (SCRUM, Extreme Programming (XP), etc.) that all try to encompass the Agile Manifesto. One advantage that the Agile process has over the Waterfall approach is it's ability to adapt to the unpredictability of requirements and changing customer needs. This is handled through the use of Iterations. From the Martin Fowler paper:

The key to iterative development is to frequently produce working versions of the final system that have a subset of the required features. These working systems are short on functionality, but should otherwise be faithful to the demands of the final system. They should be fully integrated and as carefully tested as a final delivery.

The purpose of the iterative development is to facilitate requirements changes between each iteration. Agile Methodologies in a Validated Setting by Frank Jacquette proposes some steps to accomplish the use of iterative development in a FDA regulated environment.

Let's assume that an Agile development process would be able to produce higher quality medical device software and that because of the customer focus of this process the resultant product would better meet market needs. Even with these assumptions, I think there are two major issues that need to be addressed:

  1. I just don't see how a cost-effective quality system process implementation can be accomplished. Even if (and it's a big if) the actual overall software development time was shorter, the extra costs incurred by the additional process controls, documentation and testing required for each iteration would far exceed those savings.
  2. The last point Mr. Jacquette makes is the other issue:

    Take the time to explain agile methodologies to your regulatory specialists and work hard to gain their understanding and agreement, because the burden of proof will be on them and you if an FDA auditor decides to take a peek under the hood.

    This seems like a huge risk to me.

The first question becomes: Is the possibility of an improved product (features and quality) worth the additional cost? I suppose if you had a development team with extensive Agile experience you could make the argument that it would be worth it. If not, I think the ROI (return on investment) analysis would be a difficult one to make.

The second question is a big unknown, which is why I think the risk is high. My experience with FDA auditors is generally good. They are professionals that are focused on getting a very specific job done. Since their interest is the entire quality system, the typical audit is a whirlwind affair as it is. The amount of time spent on design control (auditing the Design History File) is usually minimal. Even if you had received 510(k) approval with an Agile design control process, having to take the time to explain to an on-site FDA auditor (who in all likelihood has never seen your 510(k)) a methodology they probably have never even heard of is reason to worry.

Conclusion:

It seems to me that Agile methodologies have a long way to go before we see them commonly used in medical device software development. I've searched around and have found nothing to make me think that there is even a trend in this direction. Maybe it's that Agile processes are just too new. They seem popular as a presentation topic (I've been to several), but I wonder how prevalent Agile is even in mainstream software development?

If you are (or have ever been) part of an Agile development team for a FDA regulated product I'd love to here about your experiences and how you were able to resolve the types of issues presented here.

Thanks!

18 Responses to “Agile development in a FDA regulated setting”

  1. […] contacted Frank Jacquette regarding my previous port on this subject (Agile development in a FDA regulated setting). His experience using Agile methodologies for pure software medical device projects does not […]

  2. […] failure’, but may be close). I think the FDA “rule book” (as I briefly describe here) is a full product life cycle quality system that generally meets its intended purpose. It […]

  3. […] a related note, check out Agile development in a FDA regulated setting. Published Sep 04 2007, 05:49 AM by Jeremy D. […]

  4. […] On a related note, check out Agile development in a FDA regulated setting. […]

  5. […] Agile development in a FDA regulated setting: this post explains why agile methodologies have a long way to go before we see them commonly used in regulated environments, in this case, FDA. […]

  6. CK says:

    Very informative and thanks for the post. I was hoping you could help me figure out how a small company can do FDA approved sfw development without setting up all the processes in place. I was hoping that there are companies one could outsource the sfw work to. Companies that understand the FDA process and can design, implement, deliver and maintain the software for a device maker. Is there any one out there like this?

  7. […] not so sure about that. As I’ve pointed out before, because of the validation requirements those guidelines are much better suited for the Waterfall […]

  8. Chip Weller says:

    The FDA documentation is written as if a waterfall process is used. It does claim in various documents that a waterfall process is not required. For example http://www.fda.gov/cdrh/comp/guidance/938.html#_Toc517237956.

    We typically use a waterfall approach for requirements (although there are changes expected, often TBDs remaining, at the approval) and for the software architecture. Then we use an iterative approach on the detailed design, coding, and verification steps. Validation is done at the end. This approach is certainly not agile. Our concern in using Agile is justifying to the FDA the lack of the design input stage gate. While I think this could be done, our typical clients are somewhat regulatory risk adverse.

  9. Ilja Preuss says:

    “The purpose of the iterative development is to facilitate requirements changes between each iteration.”

    That is one of the purposes, yes. Another is to get earlier feedback on whether what we are building actually works the way it is supposed to work, and to build that learning into future iterations. Another is to more reliably measure progress, in terms of Running Tested Features.

    “I just don’t see how a cost-effective quality system process implementation can be accomplished.”

    Anecdotal Evidence seems to suggest that an Agile approach actually is more cost effective than a waterfall approach. I think the most important reason is that the earlier we get feedback – for example from a failing test – the less costly it is to incorporate that feedback into the product.

    “If you are (or have ever been) part of an Agile development team for a FDA regulated product I’d love to here about your experiences and how you were able to resolve the types of issues presented here.”

    I suggest you do some research on PatientKeeper.

  10. Clibiaanelt says:

    I wonder if web industry affected by crisis as well? and to what extend? Will the admins continue this web?

  11. Vince Delmonte says:

    My friend on Facebook shared this link with me and I’m not dissapointed at all that I came here.

  12. Hitchhiker says:

    This is an interesting subject for me as I am charged with taking an agile software organisation into medical device land. Agile techniques have been remarkably successful in the non-medical world for this organisation who develop instrumentation software. Does anyone know of any specific FDA comments on the use of agile methodologies?

  13. Tom Gradel says:

    The best we have been able to do is prototype(s) while developing the requirements and architecture, followed by design, coding, and testing. Feedback from prototypes can result in evolving requirements and architecture. However, once the the SRS goes through the ‘final’ design review and is put under stricter ECN control, agility stops.

    I think more research needs to be done to access risk versus software methodology outside of the FDA arena. If Agile became the predominant methodology used to develop control systems for cars, for example, and we could measure pre- and post-Agile risk in that scenario, we would be better able to change the FDA requirements. These changes won’t be made on the basis of efficiency or cost.

  14. […] been three years since I wrote Agile development in a FDA regulated setting.  I’ll be interested to see if the application of “agile, high assurance […]

  15. […] Agile development in a FDA regulated setting […]

  16. […] device manufacturers should fear Agile. I do, however, see some stipulations that need to be made. Here is a rather dated article on the subject (from 2007) : Agile Development in an FDA Regulated […]

  17. Tushar says:

    Hi, thank you for this post I agree with you that In order to sell your device you not only have to prove safety and effectiveness, but you also have to demonstrate that the design and development of the device — including the software — follow Quality System Regulations

Leave a Reply