User:IveGoneAway/sandbox/AC 20-189

Management of Open Problem Reports (OPRs)

EASA/FAA Publication
Abbreviation	AMC 20-189 / AC 20-189 with AC 00-71
Status	Active
First published	2020/2022
Domain	Airworthiness certification
Website	AMC 20-189 AC 20-189 AC 00-71

AC 20-189 and AMC 20-189, both entitled Management of Open Problem Reports (OPRs), are harmonized advisory publications of the Federal Aviation Administration (FAA) and European Union Aviation Safety Agency (EASA), respectively.

These applicant advisories cover the importance of reporting and taking care of aircraft design problems known during certification or discovered any time after certification. They suggest that establishing effective problem reporting and correction methods early in development improves confidence in problem management during and after certification.

Applicability

These AC/AMCs describe acceptable means for assisting both design approval applicants and holders of design approvals as they demonstrate compliance with airworthiness regulations for the management of open problem reports. Their intent is to provide consistent problem report guidance across the domains of systems (ARP4754), software (DO-178/ED-12), and hardware (DO-254/ED-80).

Structure

The two publications have nearly identical content; however, the EASA AMC has an appended section entitled "Guidance Material", regarding problem report lifecycle management, open problem report classification, and additional guidance related to one of the problem report classifications. The FAA did not include this appended material in AC 20-189 but has published it separately in AC 00-71, Best Practices for Management of Open Problem Reports (OPRs). For some background to the distinction between "Guidance Material" and "Best Practices", refer to the RTCA DO-178C clarification between guidance and guidelines.

Overview

Businesses developing equipment and system designs for transport aircraft normally record evidence to assure the FAA and EASA that their designs are safe enough to fly in public airspace. However, whenever a mistake or hazard is discovered or suspected in such designs, a problem report (PR) is initiating and managed until the problem is completely fixed as needed and the problem report is "closed". Any time between initiating a problem report and closing it, the problem report is usually said to be "open". Ideally, all problems are fixed and closed before the FAA or EASA certifies a design, but these authorities give specific considerations to problem reports that intentionally remain open at the time of certification, designating them as Open Problem Reports (OPR).

By publishing AC/AMC 20-189 to communicate how the FAA and EASA want to see open problem reports handled during and after certification, they hope to inform developers how to best prepare and manage their problem report systems for successful and effective certification. Evidence of applicants managing effective problem report systems during development provides a degree of confidence to the authorities that applicants can effectively manage OPRs during and after certification. In this manner, the AC/AMC 20-189 guidance for OPRs during and after certification can be accepted by applicants as guidance for their development life-cycle problem report systems.

Problems and reporting

Problem reporting is an important activity in the creation of safe equipment and systems for hazardous industries.  As a discipline, problem reporting conducts a life cycle of activities; beginning by recording the problem, hazard in the design of a product or system,

Open problem reports (OPR)

Broadly, problem reports are open from the moment they are started to the moment they are closed. More narrowly, OPR can refer to problem reports that are identified as open at the time of a major milestone or certification review.  In the narrow case, the intention is for the OPR to remain open past completion of the baselined review to allow work and certification effort to continue before the issues are closed. The FAA is particularly concerned that OPR at the time of certification are adequately managed so that they represent no unacceptable flight hazard.

History

In 1987, the International Organization for Standardization introduced problem reporting to quality system management for the development of new products, through the first edition of their standard ISO 9001, Model for quality assurance in design, development, production, installation, and servicing. From this origin, problem reporting entered aviation software safety considerations through DO-178B:

Background publications

• 1992: The RTCA defined airborne software certification objectives for problem reporting within DO-178B.
• 1998: The SAE established problem reporting and change control for aircraft system certification through publication of ARP4754, Guidelines For Development Of Civil Aircraft and Systems.
• 2001: The RTCA's Discussion Paper #9 (DP #9) "Assessment and Classification of Open Software Problems" published in DO-248B, Supporting Information for DO-178C and DO-278A, discussed how to proceed through software certification with "Known Software Problems" (soon to be called Open Problem Reports in following publications). The focus was clarification on how to report known problems in the Software Accomplishment Summary (SAS). The discussion cited DO-178B objectives and activities for normal life cycle problem reporting, suggesting that problem reporting in the SAS should be continuation of effective life cycle problem reporting.

• 2001: The Certification Authorities Software Team (CAST) published a 6-page position paper, CAST-7 Open Problem Report Management for Certification. With this position paper, CAST sought to

• clarify the role of manufacturers and suppliers in managing and communicating known problems at the time of certification and the use of the Software Accomplishment Summary as a means of this communication.
• add the term "defect" to the progress of Design Error ⇒ Software Defect ⇒ Software Fault ⇒ System Failure

Note: Defect is not picked up in the following problem reporting publications.

• propose standard classification levels, from "3" where the software is presumed defective because a requirement is wrong to "1" where there is no evidence of software defect, but some level of assurance of absence of error has been lost

Note: This numerical rank is reversed from later numerical type classifications.

• 2010: The FAA's Notice 8110.110 Chapter 2 "Software Problem Reporting" (three pages) provided guidance to applicants and certification authorities in management and oversight of software problem reporting down through all supplier levels, stating how software problem reports

• should be categorized according to potential safety or confidence impact into one of five categories (safety, functionality, performance, operation, or design assurance) and
• how they will be communicated from developers to certification authorities.

• 2010: In CM-SWCEH-002 Software Aspects of Certification, which is effectively EASA's version of FAA's original Mega Order 8110.49, EASA makes many recognizable problem reporting provisions, but its guidance is worded and structured quite differently from Notice 8110.110 and the following Order 8110.49 revision.

• 2011: FAA Mega Order 8110.49 Revision 1 directly incorporated the content of Notice 8110.110 Chapter 2 paragraph-by-paragraph.

• 2011: The RTCA's updated Discussion Paper #9 published in DO-248C defines a complete replacement of the prior DP #9 clarifications for problem reporting. The discussion's four pages are similar to but less developed than 8110.110. However, paper notably introduces the "5 types" classifications of OPR with more details on expectations of communicating the life cycle of each problem report. These types are assigned numbers 0-4, broadly following the classifications introduced in Notice 8110.110 above (Type 0 for safety impact, Type 1 for functional defects, Type 2 for non-functional documentation defects, Type 3 for design confidence issues, and Type 4 for issues having "no functional consequence").

• 2012: DO-178C clarified and reinforced applicant objectives for problem reporting.

• 2013: Citing Order 8110.49A and CM-SWCEH-002 her book Developing Safety-Critical Software, Leanna Rierson provided over 8 pages of recommendations for problem reporting, including the "5 types" classifications of DO-248C DP #9.

• 2017: The FAA removed all software developer guidance from Order 8110.49. Much of the removed guidance was then covered in greatly reduced form as "Best Practices" in AC 20-115D and AC 00-69. However, the problem reporting guidance was the only removed topic not covered by any new best practice advisory.

• 2018: Lion Air Flight 610 crashed on 28 October 2018, killing 189 passengers and crew. The root cause was determined to be an Open Problem Report that had not been effectively communicated to the FAA and aircraft operators.

Present publications

The present FAA/EASA guidelines for managing problem reporting are now found in these later advisories:

• 2020: EASA AMC 20-189 Management of Open Problem Reports (OPRs)
• 2022: FAA AC 20-189 Management of Open Problem Reports (OPRs)

• These AC/AMCs emphasize both the requirements for suppliers of compliance data and, when the applicant is not the supplier, the requirements for applicants to flow those requirements to suppliers.
• Problem Reports should have managed life cycles (i.e., recorded, classified, resolved, closed) with extent of assessment, disposition, and reporting influenced by classification according to 'Significant', 'Functional', 'Process', 'Life Cycle Data', or any 'other' tailored classification.

• 2022: FAA AC 00-71 Best Practices for Management Open Problem Reports

The classifications "Significant", "Functional", "Process", and "Life Cycle Data" are correlated to the prior "5 types" classifications of DO-248 DP #9.

"Advice"

4.2 States of PRs/OPRs

• Recorded – A problem that has been documented using the problem reporting process.
• Classified – A problem report that has been categorized in accordance with an established classification scheme.
• Resolved – A problem report that has been corrected or fully mitigated, for which resolution of the problem has been verified but not formally reviewed and confirmed.
• Closed – A resolved problem report that underwent a formal review and confirmation of an effective resolution of the problem.

4.3 Classification of PRs/OPRs

• Significant – assessed at the product, system, or equipment level, a PR that has an actual or potential effect on the product, system, or equipment function that may lead to a Catastrophic, Hazardous or Major failure condition, or may affect compliance with the operating rules.
• Functional – a PR that has an actual or a potential effect on a function at the product, system, or equipment level.
• Process – a PR that records a process non-compliance or deficiency that cannot result in a potential safety, nor a potential functional, effect.
• Life Cycle Data – a PR that is linked to a deficiency in a life cycle data item but not linked to a process non-compliance or process deficiency

References

• "EASA Publishes New AMCs on Development Assurance for Airborne Electronic Hardware and Management of Open Problem Reports". AEROSPACE TechReview. August 26, 2020. Retrieved 2023-05-23. AMC 20-189 provides guidance on management and classification of Open Problem Reports (OPR) at time of product approval or ETSO authorization. This material is intended to be used when showing compliance for and across airborne electronic hardware, software and system development.
  These documents have been developed in close coordination with FAA, in order to achieve harmonization in these domains. They result from four years of work in coordination with FAA, including a joint EASA/FAA public consultation phase, and with the support of the US and European industry associations ASD, GAMA, AIA according to EASA.

• "AC/AMC 20-189". DO-254, DO-178C & PACT Tools Glossary. Airworthiness Certification Services LLC. September 4, 2020. Retrieved 2023-05-23. Published July 2020, this document compliments current airborne software, hardware and systems guidance by covering the topic of "The Management of Open Problem Reports.

• "Managing Problems with AMC/AC 20-189 for DO-178C or DO-254 programs". Airworthiness Certification Services. Airworthiness Certification Services LLC. April 24, 2023.

• Umut Pisken (August 10, 2022). "AMC 20-189 Management of Open Problem Reports". Safety First. Retrieved 2023-05-23. In the document "DO-248C Supporting Information for DO-178C and DO-278A" there is a title "Assessment and Classification of Open Software Problems" [3]. Under this heading, there is also a warning that having a large number of open problem reports during the certification approval phase can pose a risk in terms of obtaining approval. The main reason for this is that the large number of open problem reports is an indication that the software is not mature enough and that the process assurance approach that should be applied while developing the software is not properly implemented[4]. An example chart that can be used to classify open problem reports is also given under the title.
  AMC 20-189 "The Management of Open Problem Reports (OPRs)" published by EASA on 29 July 2020 proposes a different classification from the classification in DO-248C. AMC 20-189 provides a common problem reporting management system for software and complex-electronic hardware. It is emphasized in the document that one of the main objectives of AMC 20-189 is to provide compatibility in different open problem report management methods currently recommended by different guidance documents for software and complex-electronic hardware.

• "How AMC/AC 20-189 Stops "Un-Accomplishment Summaries" in DO-178C or DO-254 programs". PATMOS Engineering Services, Inc. Retrieved 2023-05-23. Both DO-178C (airborne software) or DO-254 (airborne electronic hardware) standards allow for a listing of open PRs in the accomplishment summary document. However, many applicants have abused this and create long lists of unresolved and uncategorized PRs in the Software/Hardware Accomplishment Summary documents. This makes it difficult for applicants to show compliance and regulators to find compliance. The result, as one would guess, is equipment and aircraft level functional issues and airworthiness directives.
  The New AMC/AC 20-189 provides guidance on a means of compliance when applicants have unresolved (i.e., open) PRs at the end of a TC, STC or TSO project. The reason for this is to provide a consistent expectation related to the communication, review and assessment of open PRs (OPRs) by all possible stakeholders who are integrating the software or AEH into their aircraft programs.

• Prepared by the Democratic Staff of the House Committee on Transportation and Infrastructure (March 2020). "The Boeing 737 MAX Aircraft: Costs, Consequences, and Lessons from its Design, Development, and Certification" (PDF). Preliminary Investigative Findings. Retrieved 2023-05-23. In August 2017, five months after the 737 MAX was certified by the FAA and three months after it entered revenue service, Boeing issued a problem report to its supplier complaining that the 737 MAX's AOA Disagree alert was tied to an optional AOA Indicator display and therefore was not functioning on the vast majority of the 737 MAX fleet worldwide.
  40 ▪ Rather than immediately informing the FAA and Boeing customers about this issue, and advising Boeing to fix the problem via a software update as soon as possible, a Boeing AR consented to Boeing's plan to postpone the software update until 2020, three years later, so it could be done in conjunction with the rollout of Boeing's planned 737 MAX-10 aircraft.
  41 ▪ Although Boeing prepared a "Fleet Team Digest" to inform its customers about the inoperable AOA Disagree alert, the company never sent it, keeping its customers in the dark about the inoperable alert.
  42 ▪ Boeing provided Lion Air a Flight Crew Operations Manual (FCOM) on August 16, 2018, one year after learning that the AOA Disagree alert was not functioning on most 737 MAX aircraft, highlighting the operation of the AOA Disagree alert. Boeing failed to indicate that it knew the AOA Disagree alert on the Lion Air 737 MAX aircraft was not operational.

External links

• AMC 20-189, The Management of Open Problem Reports (OPRs), EASA.
• AC 20-189, Management of Open Problem Reports (OPRs), FAA.
• AC 00-71, Best Practices for Management of Open Problem Reports (OPRs), FAA.

Category:Avionics Category:Safety Category:Software requirements Category:Computer standards