Publication:Navy Engineering Bulletin March 2006/Objective Quality Evidence
Following the incident aboard HMAS WESTRALIA in 1994 the RAN has changed the way in which it manages Objective Quality Evidence (OQE). During this time the RAN has given the Systems Program Offices (SPO) more responsibility for implementing the RAN OQE policy.
This paper will discuss how the RAN uses OQE to manage risk with consideration to the burden it places on the RAN. The paper will focus on OQE requirements for external maintenance. It will also make a comparison to the quality assurance systems in the Royal Thai Navy, the Royal Malaysian Navy and civilian industry for consideration.
Contents |
AIM
The aim of this paper is to analyse the current RAN OQE policy for external maintenance and its implementation, so as to determine whether the RAN is creating an administrative burden for itself.
BACKGROUND
Reasons for OQE
OQE is part of the RAN's current quality assurance system. It forms part of the technical regulation policy[1]
stemmed from the corrective measures taken in response to the WESTRALIA fire incident. To a large extent WESTRALIA's fire was believed to be caused as a result of failure of an inappropriate fuel hose.[2]
The hose was fitted by a contractor without being subjected to the configuration change process.
It is now a requirement for all contractors to provide a signed certificate (OQE) to state that the part or service that has been provided is fit for service. Generally certification for contracted labour covers the goods or service provided, the materials used, how the work was done and the qualifications of the personnel who worked on the maintenance contract. In doing this the principles of Technical Regulation are followed.
The question of how much OQE is necessary
It is accepted that increased levels of OQE are an appropriate corrective measure for the WESTRALIA incident, but the question remains about much OQE is necessary. Currently, contractors must provide the following OQE without being specifically instructed by a project manager:[3]
- Pressure vessels, flexible hoses /pipes, relief valves and safety valves, require a pressure test certificate.
- High points, lift points, lifting appliances/equipment, guardrails, safety nets and any other load carrying items, equipments or structures that shall be tested under naval, Australian or Classification Society Standards/ Regulations, require a static load test certificate.
- Temperature, pressure and speed sensors and meters, governors, over speed trips and all other fixed safety devices, require a calibration certificate.
- Electrical equipment, components and cables, require applicable certification of cable continuity, insulation resistance and/or phase rotation.
- Equipment installations require confirmation that that the equipment has been inspected to ensure it is installed correctly and safely in accordance with good engineering practice.
Maritime Capability Support administers and manages the Ship Repair Panel Agreement for Major Surface Ships. In their opinion it is impossible to label tasks as not requiring certification,[4]
because in reality there is always some risk of a hazard being created as a result of contractors failing to perform their work in accordance with the specifications. Following this philosophy the COLLINS Class build program demanded certification for all stores received. This was considered over-stringent, causing an increase on the cost of materials and ultimately a significant increase in the logistic cost of ownership of the COLLINS Class. In 2003 there were 1900 items, valued at $13M, quarantined due to OQE deficiencies. Due to budget constraints, caused by the deficiencies, materials in stock had to be reduced. This severely impacted submarine capability and availability.[5]
As an alternative to standard requirements for OQE Hazard Risk Assessments (HRAs) could be used to determine if certification is necessary. Under the technical regulation policy, maintenance can only be contracted to external companies that meet the requirements of being an Authorised Engineering Organisation (AEO). Such requirements include having a Quality Management System (QMS), and a system for the Delegation of Engineering Authority.[6]
So there is already regulation before contracted work begins. We are creating an administrative burden by requiring a system check after contracted work has been done well.
The confusion of the delegation of responsibility for determining OQE requirements
If there is uncertainty in regards to how much OQE should be required, then we need to turn to the delegated authority for advice on making that determination. However, there is also disagreement over who has that authority.
According to DI(N) LOG 63-5 the SPOs are responsible for ensuring that certification requirements for external maintenance projects are properly defined. They must be documented in the work contract,[7]
while Engineering officers on board RAN ships are responsible for ensuring that any certification documentation is provided to the ship, certifying that work of a safety or mission critical nature is correct. Engineering officers are also responsible for retaining all certification documentation in a technical data management system.[8]
There have been cases of SRCOs, Ships' Staff (SS) and contractors rejecting Government Furnished Materiel (GFM) stores that had been received without accompanying OQE, despite the stores having not had OQE requirements specified by the SPO.[9]
The engineering officers rejecting the stores believed they were within their right to do so according to their interpretation of the technical regulation policy. They believed that they could demand OQE for stores to assist in their responsibility to ensure technical integrity and safety of their systems. This was particulary important when they were accepting store for tasks was outside their level of competence. The SPOs believed the engineering officers were acting outside their delegated authority.
CURRENT PROBLEMS WITH OQE
Confusion as to whether or not engineering officers can refuse stores if they are unsatisfied with the level of OQE set for them has caused financial and administrative burdens. Rejected underwater valves supplied to HMAS ARUNTA required quality assurance testing at a cost of around $60 000 due to a lack of OQE, despite the SPO not having specified that OQE was required.[10]
Similar problems have lead to some doubt that SPOs can list OQE requirements for all items.[11]
Under the current policy laid down in DI (N) LOG 63-5 the SPO's are responsible for determining the levels of OQE required when conducting external maintenance. The contractors carrying out the external maintenance are not submitting more OQE to the SPO's than they had previously done before the WESTRALIA incident.[12]
The documentation is examined by SPO staff in greater detail on items that would previously been deemed to have a lower HRI. In the case of the FFGSPO this burden has necessitated the contracting of additional staff for individual IMAV's to determine the level of OQE required, ensure that the documentation is received as required to ensure the documentation is recorded and retained in an appropriate manner I.e. in AMPS. Twenty thousand dollars was recently spent dealing with OQE for HMAS ADELAIDE.
It was estimated that $100,000 p.a. is being spent by the FFGSPO hiring personnel to deal with the burden OQE has placed on the SPO's.[13]
This has also had a direct bearing on ships as they cannot sail unless the OQE is entered into AMPS and the job completed. An example of this is the discrepancy between HRMI data in the ANZAC SPO HRMI team and the data on AMPS over Halon Hoses[14]
In this case the data entered into AMPS was not adequate for a HRMI however the documentation provided and work done was adequate. This outlay by the SPO's has alleviated some of the administrative burden of managing OQE from SS, as such the SPO and the ship have an electronic record of the same information.
In many cases it has been perceived that the requirements for OQE have prompted a culture of risk aversion in the technical community of the Navy. Even brand new parts that are obviously without flaw have been doubted if they are received without OQE.[15]
In one instance a propeller was found to be faulty. Its replacement, delivered as GFM was still in its original packaging. Because the OQE for the propeller was out of date the Engineer proceeded to hire a company to provide new OQE.[16]
In this instance the assessment made was to carry out a visual inspection, considering the data on the blade stamping and recommend its use. This is an example of an engineer demanding OQE to alleviate their responsibility with respect to risk management and to provide greater assurance outside of their self assessed competence.
POMT ERIC CROMB ON THE MCR CONSOLE ON HMAS ARUNTA. PICTURE: LSPH DAMIAN PAWLENKO
Project directors have difficulty managing and storing all the certification documentation for audit trails. They also have difficulty providing it to the Directorate of Naval Certification in the timeframe it is required.[17]
The documentation requirements are laid down in the contracts written for the maintenance. Problems have arisen when there has been inconsistency between the requirements of Technical Regulation and those laid down in contracts. In the Case of HMAS STUART it almost prevented the ship from undocking. The rectification of this issue took a great deal of time, effort and money that could have otherwise been avoided. The MEO, upon seeing the lack of an audit trail for the underwater valves refused to accept the work, preventing the ship from undocking.[18]
There is also a lack of consistency across the navy, with each SPO and ship managing OQE differently.
A POSSIBLE SOLUTION - MARITIME SYSTEMS DIVISION'S OQE POLICY
Maritime Systems Division (MSD) have recently drafted a policy intended to target some of the fundamental issues with the OQE problem. It clarifies the delegation of responsibility for the determination of OQE requirements, stating that both the need and form of certification needs to be determined through risk assessment by the SPO.[19]
This clarification alleviates the administrative burden resulting from contractual conflict.
MSD's policy also states that in most cases appropriate OQE for GFM would be evidence that the item is registered with a NATO Stock Number (NSN). Unique OQE required over and above this must then be defined in the configuration record,[20]
but this would only occur when the intended use for the GFM goes beyond what was originally designed for. This proposal alleviates some of the administrative burden resulting from the sheer volume of parts requiring some form of OQE.
HOW OTHER ORGANISATIONS HAVE DEALT WITH QUALITY ASSURANCE AND OQE
The Royal Thai Navy (RTN) attempts to get its external maintenance certified to approved standards where they exist. These standards are laid down in technical manuals and other documents. This is not always possible as the personnel at Authorised Engineering Organisations are not necessarily certified as competent when undertaking minor maintenance. The RTN does not have its own system for certifying the competency of its workers. It does however use the Personnel Qualification Standards from the United States Navy (USN) for a number of personnel at some organisations.
The 3 major naval dockyards have a Quality Control Division to develop and implement Quality Control Plans for the work undertaken and items used. The RTN has no formal risk assessment system however experienced senior engineers meet to discuss issues as they arise. There is no emphasis on the importance of High Risk Maintenance Items (HRMI) however there are a few items that require additional documentation. One such set of items are the high pressure components on steam ships. These require documentation such as weld forms, Hydrostatic test forms and technical work documents. Due to the numerous classes with few ships of each type there are many different standards throughout the RTN.
The RTN Supply Division is responsible for the tender and purchase of items used during external maintenance. When doing so they follow the requirements laid down for use in the Royal Thai Dockyard and Technical Manuals. When assessing a company for use as a Certified Company the RTN Supply Division may call upon specialists from the dockyard.
For routine defects the repair requirement form is filled out for each system; hull, mechanical, electrical and weapons electrical. Then the report is submitted to the squadron and technical repair unit responsible for the job, specifying the capability of the repair unit needed. A single form is used for this however observations may require measurements taken, further detail may be found in Technical Manuals.
Currently there are no standardisation factors set by RMN in governing the sets of documentation required for maintenance; i.e. each dockyard or contractor produces their own documentation as a proof of work being done. Distribution of equivalent documents in RMN will be divided amongst the ship as a record, FRA for repair history during ship's whole life, Engineering Dept in Naval HQ for budget overseeing and FMD's for the contractors work.
Each HQ and FMD maintains the certification documents of all the registered contractors. Any companies that adopt ISO requirements in their work culture have a higher chance to receive or awarded with any maintenance work from the government. Every contractors should produce to the RMN, sorts of documents to prove that work is correctly done, with appropriate tools, trained personnel, test and testing equipments, witnessed by RMN or ship's staff and finally the acceptance certificate.
Although there are no specified guidelines, every RMN engineer knows there are requirements for proper certification documentations in every maintenance task, whether onboard ship or shore establishment. RMN is still assessing other navies approach in terms of repair management as a benchmark.
RMN apply basic certification documentation requirements for maintenance which are managed by relevant engineering authorities in the fleet. However, they are moving towards improving the system, and are currently assessing other navies' approach in managing systematic and proper documentation in maintenance. Hazard Risk Assessment will soon be introduced in the RMN, together with proper policy to support its application for better handling of risk management in the fleet.
Civilian Quality Assurance
In the civilian sector there is a wide range of OQE demanded or in many cases not demanded. Walther Construction a subcontractor providing conveyors at a mining pit in the hunter valley, provided OQE for work in the form of a commissioning sheet.
This is signed by the immediate supervisor of the personnel carrying out the contracted work, the sub-contractors engineer and the mines engineer.[21]
This is in contrast to manufacturers such as Bradken, a company providing digging teeth on mining equipment operating in the hunter valley[22]
who will test and certify their products if requested but usually only test the initial batch and a random sampling thereafter.
CONCLUSION
OQE certification for stores and labour contracted to external companies is a necessary component of quality assurance, as part of the navy's technical regulation.
OQE certification requirements and management of documentation have been a cause for administrative burden, frustration and an increased workload for departments in the Navy. There are many valid complaints regarding OQE administration.
The foreign navies and the civilian industry investigated do not have formalised quality assurance processes like the RAN. However, in the case of the foreign navies they are in the process of establishing a more formalised system. Civilian industries do not find it a burden to provide OQE and in many high risk areas such as mining do so when dealing with HRMI. However the quality assurance procedures undertaken ISO9000 companies should by their nature make it simple to produce such documentary evidence if required.
RECOMMENDATIONS
A significant portion of the trouble caused with OQE certification requirements arises out of the sheer magnitude of stores that require certification documentation, and the management of the paperwork that follows. MSD's policy on OQE provides a solution for a large component of this problem, suggesting that for most cases the OQE required need not be more than the NSN.
Another significant cause for the administrative burden of OQE arises out of the confusion of delegated authority to determine OQE requirements, with stores getting refused based on risk assessments for decisions that were made by personnel not authorised to do so. MSD's policy clarifies the delegation of responsibilities for determining OQE requirements to the SPO. Meanwhile, it is the responsibility of the Engineering officers onboard ships to first recommend what levels of OQE they believed is required, and then to ensure that what is agreed upon is received with the stores and / or labour.
In order to implement these corrections to the running of the OQE system in the Fleet it is necessary to educate officers and departments of their roles and responsibility. An education system is suggested for Engineering officers at some point during their shore time before they start their posting as Deputy Engineer.
Furthermore, it is recommended that OQE requirements for services provided by contractors to be reduced. Because the RAN already requires that companies have a quality assurance system and be registered as AEOs. It seems over-stringent to quality check their output to the extent we currently do. Periodic quality audits are nonetheless endorsed and recommended. In the same way there is a stores system for GFM reducing the requirement for OQE, we suggest there could be a services system for AEOs that also limits the requirements for detailed OQE.
While it is necessary to lighten the burden of OQE placed on ships staff and the SPO's, adequate resources need to be allocated to ensure that our OQE systems operate smoothly. If properly resourced, the SPO's can better ensure that the AMPS data is correct. Additionally ther is still a need to ensure that the specifications laid down in the external maintenance contracts are correct.
- G. De Visser
Sub Lieutenant, RAN WEEAC
- K.B. Brown
Sub Lieutenant, RAN MEAC
- Wasanasit
Commander, RTN MEAC
- Rosnizam
Lieutenant Commander, RMN MEAC
22 October 2005
Footnotes
- ↑ ABR 6492 Volume 1, Chapter 6 Technical Regulations Manual
- ↑ Executive Report on the WESTRALIA Board of Inquiry.
- ↑ Ships' Repair Contracting Panel Agreement of Logistics Management Group's General Conditions of Contract with respect to Objective Evidence.
- ↑ E-mail CMDR Worstencroft/SBLT De Visser 1418 h, 05 Oct 05
- ↑ CAPT Idrus Minute to the Director General Naval Certification Safety and Acceptance: Submarine Objective Quality Evidence Rationalisation.
- ↑ ABR 6492 Volume 2, Sect 5, Chap 2, p 3, para 2.14 - Technical Regulations Manual
- ↑ DI(N) LOG 63-5 p 1, para 7.a. _ Quality Assurance of Procured Supplies and Services
- ↑ DI(N) LOG 63-5 p 2, para 9.d,e. _ Quality Assurance of Procured Supplies and Services
- ↑ Email Mr. William Coombes/SBLT De Visser 1517 h, 05 Oct 05
- ↑ Email Mr. William Coombes/SBLT De Visser 1510 h, 10 Oct 05
- ↑ Email CMDR Worstencroft/SBLT De Visser 1328 h, 10 Oct 05
- ↑ Telecon SBLT Brown/LEUT Brown of 22 Oct 05
- ↑ Telecon SBLT Brown/LCDR Anderson of 22 Oct 05
- ↑ Email CMDR Coyle/SBLT De Visser 1304 h, 19 Oct 05.
- ↑ E-mail LCDR Williams/LCDRCrawford 1705 h, 30 Jul 05
- ↑ Rolls Royce Service Report
- ↑ Email Mr. Michael Mechianos/SBLT De Visser 1437 h, 28 Sep 05
- ↑ Email LCDR Williams/LEUT McCracken 1529 h, 28 Jul 05
- ↑ MSD OQE Policy Ver 1.0 - 20Sep05 para 22.
- ↑ MSD OQE Policy Ver 1.0 - 20Sep05 para 26.
- ↑ Telecon SBLT Brown/Mr. Scott Wyborn of 17 Oct 05
- ↑ Telecon SBLT Brown/Mr. Simon Wyborn of 17 Oct 05
BIBLIOGRAPHY
- ABR 6492 Technical Regulations Manual, Volumes 1 and 2
- DI(N) LOG 63-5 Quality Assurance of Procured Services and Supplies
- DI(N) LOG 47-3 Regulation of Technical Integrity of ADF Maritime Materiel
- DI(G) LOG 08-15 Regulation of Technical Integrity of ADF Materiel
- TM181 051-04 High Risk Management Items
- TM180 12-04 Quality Assurance of Procured Supplies and Services
- Maritime Systems Division Policy: Identification of Objective Quality Evidence (OQE) Versions 1 and 1.0
- CAPT Idrus' Minute to the Director General Naval Certification Safety and Acceptance: Submarine Objective Quality Evidence Rationalisation.
