Introduction

A software testing framework is a fundamental pillar in any software development organization. In general, it enables a structured and organized approach to testing software applications.

Software testing plays a crucial role in ensuring the quality and reliability of software systems. It is never a goal in itself and is always motivated by a quality need. For example, the quality need ‘functionality’ will lead us to do functional testing.

To achieve effective and efficient testing processes, organizations often rely on industry-recognized standards and frameworks. ISO 29119, an international standard for software testing, provides guidelines and best practices for planning, executing, and managing software testing activities. A software testing framework provides:

1. Consistency and Standardization: ISO 29119 offers a standardized approach to software testing, enabling organizations to establish consistent testing processes across projects, teams, and departments. By adhering to this framework, the organization can ensure that testing activities are performed in a consistent manner, reducing variability and improving the overall quality of the software being developed. Consistency in testing practices also facilitates collaboration, knowledge sharing, and smoother transitions between team members.
2. Improved Test Planning and Documentation: ISO 29119 emphasizes the importance of thorough test planning and documentation. The standard provides detailed guidance on creating test plans, test designs, and test cases, ensuring that testing activities are well-documented and aligned with project objectives. With a standardized approach, organizations can systematically identify and address risks, dependencies, and potential issues in the software being tested. Comprehensive documentation also aids in traceability, making it easier to track the progress of testing activities and assess their effectiveness.
3. Enhanced Test Coverage and Risk Mitigation: ISO 29119 promotes a holistic approach to testing by emphasizing the need for comprehensive test coverage. The framework encourages organizations to identify and test various dimensions of quality, including functional, non-functional, and structural aspects. By following this approach, software testers can verify that the system meets the intended requirements and functions as expected under different scenarios. Additionally, ISO 29119 emphasizes risk-based testing, enabling organizations to prioritize test activities based on the potential impact and likelihood of risks, thus optimizing testing efforts.
4. Increased Stakeholder Confidence: Implementing a software testing framework based on ISO 29119 demonstrates a commitment to quality and best practices. It provides assurance to stakeholders, including clients, customers, and regulatory bodies, that the software has undergone a rigorous testing process. This, in turn, enhances stakeholder confidence and strengthens the organization’s reputation for delivering reliable and high-quality software products.
5. Compliance: Our organization is required to comply to a number of regulations. A software testing framework makes it easier to detect we are not fully compliant, but also makes it easier to fix this. By adopting a testing framework based on ISO 29119, organizations can demonstrate compliance with these regulations, making it easier to meet regulatory requirements and participate in projects where adherence to specific standards is a prerequisite.

Iso 29119 Framework or Euroclear custom-made framework?

In general, the framework we will propose in this document aligns closely with the ISO standard 29119, encompassing about 99.95% of its prescribed guidelines. However, Euroclear, being a distinct entity with its own unique priorities, sought to accentuate certain aspects that hold particular significance to Euroclear. In these specific areas where deviations from the standard are observed, Euroclear consciously made the decision to exclude them due to the presence of other well-established processes that adequately address those particular domains. By doing so, Euroclear aims to streamline its approach, ensuring that resources are efficiently allocated to areas where its specific requirements are best addressed, while still adhering to the fundamental principles outlined by the ISO standard 29119.

What does Euroclear consider to be part of the software testing framework?

What are the principles Euroclear uses for the framework?

• Software testing is not a goal in itself. It is a means to establish the ‘quality’ of a product. Hence, the motivation and software testing (framework) models are linked.
• Motivation comes from influencers, both internal and external to the organization.
• Influences can be quality-driven, requirements, or remediations to something that is not optimal or can be part of a continuous improvement effort.
• Every influence is connected to a specific test type and its corresponding sub-process.
• Software testing covers the full life cycle of a product, from the idea to the decommissioning of the product.
• A way to logically group and organize all software testing topics in the organization.
• Software testing covers organizational test processes, test management processes, and (static & ) dynamic test processes.
• The detailed framework must be seen as a checklist.
• Additional frameworks/models are required to detail the software testing framework.

What does Euroclear consider NOT to be part of the software testing framework but strongly related to it?

The software testing framework’s focus is on what is important for software testing, but it is not enough to fully understand and evaluate the impact on the organization, therefore it is important to understand the dependencies the software testing framework has to embed it in the Euroclear organization.

The software testing framework will be built around many frameworks/processes related to software testing, which we call the ‘Test Practices.’

The dependencies of the software testing framework will cover the other (non-testing) aspects of these test practices: the life cycle relevant to the method, qualitative/quantitative techniques (frameworks and/or reference models) to describe all aspects of how the method fits in the organization, who is involved, what to do, what products to use and produce, how to gradually introduce the method in the organization, how to assess the method is correctly applied and objectively measure this, what quality needs it covers and how it fits in the quality management (system) framework of the organization, and the method also contains its paradigm: referenced material, templates, guidelines, checklists, examples, tooling, etc.

What is the high-level structure of the software testing framework?

If we link all methods to a “software testing framework,” we would create a flat structure, meaning that we end up with a library with many method descriptions, without structure, without organization. For that reason, Euroclear follows the layering from ISO 29119.

It is important to notice that a method often will be part of several ‘nodes.’ E.g. a testing process (method) impacts the organizational test processes, the test management processes, and the test sub-processes.

Before moving on to the next chapter, it is important to note that we will provide an example of what it means to use a software testing framework in the description of test practices. It is NOT the intention for the example to be complete. It merely intends to provide a non-theoretical view (more pragmatically).

Example: Disaster recovery testing.

Main sections explained

Test Motivation Processes

Software testing is a collection of essential processes in determining the extent to which a test item meets the required quality criteria. These quality criteria can be identified and categorized based on the specific characteristics they aim to assess, which can be mapped to various test types. The need for software to fulfill certain quality requirements can arise from multiple sources, both within and outside the Euroclear organization. These sources, which we refer to as influencers, can include internal factors such as organizational policies, procedures, and standards, as well as external factors like regulatory compliance and customer demands. Understanding and identifying these influencers is crucial in ensuring that software testing effectively addresses the diverse quality needs of the organization.

Quality needs often (always) come from a quality framework as described in ISO 25010.

Example: Disaster recovery testing is a test type that covers the quality need (characteristics) ‘reliability – recoverability.’ Definition: The purpose of disaster/recovery testing is to determine if, in the event of failure, the operation of the test item can be transferred to a different operating site and whether it can be transferred back again once
the failure has been resolved. Disaster/recovery testing uses a model of the test item (typically a disaster recovery plan) that specifies its disaster recovery requirements, including any required disaster recovery design standards to which the test item must conform. The test item in disaster recovery testing may be an entire operational system with associated facilities, personnel, and procedures.
Disaster recovery testing may cover factors such as procedures to be carried out by operational staff, relocation of data, software, personnel, offices, or other facilities, or recovering data previously backed up to a remote location.
Failover/recovery testing is a form of disaster/recovery testing limited to moving to a backup system in the event of failure without transfer to a different operating site.
Back-up/recovery testing is a form of disaster/recovery testing limited to restoring from backup memory in the event of failure without transfer to a different operating site or back-up system.

Organizational Test Processes

The organizational test process encompasses critical activities to establish, evaluate, and uphold organizational test specifications. These activities include the development, thorough examination, and ongoing upkeep of these specifications, ensuring they align with the organization’s goals, objectives, and industry standards. Additionally, the process involves vigilant monitoring to ensure consistent adherence to these specifications throughout the organization.

The first step in the organizational test process involves the creation of comprehensive test specifications that outline the desired testing procedures, methodologies, and criteria. These specifications serve as a blueprint for testing activities across various departments, projects, or systems within the organization. They establish a standardized framework that enables consistent and efficient testing practices, ensuring reliable and accurate results.

Once the specifications are established, a meticulous review assesses their comprehensiveness, clarity, and relevance. This review involves engaging relevant stakeholders, such as testing professionals, subject matter experts, and project managers, to gather valuable insights and feedback. Through this collaborative review, any potential gaps, ambiguities, or inconsistencies in the test specifications are identified and rectified, enhancing the overall quality and effectiveness of the testing process.

However, the organizational test process does not end with creating and reviewing test specifications. It also entails maintaining these specifications to accommodate technological changes, industry regulations, and organizational needs. Regular updates and revisions ensure the test specifications remain relevant, up-to-date, and aligned with the evolving testing landscape. This dynamic approach to maintaining test specifications guarantees that the organization continues to employ the most effective and efficient testing practices.

Furthermore, the organizational test process is crucial to monitoring organizational compliance with the established test specifications. This involves regularly assessing and verifying whether the testing activities conducted within the organization adhere to the defined specifications. Compliance monitoring ensures that all stakeholders involved in testing activities follow the standardized procedures and guidelines, promoting consistency, accuracy, and reliability in the testing process. Any non-compliance issues identified during this monitoring phase are promptly addressed and rectified to maintain the integrity and effectiveness of the testing process.

In conclusion, the organizational test process encompasses a range of activities, including creating, reviewing, and maintaining test specifications and monitoring organizational compliance with these specifications. Organizations can ensure consistent, reliable, and high-quality testing practices by establishing standardized testing procedures and continuously evaluating their effectiveness.

Typical inputs to this process include views of primary stakeholders, knowledge of the current test practices within the organization, organization’s mission statement, IT policy, IT project management policy, quality policy, organizational test policy, organizational test practices, feedback on test policy, feedback on test practices, typical test plans from the organization, and industry and/or government standards.

Euroclear bundles this information in two documents:

1. (Organizational) Test Policy: Test policy documents explain the ‘why do we do which tests.’ Organizational means at group level, division level, or tribe level, where top layer documents contain ‘common parts’ or parts applicable by most of the lower level organization. Test policy documents on lower echelons provide information solely applicable to that echelon but also mention deviations from the higher echelon test policy.
2. (Organizational) Test strategy (practices): Contains the ‘what do we do to implement the tests mentioned in the test policy, who is involved, when (in the life cycle/test levels) are they executed, where (test environments) are they executed, how are they executed (manual, automated, externally, etc.) and which controls do we have for these tests.‘ The same remarks apply to the organizational part as to the test policy documents.

Example: The test policy for ‘Disaster recovery’ would mention that both the management as well as external regulators expect Euroclear to have processes in place to handle business interruptions of the Euroclear IT systems and to test that reactivation of these systems is done swiftly, correctly and without a loss of data.

The testing strategy would mention that a disaster recovery test is executed on a regular basis, involving infrastructure teams, business teams, etc. to (1) cause a business disruption, (2) to trigger business recovery, and (3) verification an validation that systems are up and running with loss of data.

Develop organizational test specification

• Requirements for the organizational test specifications shall be identified from the current testing practices within the organization, from stakeholders and/or shall be developed by other means.
• The organizational test specification requirements shall be used to create the organizational test specification.
• Approval on the content of the organizational test specification shall be obtained from the stakeholders.
• The availability of the organization test specification shall be communicated to the stakeholders in the organization.

Monitor and control the use of organizational test specification

• Usage of the organizational test specification shall be monitored to determine whether it is being used effectively within the organization.
• Appropriate actions shall be taken to encourage alignment of stakeholders to the organizational test specification.

Update organizational test specification

• Feedback on the use of the organizational test specification should be reviewed.
• The effectiveness of the use and management of the organizational test specification should be considered and any feedback and changes to improve its effectiveness should be determined and approved. NOTE: This can be achieved by reviewing feedback, through workshops, interviews and other suitable means.
• Where changes to the organizational test specification have been identified and approved, these changes shall be implemented.
• All changes to the organizational test specification shall be communicated throughout the organization including to all stakeholders.

Test Management Processes

Test management processes refer to a series of actions or steps taken to manage and control the testing activities in a software product lifecycle. These processes include planning, designing, executing, and monitoring tests, as well as recording and analyzing the results. The primary aim of test management processes is to ensure the quality and reliability of the software product. They involve coordinating resources, setting up schedules, managing risks, and creating and maintaining test documents. Test management processes also involve communication with various stakeholders such as developers, project managers, and clients to ensure that the product meets the desired quality standards. Effective test management processes can lead to an efficient testing phase, reducing errors and defects in the final product.

The management of the testing to be performed is described by the test management processes. Initially, a test plan is created in the test strategy and planning process based on an analysis of identified risks and project constraints, and taking account of the organizational test practices document, where one exists. The test plan includes the project-specific test strategy, the staffing, and the scheduling of the testing.

The test plan drives the testing, which is managed through the test monitoring and control process. When testing is complete, the test completion process includes activities to generate a test completion report, clean up the test environment, archive testware and perform lessons learned, among others.

The link between organizational test processes and test management processes lies in the integration and alignment of the broader organizational testing practices with the specific activities and responsibilities carried out in test management. Organizational test processes provide constraints, a framework, and guidelines for how testing should be conducted across various projects within the organization.

Test management processes, on the other hand, focus on the planning, monitoring, control, and completion of testing activities within a specific project and provide useful feedback to review the organizational test processes regularly.

Test strategy and planning

Test strategy and planning is a critical facet of the test management process. This process involves outlining the approach and resources needed to effectively conduct software testing. Essentially, it identifies the testing objectives, methods, and tools, as well as the resources and timeline needed to accomplish the testing goals. Effective test strategy and planning necessitate a thorough understanding of the software under scrutiny, including its functionality and potential areas of risk. This process is vital in ensuring that quality control measures are in place and that any software issues are identified and addressed promptly, thereby preventing future complications. It is the blueprint that guides the entire testing process, ensuring that the software meets the desired quality standards.

The test plan an strategy for ‘disaster recovery testing’ involves a risk based assessment of all assets for which disaster recovery needs to be tested. The schedule for disaster recovery testing is to plan this <m> number of times and effectively executing disaster recovery <n> number of times. The necessary IT staff as well as business representatives will be made available for validating that disaster recovery was successful.

The purpose of the test strategy and planning process is to develop, agree, record and communicate to relevant stakeholders the scope and approach that will be taken to testing, enabling early identification of resources, environments and other requirements of testing.

Understand context

It involves considering the overall environment in which the software or system will be used and the specific conditions under which it will operate. This includes factors such as the user’s needs and expectations, the system’s intended purpose, and the technical and business constraints that may affect its performance. By understanding the context, test managers can develop a more effective and efficient testing strategy, prioritize testing activities, identify potential risks and issues, and make better decisions about resource allocation. It allows them to align the testing process with the project’s objectives and ensure that the system meets the requirements of its intended users and environment.

Organize test plan development

This process usually entails identifying the required resources, setting up timelines, defining objectives, and detailing the methodology or techniques to be used. It also includes identifying potential risks and planning mitigation strategies. Organizing a test plan development ensures a structured approach to testing, leading to efficient utilization of resources, and ultimately, a high-quality software product. It also aids in tracking, monitoring, and controlling the testing process.

Identify and analyze risks

The identification of potential risks involves anticipating possible problems that might occur during the testing phase, which could hinder the achievement of the project’s objectives. This could include technical glitches, resource constraints, timeline issues, or even human errors. Once these risks are identified, they are then analyzed to determine their potential impact on the project. This analysis provides a basis for developing strategies to mitigate these risks, thereby ensuring the smooth execution of the project. Therefore, risk identification and analysis are instrumental in enhancing the effectiveness and efficiency of the planning and strategy process in test management.

Identify risk treatment approaches

These approaches aim to address potential risks that may occur during the testing phase of a project. They involve identifying the risks, evaluating their potential impact, and determining the best strategies to mitigate or completely eliminate these risks. The treatment of risks may involve avoiding the risk, reducing the negative effect or probability of the risk, transferring the risk to another party, or even accepting some or all of the potential or actual consequences of a particular risk. This systematic approach to dealing with risks is instrumental in ensuring that the testing phase of a project goes smoothly, thus avoiding costly and time-consuming setbacks.

Design test strategy

This strategy serves as a blueprint for the entire testing process, outlining the testing objectives, methodologies, test levels, and the resources required. It includes details such as what needs to be tested, the different types of testing to be done, the order of testing, and the personnel responsible for each task. The test strategy should align with the project’s goals and objectives and should be flexible enough to accommodate any changes or issues that may arise during the testing process. This strategic approach to testing ensures that the process is efficient, effective, and aligned with the project’s overall objectives.

Determine staffing and scheduling

It involves identifying the required personnel necessary for the testing process, their roles, responsibilities, and the timeline within which they are expected to complete their tasks. The objective is to ensure that there are enough resources to carry out the testing activities efficiently and effectively. This step is not just about assigning tasks; it also involves considering the skills and expertise of the team members, their availability, and the project deadlines. The output of this process is a well-defined and structured test plan that guides the entire testing process to ensure project success.

Record test plan

This plan serves as a strategic blueprint, outlining the approach, resources, schedules, and potential risks involved in the testing process. It provides a comprehensive guide on how to test a system or product effectively and efficiently, ensuring that all potential issues are addressed and mitigated. This record test plan is not static but evolves as the project progresses, incorporating changes and adjustments as necessary. Thus, it plays a pivotal role in maintaining the quality and reliability of the software or system being developed, making it a vital component of the overall test management processes.

Gain consensus on test plan

This consensus-building step involves gathering input and agreement from all relevant stakeholders, including project managers, software engineers, quality assurance teams, and potentially even clients or end-users. The test plan outlines the approach and resources required for validating the quality and functionality of a software product or system. By reaching a consensus on the test plan, potential misunderstandings or miscommunications can be avoided, ensuring that all parties have a clear understanding of the testing objectives, methodologies, timelines and expected outcomes. This collaborative approach not only fosters transparency and trust but also promotes accountability and efficiency in the overall testing process.

Communicate the test plan and make it available

This communication involves articulating the testing objectives, the methods to be used, the resources needed, and the schedule to be followed. By making this test plan readily available, everyone involved in the project can gain a clear understanding of the testing approach, facilitating greater coordination and efficiency. Moreover, this transparency helps to ensure all stakeholders are aligned, minimizing potential misunderstandings and conflicts down the line. Ultimately, effective communication of the test plan contributes to a robust and well-structured testing strategy, enhancing the overall effectiveness of the test management processes.

Test Monitoring and Control as part of test management processes

This process ensures that the testing work is proceeding as per the defined plan and is meeting the predefined objectives. It aids in identifying any deviations or issues in the testing process early, allowing for timely corrective actions. It involves the continuous assessment of the test progress, coverage, potential risks, and issues. Test Monitoring and Control also helps in improving the testing process by providing valuable insights and feedback, thereby contributing to enhanced product quality and delivery.

Set-up, monitoring, control, and reporting are done using the general principles set out in this document. Stakeholders are internal to the company, but also regulators.
A template for the report exists and is delivered within 1 day after finishing the disaster recovery testing.

Set-up

This process is established to meticulously monitor and control the activities and resources used in testing. It involves tracking the test progress, comparing the actual outcomes with the planned ones, and adjusting the process as necessary to meet the desired goals. The set-up of this process is integral to ensuring that any issues or deviations are identified and addressed promptly. It helps in maintaining the efficiency and effectiveness of the testing phase, thus enhancing the overall quality of the software or system under test.

Monitor

The primary purpose of test monitoring is to provide ongoing insight into the test activities’ performance, progress, and quality. This includes tracking and documenting test execution, evaluating against planned objectives, and making necessary adjustments to stay aligned with the test plan. The monitor provides real-time data and feedback, enabling stakeholders to make informed decisions and take timely corrective actions. It helps ensure that the testing process remains on track, maintains the desired quality standards, and delivers expected outcomes.

Report

It involves a detailed report that provides comprehensive insights into the ongoing testing activities. It is instrumental in ensuring that the test plans and strategies are being executed correctly and that the testing goals and objectives are being met effectively. The report offers vital information such as the status of the testing efforts, the testing progress, any issues or risks identified, and the overall quality of the product under test. This crucial data aids stakeholders in making informed decisions, taking corrective actions if necessary, and keeping the project on track. Therefore, the reporting in the Test Monitoring and Control process plays a pivotal role in the success of the entire test management process.

Control

The control aspect involves the management and adjustment of the ongoing testing activities to ensure they align with the planned objectives and strategies. It involves making decisions based on the test monitoring data, which may include altering the testing schedule, resources, or strategies to meet the set goals. Control is essential in managing risks, optimizing resources, and ensuring the effectiveness and efficiency of the testing procedures. Without effective control mechanisms, it would be difficult to align the testing activities with the project’s objectives, which could lead to inefficiencies and sub-optimal outcomes.

Test Completion

Test Completion is a critical component of the test management process, signifying the end of the testing phase. It involves several activities such as evaluating the quality of the product based on the test results, identifying areas of improvement, archiving test assets for future reference, and documenting lessons learned during the process. This phase ensures that all testing objectives have been achieved and that the product is ready for delivery or deployment. The insights gained from the Test Completion stage can also help in improving the efficiency of future testing processes. It is essential that the Test Completion process is carried out meticulously to avoid any overlooked errors that might affect the performance of the software or product.

Disaster recovery testing can be considered as completed once:
– All critical and high-importance assets are operational;
– Validation is finished that no data has been lost;
– All necessary logging, recording, reporting is created;
– All necessary deliverables have been communicated to the relevant stakeholders;
– Formal sign-off on the disaster recovery procedure has been received.

Archive test assets as part of the test completion process, which is part of the test management processes

The process of archiving test assets is a vital component of the test completion process, forming a crucial part of the overall test management procedures. Test assets, such as test plans, test cases, test scripts, and data sets, are significant resources utilized during the software testing process. Once the testing phase is completed, these assets are archived for future reference and reusability. Archiving test assets not only ensures the preservation of valuable data but also promotes efficiency and consistency in subsequent testing procedures. It enables the testing team to analyze previous test results, understand the methodologies used, and learn from past mistakes or successes. Therefore, the archival of test assets is an essential practice in the test completion process in the realm of test management processes.

Clean up test environments

This step involves restoring the test environment to its original state before the commencement of the test, clearing any test data or configurations that were set up, and ensuring that the environment is ready for any subsequent testing. This process is crucial as it helps prevent any discrepancies or conflicts that might arise due to residual data or settings from previous tests. It also ensures that every new test starts in a standardized and controlled environment, thereby improving the accuracy and reliability of the test results.

Identify lessons learned

This phase allows the team to evaluate the effectiveness of their testing strategies, identify any issues or difficulties they encountered, and determine ways to improve their processes in the future. It is also an opportunity to document any identified bugs or issues for future reference. Lessons learned could include the need for more rigorous testing procedures, better communication among team members, or improvements in bug tracking and reporting. These insights are valuable for improving the efficiency and effectiveness of future testing processes, thereby enhancing the overall quality of the software or system being developed.

Report test completion as part of the test completion process, which is part of the test management processes

This involves documenting and communicating the results and findings of the conducted tests to relevant stakeholders, such as the project team, clients, or management. The report typically includes details about the tasks executed, the methodology used, the resources involved, and most importantly, the outcomes of the tests. This final report serves as a formal declaration that the testing phase has been concluded, and it’s instrumental in guiding subsequent actions, such as debugging, quality assurance, and product deployment. It also provides valuable insight into the system’s performance and potential issues that need to be addressed.

Test Sub-Processes

ISO 29119 calls them dynamic test processes, but we at Euroclear, prefer to call them Test sub-processes, because they can be dynamic (executing code, changing data), but they can also be static (peer-to-peer reviews, demos, sprint planning, quarterly planning, etc.)

Test subprocesses are integral parts of a broader testing process in software development. These are individual elements or steps that together form the comprehensive testing procedure. More often, they include planning, preparation, evaluation, execution, and result analysis. Each subprocess has a unique role, contributing to the overall objective of ensuring the software or application is free from defects, operates as expected, and meets the set quality standards. By segmenting the testing process into various subprocesses, developers can effectively manage and control each stage, enhancing efficiency and accuracy in identifying and resolving potential issues.

Test sub-processes are integral components of the overall testing methodology, encompassing various aspects such as test life cycle phases, test levels, and test types. These sub-processes deal with different stages in the test life cycle, such as test planning, test design, test execution, and test closure. They also cover various test levels including unit testing, integration testing, system testing, and acceptance testing. Additionally, they encompass diverse test types like functional testing, non-functional testing, regression testing, and smoke testing. Essentially, test sub-processes provide a structured approach to testing, ensuring that each aspect of the software or system is thoroughly examined for potential flaws or issues.

Describing test sub-processes should cover at least:

Static test sub-process
All new and updated software will deliver a document describing the impact on the disaster recovery testing for that software. This is reviewed by IT operations and the business.
The disaster recovery process is reviewed on a yearly basis, based on a list of critical and high-importance assets. A risk-based approach is used. These reviews are done with IT Development, IT operations, and the business. Result is a list of observations and recommendations and may result in an update of the disaster recovery (testing) process.

Dynamic test sub-process
Disaster recovery testing is a testing type and it is executed in the operational testing phase of the testing life cycle. It is executed as part of the operational testing level. The environment in which this testing is done is the production environment and it involved an operational swap from one production system on location A to another production system on location B. After returning operations from B to A, disaster recovery testing is done. All data is production data related. Tests are not automated and are performed by resources from the business. For a part, the testers use error guessing (based on experience), non-invasive regression tests, and exploratory testing as testing techniques. Traceability is established on the list of critical and highly important assets. Entry criteria (DOR) to start disaster recovery testing is the signal from the infrastructure department that activity has resumed on location A and intake testing (technical) has been successful. Exit criteria (DOD) for disaster recovery testing:
– All critical and high-importance assets are operational;
– Validation is finished that no data has been lost;
– All necessary logging, recording, and reporting is created;
– All necessary deliverables have been communicated to the relevant stakeholders;
– Formal sign-off on the disaster recovery procedure has been received.

Note: real documentation is of course more extended, but this is in a nutshell the disaster recovery testing sub-process.

Every test sub-process can be fitted into:

The terminology used comes from ISO 29119 and since the ISO standard refers to a dynamic test sub-process, but Euroclear also includes static test-sub-processes, we adapt the above to:

Test design and implementation – Test context and concept

These processes involve the construction of a test model, which serves as a blueprint for the testing strategy. Following this, test coverage items are identified, outlining the areas and components that need to be tested to ensure comprehensive quality assurance. Based on these coverage items, specific test cases are determined to systematically evaluate the functionality, performance, correctness, accuracy, reliability, etc. of the system or product under test. Finally, test procedures are created, providing detailed instructions on how to execute the test cases, record results and handle any potential issues or anomalies. These systematic and structured steps ensure that the testing process is thorough, effective and efficient.

Create test model

This model serves as a blueprint or a guide that outlines the testing strategy to be followed. It contains the detailed plan of the test cases, scenarios, test data to be used, and/or meetings, demos, etc. to organize as well as the expected outcomes. It also provides a visual representation of the system’s functionality and its interaction with users, which helps in identifying potential areas of risk and vulnerability. providing an overview of the involved stakeholders heps to identify who to invite to the ‘reviews’. This model aids in enhancing the efficiency and effectiveness of the testing process by ensuring that all aspects of the system are thoroughly examined, thereby improving the overall quality of the software.

Identify test coverage items

These items are specific features, functionalities, or aspects of the system or software that need to be tested as well as sources on which the software development was based. They are identified based on the test basis and the test objectives. The test coverage items ensure that all the important areas of the software, as well as its sources, are tested, thereby reducing the risk of undetected bugs or errors. They are also used to measure the extent of testing performed by a set of tests. Identifying these items is crucial to designing effective test cases, organizing static testing, and ensuring a comprehensive and thorough testing process.

Derive test cases

This is a critical stage in the testing process, where specific scenarios or conditions are developed to validate the functionality and performance of software or systems under test.

Static testing, it drills down to the selection of which static testing techniques to use on which test items. These test cases are derived based on the defined test objectives, test conditions, software requirements, and requirement sources. They consist of test inputs, execution conditions, and expected outcomes, which are meticulously designed to cover as many potential bugs or issues as possible. The derivation of test cases is not a one-time activity but a continuous process that evolves with the changes and enhancements in the software or system under test. As such, the quality, accuracy, and comprehensiveness of these test cases significantly influence the effectiveness and efficiency of the overall testing process.

Create test procedures

This involves the meticulous crafting of specific sets of instructions or steps that will be followed during the testing phase. The goal of these procedures is to verify and validate that the product or system meets its predetermined specifications and operates correctly within its intended environment. These procedures also enable a consistent and systematic approach to testing, thereby enhancing the efficiency and effectiveness of the testing process. Hence, the formulation of test procedures plays an instrumental role in ensuring the quality, reliability, and performance of the end product or system.

Test environment and data management – setup and content

Test environment and data management/setup and content form a crucial part of the test sub-processes.

The test environment refers to the hardware, software, and network configuration set up to conduct the testing process. It is crucial as it can significantly impact the functionality and effectiveness of the software being tested.

Data management, on the other hand, refers to the process of managing, controlling, and validating the data used during the testing process. It ensures that the data used is accurate, consistent, and reliable for the tests being carried out.

Setup and content refer to the process of static testing. It ensures that test items are verified for completeness, accuracy, correctness, etc. before code is created/executed.

Establish test environment

This is where the conditions under which a product or software is tested are created and controlled. The test environment is designed to mimic the final environment where the product will be used, thus ensuring that any potential issues or faults are identified and rectified before product release. Hence, it’s a critical measure in ensuring the quality, functionality, and reliability of the end product.

Maintain test environment

The test environment refers to the controlled conditions under which testing is conducted, including hardware, software, network configurations, and other specific conditions. It is a stage where the developed software is tested for its functionality, performance, and stability before deployment. The maintenance of this environment ensures that it remains conducive and effective for all necessary testing procedures, thus enabling accurate and reliable results. It includes setting up the environment, updating software and hardware, monitoring performance, troubleshooting issues, and ensuring data integrity. Therefore, sustaining a functional, efficient and secure test environment is key to successful software development and deployment.

Prepare test data

This process involves creating, selecting, and maintaining data for testing the functionality and performance of software applications. It often requires the collaboration of both testers and developers to ensure the test data aligns with the system’s requirements and scenarios. It may include creating fictitious customer profiles, transaction histories, or any other data patterns that the application may encounter in a real-world scenario. The role of test data in this context is to simulate real user behaviour and inputs, allowing the team to identify potential issues and optimize the system’s performance.

Maintain test data

This involves ensuring that the data used for testing is consistent, reliable, and secure to facilitate accurate and comprehensive testing results. This process includes the creation, storage, manipulation, and protection of test data, which must be accurately updated and managed for effective test execution. Therefore, test data management plays a pivotal role in enhancing the efficiency of the testing process, mitigating risks associated with data-sensitive information, and ultimately improving the quality of the software product.

Setup static testing sessions

Static testing is a form of software testing that involves the examination of a program’s code and documentation but does not require the program to be executed. By setting up regular static testing sessions, potential issues can be identified and addressed early in the development process, thereby enhancing the efficiency and effectiveness of the overall testing strategy. This proactive approach to testing not only helps to maintain the integrity of the test environment but also ensures that the data being used for testing purposes is robust and reliable.

Follow-up static testing sessions

Static testing, which involves the examination of software’s non-runtime components, serves as a pivotal technique for early error detection. After conducting these sessions, it is essential to review the outcomes thoroughly and address any discovered defects or issues. This follow-up process ensures that the test environment remains robust and efficient, enabling accurate test data management. Moreover, it aids in the optimization of the testing procedures and their subsequent sub-processes, fostering overall software quality and reliability.

Test execution as part of the test sub-processes

This process is designed to detect flaws, bugs, and other issues that may affect the performance, functionality, or usability of the system. Test execution not only helps in ensuring that the software meets its specified requirements but also ascertains its reliability under different scenarios. Therefore, it plays a significant role in enhancing the quality of the software and ensuring it delivers the desired output.

Execute test procedure as a part of test execution, which is a part of the test sub-processes

This procedure involves systematically conducting tests on a system or application to identify any potential flaws or issues. It is a meticulous process that requires precision, as every single document, every process, and every single function of the system or application must be tested statically or dynamically, to ensure optimal performance. The test procedure execution is designed to scrutinize the documentation, processes, and system under various conditions and scenarios to assess its reliability, functionality, correctness, accuracy, completeness, and efficiency. This integral part of the test sub-processes is pivotal in guaranteeing the quality and reliability of the final product.

Compare test results as a part of test execution, which is a part of the test sub-processes

The comparison of test/review results involves analyzing the actual output against the expected outcome, and also the review observations. This analysis helps in identifying any discrepancies or defects, thus ensuring the product requirements are in order as well as the software’s functionality aligns with the specified requirements. The test execution process includes execution, capturing the results, and tracking the identified bugs. All these tasks cumulatively contribute to efficient software testing, with comparing test results playing a pivotal role in determining the software’s quality and reliability.

Record test execution as a part of test execution, which is a part of the test sub-processes

The purpose of recording test execution is to document the detailed steps taken during testing, the observations, the outcomes of these steps, any issues encountered, and how those issues were resolved. This recorded data serves as a valuable resource for future reference and can help in troubleshooting, improving test strategies, and ensuring repeatability of tests. It contributes to the overall effectiveness and efficiency of the testing process, thereby enhancing the quality of the final software or product.

Test incident reporting as part of the test sub-processes

It is the process of documenting any anomalies or discrepancies found during the testing/reviewing phase. These could range from minor glitches to significant system failures and bugs that affect the functionality of the software. The objective of this process is to provide a detailed account of the incident, including the steps to reproduce it, the expected outcome, and the actual result. This report facilitates the troubleshooting process, allowing analysts and/or developers to identify and rectify the issue more efficiently. It also aids in improving the overall quality of the software by ensuring that no defect goes unnoticed and unaddressed.

Analyze test/review results

This analysis typically involves a comprehensive assessment of the data gathered from conducting tests and reviews, with the aim of identifying any incidents or discrepancies that occurred. These could be anything from unexpected behaviors to outright errors. Once these incidents/observations are identified, they are documented and reported accordingly. This process not only aids in rectifying the identified issues but also contributes to the overall improvement of the product or system being tested. Additionally, the analysis can provide valuable insights for future testing strategies, thereby enhancing the effectiveness and efficiency of subsequent test cycles.

Create/update incident/observations report

The goal is to meticulously document any incidents or observations made during the testing/reviewing phase. Any bugs, errors, observations, or unexpected outcomes are recorded in detail to facilitate a comprehensive understanding of the system’s performance. This not only aids in identifying the root causes of any issues but also helps in formulating solutions. Regular updates to these reports ensure that all information is current and accurately reflects the ongoing status of the test processes.