Massachusetts Institute of Technology Touchstone Performance Test Plan Plan
Abstract
This test plan is intended to prescribe the scope, approach, types of performance testing, resources and high-level schedule of the testing activities to be performed in the Touchstone project. This plan will identify the use cases, data, and related systems to be included in the testing process.
...
Risk | Contingency |
Production-like test environment not available | Utilize development or production environment. Results may not be indicative of production and therefore cannot be used as a benchmark. Production performance issues may not be identified during testing. |
Production-like setup and settings not available. | Use the closest setup and settings we can. Results may not be indicative of production and therefore cannot be used as a benchmark. Production performance issues may not be identified during testing. |
Fully operational test tools not available. | Wait until the test tools are availalbe available or find and use another test tool(s). This will extend the time required to perform testing. |
Test time increases due to changes in scope requiring additional test analysis and/or test case creation | If test time cannot be increased, reduce/cut performance testing scenarios and execute highest priority scenarios priority scenarios initially followed by lower priority tests until test time runs out |
Involvement of subject matter experts (SMEs) for all stages of the testing effort not sufficient. | If test time cannot be increased, reduce/cut performance testing scenarios and execute highest priority scenarios priority scenarios initially followed by lower priority tests until test time runs out |
Inadequate Non-functional Requirements | Missing pass/fail criteria invalidates benchmarking. Missing load modeling invalidates all scenarios. Perform only a brute stress test to try and flush out major bottlenecks and functionality under load issues. Additionally an endurance tet test can be run to attempt to identify memory leaks. All tests will be less indicative of real world usage scenarios. |
Insufficient access to systems in order monitor (This includes any necessary server side scripts which may need to be developed in order to capture desired metrics.) | Root cause analysis will be difficult is possible. Testing time will most likely need to be extended and scenarios may be abbreviated due to time constraints. |
Substantial issue(s) which requires significant modifications to the application or re-configuration of the system are encountered. | Some testing may need to be re-done, possibly including re-scripting etc. This would extend testing time. |
Excessive number of bottlenecks encountered and/or issue correction time. | Extend testing time. |
Test time increases due to changes in scope requiring additional test analysis and/or test script/scenario creation | If test time cannot be increased, reevaluate priorities and risk and test according to new priorities. |
...
The overall strategy for performance testing the Touchstone project is goal based. There are four main goals whe we hope to acheiveachieve:
- Performance - Benchmark the system to ensure it meets all non-functional requirements related to performance.
- Stress - Push the system to it breaking point and beyond to identify how and under what level of load the system fails as well as the ramifications of such a failure.
- Endurance - Place the system under a heavy, yet manageable, load for a protracted period of time to identify any performance degradation and/or memory leaks.
- Fail-over - Place the system under a heavy, yet manageable, load, wait for it to stabilize and then disconnect the servers from their network connections to identify how the system handles the sudden loss of a server. This will help satiate any up-time SLAs or non-functional requirements.
Scripts will be designed to model various user interactions with the system. While most of the user interactions will be scripted, some may be omitted according to the 80/20 rule and/or any time constraints which may exist.
...
The tools we will employ are yet to be determined. A proof of concept (PoC) is under way on a performance tseting testing tool. If it is acceptable to MIT then this tool will be identified here. Otherwise furhter further PoCs may need to be conducted until a satisfactory tool is identified and accepted by MIT.
...
The following scripts will be used during the performance testing effort. When the design steps have been provided by MIT all of the to be determined (TBD) values will be replaced with the actual values.
4.1 CAMS 1 CAMS Account Creation
Precondition: TBD
...
A performance test is designed to benchmark the system under test under a realistic load scenario that mimics what we anticipate real world usage will be at its peak. References to non-functional requirements marked as TBD will be updated once the non-functional requirements are provided by MIT.
5.1.1
...
IdPi Only
The objective of this scenario is to benchmark just the internal IDP.
5.1.1.1 Load Model
Desired Transaction Rate: TBD
Script | % of Load |
|---|---|
Site Access - Kerberos w/ticket | TBD |
Site Access - Web Auth | TBD |
5.1.2
...
IdPe Only
The objective of this scenario is to benchmark just the exzternal external IDP.
5.1.2.1 Load Model
Desired Transaction Rate: TBD
Script | % of Load |
|---|---|
CAMS Account Creation | TBD |
CAMS Association - OpenID | TBD |
CAMS Association - Kerberos | TBD |
Site Access - CAMS Account | TBD |
Site Access - OpenID | TBD |
...
The objective of this scenario is to benchmark both IDPs IdPs concurrently.
5.1.3.1 Load Model
Desired Transaction Rate: TBD
Script | % of Load |
|---|---|
CAMS Account Creation | TBD |
CAMS Association - OpenID | TBD |
CAMS Association - Kerberos | TBD |
Site Access - CAMS Account | TBD |
Site Access - OpenID | TBD |
Site Access - Kerberos w/ticket | TBD |
Site Access - Web Auth | TBD |
5.2 Stress 2 Stress Testing Scenarios
5.2.1
...
IdPi Only
The objective of this scenario is to stress only the internal IDP. We plan to push it gradually up to its breaking point and then beyond to determine how and at what load it fails.
...
Desired Transaction Rate: OPEN
Script | % of Load |
|---|---|
Site Access - Kerberos w/ticket | TBD |
Site Access - Web Auth | TBD |
5.2.2
...
IdPe Only
The objective of this scenario is to stress only the external IDP. We plan to push it gradually up to its breaking point and then beyond to determine how and at what load it fails.
...
Desired Transaction Rate: OPEN
Script | % of Load |
|---|---|
CAMS Account Creation | TBD |
CAMS Association - OpenID | TBD |
CAMS Association - Kerberos | TBD |
Site Access - CAMS Account | TBD |
Site Access - OpenID | TBD |
...
The objective of this scenario is to stress both IDPs IdPs concurrently. We plan to push it gradually up to its breaking point and then beyond to determine how and at what load it fails.
...
Desired Transaction Rate: OPEN
Script | % of Load |
|---|---|
CAMS Account Creation | TBD |
CAMS Association - OpenID | TBD |
CAMS Association - Kerberos | TBD |
Site Access - CAMS Account | TBD |
Site Access - OpenID | TBD |
Site Access - Kerberos w/ticket | TBD |
Site Access - Web Auth | TBD |
5.3 Endurance 3 Endurance Testing Scenarios
5.3.1 Integrated IDP External & Internal
The objective of this scenario is to run both IDPs IdPs concurrently for a protracted period of time (multiple days) to determine stability and check for memory leaks. We plan to load the system with 80% of the capacity as determined by the integrated stress test scenario and hold it. During this time special attention wil will be paid to memory and general system stability. There should also not be any appreciable deterioration in end-user response times.
...
Desired Transaction Rate: 80% of capacity
Script | % of Load |
|---|---|
CAMS Account Creation | TBD |
CAMS Association - OpenID | TBD |
CAMS Association - Kerberos | TBD |
Site Access - CAMS Account | TBD |
Site Access - OpenID | TBD |
Site Access - Kerberos w/ticket | TBD |
Site Access - Web Auth | TBD |
5.4 Fail4 Fail-over Testing Scenarios
5.4.1 Integrated IDP External & Internal
The objective of this scenario is to check how both IDPs IdPs handle a sudden iteruption interruption in connectivity by pulling the network plug from 1 of the servers (at a time)
...
Desired Transaction Rate: TBD
Script | % of Load |
|---|---|
CAMS Account Creation | TBD |
CAMS Association - OpenID | TBD |
CAMS Association - Kerberos | TBD |
Site Access - CAMS Account | TBD |
Site Access - OpenID | TBD |
Site Access - Kerberos w/ticket | TBD |
Site Access - Web Auth | TBD |
...
The following metrics will be collected from each Touchstone server during the performance tests to assit assist in diagnostics
- CPU %
- System Load
- System Memory
- JVM Memory (For each JVM)
- JVM Processor % (Hopefully we can get this through JMX) (For each JVM)
- JVM Garbage Collections (For each JVM)
- Apache httpd processes (memory, CPU, and open files for each process)
- Number of open files.
- Network Connections
- LDAP Connections (This would be applicable to Core IdP testing only)
- DB Connections (This would be applicable to CAMS testing only)
...
MIT has not yet provided the non-functional Requirements. The link below will be udated updated if a web page is created to house them, otherwise they will be specified here.
...
Architectural diagrams will be refenced reverenced here as MIT proved them.
8.1 Physical
Touchstone Production Physical ArchitecturePhysical Architecture
8.2 IdPi Logical
Touchstone Production IdPi Logical Architecture
...
Touchstone Production IdPe Logical Architecture
9.0 Schedule of Deliverables and Resources
9.1 Deliverables
This section identifies the deliverables, delivery date and resource responsible for each deliverable.
Key Deliverables | Description | Expected Delivery Date | Resource |
Performance Test Plan | This document. | After all non-functional requirements and other needed data is delivered | Questcon |
Performance Test Scripts | Automated scripts used to deliver load. | 36 business days after test plan finalization and environmental needs are met. | Questcon |
Performance Test Scenarios | Automated execution designs used to conduct performance tests. | 5 business days after scripts are developed. | Questcon |
Status Reports | Accomplishments, issues and plans. | Weekly | Questcon |
Defect Reports | Entered in Jira as they are discovered. | Ongoing during test execution | Questcon |
Performance Test Summary Report | Details the results of the testing effort. | 3 business days after the last performance test is completed. | Questcon |
4.2 Test Schedule
The planned test schedule of the Touchstone project has an anticipated start date of //2008 and completion date of //2008. The estimated completion date is based on several assumptions, some of which have been identified in 2.3 Risks & Contingencies.
Milestone | Target Timeframe | Summation of Activities |
Develop performance test plan | 01/15/2008 - 02??/05{}/2008?? |
|
Review performance test plan | 02/05/2008 - 02??/11{}/2008?? |
|
Build Performance test scripts | //2008 - //2008 |
|
Build Performance test scenarios | //2008 - //2008 |
|
Setup test data | //2008 - //2008 |
|
Execute performance tests | //2008 - //2008 |
|
Create test summary | //2008 - //2008 |
|