Versions Compared

Old Version 31

changes.mady.by.user Will Smithee

Saved on

compared with

New Version Current

changes.mady.by.user Paul B Hill

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Corrected links that should have been relative instead of absolute.

...

MIT Touchstone is a new suite of technologies for authenticating a variety of web applications, being introduced by IS&T. MIT Touchstone does provide provides a single sign-on solution for applications that have been coded and configured to use the system. Within the context of Touchstone enabled applications, users will be able to seamlessly transition between systems without being prompted for additional authentication information.
The intended audience of this document includes all IT personnel involved in the development, testing, and support of Touchstone.

...

Risk

Contingency

Production-like test environment not available

Utilize development or production environment. Results may not be indicative of production and therefore cannot be used as a benchmark. Production performance issues may not be identified during testing.

Production-like setup and settings not available.

Use the closest setup and settings we can. Results may not be indicative of production and therefore cannot be used as a benchmark. Production performance issues may not be identified during testing.

Fully operational test tools not available.  (i.e. tools capable of generating the desired load and capturing the desires metrics)

Wait until the test tools are available or find and use another test tool(s). This will extend the time required to perform testing.

Test time increases due to changes in scope requiring additional test analysis and/or test case creation

If test time cannot be increased, reduce/cut performance testing scenarios and execute highest priority scenarios initially followed by lower priority tests until test time runs out

Involvement of subject matter experts (SMEs) for all stages of the testing effort not sufficient.

If test time cannot be increased, reduce/cut performance testing scenarios and execute highest priority scenarios initially followed by lower priority tests until test time runs out Extend testing time.

Inadequate Non-functional Requirements

Missing pass/fail criteria invalidates benchmarking. Missing load modeling invalidates all scenarios. Perform based on stated non-functional requirements will invalidate performance benchmarking. Missing load modeling will invalidate all test scenarios. The contingency is to perform only a brute stress test to try and flush out major bottlenecks and functionality under load issues. Additionally an endurance test can be run to attempt to identify memory leaks. All , although these tests will be less indicative of real world usage scenarios.

Insufficient access to systems in order to monitor the tests (This includes including any necessary server side scripts which may need to be developed in order to capture desired metrics). )

Root cause analysis will be difficult is if possible. Testing time will most likely need to be extended and scenarios may be abbreviated due to time constraints.

Substantial issue(s) which requires significant modifications to the application or re-configuration of the system are encountered.

Some testing may need to be re-doneexecuted, possibly including re-scripting etc. This would extend testing time.

Excessive number of bottlenecks encountered and/or issue correction time.

Extend testing time.

Test time increases due to changes in scope requiring additional test analysis and/or test script/scenario creation .

If test time cannot be increased, reevaluate priorities and risk risks and test execute tests according to new priorities.

...

Conflicting information provided.

...

Project manager will work to resolve any conflicting information, goals, and/or instructions received.

3.0 Approach

3.1 Testing Strategy

3.1 Testing Strategy

The overall strategy for performance testing the Touchstone project is goal based. There are four main goals we hope to achieve:

  1. Performance - Benchmark the system to ensure it meets all documented non-functional requirements related to performance.
  2. Stress - Push the system to it its breaking point and beyond to identify how and under what level of load the system fails as well as the ramifications of such a failure.
  3. Endurance - Place the system under a heavy, yet manageable, load for a protracted period of time to identify any performance degradation and/or memory leaks.
  4. Fail-over - Place the system under a heavy, yet manageable, load, wait for it to stabilize and then disconnect the servers from their network connections to identify how the system handles the sudden loss of a server. This will help satiate any up-time SLAs or non-functional requirements.

...

Tool

Purpose

Used By

Atlassian Jira

Web-based defect tracking system accessed by http://mv.ezproxy.com.ezproxyberklee.flo.org/jira

Touchstone Project Team (MIT & Questcon)

  StressTester

Performance testing / load generation tool

  (MIT & Questcon)

3.3 Environmental Needs

We will need the following:

...

The following scripts will be used during the performance testing effort.

When the design steps have been provided by MIT all of the to be determined (TBD) values will be replaced with the actual values.

4.1 CAMS Account Creation

Precondition: TBD

Data Needed: TBD

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

4.2 CAMS Association - OpenID

Precondition: TBD

Data Needed: TBD

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

4.3 CAMS Association - Kerberos

Precondition: TBD

Data Needed: TBD

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

...

** Design steps for the following scripts have not been provided by MIT yet.

Script

Objective

Priority

CAMS Account Creation

End user creates a Touchstone account.

High

CAMS Association - OpenID

End user associates their Touchstone account with an OpenID account.

Medium

CAMS Association - Kerberos

End user associates their Touchstone account with a Kerberos account.

High

Site Access - Kerberos w/ticket

Precondition: TBD

Data Needed: TBD

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

4.5 Site Access - Web Auth

Precondition: TBD

Data Needed: TBD

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

4.6 Site Access - CAMS Account

Precondition: TBD

Data Needed: TBD

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

4.7 Site Access - OpenID

Precondition: TBD

Data Needed: TBD

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

4.8 Password Reset

Precondition: TBD

Data Needed: TBD

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

4.9 Admin - Password Reset

Precondition: TBD

Data Needed: TBD

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

4.10 Admin - De-Activate Account

Precondition: TBD

Data Needed: TBD

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

4.11 Admin - Delete Account

Precondition: TBD

Data Needed: TBD

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

4.12 Admin - Activate Account

Precondition: TBD

Data Needed: TBD

End user access a site using an MIT machine that has a Kerberos ticket.

High

Site Access - Web Auth

End User access a site using Web Auth.

High

Site Access - CAMS Account

End user access a site using their CAMS accont only.

High

Site Access - OpenID

End user access a site using their OpenID.

Medium

Password Reset

End User resets their password.

Low

Admin - Password Reset

Administrator resets an end user's password.

Low

Admin - De-Activate Account

Administrator de-activates an end user's account

Low

Admin - Delete Account

Administrator deletes an end user's account.

Low

Admin - Activate Account

Administrator activates a de-activated account.

Low

Transaction Name

Step(s)

Expected Result

95th % Response Time

TBD

 

 

 

5.0 Scenarios

5.1 Performance Testing Scenarios

...

Script

% of Load

Site Access - Kerberos w/ticket

TBD

Site Access - Web Auth

TBD

We will be able to specify the transaction rate, and load modeling once the non-functional requirements have been provided by MIT.

5.1.2 IdPe Only

The objective of this scenario is to benchmark just the external IDP.

...

Script

% of Load

CAMS Account Creation

TBD

CAMS Association - OpenID

TBD

CAMS Association - Kerberos

TBD

Site Access - CAMS Account

TBD

Site Access - OpenID TBD - OpenID

TBD

We will be able to specify the transaction rate, and load modeling once the non-functional requirements have been provided by MIT.

5.1.3 Integrated IDP External & Internal

...

Script

% of Load

CAMS Account Creation

TBD

CAMS Association - OpenID

TBD

CAMS Association - Kerberos

TBD

Site Access - CAMS Account

TBD

Site Access - OpenID

TBD

Site Access - Kerberos w/ticket

TBD

Site Access - Web Auth

TBD

We will be able to specify the transaction rate, and load modeling once the non-functional requirements have been provided by MIT.

5.2 Stress Testing Scenarios

...

Script

% of Load

Site Access - Kerberos w/ticket

TBD

Site Access - Web Auth TBD - Web Auth

TBD

We will be able to specify the transaction rate, and load modeling once the non-functional requirements have been provided by MIT.

5.2.2 IdPe Only

The objective of this scenario is to stress only the external IDP. We plan to push it gradually up to its breaking point and then beyond to determine how and at what load it fails.

...

Script

% of Load

CAMS Account Creation

TBD

CAMS Association - OpenID

TBD

CAMS Association - Kerberos

TBD

Site Access - CAMS Account

TBD

Site Access - OpenID

TBD

We will be able to specify the transaction rate, and load modeling once the non-functional requirements have been provided by MIT.

5.2.3 Integrated IDP External & Internal

...

Script

% of Load

CAMS Account Creation

TBD

CAMS Association - OpenID

TBD

CAMS Association - Kerberos

TBD

Site Access - CAMS Account

TBD

Site Access - OpenID

TBD

Site Access - Kerberos w/ticket

TBD

Site Access - Web Auth TBD

TBD

We will be able to specify the transaction rate, and load modeling once the non-functional requirements have been provided by MIT.

5.3 Endurance Testing Scenarios

...

Script

% of Load

CAMS Account Creation

TBD

CAMS Association - OpenID

TBD

CAMS Association - Kerberos

TBD

Site Access - CAMS Account

TBD

Site Access - OpenID

TBD

Site Access - Kerberos w/ticket

TBD

Site Access - Web Auth

TBD

We will be able to specify the transaction rate, and load modeling once the non-functional requirements have been provided by MIT.

5.4 Fail-over Testing Scenarios

...

The objective of this scenario is to check how both IdPs handle a sudden interruption in connectivity by pulling the network plug from 1 from one of the servers ( at a time).

5.4.1.1 Load Model

Desired Transaction Rate: TBD

Script

% of Load

CAMS Account Creation

TBD

CAMS Association - OpenID

TBD

CAMS Association - Kerberos

TBD

Site Access - CAMS Account

TBD

Site Access - OpenID

TBD

Site Access - Kerberos w/ticket

TBD

Site Access - Web Auth TBD - Web Auth

TBD

We will be able to specify the transaction rate, and load modeling once the non-functional requirements have been provided by MIT.

6.0 Monitoring

The following metrics will be collected from each Touchstone server during the performance tests to assist in diagnostics

...

Architectural diagrams will be reverenced referenced here as MIT proved provides them.

8.1 Physical

Touchstone Production Physical Architecture

...

Key Deliverables

Description

Expected Delivery Date

Resource

Performance Test Plan

This document.

After all non-functional requirements and other needed data is delivered

Questcon

Performance Test Scripts

Automated scripts used to deliver load.

30 business days after test plan finalization and environmental needs are met.
(2.5 days for each /script * 12 scripts)

Questcon

Performance Test Scenarios

Automated execution designs used to conduct performance tests.

5 ~5 business days after scripts are developed.
(~.75 days for each .65 days/scenario * 8 scenarios)

Questcon

Execute Performance Tests

Running of scenarios.

10 business days
(~1 day for 7 scenarios and 3 days for the  endurance scenario)

Questcon

Status Reports

Accomplishments, issues and plans.

Weekly

Questcon

Defect Reports

Entered in Jira as they are discovered.

Ongoing during test execution

Questcon

Performance Test Summary Report

Details the results of the testing effort.

3 business days after the last performance test is completed.

Questcon

...

Milestone

Target Timeframe

Summation of Activities

Develop performance test plan

01/15/2008 - 02/05/2008
15 Business Days Days

  • Analyze existing design documents, notes, and other available materials
  • Develop test plan document

Review performance test plan

02/05/2008 - 02/11/2008
3 Business 3 Business Days

  • Review, clarify, correct, and update the test plan
  • Client approval of test plan

Build Performance test scripts

//2008 - //2008
30 Business Days

  • Author test scripts in automated tool

Build Performance test scenarios

//2008 - //2008
5 Business Days

  • Setup web server and database server
  • Load application under test
  • Setup logins and authorizations

Setup test data

//2008 - //2008
1 Business Day

  • Review & analyze test cases to target data to load in test environment
  • Load initial test data set

Execute performance tests

//2008 - //2008
10 Business Days

  • Execute documented performance test scenarios
  • Communicate with the development team when issues are found
  • Maintain a test run log
  • Track test metrics

Create test summary

//2008 - //2008
3 Business Days

  • Create and deliver a test summary report to include:
    • Summation of planned/actual test activities
    • Deviation from planned activities
    • Summary of defects (open defects)
    • Summary of test metrics

TOTAL:

73 Business Days