 
|
|||||||||
Athena Release Testing Cycle
Annual Release
The Annual Release testing timeline is synchronized with the annual renewal of Athena General Use Cluster hardware. Often a new year's generation of hardware requires significant updates within the operating system. The Development phase of the Athena Relese aligns with the acquisition of prototype hardware. The Release Deployment phase aligns with the deployment of the hardware acquired in each year's big buy for renewal.
The deltas in the Athena release updates are driven by:
- migration off OS versions the vendor no longer will support.
- patches the OS vendor recommends.
- changes are required to keep existing applications and services going across the OS update.
- additional functionality requested by the community.
There is no "test suite" as such for the Athena release. Rather than employing a manual or automated process to exhaustively test the myriad of operating system features, utilities, applications, and services, we rely on a cycle of testing where progressively larger communities adopt the release for daily use and identify anything that broke in the update. The success of such a testing process depends on a diverse community who can cover an adequate amount of system functionality, a responsive group of developers who can investigate and resolve issues, and a robust communication system to join the developers with the community.
The testing proceeds from March through July with four phases (Crash & Burn, Alpha, Beta and Early) before the release is deployed to all users.
Timeline
Phase |
Begins |
Description |
Crash & Burn |
March 1 |
The release engineer has integrated the updates, confirmed that everything builds without error. During Crash & Burn, the Release Engineer and the designated "platform expert" developers attempt to update a test system, and try basic functions such as logging in and reading mail. Prototypes of new hardware should arrive at this point for integration development and testing. |
Alpha |
April 2 |
With some confidence that the update will probably succeed and probably not destroy data, a small group of volunteers attempts to update to the new release and give it daily use. At this point there is the very real possiblity of a mistake in the update, or in some subsystem that will destroy user data or will mess up the OS install necessitating a re-install. |
Beta |
May 1 |
When "show stopper" issues discovered in Alpha testing are resolved, Beta testing begins and a larger community adopts the release for daily use. Since the testing coverage has been small, we expect to see many issues in the early days of Beta. Maintainers of third party software in Athena lockers should be Beta testers. They should work at migrating their software to the new release during Beta and get problems resolved so that both OS and third party software can be tested by users during Early. |
Early |
June 1 |
The Release is made available to end user early testers, and deployed to some clearly labeled Early Test machines in the General Use Clusters. We expect there will be no new show-stoppers, but here is where large scale testing can unearth them in time to fix them without disrupting ordinary users. Halfway through Early testing, (around June 15) a discussion at the Owls team takes place to confirm we are on track for a Release on July 1. If there are risks to that roll-out date, the mitigation plans are made during that discussion. |
Release |
July 1 |
The release goes to the general public with the expectation that sufficient functionality has been covered by the expanding community of testers to offer a reasonably robust update. At this time faculty use this platform to prepare for returning students. The big buy of new hardware can, and should occur now to allow for ample time to deploy new hardware, and to give time in the summer in case large-scale deployment uncovers surprises. |
Patch Releases
Patches to the release have their own test/deploy cycle:
- The developer crafts a change and tests it.
- The developer sends email to source-reviewers detailing the change.
- source-reviewers members suggest amendments, or agree that the change is correct.
- After successful review, the change is committed to the Athena source tree.
- The Athena Release Team decides when to schedule deployment of a patch release balancing risk and need, scope and value.
- To build a patch release, changes are pulled up into the source tree version control branch, and built.
- The updated packages deployed into a read/write AFS volume and sanity checked. If there are problems, amendments may be made or the change gets backed out, and the cycle begins again at step 1.
- If the sanity check succeeds, the read/write volume is released to the read-only volumes in the dev.mit.edu AFS cell for beta testers.
- After two weeks of Beta testing with no reported problems, the patch release is deployed to the general public. If there are problems, the cycle may restart at step 1. If the resolution is of small scope, the correction is made and testing continues.
- If a problem is uncovered after deployment to the general public, either an emergency patch is crafted, or some other action is taken to disable the change.
