Proposed test of longjobs service

• Provides an overview of the proposed service design:
  • The problem
  • The proposed solution
    • The goal
    • General architecture
    • Queues and Scheduling
    • Job execution
    • Tickets/tokens
    • Implementation
    • Registration, reservations, and support
• Discusses the major concerns that have been expressed over IS' providing a longjobs service:
  • Over-subscription
  • Costs
  • Security
  • Scale, Robustness
• Lists open issues we have identified:
  • Billing
  • SGI platform support
  • Suppressing interactive capabilities
• Describes the proposed test, including its specific size and scope, goals, and conditions:
  • Questions to be answered by the test
  • Participants and Expectations
  • Timeline
  • Scale Factors
  • Peak Usage Calculation
  • Number of machines: Three Options

The major problem we are addressing is the requirement, as stated in the Athena Rules of Use, that users remain present at the console of an Athena workstation for the duration of a session. This presents a dilemma for users who need to execute long-running, typically non-interactive jobs for legitimate academic purposes, which has led to frustration and abuse.

Our proposed service would give users the ability to run non-interactive, unattended jobs within the Athena environment, by providing a pool of dedicated, centrally-managed Athena server machines on which users can execute such non-interactive jobs remotely, in as regulated, ordered, and predictable a manner as possible.

The goal

General architecture

A registered user could submit a job from any Athena workstation, specifying the type of Athena machine they want the job to run on, and other desired job parameters. The job would be directed immediately to a central "master" server, which would dispatch jobs to "slave" execution machines, as they become available. An execution slave would only run one job at a time, so a job would remain queued at the master until a suitable slave machine became available. Authentication and authorization would be based on the user's Kerberos principal.

Queues and Scheduling

Scheduling of all jobs will be based on a first-in, first-out scheme. A slave machines will serve one or more queues (e.g. the "short" or "long" queue, or both) and may be migrated between queues to satisfy varying demands on load.

Users can query the server to obtain job status, and cancel jobs that were no longer wanted. Users would optionally be notified, via e-mail and/or Zephyr, when a job begins and/or ends.

Job execution

Tickets/tokens

Implementation

Athena Development will make the necessary enhancements and additions to PBS, including the support for Kerberos authentication and authorization, the management of renewable tickets and tokens, compatibility with the Athena login system, Zephyr support, and integration with a group-based quota system. The server machines will be administered by Athena Server Operations, and registrations by the appropriate support group (see below).

Registration, Reservations, and Support

To ensure that sufficient resources are available for their classes, instructors would be able to request a reservation from the Faculty Liaisons for a particular number and type of machines during a specific time period. This would be implemented via a separate queue mapping to an appropriate subset of machines in the pool, ACLd to the group for that class. We would require advance notice for such requests (in order to ensure sufficient machines are available, to transfer general queues to non-reserved machines, etc.). Similarly, reservations would probably need to be staggered to build in some free time for handling transitions.

See Over-subscription for more details on the registration process.

Over-subscription

Methods to prevent over-subscription need not involve simply some charge for using the service. There may also be barriers to usage, or throttles that control the amount of usage. The methods available to this service involve registration, quotas that limit usage, billing for actual use, and the queueing system.

For a service test, we propose the use of quotas and registrations. The issues surrounding billing are discussed in the next section.

The service will be available only to registered users. Registration will be based on the user's Kerberos principal, and will require some description of the intended use, and an estimate of the quota needed. (If we decide to implement billing, a billable account would also be required).

Registrations would typically be done on a group basis, by departments and/or faculty members on behalf of students in a class. Registered users will be assigned one or more group names, which is intended to classify the intended uses. A request for registration should be denied if the service is already at its subscription limit. The user will have to provide a valid (user, group-name) pair to be able to submit a job to the service.

Registrations will expire. The expiration time will either be specified/negotiated at registration time, imposed by policy (e.g. no registration may last more than the current year or current semester), or some combination thereof. Expiration of registrations is necessary for reasonably accurate estimates of quota likely to be used.

The quota available to a user is the sum total of quotas available for all of the (user, group-name) pairs for that user. The quota provided is not necessarily what is requested at registration, it may be less due to constraints on the actual available quota hours available. Quotas will be valid until a registration expires or is renewed - in the case of renewal, a new quota will be assigned. Quotas may be valid only on certain types of machines (e.g. SGI only, older sparcs only), to allow for better estimation of future service usage.

The queueing system acts to help keep the service from being over-subscribed by providing both positive and negative re-inforcement to the user. Specifically to teach the users to plan for submitting their jobs at non-peak times in order to get their jobs running and finished faster.

Since delays in running jobs may manifest as the highest apparent cost to the user, there is an incentive on our part to limit registrations and quota assigned. This, however, will be at the expense of turning people away from the service and pushing them toward use of cluster machines. Balancing these costs will take some experience.

Costs

For an eventual service roll-out, we propose a 2 year renewal cycle for the slaves, in order to keep pace with the cluster machines. Old slaves could be recycled for other services, or kept in the pool, with a factor to equate them with a job-hour on the faster slaves.

For the master server machine to be used in an eventual production service, our ballpark estimate is a cost of up to $15000; this includes the optional cost of some form of hardware-based redundancy. This would not be needed for the proposed test, for which we estimate a cost of about $3900 (Ultra 5/400, 512MB RAM).

Finally, we estimate that the following staff costs would be incurred:

• Dev: 6 FTE mos. initial development, plus 10% FTE for ongoing maintenance.
• Ops: 1-2 FTE mos. initial, 1/3 FTE ongoing.
• Support: 1 FTE month initial, 15-25% FTE ongoing.

We will track the staff time spent maintaining the service, to make sure that it does not become an inordinate drain on IS resources.

Security

• The system will perform Kerberos 5-based authentication and authorization.
• Users will not be given root access to the execution machine.
• Users will not be allowed to login remotely to the execution machine. However, we do not plan to implement additional measures to suppress the ability to run interactive commands (e.g. xterm), as is done on the dialup machines, since system load is not an issue, and such measures would likely be ineffective, given that the user would already have the ability to execute arbitrary commands on the slave. (See Open Issues).
• Users will have the option of running jobs without tickets, and with no or default tokens, if they do not want to create a long-lived ticket for their jobs.
• Regular integrity checks will be performed on the slave and master machines. Slaves will only run one job at a time; any user processes still running at the end of a job will be killed.
• No jobs will run on the master.

Scale, Robustness

• One suitably-configured master server should be able to support many times the number of slaves currently being considered.
• The number of jobs being managed by the master is probably a better measure of size than number of slaves, and this number will probably be relatively low in our environment, compared to other sites' use of batch facilitites. The strict regulation of the service provided by use of registration and quotas will also tend to keep the scale smaller than might be feared.
• We envision the use of some form of redundancy, either hardware or software-based, so that the master server will not be a single point of failure. This would not be done for the proposed test, however.
• Should the service grow sufficiently, additional masters could be deployed, such that each master had its own set of slaves. We could also consider implementing a multiple master scheme.
• Registration could be automated, if we decide to make the service available to a wider community.

Billing

Given the presumption that at least some fraction of the service will be subsidized, i.e. so that users would not be billed directly for such use, the lack of supporting infrastructure to do billing, and the controversy surrounding the issue for a service viewed as meeting a basic academic computing need, we recommend that no billing system be implemented for our proposed test of the service.

IS could decide to add a billing component, possibly as part of an actual roll-out of the service, or later, as a basis for expanding the service, or to recover the costs of maintaining the service. Regardless, the system will always write accounting records, containing any fields (e.g. account number, type) which might be required to interface to an eventual billing system. This data would be derived from the group under which the user submitted the request. Such records will also be critical in our determining how well the system performs; we will write tools to extract information such as average job and queue wait times.

We expect that the proposed service test would help to suggest what eventual billing model, if any, might be appropriate.

SGI platform support

Suppressing interactive capabilities

Questions to be answered by the test

• Does a batch system satisfy the preponderance of user needs?
• How well does PBS work in our environment (including local enhancements)?
• Are there any issues with particular 3partysw applications?
• Is the basic scheduler implementation satisfactory?
• Is our security model appropriate and adequate?
• Does the service's automatic renewal of tickets and tokens work?
• Is the master server configuration satisfactory?
• Is the user interface satisfactory?
• Are our provisions for dealing with failure modes satisfactory?
• Are our accounting records satisfactory, both for analysis and SAP?
• What are the usage patterns?
  • usage level in idle periods vs. peak periods
  • wait times and queue length
  • usage during reserved times vs. other periods
  • collisions between classes
  • implications of these patterns for Ops and support
  • implications for capacity planning and budget
• How well does the provided capacity meet the actual usage?
• Is the proposed quota scheme workable?
  • quota calculations vs. actual usage
  • user behavior, requests for adjustments, other support questions
• Is the proposed reservation system workable?
• How accurate were our operational and support estimates?
  • Support burden of registrations, quotas, reservations
  • Burden of additional server platforms for ops
• How much did the test really cost?
• What are likely additional costs for Ops and support with scale?
• What billing model is suggested by the test experience?

Participants and Expectations

We recommend also including some individual students with legitimate academic needs outside of classes, to gather as much data as possible on the continuing cases of students who appear to be violating the Rules of Use (against leaving a machine unattended or using multiple machines) in the clusters because they have no other means of running non-interactive jobs.

We will explain to the participants that we are conducting the test to determine whether such a service meets their needs, to answer technical questions, and to gather data on what such a service would cost IS as the basis for determining whether an acceptable billing model could be applied to it. All participants would be required to agree to some basic conditions:

• this is a feasibility test, not a beta; there is no guarantee of service and participants must have backup plans ready in case of problems
• there is no implied promise of a future service roll-out, and no commitment to subsidize usage costs if a service is established
• testers will be expected to provide requested feedback in a timely fashion

Timeline

• extend the test (if additional data is needed)
• begin implementing an actual service (perhaps a staged pilot)
• abandon the longjobs effort, explain why to the testers and other users awaiting a decision, and decide how else to address their needs

• write software required for the test
• line up the participants
• buy machines (requires two months lead time)
• code review (at least with respect to security issues)
• usability test
• documentation

Scale Factors

1. Peak usage periods: the calculations below give examples of the relation between number of machines and time to process the set of jobs for a single class at a given peak period.
2. Collisions between classes: in order to learn about behavior of both the system and users when more than one class is submitting jobs, we would ideally include 3 or more classes in the test.
3. 3partysw applications: A large proportion of classes are interested in longjobs for BioSym/MSI software, which is currently available only for SGI. While this software may migrate to other platforms over the next several years and we plan to phase SGIs out of the clusters by 2003, given that we aim to support software that runs in the clusters and would renew equipment on a 2-year cycle, it does not seem reasonable to rule out SGI support at the outset. As we near implementation, future platform support for these applications may be clearer and perhaps lead to revised recommendations.
4. Machine cost: For slave machines, we estimate hardware will cost:
   • $2800 per Sun (Ultra 5/400, 256MB RAM)
   • $1700 per Linux PC (Dell GX110, 256MB RAM)
   • $4800 per SGI (rackmount O2, 128MB RAM)
   and for the master server, the estimated cost is $3900 (Ultra 5/400, 512MB). A more complete discussion is in the Costs section above.

Peak Usage Calculation

Observations:

• given M dedicated machines, the time span to complete a total of J job-hours will be J/M hours, or J/24M days
• M of the students can have their jobs complete in real time (e.g., 24 hrs for a 24-hr job); the remainder would wait some multiple of the job length, up to the aggregate time span (e.g., next M students wait 48 hrs, next M wait 72 hrs, ..., last M wait J/M hrs).

Case 1

This corresponds to estimates from Ceder (MatSci, 30 students x 20 hrs/job) and Rutledge (ChemEng, 25 students x 24 hrs/job), both on SGI.

Case 2

This corresponds to estimates from Cesnik (AeroAstro) and Gupta (Sloan), who each estimated 50 students for "a few hours", both on Sun.

Number of machines: Three Options

• Small number of Suns and SGIs (e.g. 5-7)
  pro:
  • tests applications on both platforms
  • tests operational and support issues for multiple platforms
  
  con:
  • fewer classes each platform ==> fewer collisions to observe

• Larger number of Suns, no SGIs (e.g 10-15)
  pro:
  • more classes ==> more chances to observe collisions
  
  con:
  • no data on SGI applications
  • no data on supporting multiple platforms

• Larger number of one platform, small number of other (e.g. 10, 5-7)
  pro:
  • compromise between the above
  
  con:
  • higher equipment cost