Assessment and Evaluation Process

Setting Goals and Objectives for Learning

The assessment and evaluation process begins by helping participants identify the goals and objectives (cognitive, attitudinal, and behavioral) of a class or a project. After identification, goals are refined and operationally defined to make them measurable.

Making Decisions about Research Methods

With well-defined goals, suggestions for metrics and methods are easier to make. If quantitative methods are used, published tests and questionnaires are recommended. These instruments have the advantage of established validity and reliability and can be discussed in terms of previous uses and findings. It also is possible to custom tailor and/or create measures for the needs of specific research questions or educational goals. For example, TLL educational researchers have developed a set of Learning Behavior Surveys designed to measure student attitudes about those aspects of the educational environment that contribute to their learning.  In many cases, qualitative measures are appropriate. These include focus groups, interviews, classroom observations, and “think-aloud” protocols.

A mixed method (quantitative and qualitative) approach is often the most useful, and both direct and indirect approaches can be employed to assess learning.  The direct approach uses quantitative performance assessments such as:

• Portfolios
• Oral presentations
• Exams
• Problem sets
• Pretest-posttest comparisons of learning

The indirect approach uses qualitative and quantitative approaches, such as:

• Naturalistic descriptive observations
• Focus groups
• Journals
• Structured and open-ended interviews
• Surveys 

Where it is possible, historical or matched comparison groups and experimental procedures are used.  However, because we are an applied laboratory and our research settings are generally “real” classrooms, the preferred measures and methods adjust to whatever the best approach is for answering the questions being asked in a specific situation.

Analyzing and Interpreting the Data

There are many well-developed methods available for conceptually or statistically analyzing the different kinds of data that can be gathered.  When analyzing qualitative data, one can develop taxonomies or rubrics to group student comments collected by questionnaires and/or made in classroom discussions. The frequency of certain types of comments can be described, compared between categories, and investigated for change across time or differences between classes. Frequency data and chi-square analysis can supplement the narrative interpretation of such comments. For the analysis of quantitative data, a variety of statistical tests are available, ranging from the simple (t-tests) to the more complex (such as the use of factor analysis to develop scales).

Making Recommendations and Disseminating Findings

The assessment and evaluation staff can make recommendations based on interpretation of the results.  Since our activities are conducted at the request of a faculty member or investigator, the information gathered belongs to that person.  It is the investigator’s choice as to whether to convey information to oversight authorities internal or external to the Institute or to publicize findings. Assessment results are not used in any way for faculty or staff evaluation.

Assessment and evaluation staff can assist and/or collaborate with faculty members and investigators in the preparation and presentation of written reports, conference presentations, posters, and manuscripts for publication. The degree of involvement can range from simple suggestions to co-authorship. With the permission of the instructors and investigators, TLL can disseminate results about educational and technological innovations both within and outside the Institute through written reports, journal articles, and oral presentations.