Executive Summary

Overview

This Michigan Test for Teacher Certification (MTTC) Technical Manual provides information on test development, updating, administration, scoring, and reporting activities that have been undertaken since 1992 by the Michigan Department of Education (MDE) and Evaluation Systems group of Pearson (formerly National Evaluation Systems). The report highlights the steps taken to establish the validity and reliability of the MTTC program.

The MTTC program is designed to ensure that individuals who wish to become teachers in Michigan schools have the level of knowledge and skills required to perform effectively the job of a qualified Michigan educator in their content area. The test development, updating, administration, scoring and reporting processes employed have been effective in establishing that the MTTC tests are appropriate instruments for that purpose.

The information summarized in this report is designed to show the following about the MTTC tests:

• The MTTC tests provide important information directly relevant to the licensure of Michigan educators who meet the knowledge and skill requirements mandated by the state.
• The tests are valid—the information they provide is an accurate measure of the knowledge and skills—and have been validated throughout the development of the program.
• The tests are reliable—the information they provide has sufficient consistency to assure the Michigan public that licensure decisions are being made on reasonable grounds.

The Michigan Department of Education and Evaluation Systems have created a testing program designed to meet the unique educational needs of the State of Michigan. Both organizations are committed to refining the program further as it progresses and as the Michigan education environment changes.

Background

Section 1531 of Public Act 451 (1976) , as amended by Public Act 267 (1986), Public Act 282 (1992), and Public Act 289 (1995), and most recently amended in September 2018, mandates a testing program as part of Michigan's teacher certification requirements. In June 1991, the Michigan Department of Education awarded National Evaluation Systems a contract to develop and administer the testing program. The purpose of the tests is to ensure that each certified teacher has the level of knowledge and skills required to perform effectively the job of a qualified Michigan educator in their content area. The tests are not the only basis on which prospective teachers are judged in Michigan, nor are the knowledge and skills covered by the tests the only types of knowledge and skills, or the only professional and personal qualifications, those teachers must have. The tests represent one prerequisite for obtaining a teaching certificate or endorsement in Michigan.

Initial development of the MTTC took place in 1991–1992. During initial development tests were prepared in 76 fields. A Basic Skills (reading, mathematics, and writing) test was developed that was required of candidates seeking a Michigan provisional teaching certificate. Academic content-area tests were developed in 75 fields. The academic content area tests are required of candidates seeking a secondary-level teaching certificate or those teachers in grades K–8 who teach in specific subject areas. Beginning fall 2013, and through the 2016–2017 test administration period, the MTTC program had included the Professional Readiness Examination (replacing the Basic Skills test) and 61 tests in various content areas. By decision of the MDE, the Professional Readiness Examination was last administered on September 23, 2017.

A multi-step process involving the participation of Michigan teachers and teacher educators was used to develop the test materials. Pertinent state regulations and policies, curriculum materials, and information from Michigan teacher preparation programs were reviewed to assist in the preparation of test frameworks and objectives. Before the frameworks were finalized, content validation surveys of test objectives for each test field were conducted with additional teachers and teacher educators. Test items were prepared and field tests were conducted with hundreds of students in Michigan. Test items were reviewed by committees of Michigan teachers and teacher educators who participated in Content Validation and Standard Setting activities. The recommendations of Michigan teachers and teacher educators as well as examinee performance on the first operational test form for each test were presented to the State Board of Education. In June 1992, the State Board of Education set the initial passing standards for the MTTC.

Advisory Panels

The MDE constituted two committees, the Teacher Examination Advisory Committee (TEAC) and the Standing Technical Advisory Council (STAC). The TEAC meets on an as-needed basis to advise MDE staff on general policy issues regarding the assessment program. In 2014, the STAC was reconstituted as the Technical Advisory Committee (TAC), which continues to meet annually, or more often if needed, to review MTTC results and to provide advice on technical issues.

Ongoing Test Development and Updating Activities

An important feature of the MTTC program is an ongoing review and updating of test materials. These periodic reviews are designed to ensure that the testing program continues to address the knowledge and skills needed by Michigan teachers and to reflect current practice in Michigan schools. Test updating began in 1992. Each year the Michigan Department of Education and Evaluation Systems determine the fields that will be reviewed and updated. The updating process has included the following steps:

• Collection and review of Michigan policy documents and curriculum materials
• Development of test objectives that appropriately reflect Michigan curriculum and classroom practice
• Review of draft test frameworks (including subareas, objectives, and descriptive statements) by committees of Michigan teachers and teacher educators
• Conduct of Content Validation Surveys of Michigan teachers and teacher educators to evaluate the importance of the content of the test objectives to the job of a Michigan educator
• Development of test items that appropriately reflect Michigan curriculum and classroom practice
• Review of draft test items by committees of Michigan teachers and teacher educators
• Field testing of new items
• Standard setting activities with Michigan teachers and teacher educators
• Determination of passing standards by the Michigan Department of Education

Test Validation Process

The "validity" of a test refers to the ability of the test to support the inferences that are to be drawn from it. Test validation is the process of gathering evidence that the test measures what it is supposed to measure and that the inferences to be made from the test scores are supported by evidence. Various types of evidence may be considered in establishing the validity of a test, and a number of methods are typically used to gather such evidence.

The focus of the validation efforts for the MTTC was on establishing that the content measured by the tests was directly related to the specific knowledge and skills required for teacher certification in Michigan. Materials were collected and reviewed from various sources, including approved Michigan teacher preparation programs; state certification statutes, regulations, and minimum teacher qualifications; state curriculum regulations and guidelines; curriculum materials used in Michigan classrooms; textbooks in use in Michigan classrooms and Michigan teacher preparation programs; professional publications; MDE curriculum guides and goals; and Michigan student assessment materials.

Several levels and types of validity evidence were gathered. Validity evidence was gathered pertaining to the overall purpose and goals of the testing program, the actual test content to be measured, the structure and content of the test objectives measured by each test, and the specific test questions measuring each objective. In this way, validation was a central part of each step in the development of the tests, proceeding from the general level of program authorization to the specific level of the test questions themselves.

The validation process for the MTTC followed professionally accepted procedures for the validation of certification tests, including the following steps:

• The purpose of the testing program was established by the mandating legislation.
• Test objectives—which define eligible test content—were based on Michigan laws, Michigan standards for teacher certification, and other state documents/policies provided by the Michigan Department of Education.
• The test objectives were reviewed by MDE staff, a Michigan Bias Review Committee, and a Michigan Content Advisory Committee for each test field. Revisions were made based on the feedback received.
• For each test field, a Content Validation Survey of the proposed test objectives was conducted among Michigan school teachers and college faculty at Michigan colleges and universities with approved teacher preparation programs. The survey asked Michigan educators to rate the importance of the test objectives to the job of a Michigan school teacher. Test objectives rated important were eligible for inclusion on the tests.
• Draft test items for each field were reviewed by a Michigan Bias Review Committee and a Michigan Content Advisory Committee.
• Draft test items were field tested on operational test forms and with students enrolled in Michigan teacher preparation programs.
• Michigan school teachers and college faculty participated in item validation and standard setting activities. Test items were reviewed to ensure that they matched the validated test objectives and were accurate, free from bias, and related to the job of an educator in Michigan.

In these ways, the content of the tests—from the most general level to the most specific—was repeatedly and systematically judged by qualified reviewers to be valid for the MTTC program.

Bias Prevention

Prevention of bias in the MTTC is important as a matter of fairness and as an aspect of validity. Guarding against bias in the MTTC materials involved the collaboration of educators and reviewers focused on excluding language, content, or perspectives that might disadvantage examinees based on background characteristics irrelevant to the purpose of the test, and on including content and perspectives that reflect the diversity of the Michigan population.

The review of MTTC materials from the standpoint of bias prevention is not a single activity that is conducted at a particular point in the development process. Rather, it is an ongoing aspect of all components of the development process, from the definition of test content through the review of all test items that may appear on all test forms.

Educators from diverse backgrounds were invited to participate throughout the development and updating activities of the MTTC. They served as members of Content Advisory Committees, reviewing draft test frameworks and items, and serving on item validation and standard setting panels. Bias review was a responsibility of every educator who reviewed test materials. In addition, a separate Bias Review Committee, composed of a diverse group of Michigan educators, had the specific responsibility of examining test materials for potential bias.

Standard Setting

Committees of Michigan educators reviewed test questions in their fields of expertise and made judgments concerning the level of knowledge required to perform the job of an entry-level teacher in Michigan. Their item-level judgments were compiled to determine a recommended passing score for each test. The Michigan Department of Education considered the committee recommendations in setting the passing score for each test.

Reliability of Test Results

Reliability refers to the consistency of scores received by a group of examinees. The Appendix presents a number of statistical estimates of the reliability for scores on the tests in the program. For the MTTC tests, as with other tests used in a licensing environment, a relevant reliability statistic is one that provides an estimate of the decision consistency of the tests (i.e., the consistency of the pass/fail decisions that examinees receive).

The meaning of specific reliability estimates derived from the scores of a given group of examinees on a given occasion is a matter of interpretation and judgment that depends on several factors, including the nature of the tests for which the estimates are provided and the purpose for which the results will be used.

Statistical estimates of reliability may be affected by several factors related to the test itself and the group of examinees on which they are based. Key factors that influence reliability estimates include the following:

• Number of examinees—In general, reliability estimates based on larger numbers of examinees are more stable than estimates based on smaller numbers.
• Test length—Reliability estimates tend to be higher for tests with greater numbers of questions.
• Composite tests—Reliability estimates for tests with both multiple-choice and performance components (e.g., Professional Readiness Examination (formerly the Basic Skills test) and many of the World Languages tests, such as Spanish) are more meaningful when they are based on the combined, total test than on either of the two components alone.
• Test content—Reliability estimates are typically higher for tests that cover narrow, homogeneous content than for tests, such as teacher certification tests, that cover a broad range of content.
• Examinees' knowledge—Reliability estimates tend to be higher if examinees in the group have widely varying levels of knowledge and lower if they tend to have similar levels of knowledge.

Administration

The MTTC tests are administered under standardized, consistent procedures at sites across Michigan. Test administrations are designed to provide a professional, equitable, and secure testing environment for candidates, including candidates with needs for alternative testing arrangements. Test sites are screened and selected based on criteria relating to test security, accessibility, and appropriate testing conditions and facilities.

Test administrators are oriented and provided with procedural manuals to use before and during the test administration. The orientation process and the manuals are designed to facilitate secure, efficient, and professional test administrations for all candidates.

As of December 2010, computer-based testing became available as an alternative to paper-based testing for some fields. Initially, computer-based testing supplemented the four paper-based test administrations offered during the program year; candidates had access to two-week testing windows in each month during which a paper-based administration was not offered. During a testing window, testing was available every day except Sunday and select holidays. Beginning November 9, 2015, for all tests available via computer except the Professional Readiness Examination and the world languages tests, candidates have had access to computer-based testing year-round, Monday through Saturday (excluding some holidays). As of March 2016, all tests are available via computer-based testing. Candidates are able to test at any one of a number of VUE Pearson Professional Centers (PPCs) and authorized test centers in Michigan, as well as at any of hundreds of PPCs throughout the United States, and in over 165 countries.

Conclusion

This Technical Manual presents information concerning test development, updating, administration, scoring, and reporting activities conducted for the Michigan Test for Teacher Certification program since 1992. The report reviews the steps that were taken by the Michigan Department of Education and Evaluation Systems to establish and verify the validity and reliability of the tests included in the MTTC program.