Mastering the Art of Test Writing Roundtable Discussion Social Science

Mastering the Art of Test Writing Roundtable Discussion Social Science Faculty Meeting January 2010

Mastering the Art of Test Writing Mastering the Art of Test Writing Roundtable Discussion January 2010 Presenter Jesse Coraggio, Director, Academic Effectiveness Former Life Director of Test Development , SMT Director of Measurement and Test Development, Pearson Taught EDF 4430 Measurement for Teachers, USF January 8. 2009 Academic Effectiveness and Assessment 2

Purpose Mastering the Art of Test Writing Roundtable Discussion January 2010 This presentation will explain how to create effective multiple choice test questions. The presentation will provide item-writing guidelines as well as best practices to prevent students from just guessing the correct answers. January 8. 2009 Academic Effectiveness and Assessment 3

Objectives Mastering the Art of Test Writing Roundtable Discussion January 2010 Propose of a Test Advantages of Objective Tests Types of Objective tests Writing Multiple Choice Items The Test-wise Student Test Instructions Test Validity January 8. 2009 Academic Effectiveness and Assessment 4

Purpose of a Test Mastering the Art of Test Writing Roundtable Discussion January 2010 “Clearly delineate between those that know the content and those that do not.” The purpose of an assessment is to determine whether the student knows the content, not whether the student is a good test-taker. Likewise, confusing and tricky questions should be avoided to prevent incorrect responses from students who know the material. January 8. 2009 Academic Effectiveness and Assessment 5

Objective Tests Mastering the Art of Test Writing Roundtable Discussion January 2010 Measure several types of learning (also levels) Wide content, short period of time Variations for flexibility Easy to administer, score, and analyze Scored more reliability and quickly What type of learning cannot be measured? January 8. 2009 Academic Effectiveness and Assessment 6

Types of Objective Tests Mastering the Art of Test Writing Roundtable Discussion January 2010 Written-response Completion (fill-in-the-blank) Short answer Selected-response Alternative response (two options) Matching Keyed (like matching) Multiple choice January 8. 2009 Academic Effectiveness and Assessment 7

Written-response Mastering the Art of Test Writing Roundtable Discussion January 2010 Single questions/statements or clusters (stimuli) Advantages Measure several types of learning Minimizes guessing Points out student misconceptions Disadvantages Time to score Objectivity Misspelling and writing clarity Incomplete answers More than one possible correct response (novel answers) Subjectivity in grading January 8. 2009 Academic Effectiveness and Assessment 8

Completion Mastering the Art of Test Writing Roundtable Discussion January 2010 A word that that describes a person, place or thing is a . 1. 2. 3. 4. 5. 6. Remove only key words Blanks at end of statement Avoid multiple correct answers Eliminate clues Paraphrase statements Use answer sheets to simplify scoring January 8. 2009 Academic Effectiveness and Assessment 9

Short Answer Mastering the Art of Test Writing Roundtable Discussion January 2010 Briefly describe the term proper noun. Terminology – Stimulus and Response 1. Provide an appropriate blank (word (s) or sentence). 2. Specify the units (inches, dollars) 3. Ensure directions for clusters of items and appropriate for all items January 8. 2009 Academic Effectiveness and Assessment 10

Selected-response Mastering the Art of Test Writing Roundtable Discussion January 2010 Select from provided responses Advantages Measure several types of learning Measures ability to make fine distinctions Administered quickly Cover wide range of material Reliably scored Multiple scoring options (hand, computer, scanner) Disadvantages Allows guessing Distractors can be difficult to create Student misconceptions not revealed January 8. 2009 Academic Effectiveness and Assessment 11

Alternative Response Mastering the Art of Test Writing Roundtable Discussion January 2010 T F 1. A noun is a person place or thing. T F 2. An adverb describes a noun. 1. 2. 3. 4. 5. 6. Explain judgments to be made Ensure answers choices match Explain how to answer Only one idea to be judged Positive wording Avoid trickiness, clues, qualifiers January 8. 2009 Academic Effectiveness and Assessment 12

Matching Item Mastering the Art of Test Writing Roundtable Discussion Column A Person, place, or thing. Describes a person, place, or thing. January 2010 Column B a. Adjective b. Noun Terminology – premises and responses 1. Clear instructions 2. Homogenous premises 3. Homogenous responses (brief and ordered) 4. Avoid one-to-one January 8. 2009 Academic Effectiveness and Assessment 13

Keyed Response Mastering the Art of Test Writing Roundtable Discussion January 2010 Responses a. A noun b. A pronoun c. An adjective d. An adverb Person, place, or thing. Describes a person, place, or thing. Like matching items, more response options January 8. 2009 Academic Effectiveness and Assessment 14

MC Item Format Mastering the Art of Test Writing Roundtable Discussion January 2010 What is the part of speech that is used to name a person, place, or thing? A) A noun* B) A pronoun C) An adjective D) An adverb January 8. 2009 Academic Effectiveness and Assessment 15

MC Item Terminology Mastering the Art of Test Writing Roundtable Discussion January 2010 Stem: Sets the stage for the item; question or incomplete thought; should contain all the needed information to select the correct response. Options: Possible responses consisting of one and only one correct answer. Key: correct response Distractor: wrong response, plausible, but not correct, attractive to an under-prepared student January 8. 2009 Academic Effectiveness and Assessment 16

Competency Mastering the Art of Test Writing Roundtable Discussion January 2010 Items should test for the appropriate or adequate level of knowledge, skill, or ability (KSA) for the students. Assessing lower division students on graduate level material is an ‘unfair’ expectation. The competent student should do well on an assessment, items should not be written for only the top students in the class. January 8. 2009 Academic Effectiveness and Assessment 17

Clarity Mastering the Art of Test Writing Roundtable Discussion January 2010 Clear, precise item and instruction Correct grammar, punctuation, spelling Address one single issue Avoid extraneous material (teaching) One correct or clearly best answer Legible copies of exam January 8. 2009 Academic Effectiveness and Assessment 18

Bias Mastering the Art of Test Writing Roundtable Discussion January 2010 Tests should be free from bias No No No No No No January 8. 2009 stereotyping gender bias racial bias cultural bias religious bias political bias Academic Effectiveness and Assessment 19

Level of Difficulty Mastering the Art of Test Writing Roundtable Discussion January 2010 Ideally, test difficulty should be aimed a middle level of difficulty. This can not always be achieved when the subject matter is based on specific expectations (i.e, workforce area). January 8. 2009 Academic Effectiveness and Assessment 20

Level of Difficulty Mastering the Art of Test Writing Roundtable Discussion January 2010 To make a M/C item more difficult, make the stem more specific or narrow and the options more similar. To make a M/C item less difficult, make the stem more general and the options more varied. January 8. 2009 Academic Effectiveness and Assessment 21

Trivial and Trick Questions Mastering the Art of Test Writing Roundtable Discussion January 2010 Avoid trivia and tricks Avoid humorous or ludicrous responses Items should be straight forward, they should cleanly delineate those that know the material from those that do not Make sure every item has value and that it is contributing to the final score January 8. 2009 Academic Effectiveness and Assessment 22

http://www.services.unimelb.edu.au/asu/download/Study-Multiple-ChoiceExams-Flyer.pdf Test Taking Guidelines Mastering the Art of Test Writing Roundtable Discussion January 2010 When you don’t know the answer As with all exams, attempt the questions that are easiest for you first. Come back and do the hard ones later. Unless you will lose marks for an incorrect response, never leave a question blank. Make a calculated guess if you are sure you don’t know the answer. Here are some tips to help you guess ‘intelligently’. Use a process of elimination Try to narrow your choice as much as possible: which of the options is most likely to be incorrect? Ask: are options in the right range? Is the measurement unit correct? Does it sound reasonable? January 8. 2009 Academic Effectiveness and Assessment 23

http://www.services.unimelb.edu.au/asu/download/Study-Multiple-ChoiceExams-Flyer.pdf Test Taking Guidelines Mastering the Art of Test Writing Roundtable Discussion January 2010 Look for grammatical inconsistencies In extension-type questions a choice is nearly always wrong if the question and the answer do not combine to make a grammatically correct sentence. Also look for repetition of key words from the question in the responses. If words are repeated, the option is worth considering. e.g.: The apparent distance hypothesis explains b) The distance between the two parallel lines appears Be wary of options containing definitive words and generalizations Because they can’t tolerate exceptions, options containing words like ‘always’, ‘only’, ‘never’, ‘must’ tend to be incorrect more often. Similarly, options containing strong generalizations tend to be incorrect more often. January 8. 2009 Academic Effectiveness and Assessment 24

http://www.services.unimelb.edu.au/asu/download/Study-Multiple-ChoiceExams-Flyer.pdf Test Taking Guidelines Mastering the Art of Test Writing Roundtable Discussion January 2010 Be wary of options containing definitive words and generalizations Because they can’t tolerate exceptions, options containing words like ‘always’, ‘only’, ‘never’, ‘must’ tend to be incorrect more often. Similarly, options containing strong generalizations tend to be incorrect more often. Favor look-alike options If two of the alternatives are similar, give them your consideration. e.g.: A. tourism consultants B. tourists C. tourism promoters D. fairy penguins January 8. 2009 Academic Effectiveness and Assessment 25

http://www.services.unimelb.edu.au/asu/download/Study-Multiple-ChoiceExams-Flyer.pdf Test Taking Guidelines Mastering the Art of Test Writing Roundtable Discussion January 2010 Favor numbers in the mid-range If you have no idea what the real answer is, avoid extremes. Favor more inclusive options If in doubt, select the option that encompasses others. e.g.: A. an adaptive system B. a closed system C. an open system D. a controlled and responsive system E. an open and adaptive system. Please note: None of these strategies is foolproof and they do not apply equally to the different types of multiple choice questions, but they are worth considering when you would otherwise leave a blank. January 8. 2009 Academic Effectiveness and Assessment 26

Test-wise Students Mastering the Art of Test Writing Roundtable Discussion January 2010 Are familiar with item formats Use informed and educated guessing Avoid common mistakes Have testing experience Use time effectively Apply various strategies to solve different problem types January 8. 2009 Academic Effectiveness and Assessment 27

Test-wise Students Mastering the Art of Test Writing Roundtable Discussion January 2010 Vary your keys: “Always pick option ‘C’” Avoid ‘all of the above’ and ‘none of the above’ Avoid extraneous information: It may assist in answering another item Avoid item ‘bad pairs’ or ‘enemies’ Avoid clueing with the same word in the stem and the key January 8. 2009 Academic Effectiveness and Assessment 28

Test-wise Students Mastering the Art of Test Writing Roundtable Discussion January 2010 Make options similar in terms of length, grammar, and sentence structure. Different options stand out. Avoid ‘clues’. January 8. 2009 Academic Effectiveness and Assessment 29

Item Format Considerations Mastering the Art of Test Writing Roundtable Discussion January 2010 Information in the stem Avoid negatively stated stem, qualifiers Highlight qualifiers if used Avoid irrelevant symbols (“&”) and jargon Standard Set number of options (Prefer only four) Ideally, you should tie an item to reference January 8. 2009 Academic Effectiveness and Assessment 30

Test Directions Mastering the Art of Test Writing Roundtable Discussion January 2010 Highlight Directions 1. 2. 3. 4. State the skill measured. Describe any resource materials required. Describe how students are to respond. Describe any special conditions. January 8. 2009 Academic Effectiveness and Assessment 31

Ensure Test Validity Mastering the Art of Test Writing Roundtable Discussion January 2010 Congruence between items and course objectives Congruence between item and student characteristics Clarity of items Accuracy of the measures Item formatting criteria Feasibility-time, resources January 8. 2009 Academic Effectiveness and Assessment 32

Questions Mastering the Art of Test Writing Roundtable Discussion January 8. 2009 Academic Effectiveness and Assessment January 2010 33

Mastering the Art of Test Writing Roundtable Discussion Social Science Faculty Meeting January 2010