Talk:Computer-adaptive testing

From Wikipedia, the free encyclopedia

Contents 1 CAT question 2 Citations 2.1 Summary of 1 Dec 2006 edits 3 Moved page

[edit] CAT question

I'm proposing Computer Adaptive Tests for final exams in my University, but I find some coarse reaction. They argument that this kind of tests punishes the most intelligent people and unmotivate them. Has anyone had an experience at implementing CATs for academic courses? -- Mario Montoya (User:200.23.145.23)

In order to implement an adpative test, you must first have scaled the pool using item response theory. The sample size typically required for IRT scaling is at least 200 and preferrably several hundreds or thousands of people. As a consequence, it is uncommon and generally impractical for use in academic settings unless there is a dedicated staff who produce one or a few exams each year and who can manage the CAT. The way this usually evolves is to first administer the exam on computer for a period of time while the database of questions and answers accumulates. Then the scaling is performed and an adaptive version of the exam is introduced.

I haven't seen any studies that reinforce the notion that CAT's punish the most intelligent students although CAT introduces a new administration method that many students dislike. For example, item review and modification are generally disallowed in CAT because if you get an easier item, then you know that you got the previous item wrong. If you can backup and correct that answerm then your score will be biased. In fact, if item review is allowed then here's how you subvert the exam: Intentionally answer each item wrong until you finish the exam. The CAT will have served the easiest possible items to you. Now go back and answer all the easy items correctly. If you actually get 100% correct, then you should pass but if you don't then you can take them to court and have them explain to a judge why, with 100% correct, you did not pass their exam! (This is, from memory, Howard Wainer's argument against allowing item review.) Many students are frustrated that they cannot skip harder questions and return to them later.

Why would you want to introduce CAT exams in an academic setting? CAT is really only helpful if you need extremely reliable tests over a broad range of ability (e.g., for categorization or on a broad entrance exam like the GRE). CAT is too much work and, as Wainer shows, too much of a security risk for general usage. -- Amead

[edit] Citations

I think the writers of this article should not only cite the books they used in writing the article (which they seem to have done), but also that they cite which sentences/paragraphs came from which texts. As I understand, citing sources is not just listing the book, but also citing the source in context, that is, in the article text. For example, in APA style, if I were to say "Computer-adaptive tests were first used in 1980," I would follow the sentence by citing the name of the author and the year of publication, like so: "(Michels, 2000)." In this article, nothing comparable to that is done. If I question the factual accuracy of a sentence or paragraph, I don't know which source to look at in the reference list. This is why many articles, such as Normal curve equivalent, use a reference style that cites particular sources for the statements in question. That is why I added the {{references}} template to the top of the article. There may be references cited at the end of the article, but not within the text. – Chris53516 ^(Talk) 16:38, 20 November 2006 (UTC)

Chris, do you question the accuracy of any sentence in this article? I wrote most of the words in this article without reference to any work because this article relates common psychometric knowledge. There is nothing to cite (well, I should dig up the van der Linder citation for shadow testing). I disagree with your interpretation of the APA standards... feel free to cite the sentences in the 5TH ed. manual which supports your perspective. I might also be persuaded if you could come up with an example written in APA format (the Normal curve equivalent article is not). But bear in mind that APA manuscripts generally are research or review documents requiring support for assertions and conclusions--APA format was not intended for explaining common knowledge.Amead 20:20, 27 November 2006 (UTC)

According to Wikipedia policy, information must be verifiable and we must cite our sources. An encyclopedia is not written from common knowledge, but from research. See Wikipedia:Verifiability and Wikipedia:Citing sources. We need to find better sources of verifiable information, other than a particular user's views, common knowledge, or opinion.

I did not say that it needed to be in APA, that was just an example of a good way to cite sources. I used an example in a sentence above. I never said that the NCE article was in APA style, I said it used a reference style. The NCE article uses references that links a statement to a source listed at the end of the article. That was my point in using that page as an example. It uses another way of citing sources. I like and use it because it makes every statement verifiable by linking it to a citation or an online reference. Both APA and this "Wikipedia" style cite sources for every claim. And that's what we need to do in this article to make it better. So, stop removing the template until this is done. – Chris53516 ^(Talk) 20:38, 27 November 2006 (UTC)

Here are some quotes from the above policies: "Editors should provide a reliable source for material that is challenged or likely to be challenged, or it may be removed" (Wikipedia:Verifiability). I have provided the challenge. Also, "Any material that is challenged and for which no source is provided may be removed by any editor" (Wikipedia:Citing sources). I'd rather not remove dubious content. – Chris53516 ^(Talk) 20:50, 27 November 2006 (UTC)

Chris, you win. I don't give a crap about this article anymore. Amead 05:59, 28 November 2006 (UTC)

There's no need to be vulgar. I am simply trying to improve the page. – Chris53516 ^(Talk) 14:27, 28 November 2006 (UTC)

Thanks, Iulus Ascanius, for adding references. I would like to convert the references to a Wiki-style so readers can directly link to the citations. I'll probably get to it this weekend. — Chris53516 ^(Talk) 21:09, 30 November 2006 (UTC)

[edit] Summary of 1 Dec 2006 edits

Basically, I took all of the references added by Iulus Ascanius and put them in a Wikipedia reference style so readers can link right to the citation. It's a bit messy in code, but it looks nice in the end. I did some minor edits too and added some {{unreferenced}} templates for sections and paragraphs still needing references, but basically that's all I did in the multiple edits I did today. See here for the full edit history. — Chris53516 ^(Talk) 21:43, 1 December 2006 (UTC)

[edit] Moved page

Do NOT move this page without discussing it first. That is inappropriate. — Chris53516 ^(Talk) 02:22, 7 December 2006 (UTC)

Sorry about the move - didn't know the rules. I just believe it is more appropriate under the name "Computerized adaptive testing," as this is what is more often called. A quick scan of the references supports this. "Computer-adaptive" is not incorrect, just not as common. Anyone else have thoughts on that (though it doesn't appear that anyone else is contributing material at this point)?

I've tried to add more technical information for readers who are interested in the inner workings. Some figures would be helpful, but I'll have to figure out how to do that first.

-- Iulus Ascanius 18:11, 13 December 2006 (UTC)

I oppose moving this article at all. Computer-adaptive testing is the accepted terminology that appears in research, especially item-response theory research. Any other terminology came from this terminology. — Chris53516 ^(Talk) 18:21, 13 December 2006 (UTC)

It is hardly "accepted" as the term in research. A complete list of CAT research ([1]) has the term "computer-adaptive" used in titles only 43 times - "computerized adaptive" is used approximately 540 times! And it's impossible to argue with a 'sample' of that size. Computerized adaptive testing became the accepted terminology when Reckase's use of "tailored testing" died out.

Likewise, there is no hyphen in "item-response theory." It is item response theory.

--Iulus Ascanius 19:07, 13 December 2006 (UTC)