I just realized that there have been only 35 posts this year (not including this one), which is a very low number. I usually try to post during the weekends, but making up for missed weekends is a problem. Well I want to get the number to at least 40... so hopefully I'll make that number to 40 in the next ten days. :) Also, as is the tradition on the blog, I would like to invite all my readers to submit a guest post for the year-end post. Criteria: you don't have a blog or if you do, you haven't written there in at least 6 months. Well, now is the chance to scribble away at those creative quills of yours! :)
----------------------------------
I read Rashmi's blog post titled CAT: Restore the faith a few days back. Instead of posting a lengthy comment there, I thought I'd make my own blog post. Basically, I do not agree with her conclusions and have some counter-arguments. I think the heart of the CAT administrators was in the right place - they just lost sight of a few practical problems and messed up some things the first time. Please go and read that post first and also the comments if you have time - mine is rather a follow-up to that one.
Let me first begin with an analogy from cryptanalysis, or "breaking the code" as we commonly refer to it. Say there is an apartment complex, with a common gate and a number keypad combination lock in front. Every apartment in the complex is given a combination of digits and all such combinations open the gate. Now, if the combination is only 2 digits long, and there are even say 30 apartments in the complex with each having a unique combination, then 30 out of 100 possible combinations are already valid keys for the gate to open! If you are a random hawker/seller/murderer/friend/ex-boyfriend and want to get inside the apartment complex but don't know the key combination, you can just walk up there and chances are within 5-6 tries you will be able to punch in a key code that opens the gate!
So the obvious next question is: how do you prevent it? You do three things: one, you make the key combination pretty long and therefore difficult to randomly guess (say about 7 digits). Two: you install a video camera to check that someone can't stand around and keep trying codes all day. Three: you keep changing the codes every now and then so that even accidental leaks are controlled. In geek speak, this would mean increasing the size of the sample space (i.e. long combination sequence), making the cryptanalysis of the cipher-text too expensive to crack for the plaintext to be of any use after it has been decoded, and finally rotating through the sample space to add the dispersion of the combination sample points along the time dimension as well. Oh never mind - you simply know that 7 digit codes with a video camera installed and frequent key-combination changes would work. :)
So all that is very good, but how does it apply to CAT and computerized tests? Well, first of all, Rashmi quotes a bunch of surveys that proved that the ETS-organized tests were cracked by dedicated people taking the tests to just remember the question bank. Yes it is possible, of course, but you need to have access to the system numerous times and then you also need to be able to compile it together, advertise it, sell it, get people enough time to read it and practice, and then those people are to take the test. It worked for the GMAT because you can take the test whenever you want, as many times as you want. This luxury of the time window was sealed off through the administration of the CAT by making this test for only 10 days. This is basically the video camera analogy: you cannot keep trying and compiling the questions and then hope to be able to sell it to the right person who has to read and practice and remember 2,00,000 question-answers in a day hoping that 20 out of those 2,00,000 will be "common", i.e. repeat themselves. And then, they hope they don't get caught. What I am saying is, given the short window and a sufficiently large number of questions (most of which, hopefully, will change next year), the CAT can't be expected to be cracked in a 10-day window.
Second, how about questions repeating? Well, how do you judge if a question is good, in the first place, then? You give it to a large class of students and if only 5 out of 100 are able to solve it, then it is a fairly difficult question (of difficulty level 10, say). And then if there is another question which 90 out of 100 people are able to solve, then it can be deemed easy (of difficulty level 2, say). Effectively, this way you build up a question bank having all sorts of questions from grade 10 (most difficult) to grade 1 (easiest). Now, when you see that people who are consistently solving a difficulty level 7 question are struggling with a new question you are about to add to your bank, you can assume that it is of level 8 at least. That way, you don't even need to run every question past every person to find out where it stands. If just ten people of problem-solving-skill level (PSS level) 7 have struggled with one particular question at different times in the past, you can give the same question to everyone taking the test on a particular day and whoever solves it successfully is definitely a candidate for PSS level 7. What I am saying is simply: repetition is good. Repeated questions can help you classify and grade test-takers: exactly the purpose of the computerized and automated test. And if you say that the stakes are too high for something like CAT, then just make your key combination long! That is, increase the number of questions. Like in the GRE, sometimes you may be randomly administered an entire section that is being used just to build new question banks and will not affect your final score... but you cannot tell which is which: you have to take them all seriously. In our case in the CAT, you have to administer enough questions to enough volunteers, test takers, and draw from other global question banks and collaborate with such test administrators around the globe. If building the question bank is your problem, then crowdsourcing is your solution. That should not prevent the Common Admission Test from turning into a Computer Adaptive Test.
Yes of course some things were done absolutely stupidly: for example the interface being different at different times of the test, or the fact that no scalability tests were done and the software was just launched one fine morning. Marketing the word "computerized" into the CAT definitely drove a big chunk of the hasty deployment. The management had taken the "paper" out of the equation for the most part - a more scalable solution, definitely. If you are wondering whether the CAT results are relevant at all in determining your aptitude for business schools, you are asking the broader question that is outside the scope of the question: "Should the CAT be turned into a computerized test?". I think yes, the CAT should be computerized. No it wasn't implemented properly this time, but yes its future is definitely a computer-based avatar.
----------------------------------
I read Rashmi's blog post titled CAT: Restore the faith a few days back. Instead of posting a lengthy comment there, I thought I'd make my own blog post. Basically, I do not agree with her conclusions and have some counter-arguments. I think the heart of the CAT administrators was in the right place - they just lost sight of a few practical problems and messed up some things the first time. Please go and read that post first and also the comments if you have time - mine is rather a follow-up to that one.
Let me first begin with an analogy from cryptanalysis, or "breaking the code" as we commonly refer to it. Say there is an apartment complex, with a common gate and a number keypad combination lock in front. Every apartment in the complex is given a combination of digits and all such combinations open the gate. Now, if the combination is only 2 digits long, and there are even say 30 apartments in the complex with each having a unique combination, then 30 out of 100 possible combinations are already valid keys for the gate to open! If you are a random hawker/seller/murderer/friend/ex-boyfriend and want to get inside the apartment complex but don't know the key combination, you can just walk up there and chances are within 5-6 tries you will be able to punch in a key code that opens the gate!
So the obvious next question is: how do you prevent it? You do three things: one, you make the key combination pretty long and therefore difficult to randomly guess (say about 7 digits). Two: you install a video camera to check that someone can't stand around and keep trying codes all day. Three: you keep changing the codes every now and then so that even accidental leaks are controlled. In geek speak, this would mean increasing the size of the sample space (i.e. long combination sequence), making the cryptanalysis of the cipher-text too expensive to crack for the plaintext to be of any use after it has been decoded, and finally rotating through the sample space to add the dispersion of the combination sample points along the time dimension as well. Oh never mind - you simply know that 7 digit codes with a video camera installed and frequent key-combination changes would work. :)
So all that is very good, but how does it apply to CAT and computerized tests? Well, first of all, Rashmi quotes a bunch of surveys that proved that the ETS-organized tests were cracked by dedicated people taking the tests to just remember the question bank. Yes it is possible, of course, but you need to have access to the system numerous times and then you also need to be able to compile it together, advertise it, sell it, get people enough time to read it and practice, and then those people are to take the test. It worked for the GMAT because you can take the test whenever you want, as many times as you want. This luxury of the time window was sealed off through the administration of the CAT by making this test for only 10 days. This is basically the video camera analogy: you cannot keep trying and compiling the questions and then hope to be able to sell it to the right person who has to read and practice and remember 2,00,000 question-answers in a day hoping that 20 out of those 2,00,000 will be "common", i.e. repeat themselves. And then, they hope they don't get caught. What I am saying is, given the short window and a sufficiently large number of questions (most of which, hopefully, will change next year), the CAT can't be expected to be cracked in a 10-day window.
Second, how about questions repeating? Well, how do you judge if a question is good, in the first place, then? You give it to a large class of students and if only 5 out of 100 are able to solve it, then it is a fairly difficult question (of difficulty level 10, say). And then if there is another question which 90 out of 100 people are able to solve, then it can be deemed easy (of difficulty level 2, say). Effectively, this way you build up a question bank having all sorts of questions from grade 10 (most difficult) to grade 1 (easiest). Now, when you see that people who are consistently solving a difficulty level 7 question are struggling with a new question you are about to add to your bank, you can assume that it is of level 8 at least. That way, you don't even need to run every question past every person to find out where it stands. If just ten people of problem-solving-skill level (PSS level) 7 have struggled with one particular question at different times in the past, you can give the same question to everyone taking the test on a particular day and whoever solves it successfully is definitely a candidate for PSS level 7. What I am saying is simply: repetition is good. Repeated questions can help you classify and grade test-takers: exactly the purpose of the computerized and automated test. And if you say that the stakes are too high for something like CAT, then just make your key combination long! That is, increase the number of questions. Like in the GRE, sometimes you may be randomly administered an entire section that is being used just to build new question banks and will not affect your final score... but you cannot tell which is which: you have to take them all seriously. In our case in the CAT, you have to administer enough questions to enough volunteers, test takers, and draw from other global question banks and collaborate with such test administrators around the globe. If building the question bank is your problem, then crowdsourcing is your solution. That should not prevent the Common Admission Test from turning into a Computer Adaptive Test.
Yes of course some things were done absolutely stupidly: for example the interface being different at different times of the test, or the fact that no scalability tests were done and the software was just launched one fine morning. Marketing the word "computerized" into the CAT definitely drove a big chunk of the hasty deployment. The management had taken the "paper" out of the equation for the most part - a more scalable solution, definitely. If you are wondering whether the CAT results are relevant at all in determining your aptitude for business schools, you are asking the broader question that is outside the scope of the question: "Should the CAT be turned into a computerized test?". I think yes, the CAT should be computerized. No it wasn't implemented properly this time, but yes its future is definitely a computer-based avatar.
I see computerization of most competitive exams as inevitable. It saves so many time, money and paper. It also streamlines the entire process of evaluation and posting the results. The obvious and more subtle problems with technology can be sorted out relatively painlessly. It is adjusting people's attitude that can take ages.
ReplyDeleteThose who want those hand-written sheets continue to live in the past.
Online is the way to go.. whatever hurdles everyone faced this time can certainly be taken care next time by proper preparations
ReplyDeleteHari, yes of course.. good points, all of them.
ReplyDeleteShrinidhi, yes, very true.
I don't think the issue was ever about CAT going computer based. The issue was it not being conducted properly this year and that there should have been a retest only for this year. The Computer Based test can return once all the problems are sorted out.
ReplyDeleteAnkit, yeah I know. But I was actually replying to Rashmi's post here as I linked to in the beginning. I agree with your view wholeheartedly. Welcome onboard, btw!
ReplyDeleteyeah Sudipta agreed it shud b computer based avatar, but d wat glitches v faced dis tym, n den dis approximate lottery system of giving results, dis has marred d faith of students in IIM's 4 sure !!!!
ReplyDelete