Survey Sins of the Times
It is that time of the year again, where the anxious gullible is fed with enough printed material soaked in vague statistics based on “national poll” surveys that either conclude sometimes correctly with wrong reasons or conclude wrongly with correct reasons or is plain inconclusive and correctly so, because of the method of the survey.
One such survey has already been given the treatment by Prof. Madhukar Shukla of XLRI in his blog alternate perspective. What follows is not as hilarious as that one, but only a sober set of mundane doubts that I have.
I came to know of Science of the Times, the recent Outlook magazine survey for top 100 engineering colleges and top 25 medical colleges through a post in the nanopolitan, a blog by Dr. Abinandhan from IISc. By reading the cover story and the methodology of survey, one can form one’s opinion without having to read this post any further for my opinion.
From the methodology one could learn that
Cfore conducted the study in two phases. In the first, a pilot survey was conducted among students, faculty members and industry representatives to finalize the main parameters. Based on this, it identified five factors�intellectual capital, infrastructure and facilities, pedagogic systems and processes, industry interface and placements.
What is “intellectual capital” in the above survey method? How is it objectively defined and quantified for this survey to rank a particular institute?
If it is the “knowledge content” of the faculty of that institute, which could be measured by how many of them hold Ph. D. degrees, then as faculties of most of these engineering institutes would agree, a Ph. D. is not necessary for teaching the UG courses. I myself have been a faculty in one such engineering institute for three years, before my Ph. D. degree, but with reasonable success – I know this because my increments were based then on my teacher evaluation. So, how are institutions that actually have good UG faculty without Ph. D.s treated in this survey?
Next, let us say the “infrastructure and facilities” are measured by some standardized scale. One institute can prescribe a good syllabus for the UG program and may lack the infrastructure (lab equipment and machines, computers, classrooms with proper teaching aids etc.) to successfully impart the syllabus to the students. So it may not get a high rank in this category. Fair enough. But there are institutions that deliberately reduce the syllabus. For instance, a specific experiment may be supplemented with another simpler one or altogether cancelled to suit the (non)available infrastructure and (lack of) facilities. But on paper, this would mean that the infrastructure of that institute is “adequate” to teach the “prescribed” syllabus. How is this factored into this survey?
If “pedagogic systems and processes” include syllabus content and how it is taught (hopefully), how are these things quantified objectively?
The success of an institution lies solely with its output, in this case, UG degree holders educated in engineering. If the UG graduates are “successful” then the institute is successful. The only glitch is how to define success for the UG graduates. One measure is to find out whether they get a job after they graduate (or even before they do). In this context, I think the “placements” section is the only place where the actual worth of an institute is being measured. But this depends on various factors including past performance of an institute, the relevance of a particular UG program for that year’s job market, the job market itself for that year, etc. How are these factored into the survey?
Proceeding with the methodology of the survey we further learn that
In the second phase, a semi-structured questionnaire was designed and given to senior faculty members in various institutes who had more than ten years of teaching experience in different states.
Firstly, what is special about this “senior faculty” population? Has any statistics been done on how many of those “senior faculty” consulted, actually have taught UG program courses for “more than 10 years”? On the other hand, why not the opinions of “junior” faculty, who actually teach most of the UG programs in most of these institutes need not be considered?
Secondly, what about institutions with no “senior faculty” as defined by this survey, but are still producing UG graduates with reasonable “success” (they get jobs)?
Thirdly, what is “semi-structured questionnaire”? Does this mean the other remaining “semi” is “unstructured”? Why not simply make it known to the public, what the questions asked in this questionnaire are? If it is too detailed, it can be put in the internet.
Further, the methodology suggests
The respondents were asked to give weightages to different parameters and evaluate the institutes that they were aware of on a ten-point scale. The rating that the respondents gave to their own institutes was not considered.
“The respondents were asked to…..evaluate the institutes that they were aware of…” (bold-font mine)
Wait a minute: Does this mean if some institutes are not heard of or eluded the memory of the “senior faculties” at the time of the survey, they didn’t make it into the survey board at all? This is, a wee bit (only a wee bit, I assure you), confusing for me.
If I am the “senior faculty”, and since the rating that I give for my institute is not considered anyway, I could as well consciously put down any “rival institute” I chose to, by not having them in my top 10 at least. Before jumping at me that I am claiming all “senior faculty” are liars, just answer my implied question about the survey of how it avoids “human fudge factors” in an objective analysis.
Proceeding with this argument, if I were not from the IITs, I might just choose the IITs as top institutes rather than the non-IITs, since, I can safely assume that everyone else in my situation will do the same. It is no surprise then, that the IITs top the list.
The hierarchy amongst them (IITs) in the top list could as well be explained by the random perturbation in the opinions of the consulted senior faculties among the IITs itself, as the “senior faculty” from most of the rest of the institutes are going to anyway point IITs as the best in India (and so essentially doesn’t contribute to the survey to determine the top 10 at least). This argument could be proved if the survey is taken again exactly with the same population. Chances are high that there could be a redistribution in the first five, but all of them will again be IITs.
Finally, the methodology states
In all, 208 faculty members participated in the survey. The sum total of average rating score that each institute got against different parameters was used to rank the institutes. Only the engineering colleges that offered under-graduate programmes and that were evaluated by at least 10 senior faculty members got into the final list.
The cover story itself points out “there are 1,346 engineering colleges approved by the All India Council for Technical Education”. If one were to consider only 10 “senior faculty” in each of these institutes, their total population is 13460, out of which 208 is about 1.5 percentage (if it is only 1 “senior faculty” per engineering college then 208 is about 15.5 percentage and my question still is valid). What scientific statistical method is employed to pick 208 as the representative “statistically significant” group to conduct this survey?
If a magazine in a few pages cannot do a survey with all of these details in place, then why should one believe the results of the survey alone?
On the other hand, perhaps all of these questions are taken care of by the Cfore group, which conducted the survey. May be they or Outlook magazine can release the appropriate data and details in the internet so that credibility is given to their commendable efforts to provide the public with useful information and silence nit pickers and nosy parkers with some semblance of common sense.
As I am in a profession where as an academic exercise my very existence is sometimes suspected, I am unable to watch the nebulous clouds of reason serenade past my youthful and immature hot-head. Hence, I made the above post with questions about the method of survey (and only that). If most of them have been or will be answered by Cfore or Outlook, no, need not be to my satisfaction, but to the satisfaction of any objective standard, I can start rejoicing.
After all, the survey reports my department to be the top in India (and I believe in that, with or without this survey).
Tags: academics, aicte, cfore, education, engineering ranking, e-school ranking, iit, outlook, survey, top Indian engineering colleges
only for engineering and medicine, but also for fields such as law, fashion technology and hotel management. Like last year (or, perhaps, like every year), take it with a sack of salt. If you need to be convinced, all I need to do is to direct you to Arunn’s post about last year’s rankings. This year too, this issue of Outlook carries a couple of articles of interest. One of them is about the role of private sector in higher ed. The other one is about how the middle class views higher ed
nanopolitan
February 9, 2010 at 8:17 pm
Good fisking, Arunn. Thanks.
Asking 200 odd ’senior’ professors with a ’semi’ structured questionnaire is possibly the worst way of rating colleges and universities.
With all the trouble that several colleges have run themselves into (Sathyabama, SRM), they still find a place in the top 100 list!
If we take just the colleges in a state (say, Tamil Nadu), is the Outlook’s list consistent with the top colleges in terms of the academic credentials? For example, shouldn’t we expect colleges like SSN and Velammal in the list ahead of some othe others?
Abi
July 11, 2006 at 8:50 am
Thanks Abi for the comments:
These kind of surveys survive because people like us don’t find the time to question them carefully before trusting them out of our decency and goodwill.
Arunn
July 11, 2006 at 10:05 am
I do not understand why we need this constant endorsement from other sources (however dubious they are) to tell how we are doing. Surely, we know that ourselves.
V. Babu
July 11, 2006 at 11:34 am
Unfortunately, how many of those who read “Outlook” will read this blog? I think, we got to find out a better way of reaching out to people. Why don’t you write a letter to the editor of Outlook? Hopefully, he will publish it.
http://sskatte.blogspot.com
drkatte
July 11, 2006 at 4:55 pm
Unfortunately, how many of those who read “Outlook” will read this blog? I think, we got to find out a better way of reaching out to people. Why don’t you write a letter to the editor of Outlook? Hopefully, he will publish it.
http://sskatte.blogspot.com
drkatte
July 11, 2006 at 4:55 pm
Dr. Babu: Thanks for your comments.
As I say it in the last sentence of my post, Yes, we don’t need any outside endorsement to tell how we are doing – as long as we have the ability to be self-critical and correct ourselves from time to time for academic betterment.
Arunn
July 11, 2006 at 5:17 pm
Dr. Katte: Thanks for your comments.
Agreed that not many who read Outlook may read this blog. So, kindly, do spread the word in the blogosphere, if you want to, by advertising this post at your blog! Then at least, most of them who read blogs, need not have to read Outlook ever, anymore…;)
On a serious note: yes I am considering writing an article to the “open page”. Let us see.
Arunn
July 11, 2006 at 5:23 pm
I don’t think the conclusions of the survey are very important.
IIT’s are able to attract students, for whatever reason, and that is not going to change, I think, for some years to come. Whatever our views, other colleges have got to go a long way to get where the IIT’s are right now. May be IITs will be ahead by then. That depends only on the quality of the faculty and how interesting and useful the courses are made out to be.
All surveys are meaningful if there are hundreds of institutions of comparable quality.
In India, IITs are looked up to because the entry is (was) solely through merit and still affordable. Once these two conditions change we will have to worry about Outlook Survey or any other Survey to compete in the market place.
s p venkateshan
July 11, 2006 at 6:08 pm
Dear Prof. Venkateshan: Thrilled to have you comment here.
In India, IITs are looked up to because the entry is (was) solely through merit and still affordable. Once these two conditions change we will have to worry about Outlook Survey or any other Survey to compete in the market place.
That observation was thought provoking…Thanks!
Arunn
July 11, 2006 at 6:15 pm
[...] Patrix | Media, Education What is intellectual capital in the above survey method? How is it objectively defined and quantified for this survey to rank a particular institute? If it is the knowledge content of the faculty of that institute, which could be measured by how many of them hold Ph. D. degrees, then as faculties of most of these engineering institutes would agree, a Ph. D. is not necessary for teaching the UG courses. [...]
DesiPundit » Archives » Survey Sins of the Times
July 12, 2006 at 2:43 am
Yes, Outlook does this and India Today has been doing it for about ten years. Dataquest does this too.
All that these ‘rankings’ do is survey some select bunch of people. you know the rankings are dubious when you see Madras Presidency College figuring in a national top ten list. I doubt you will find many locals who will think so highly of this college.
I used to write a strong letter to the editor every year after India Todays ratings issue, but i have given up.
To their credit, India Today started sending questionnaires to the institutions rated, asking for specific data. Unfortunately, this ‘factual ranking’ was given low weightage and the survey result or ‘perception’ was given major weightage.
Same kind of rankings are done for business schools by even more sources – the stakes are higher there since there is more money to be made. So, we even have a non-aicte-recognized institute (which advertises prominently in the newspapers for their ’superiror to mba’ program, with the MBA in big bold letters and the superiror to in fine print, of course), which publishes its own business magazine. One guess which institute is ranked as best in the country, ahead even of IIM’s, by this magazine. and, of course, this magazines rating is mentioned in those full page advertisements. At least, most (all?) IIM’s now refuse to participate in these surveys.
We should stop participating too, till such a time as the ‘ratings’ are based on verifiable facts, like what is done by US News and World Report.
Prof. P. Sriram
Prof. P. Sriram
July 12, 2006 at 12:23 pm
Statisticians must consider the output as measure of standard. How many engineers are placed in their own field, should matter most. I can see many non-computer engineers getting placed at IT companies which means the college/university lacks in training the engineers to qualify in their field of specialisation.
Srinivasa Ramanujam
July 12, 2006 at 7:58 pm
Though overjoyed at the appearance of my to-be-college in the list at all (and ranked higher than SJCE or Bangalore Institute of Tech.), I must say that the fact that RV was at no.40 and lower than JNTU and Osmania was quite surprising. The criterion was strange and sketchy and too much credence was given to name, which perhaps resulted in BITS low rank of 6. When IIT profs don’t take this seriously, who am I, a half-wit philosophysicist, to comment?
Harish Alagappa
July 13, 2006 at 8:43 pm
But in the list prepared atleast one institute is not running degree courses and is a coaching centre
ramamurthy
November 22, 2006 at 6:59 pm
goog job
you write exactly that points which
came into my mind when i read outlook survey
first of all this survey is purley perception based
survey but what are the factors on which perception is based
it is cleared that it is a brand image ranking of colleges
india today survey follow a methodology which give 70%
weightage to perceptual score based on information
collected from website of institute published reports
& achivement of faculty & students in last year
parameters> reputaion,curriculam,quality of academic input,
student care,admission procedure,primary
infrastructure,job prospects
score given by panel of 480 experts from 10 metros
30& weightage to factual score based on information
like placement,student/faculty, quality of faculty
annual spending on 1 student etc.
if we seeabove factors we found the outlook survey absolutley
wrong
vishal
November 24, 2006 at 8:40 pm
[...] Ever since I heard of the wicked witch who cursed the beautiful princess to sleep for a hundred years, I have been wondering if there is a specific gender-connection to sleep? The answers to this question are varied. Most answers come from surveys such as this and this, and at nOnoscience, we tend to be a little skeptical about surveys. [...]
nOnoscience » Blog Archive » Hush !
June 21, 2007 at 2:07 pm