How Much Do Rankings Matter?

In this year's THES-QS World University Rankings 2007 of the world's top 200 universities, there have been several large changes from last year - the different grading criteria leading to the large shifts in positions of several top universities, including our beloved NUS and NTU.

33= 19= National University of SINGAPORE Singapore

69 61= NANYANG Technological University Singapore
73 77 University of ILLINOIS United States

NUS dropped from 19 to 33, and NTU dropped from 61 to 69. UIUC made slight progress from 77 to 73. (Darn! I should have stayed in Singapore instead of freezing my butt off for 3 years.)

Just last year, NUS was proudly embracing the honor of being the top 20 universities in the world. In fact, as of today, the reference to the 2006 rankings is still on its front page!


Then today's print edition (it cannot be found on the web edition) of 早报 quoted NUS President Prof Shih as saying that the reputation and stature of universities are built over tens, even hundreds of years, and should not be change significantly over just one year. He also reportedly said that NUS would not be taking this year's ranking seriously (article to come). This reminds me of Gold 90.5FM's slogan "Hear only the good stuff!" How typically Singaporean to welcome and applaud only glowing praise from others while downplaying negative reports or criticism.

The same 早报 report also stated, however, that the judging criteria have also changed, so the change in rankings should not come as a surprise. Another accusation pointed out the fact that QS is a British company and this year's report had UK universities occupying 3 of the top 5 (up from 2 in 2006). It's also notable that UK's UCL leap-frogged from 25 to 9 while traditional American powerhouses Stanford and UC Berkeley dropped from the top 10 into the 20's.

How reliable are University rankings anyway, and how seriously should we view them? In Singapore pre-tertiary schools (primary, secondary, JC), standardized tests like PSLE, GCE 'O' and GCE 'A' levels can be used for meaningful comparisons. In this respect, the rankings are meaningful and easy to understand.

At the university level, things get more complicated. The lack of standardized examinations and curriculum make for difficult objective comparisons. Rankings can only be based on subjective measures like opinion polls, where people in industry are asked to rate universities, and faculty are asked to rate other institutions. To inject a slight measure of objectivity,

• the percentage of faculty with PhD's,
• the amount (proportion) of money spent on research,
• the number of papers published (the scale of the conferences at which these papers are presented also matter),
• the number of references from papers published by other institutions

are also taken into account. At this level, results can be influenced by manipulation of numbers without much significant real world benefit to improve a university's ranking. For example, more 'popular' or productive research programs can receive more funding and administrative support, never mind if the papers produced are too esoteric, impractical, or too common-place.

Take a look at America's Best Colleges from the US News and World Report, where the methodology is heavily based on opinion polls, where college deans are asked to evaluate other schools on a scale from 1 to 5.

All that said, the usual wisdom applies - rankings should NOT be taken at face value - people should think about what they are looking for in a university, performing independent research to evaluate the schools. This applies to high-school graduates choosing a college, PhD's looking for research/teaching jobs in university, government agencies or venture capitalist evaluating schools for giving grants/funding, companies looking for academic partners for research collaboration, equipment or monetary donations, etc.

It is important to look at the specific area of interest when choosing a school. For example, a university that is ranked 73 (overall) may actually have one of the best engineering programs in the United States, being consistently in the top 5 among Electrical Engineering and Computer Science programs and being one of the feeder schools of fresh engineers to companies like Intel, AMD, HP, Texas Instruments, etc. It may also have one of the best Agricultural or Mass Communications departments. Or a relatively unknown university in California may have one of the most exciting solid state electronics research departments in the world.

Unfortunately, in 'meritocratic' Singapore, (good) rankings matter most, bad rankings are ignored, and objective evaluations fall by the wayside.

Welcome to Singapore!

Update: I just found this report, linked off Wikipedia, by the Shanghai Jiaotong University (上海交通大学) that puts UIUC at #26 in the Top 100 universities in 2007. Check out their ranking methodology. It's interesting why our Singapore institutions did not quote this report at all. NUS and NTU are ranked #110 and #329 respectively. You can download the full list here.