As a physicist of course I believe in quantitative measurements. For example physicists have measured the charge on the electron to an accuracy of more than 8 significant figures. This is great. But electrons are very simple objects. Scientists are more complex, and so a quantitative measurement of a scientist’s work is harder to do. But this does not deter some people from trying.

There is agreement that what matters is what a scientist publishes in the peer-reviewed literature, as these publications are in the public domain and so can be assessed by everyone. Also it has received at least a cursory inspection (hopefully more) by another scientist during peer review. But how to assess these publications in a simple way that will produce a number?

The easiest commonly used way is to assess not the paper directly, but the journal it is published in, via what is called the Impact Factor of the journal. The most recent 2010 Impact Factor of journal X is calculated approximately as follows. Count the total number of references or citations in the year 2010 to papers published in X in the years 2008 and 2009. Then divide this number by the total number of papers published in journal X in 2008 and 2009.

But there are problems both with using the Impact Factor to assess a journal and with using it to assess the papers. The basic problem with using it to assess a scientist is obvious. The paper may be be poor but published in a journal with a high Impact Factor because other papers by other people in the same journal are referenced a lot.

But even using it to assess a journal can be dodgey. For instance, it can be affected by the editor. For example the editors of a small journal called Folia Phoniatrica et Logopaedica got so annoyed with the Impact Factor that they wrote a paper citing every single paper published in this journal the year before – which of course increased the Impact Factor. They were deliberately doing this to highlight a problem, which I think they did in a clear way. But the journal then got banned by the company that does the Impact Factor, which seems a bit harsh to me.

Possibly a bit more dubious is a journal called the International Journal of Nonlinear Sciences and Numerical Simulation (IJNSNS). In 2008 this had the highest impact factor in its field at 8.9, according to a fun article that looked at this by Arnold and Fowler. They looked at where the citations came from that boosted this factor. First off, the guy who cited the journal more than anyone was the journal’s editor, Prof. Ji-Huan He. Hmm.

Also, a special issue of another journal referenced papers in this journal a whopping 294 times. The editor of that special issue was one Prof Ji-Huan He. Hmm again. Another special issue of a third journal cited the journal 206 times. I probably don’t need to tell you who was guest editor of that issue, as you have probably guessed.

I guess this shows one danger of relying on a single simple number to measure complex stuff like how good a scientist, or a journal, is. This is that there are almost always ways to play the system.