Tag Archives: peer-review

Knowledge production pipeline

There is an emerging problem within the scientific community – its rapid growth together with advancement of technology increases information flow to unbearable levels. In these conditions existing system for research evaluation is becoming more and more obsolete, creating a bottleneck in the “knowledge production pipeline”, which results in the loss of considerable amount of data. To illustrate this point “knowledge production” can be formalized as following:

Knowledge production pipeline

1) Science is a Machine operating the “knowledge production pipeline”. It extracts, filters and refines the information from the Universe.
2) The end product of the pipeline is structured Knowledge ready to be “consumed” by society, industry or by the Science Machine itself (knowledge reuse creates positive feedback).
3) Pipeline consists of four operational segments, represented by the corresponding sets of tools: extraction of information, documentation, evaluation and knowledge dissemination.

The main bottleneck is created in the segment of research evaluation. Scientists (operators of the pipeline) are still performing this process manually. Increased automation of the preceding “extraction machinery” and the shift to electronic “documentation machinery” results in disproportion between the amount of operators and amount of information to be handled. In other words, while we are perfecting tools for knowledge extraction (thus increasing the information flow), the machinery required for knowledge evaluation is worn out, loosing ability to handle this flow.

Another problem refers to inefficiency of the existing documentation tools – paper laboratory journals and local electronic data storages provide very limited access. In fact, this problem also points to the obsoleteness of the science evaluation practices: since no credit is given for the knowledge bypassing the peer-reviewed publishing process, scientists lack incentive to make negative results and «raw» data publicly available.

So essentially everything boils down to the necessity of delivering new metrics for research activities and partially automating the evaluation process. This does not mean elimination of the peer-review, but increase in its efficiency.

And, beyond everything else, academia has to start giving credit to researchers performing peer-review and make its results publicly available, thereby encouraging this type of scientific activity and removing bias and injustice from the process.

~ ~ ~

More reading:

~ ~ ~ ~ ~

Submit to:  Bookmark at del.icio.us | Digg it! | Stumble It! | Bookmark at Technorati | Bookmark at Reddit | Bookmark at Google | Bookmark at Facebook | Bookmark at Windows Live

Reblog this post [with Zemanta]

Mitigation of the crisis in science

In perspective of the topic discussed here:

The Real Science Crisis: Bleak Prospects for Young Researchers
Tight budgets, scarce jobs, and stalled reforms push students away from scientific careers.

It might sound presumptuous, but in my opinion policy makers are looking for solutions in the wrong places.

Departments and students must recognize that the majority of science doctorate recipients no longer become professors, and that realization should cause a shift in the culture and practice of graduate education. “There’s a mismatch between the opportunities available to students as they complete their work and their expectations and the nature of their training along the way.”

Indeed, as a rule, tenured professor is the only stable academic position, and amount of those is strictly limited. But instead of blaming the educational system for producing too many PhDs and inventing programs to prepare graduates for the job market outside of academia, one could simply devise a new type of a “tenure” academic position. A stable intermediate between a PostDoc and a Professor – with small budgets and limited responsibility, but stable! This would secure a person who is willing to stay in academia, but did not acquire enough luck to reach Professor level. In modern world of rapidly evolving methodology and extensive interdisciplinary research, those people will be of much higher value than one might expect. In fact, given higher efficiency of knowledge dissemination within small research groups, this could also resolve the problem with quality of postgraduate education.

The second problem is harder to spot and uneasy to accept. The source of it lies within the existing “peer-based” evaluation system for science and research, which forms the core of publishing and funding mechanisms. Quotation marks show up for the reason of “peer-based” in many cases functioning in celebrity-based and celebrity-biased manner.
This evaluation system worked fairly well with limited number of scientists, when global awareness of ongoing research was possible in most of the scientific disciplines. However, once electronic media accelerated the rate of scientific discovery, diversifying research world and increasing information flow to unbearable levels, the existing “peer-based” methods became obsolete and destructive. And while policy makers are trying to solve problems by scaling up the funding, the system cracks:

…Congress doubled the NIH budget from $13.6-billion to $27.3-billion between 1998 and 2003. Since then, the agency’s appropriations have not kept pace with inflation, which has eroded the actual amount available for research.

Beyond inflation, immense amount of those resources vanishes in the redundancy of our efforts. Existing evaluation system discourages sharing of knowledge and expertise, leading to unhealthy competition, secrecy, data falsification, and as a result – redundancy of research. One really has to restructure the whole system and ideology behind it, focusing on elimination of pyramidal hierarchy, giving more power to young scientists, introducing new methods for efficient knowledge dissemination and encouraging sharing of research artifacts. This restructuring in turn will release a lot of resources, which can be rerouted back to increase the amount of stable research positions available within academy.

///

There is also a bit of discussion on this topic going at friendfeed.

~ ~ ~ ~ ~

Submit to:  Bookmark at del.icio.us | Digg it! | Stumble It! | Bookmark at Technorati | Bookmark at Google | Bookmark at Facebook | Bookmark at Windows Live