Charm etc. posted an interesting discussion on Information Entropy and Experiments. The bloggers describe an attempt to evaluate the worth of experiments on a statistical basis, comparing the results they produce against a priori expectations from theory. (See arXiv:0712.3572 from Bruce Knuteson.) They point out very perceptively that this procedure relies too much, if not entirely, on those expectations from theory. One might wonder where else we should get our expectations, but this is not the point - the most important discoveries are the ones for which there was no expectation. And after the discovery was made, theory was altered radically, therefore changing the priors. See Charm etc. for a succinct discussion.

I think there is another problem with the approach. How does one evaluate the accumulated value of crucial measurements of particle properties and interactions? How much does a factor two improvement in the W mass measurement increase the value of a Tevatron experiment? What about the Bs oscillation frequency? These measurements fit in the standard model - in fact, they are important if not crucial empirical inputs to model calculations, without which one could not move beyond the standard model. I don’t see how to set a value for such information in the sense of theoretical expectations; if we discard the standard model in the next ten years, and replace it with something much better, is the value of the W mass measurement diminished or enhanced? At what point does the W mass (for example) become mundane or less than crucial, in contrast to the present time? I suppose there will be a day when the W mass can be predicted sufficiently well, or when it ceases to provide any insights into new physics, and these developments might be reflected in the nature of the theoretical priors required by this evaluation of scientific merit, but I doubt this could be made clear or concrete.

The bloggers at Charm etc. are also skeptical of this approach, interesting though it may be to talk or blog about it. But if someone takes it more seriously, then perhaps the next question would be: how do you place a quantitative value on individual, and improving, measurements, taking into account the possibility that some measurements are wrong?

Update: (5-Jan-2007) There are nice discussions of this issue at Quantum Diaries Survivor and Deep Thoughts and Silliness.