What is the value of measurement?

January 1, 2008 at 9:38 am 4 comments

I was happy to see that Evolving Thoughts picked up the discussion of judging experiments based on concepts taken from information theory. Apparently these ideas are not new and show the fault lines between the statistics community and certain scientific groups, including HEP.

Today I would like to iterate an earlier point: what is the value of measurement? (in the context of judging experiments)

The use of “surprisals” and the like limit the scope to discoveries only, with the idea that more surprising discoveries are worth more than “ordinary” discoveries. (One goes on from there to develop the notion that an experiment, or analysis, which is better directed toward making a surprising discoveries is inherently worth more than one which makes “expected” discoveries – the problems in this formulation seem pretty clear when stated this way…) But this overlooks completely one of the main roles of experiment, which is to measure. We know that the standard model is successful not only because someone discovered the W and the Z bosons, the top quark, etc., but also because a host of precision measurements when taken together conform to the expectations of the standard model. No one would argue that experimental (or theoretical) particle physics would be just fine without this corpus of measurements, and I doubt that many people would say that looking for new physics beyond the standard model doesn’t need concrete and precise knowledge of standard model particle properties and interactions. After all, we know in which range the mass of the standard model Higgs boson must lie, and some people are enthusiastic about supersymmetry because of the way it conforms to measurements of precision electroweak observables. What situation would we be in today if we lacked those measurements, or they were considered unimportant? (Another example might be the current empirical knowledge of CP violation, culminating in the beautiful sets constraints on the CKM matrix and the unitarity triangle, but I am more familiar with electroweak physics, myself.)

top quark mass measurementsHiggs blue-band plot

(The top quark mass plot comes from the Tevatron Electroweak Working Group, and the “blue-band” Higgs mass plot comes from the LEP Electroweak working group.)

Of course I hope for a surprising discovery from the LHC or Tevatron data, one which is not foreseen by theorists. And I would be happy for clear signs of new physics even if it corresponds to one of the many available excellent theoretical ideas. But if people like the idea of judging experiments in some semi-quantitative way, how might one incorporate the value of measurement? Any ideas?


Entry filed under: Particle Physics.

Judging experiments by a priori theoretical expectations Judging a theoretical speculation by data

4 Comments Add your own

  • 1. Dmitriy  |  January 1, 2008 at 2:20 pm

    “an experiment, or analysis, which is better directed toward making a surprising discoveries is inherently worth more than one which makes “expected” discoveries” – this formulation doesn’t capture the core of the idea, which is not in the attempt to use statistics to measure importance of a discovery, but rather about what new knowledge a discovery brings in. Clearly a new discovery that doesn’t fit in a standard model worth more than a discovery that fits perfectly. At the same time there is nothing “inherently” valuable in being directed away from the current theoretical expectations, since the phase space of wrong directions is enormous and failed attempt to find something unexpected carries very little weight. At most it will kill some exotic model, which will be replaced with something else or by the same model with different parameters in no time.

  • 2. Michael Schmitt  |  January 2, 2008 at 7:21 am

    Privet Dmitriy!

    Thanks for your comment making a distinction beween measuring the importance of a discovery, and what new knowledge it brings in. Could you elaborate on that? For me, in the context of this discussion, spurred by Bruce’s paper, they are the same. We could consider other points of discussion, such as the advance of detector technology (and its spin-offs for the world at large), or the evolution of our scientific ideas (which depends on how entrenched they are, and how well a surprising result is communicated), but right now we were considering a restricted definition of scientific merit, so we’ll set aside those more sociological issues, for now.

    You’re quite right that a result that does not fit into the standard model (say, the production of mini black holes) will have a much greater impact on our thinking than one that is expected (such as the discovery of a SM-like Higgs boson). But who knows what would follow upon the latter case – perhaps something totally unimaginable, such as charge non-conservation (to use an almost absurd example). Furthermore, the discovery of a SM-like Higgs boson would not be revolutionary only because all those precision measurements confirmed the standard model, leading us to very strong expectations that there should be a Higgs boson. This is, in fact, the starting point of Bruce’s ansatz.

    Finally, I absolutely agree with you that ruling out an exotic model is not very worthwhile, unless there already is strong evidence supporting that model. So ruling out the MSSM [Minimal Supersymmetric extention of the Standard Model] by excluding SM-like Higgs bosons up through 150 GeV would be more exciting than ruling out a fourth generation, at least for me.

  • […] from our priors for how successfully an experiment guides us to a better understanding. Aside from overlooking the role played by measurement, this approach would not have helped us to overcome the quagmire of the Bootstrap Model (a.k.a. […]

  • 4. Dmitriy  |  January 2, 2008 at 12:02 pm

    Michael, fundamentally it’s a rather hopeless idea to measure humanity with math. There is no unique and reliable way to define what is good or bad (significant/not significant). Without mathematically correct definitions application of mathematical tools makes little sense. So I would never take seriously any mathematical measure of scientific merit of one discovery or another. Only when it’s clear to everyone that we are just playing games that may have some relevance to the real world, than we can think of different ways of estimating scientific merit of a discovery. From this point of view using the information entropy as a merit sounds interesting.

    The biggest issue for me with this game is in assigning probabilities. Probability of one outcome or another is just a measure of our own knowledge, prejustices and biases, which are very subjective. What is probability to find new physics at LHC? One would say 5%, another 95%. So instead of listening to expert opinions, we will assign numbers, which will carry nothing more than a cryptic representation of these opinions (if used properly). We already have some experience on how to judge expert claims, whereas with a new system we will need time to learn this from scratch, which may lead to wrong decisions on financing or not different experiments.

    At the end no one will ever judge an experiment based on a single number and even if it happens, most likely it will be not a merit estimate, but rather a price tag 🙂


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed

January 2008
« Dec   Dec »

%d bloggers like this: