Tags:
create new tag
, view all tags, tagging instructions

# Schervish coherence criterion

#### PeterBacchetti - 24 Jun 2012 - 00:22

To clarify and emphasize, my main point is just that if observed data x makes H more likely and makes H’ less likely, then x has supported H more than H’. And this can happen even when H implies H’. I used a simple, Bayesian example for clarity, but I think the problem is general. An observation x can support H more than H’ if it is very incompatible with (H’ and not H).

I believe that Jeff is mistaken when he says that x is neutral by likelihood ratio for H vs H’ in my example. The likelihood ratio P(x|H)/P(x|H’) is 2, and it cannot be calculated without using the priors (or at least the ratio P(A)/P(A or B)).

#### JeffreyWelge - 03 Jun 2012 - 10:10

Peter’s example is very interesting, and he proposes a criterion for a sensible measure of support (evidence) in terms of prior versus posterior probabilities. As these are related via the likelihood function, I think he is on the right track. A candidate measure of evidence is the likelihood ratio between two hypotheses. If we have prior information on the probabilities of A, B and C it may be helpful (and sometimes necessary) to use it – but such information is not necessary in this example to describe the relative evidence conveyed by the observation x about the hypotheses considered. Suppose we do not have the priors. The probability of x under each hypothesis is sufficient: 1) Having observed x, we know that B is false since P(x|B)=0 2) Now H=H’=A, so the observation x is clearly neutral evidence for H v H’: It supports H’ at least as much as H, as required and as Peter noted. It is also neutral regarding the hypothesis “C” with respect to “A”. (the probabilities of C and A have both increased by the factor 4/3 – but is x evidence “for” either in isolation? It IS definitive evidence for either over B) Regardless of the prior probabilities of the hypotheses under consideration, if we can compute the probability of x under each hypothesis, we might measure the relative evidence for H1 v H2 by the likelihood ratio P(x|H1)/P(x|H2). In Peter’s example, before taking the observation we would not have been able to compute P(x|H’) without the prior information – it was fortunate that x ruled out B. The p-value fails as a measure of support because it refers to only one hypothesis, and I suggest that a sensible measure of evidence must reflect relative support for >1 hypothesis. The rule “if P(H) < P(H|x) then x is positive support for H” conforms: Though it refers directly only to H, P(H|x) depends on the probability of x under all alternatives in ~H as well as the priors on those alternatives. If we change the example only by making P(x|B)=0.01 rather than 0, we DO need the prior odds on A v B to determine that P(x|H’)=0.505. The likelihood ratio now tells us that x supports H over H’ by a factor of nearly 2, but that seems very wrong. The problem is that the logical relationship between H and H’ is not encoded in the data. The Bayesian calculation shows {P(H’|x)=0.3355}>{P(H|x)=0.3333}. Yet as before, P(H)<P(H|x), while P(H’)>P(H’|x). Wow – is Peter right? Is the search for a measure of support that will always conform to logic for this problem doomed, even if we commit fully to Bayesianism?

#### PeterBacchetti - 01 Jun 2012 - 10:38

Schervish proposes a “simple logical condition” that if hypothesis H implies hypothesis H’ then any observed data x must support H’ at least as much as it supports H. I believe that this is mistaken.

If H implies H’, then we do have
P(H’|x) >= P(H|x),
but these posterior probabilities are not measures of how much x has supported H or H’. I propose that such a measure should reflect how the probabilities given x differ from the prior probabilities before x was observed. Specifically,
if P(H) > P(H|x) then x has provided negative support for H, and
if P(H) < P(H|x) then x has provided positive support for H.

Even if H implies H’, it can happen that
P(H|x) > P(H) and
P(H’|x) < P(H’).
In this case, x supports H (positive support) more than it supports H’ (negative support), violating Schervish’s proposed condition, but no measure of support can meet it and still make any sense. This can happen in situations where x is fairly incompatible with the event (H’ & not H) but fairly compatible with H.

A simple example is the following:

3 possible states of nature: A, B, or C
H: A
H’: A or B
H implies H’

Prior probabilities:          P(A) = 1/4     P(B) = 1/4     P(C) = 1/2
Observed data probabilities: P(x|A) = 1 P(x|B) = 0 P(x|C) = 1

Simple calculations show:

P(H|x) = P(A|x) = 1/3 > 1/4 = P(H)
Any sensible measure of support must show positive support of x for H, because P(H|x) > P(H)

P(H’|x) = P(A or B | x) = 1/3 < 1/2 = P(H’)
Any sensible measure of support must show negative support of x for H’, because P(H’|x) < P(H’).

(Note that we do still have P(H’|x) >= P(H|x), which is the condition that must hold if H implies H’.)

Therefore, no sensible measure of support can be “coherent” by the Schervish criterion. Showing that a measure violates this condition is therefore not necessarily a flaw in that measure, and any measure that meets this condition will not make sense in situations like the above.

This, of course, does not mean that P-values are sensible measures of support for the null hypothesis, but I believe that exposition based on Schervish’s criterion will not be helpful.

The distinction between the posterior probability P(H|x) and the support that x provides for H is why the analogy with the coherence criterion for multiple comparisons does not hold. If H implies H’, then it would not make sense to reject H’ but not reject H, because we must have P(H’|x) >= P(H|x). The multiple comparisons coherence criterion follows from the posterior probabilities, and does not depend on the support that x provides for H and H’.

### OnlineJournalDiscussionFormedit

 Title_Discussion_Topic Schervish coherence criterion Name_Topic_Initiator Online_Journal_Club_Meeting Meeting 2 Description - Problem to be explored I believe that this criterion is impossible rather than necessary. See Also Disclaimer The views expressed within CTSpedia are those of the author and must not be taken to represent policy or guidance on the behalf of any organization or institution with which the author is affiliated.
Topic revision: r5 - 02 Dec 2013 - 11:48:44 - MaryBanach

Copyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding CTSPedia? Send feedback