When Can One Detect Overdominant Selection in the Infinite Alleles Model?

Paul Joyce, Stephen M. Krone, and Thomas G. Kurtz

Abstract

One of the goals of this paper is to show that the infinite alleles model with overdominant selection ``looks like'' the neutral infinite alleles model when the selection intensity and mutation rate get large together. This rather surprising behavior was noticed by Gillespie (1999) in simulations. To make rigorous and refine Gillespie's observations, we analyze the limiting behavior of the likelihood ratio of the stationary distributions for the model under selection and neutrality, as the mutation rate and selection intensity go to infinity together in a specified manner. In particular, we show that the likelihood ratio tends to one as the mutation rate goes to infinity, provided the selection intensity is a multiple of the mutation rate raised to a power less than 3/2. (Gillespie's simulations correspond to the power 1.) This implies that we cannot distinguish between the two models in this setting. Conversely, if the selection intensity grows like a multiple of the mutation rate raised to a power greater than 3/2, selection can be detected; i.e., the likelihood ratio tends to 0 under neutrality and infinity under selection. We also determine the nontrivial limit distributions in the case of the critical exponent 3/2. We further analyze the limiting behavior when the exponent is less than 3/2 by determining the rate at which the likelihood ratio converges to one and by developing results for the distributions of finite samples.