publish in Austrian Journal of Statistics

[RobinHankin]

this is the ideas in inst/very_simplified_likelihood.Rmd. See also issue #134

[RobinHankin]

submitted to the Austrian Journal of Statistics just now. Here is the email from the server:

Robin Hankin:

Thank you for submitting the manuscript, "A Simplified Plackett-Luce Likelihood Function for Rank Orders" to Austrian Journal of Statistics. With the online journal management system that we are using, you will be able to track its progress through the editorial process by logging in to the journal web site:

Manuscript URL: https://www.ajs.or.at/index.php/ajs/authorDashboard/submission/2129
Username: robinhankin

If you have any questions, please contact me. Thank you for considering this journal as a venue for your work.

Matthias Templ

Austrian Journal of Statistics

[RobinHankin]

Well that got rejected:

Dear Robin Hankin

We have reached a decision regarding your submission to Austrian Journal of Statistics, "A Simplified Plackett-Luce Likelihood Function for Rank Orders".

Our decision is to: Decline submission

While there is potential in your submission, the model seems unrealistic and misses sensible real life use cases. Furthermore, there are several errors in the manuscript, including an incorrect likelihood, incorrect presentation of results and conclusions. Please find attached the referee's report. I am sorry for the bad news and wish you the best in revising and publishing the manuscript.

Dr. Thomas Kuenzer
Editor, Austrian Journal of Statistics

The attachment was a PDF which I retrofitted to latex:

\documentclass{article}
\usepackage{amsmath}
\begin{document}

Comments on "A Simplified Plackett-Luce Likelihood Function for Rank Orders"

Overview

This short paper considers the problem of estimating the strength a of
a focal competi- tor based on their observed ranks in a series of
"competitions". The simplification to the

Plackett-Luce likelihood function arises through the assumption that
the other $n-1$ competitors in a competition are clones of one another
and therefore share strength parameter $b$, with the constraint that
$a+b=1$. The likelihood function for this simplified model is derived
and several properties are discussed. The model is applied to three
datasets.

General comments

I found this paper to be interesting and generally well written.
However, the assumption that all non-focal competitors have the same
strength is unrealistic and I cannot think of a scenario where this
would be a sensible model. This is perhaps most readily appreciated
by considering the predictive distribution for a new observation.

Estimating $a$ to be greater than $b$ implies that the focal
competitor is the strongest competitor and the modal outcome from the
predictive distribution is that the focal competitor will come first
(i.e. will have rank 1). Conversely, estimating $a$ to be less than
$b$ implies that the focal competitor is the weakest competitor and
the modal outcome from the predictive distribution is that the focal
competitor will come last (i.e. will have rank n). Finally, estimating
a to be equal to b implies that all n competitors have equal strength
and so the predictive distribution is uniform; the focal competitor is
equally likely to come first as last (as well as all ranks in
between).

For example, the Parkrun data for 2023 lead to estimates $\hat{a} =
0.485$ and so $\hat{b} = 0.515$ indicating that the focal competitor
is of roughly similar ability to the non-focal clones. Figure 1 shows
an estimate of the predictive distribution of the focal-competitor's
finishing position in their first Parkrun of 2024 (with 193
competitors); the estimate is based on one million simulations from
the simplified Plackett-Luce model with $\hat{a} = 0.485$ and
$\hat{b}=0.515$. The predictive distribution based on these maximum
likelihood estimates suggests that even though the focal competitor's
most likely finishing position was last, they had a reasonable chance
of winning it, which, with all due respect, does not seem realistic.
(Indeed, the focal competitor finished 145th out of 193.)

I would suggest that a model which gives a predictive distribution
that is unimodal with the mode at a more central rank would be more
realistic. Unfortunately the simplified Plackett-Luce model will never
lead to such a predictive distribution.

Histogram of predicted ranks here; almost uniform with slightly more
density at the right than the left

Figure 1: Predictive distribution of the focal-competitor's finishing
position in the first Parkrun of 2024

Specific comments

1. Page 1: Keywords have not been provided.

2. Page 2: The second and third lines in the equation for the
   simplified Plackett-Luce likelihood (up to a multiplicative constant)
   $\mathcal{L}_{r,n}(a)$ are incorrect and should be

\begin{eqnarray}
\mathcal{L}{r,n}(a) &=& \frac{B-1}{(B+n-2)\cdots (B+n-r-1)}\
&=& \frac{B-1}{\prod{j=1}^r(B+n-j-1)}
\end{eqnarray}

$$\frac{B-1}{(B+n-2)\cdots (B+n-r-1)} = \frac{B-1}{\prod_{j=1}^r(B+n-j-1)}$$

Also the third line of the author's equation uses the factorial function which is only
defined for non-negative integer arguments (and here $B$ may not be an integer).

3. Page 4: The notation $p$ hasn't been introduced. I think $\hat{p}$
   should be $\hat{a}$. Also based on the data given at the top of page
   4, I calculate $\hat{a}{2023}=0.485$ and $\hat{a}{2024}=0.413$
   which is the other way round to your estimates. This implies that the
   axis labels in the plot in Figure 3 should be swapped (unless the data
   is labelled incorrectly at the top of page 4).

4. Page 4: For the 2024 data the rank and runners should be the other way round (i.e.,
   193 runners and rank 145 in the first race).

5. Page 4. The Parkrun example is interesting but the assumption of
   non-focal competitors with equal ability is not realistic. Were all
   the Parkruns run at the same course? Even at the same course there
   will be differences in average strength of the field from run to run.
   The author's rank in these Parkruns is a rather crude measure of
   performance. A better indicator of performance might be the author'
   times in these Parkruns, from which a better assessment of a change in
   performance between the two years might be made.

6. Page 5: In the education example you write that you are "justified
   in concluding that the student is indeed more competent at applied
   than pure mathematics". The pure classes may attract a stronger group
   of students than the applied classes and so there is no basis on which
   to come to this conclusion. It would be justified to conclude that
   the student generally has a higher ranking in the applied modules than
   the pure modules. A difference in binomial proportions (of rivals
   beaten) might be a simpler way to get to the same conclusion.

7. Page 5: It would be useful to provide the Formula 1 data, although,
   this is another example in which the assumption of non-focal clones is
   unrealistic.
   \end{document}

[RobinHankin]

The histogram that the referee gives can be produced with something like

hist(replicate(1e4,which(rrace3(pn=c(a=99,b=1),ps=c(a=0.515,b=1-0.515))=="b"))))

(although this is flatter than the one in the review).

[RobinHankin]

Wrote back to the journal:

Dear Professor Kuenzer

thank you for your email of April 1st.

I'm writing to thank your reviewer for the constructive comments. I have had a chance to consider them.
I agree that his root concern is valid: specifically the assumption of identical non-focal competitors is not realistic.

One model I have been experimenting with is one in which the non-focal competitors have geometrically distributed strengths.
Although this is much more computationally demanding, and lacks the elegance of the now-rejected submission, it does
address the reviewer's concerns.

I note that the review included a number of positive statements. Would you allow me to submit a manuscript detailing this geometric model to your Journal?

Best wishes

Robin

[RobinHankin]

the email I sent to Kuenzer bounced back from google:

"Maximum number of MX connection attempts exceeded"