Grand averages for 21 classes – 2004 Celebration Open Classes
Placed an 8,9,10 Placed a 1,2,3 Was the only one
as 1, 2, or 3 as 8, 9, or 10 to place a
particular horse
Judge
c 0.00 0.16 0.32
s 0.33 0.33 0.42
m 0.28 0.33 0.39
e 0.10 0.19 0.24
x 0.24 0.06 0.29
w 0.22 0.11 0.22
Average theta:0.38
Distance Variance classes
judged
Judge-c: 2.4 1.1 19
Judge-s: 5.7 15.2 12
Judge-m: 4.2 17.6 18
Judge-e: 3.8 7.3 21
Judge-x: 3.8 9.6 17
Judge-w: 3.6 8.8 18
Average 3.9
The above results are comparable to the performance of the 2003 Celebration judges. The summary of 15 open classes from
the 2003 Celebration resulted in a theta estimate of 0.42 compared to 0.38 above. The average distance for the 2004 judges
was 3.9 while the 2003 judges averaged 4.5. Both measures for 2003 are somewhat larger than the 2004 values indicating
the 2004 judges performed somewhat better. What helped the average distance for the 2004 judges is the very small value
(2.4) for judge c. Judge c did very well in agreeing with the choices of his peers while judges s & m had problems in many
classes as is evidenced by their large average distances (5.7 & 4.2, respectively). Their large variances indicate that they
were erratic relative to their peers agreeing on some classes and agreeing on almost nothing in other classes. Note that these
two judges placed horses their peers placed 8, 9, or 10 as one of their 1, 2, or 3 choices and placed horses their peers placed
as 1, 2, or 3 as their 8, 9, or 10 pick more frequently than did their peers. They also were the only one to place a particular
horse in a class more frequently as well. Either their judging criteria did not coincide with those of their peers or they were
not as capable of recognizing the more important characteristics in horses as they go by. These kinds of judge performances are
what makes using the Olympic judging system really worthwhile.
The most interesting class included in the 2004 summary was the WGC Four Year Old class, Class 143, which is given below:
Class 143 - WGC 2004 - 4 year olds
Rankings are: Orderings based on all 5 judges:
w s e m c w s e m c
475 475 475 475 475 1 1 1 1 1
182 182 182 182 962 2 2 2 2 3
962 535 962 526 182 3 5 3 4 2
535 962 973 962 526 5 3 7 3 4
526 526 526 535 535 4 4 4 5 5
617 617 617 973 617 6 6 6 7 6
973 600 247 617 973 7 8 9 6 7
600 247 535 600 247 8 9 5 8 9
247 973 600 247 600 9 7 8 9 8
442 442 442 442 442 10 10 10 10 10
Orderings from deleting one judge at a time:
w s e m c
1 1 1 1 1
2 2 2 2 3
3 5 3 4 2
5 3 7 3 4
4 4 5 5 5
6 7 6 7 6
7 9 9 6 7
9 8 4 9 9
8 6 8 8 8
10 10 10 10 10
Average
Distance 0.0 2.0 0.0 1.0 1.4 0.88
Theta hat for class: 0.17
The above results are extremely good and what we would really like to see more often but in fact
seldom do see. Not having seen this class it is impossible to know why the judges were so consistent
in judging it but the judges performance for this class is extremely impressive. The average distance for
this class is 0.88 and theta is only 0.17.
The class that caused the biggest challenge for the judges was Class 81 which is given below:
CLASS #81 - Ladies 3 Yr Old Mares
Rankings are: Orderings based on all 5 judges:
w m e x s w m e x s
111 198 585 184 585 1 5 2 4 2
585 252 111 111 111 2 6 1 1 1
287 557 636 975 184 7 12 8 10 4
120 111 198 198 128 11 1 5 5 14
184 277 277 277 152 4 3 3 3 15
252 10 3 585 277 6 13 9 2 3
277 120 252 108 3 3 11 6 16 9
636 636 287 183 975 8 8 7 17 10
63 3 10 287 287 18 9 13 7 7
79 585 975 3 183 19 2 10 9 17
Orderings from deleting one judge at a time:
w m e x s
1 6 2 6 2
2 10 1 1 1
13 19 14 15 6
16 1 5 5 19
5 4 4 3 18
6 16 12 2 4
4 11 6 19 12
9 8 7 17 10
19 9 15 8 8
18 2 9 9 15
Total
1-3 Distance 10.0 18.6 11.1 13.0 3.3 11.2
Theta hat for class: 0.63
Judge m had the worst problems with this class having placed a horse that no other judge picked in their top 10 choices as his 3rd place horse and placed the horse that the consensus of his peers placed 2nd as his 10th place horse. Judges w, e, and s did a good job picking the 1st and 2nd place horses but the 3rd place picks for all the judges look like they came from a lottery drawing. The average distance for this class was 11.2 and theta is 0.63. It would be of interest to know what made this class so difficult to judge.
The 2003 judges did a very poor job judging the WGC 5 Year Old and Older class, Class 132. The 2004 judges had problems agreeing how to place the comparable 2004 class which was Class 146 and is presented below:
Class 146 - 2004 MFTHBA WGC - 5 Years and older
Rankings are: Orderings based on all 5 judges:
c s m e x c s m e x
571 419 658 658 419 4 2 1 1 2
658 658 279 279 279 1 1 3 3 3
419 279 571 411 411 2 3 4 5 5
604 561 411 571 658 6 8 5 4 1
605 411 419 419 571 7 5 2 2 4
279 605 604 561 604 3 7 6 8 6
411 571 227 604 605 5 4 9 6 7
561 227 605 605 227 8 9 7 7 9
227 57 561 227 561 9 10 8 9 8
57 604 273 485 57 10 6 12 11 10
Orderings from deleting one judge at a time:
c s m e x
5 4 1 1 3
1 1 3 3 2
3 2 4 5 5
7 8 5 4 1
8 5 2 2 4
2 7 8 8 6
4 3 9 7 7
6 9 6 6 9
9 10 7 9 8
10 6 12 12 10
Total
Distance 4.1 3.3 1.4 2.2 2.8 2.8
Theta hat for class: 0.33
The above results indicate that the 2004 judges did a better job of agreeing on the placements for this class than was the case
for the comparable class, Class 132, in the 2003 Celebration. The value of theta was 0.33 and the average distance was 2.8 for
this class. However, at this level of competition, it does seem that the judges would be in closer agreement as to what constitutes
a breed champion. We can assume that the horses and riders in this class were performing consistently so the differences in the
judges’ preferences must be due to a lack of agreement as to what they regard as the most important traits in the MFT breed, i.e.
they don't all follow the breed standard, or some were just not able to recognize what they saw in front of them. While they were
able to agree on the Four Year Old WGC, they had more diverse opinions about which were the best three horses in this class.
It is important to keep in mind that the 2003 results of the comparable class at the TWH Celebration had a value of theta of
0.16 and an average distance of 1.4. That is the kind of performance we should strive to get from MFTHBA judges.