Grading Age Grading: Evaluating Methods for Handicapping Competitive Runners

Introduction

Several methods have been proposed for age handicapping distance runners (5K through Marathon). The purpose of handicapping is two-fold:  it allows runners of different ages to compete on a “level playing field”, and it gives us some insight into the aging process itself.

In this article, I give letter grades (A, B, C, D, and F) to four different handicapping methods based on the accuracy of each when it is applied to runners competing in eleven different venues representing the local, state, and world competitive levels. As you will hopefully agree, I have taken care to ensure that the grading system is both fair and objective. Three of these methods are separately applicable to both males and females and one (YALE_AG) has been developed using data for males only.  Consequently, with 11 venues and 2 genders, three of these methods receive a letter grade in 22 different courses, and one method receives a letter grade in 11 courses.  Finally, after grades have been calculated for all of the courses, the overall GPA is calculated for each of the four methods.

For those desiring a more conventional approach to evaluating these four models, I also fit each model to the eleven venue test data and calculate generalized Coefficients of Determination (R2) for each model across the various venues. As you will see, there is excellent agreement between the letter grade and the Coefficient of Determination.

The four methods and links to references and principal authors are as follows:

Although they use different approaches, the first three of these methods are all based on the single age world records maintained by the Association of Road Racing Statisticians at arrs.run.  The 2017 factors for the YALE_AG method are given for ages 40 and above.  However, the authors recommend an adjustment to extend the age range back to 35 years.  This article uses the recommended adjustment.

The fourth method, UD_5KH, is unique in that it is based on the maximal oxygen uptake during treadmill tests using subjects of different ages. Originally it was developed for only the 5K but, more recently, it has been extended for distances up to the Marathon.  In addition to age, UD_5KH also includes an adjustment for bodyweight.  In this evaluation, the CDC 10th percentile for bodyweight by age was used for UD_5KH since this provided a slightly better fit than did higher percentiles and since competitive runners typically are leaner than the general populations.  Additional observations on each of these methods are discussed later in this article.

There are many age handicapping or “age grading” calculators available on-line; however, virtually all are derivatives of one of the four methods evaluated here (particularly of WMA_AG or its earlier versions sometimes denoted with the acronym “WAVA”).

Test Data: World, State, and Local Competition

The accuracy of the four methods is evaluated by measuring their performance against eleven different test venues representing a wide range of competitive abilities. For each gender, these eleven venues were split into 3 distinct classes of competitors representing the world, state, and local competitive levels.

Between the ages of 35 and 85, male and female single age speeds were obtained for each venue.   (Some venues did not have single age data that went up to age 85, in which case I used the maximum upper age available; only data for runners, avg. speed > 4.5 mph, were used.)

To insure that the speeds were equitable among the various ages within each venue, speeds were adjusted for the size of the relevant population as described in Age Handicapping Competitive Runners, Part1: Quantifying the Population Effect.  The data were also smoothed as described in the appendix to Age Handicapping Competitive Runners, Part 2: Tables for Speed Handicaps.

The test venues were as follows:

  1. The World single age records were obtained from the Association of Road Racing Statisticians, arrs.run, as follows:
  •       World Single Age Records for the Marathon
  •       World Single Age Records for the Half Marathon
  •       World Single Age Records for the 10K
  •       World Single Age Records for the 5K
  1. State Single Age Records for were obtained or linked from the StateRunningRecords.com for the following four venues:
  •       Marathon (AL,MO,MN,MS,NH,OK,TN)
  •       Half Marathon (AL,MO,MN,MS,NH,OK,TN)
  •       10K (AL,MO,MN,MS,NH,OK,TN)
  •       5K (AL,AZ,MO,MN,MS,NH,OK,TN)
  1. Local 5-year Age Group Winners in the 5K were obtained from data summarized in Racing Among the Ages. Based on race size, there were three Local venues:
  •     Very Large 5K races (302 local races, each with 1000 and 8641 finishers)
  •     Large 5K races (356 local races, each with 500 and 999 finishers)
  •     Medium 5K races (313 local races, each with 300 and 499 finishers)

[Note that the AZ state records were available for only the 5K due to a broken link at the time these data were compiled. More information about the local races can be obtained at Median Times for Top Finishers in 5K Road Races.]

I did not include local races with fewer than 300 finishers because of concerns that too many of the age groups would be won by non-competitive individuals (i.e. recreational walkers and runners) due to small age group sizes and the absence of a competitive runner in some groups.

Final GPA for Each Handicapping Method

Before we get too far into the weeds, here are the final Grade Point Averages (based on a 4 point scale) and overall letter grades for each of the four methods:

  •    YALE_AG:   1.27 D+
  •    WMA_AG:   2.45 C+
  •    BDR_AH:       3.64 A-
  •    UD_5KH:       0.00 F

Later I will provide more detail on the methodology to calculate grades and will break down the competitive levels where each method did or did not do well. I also discuss the technical reasons for the performances of each.

Lest the proponents of the WMA_AG methodology take some comfort in the “gentleman’s C+” received by their candidate, I point out here that the WMA_AG method received an “F” in three required courses. Also, our valedictorian, BDR_AH, will graduate with only an “A-“ average, indicating that there may be still room for improvement.

Coefficients of Determination, R2

Because the observed speeds for all of the single year data points within each of the eleven venues have been adjusted for the population at each age, these speeds represent equivalent (same population percentile) performances among the various ages (typically 35-85) within each venue.

Similarly, each of the four age-grading/handicapping models defines a family of performance isoquants, where an isoquant is a series of combinations of speed and age representing an equal level of performance. Thus, for example, using WMA_AG, a 35 year old male who completes a 5K road race at 11.3 mph (time=16:30) is on the same isoquant and has a performance equal to an 85 year old man running at approximately 6.3 mph (time=29:40).  On the other hand, using the isoquants from the BDR_AH method suggests approximately 5.1 mph (36:35) for an 85 year old is equivalent to 11.3 mph at age 35.

So the question becomes, how well does each of the four models fit the data in each of the 11 venues? To answer this question, the isoquant with the least mean squared error from the observed speeds (population adjusted) was determined for each model and each venue. Since YALE_AG is only applicable to males, this gives a total of 77 families of isoquants.  (Even though these models are, in general, non-linear in speed, each of the 77 families of isoquants is defined by a single parameter, so finding the optimum fit is relatively straight forward.)

The Coefficient of Determination (R2) was used to measure the fit of each method across the eleven venues. Tables 1a and 1b show these Coefficients of Determination.  The Coefficients of Determination are color coded as under 90% = Red, between 90% and 95% = Yellow, and over 95% = Green.  Note that only BDR_AH was Green across all venues.

The second best Coefficients of Determination were obtained by WMA_AG. Even though this method did well at the world and state levels, its performance was not very good at the local level. For example, in Table 1a, the Coefficient of Determination using the WMA_AG method for Age Group Winners in Local races with 500-999 finishers is 89%.  Thus the WMA_AG method explains only 89% of the variability in running speed across the various ages.  Figure 1 shows the best fit of the WMA_AG method to the male data for age group winners in Local races with between 500 and 999 finishers as well as for 5K world and state records.  As you can see from this figure, at the Local level, the WMA_AG method gives an “unfair” advantage to runners between the mid-forties and early fifties.  On the other hand, starting in the late sixties, this method progressively disadvantages the older runners.

Nevertheless, WMA_AG is substantially better than UD_5KH, which is the worst method across all venues. Figure 2 gives an example of the fit of this method.

Grading Age Grading

In the previous section we looked at the performance of four handicapping methods in terms of how well each fits the observed speeds between the ages of 35 and 85 and across eleven different venues representing a wide range of competitive ability.  In this section, we look directly at the handicapped speeds generated by each model.

Quite simply, an ideal handicapping system will meet two requirements:

  1. Performances which are equal, as they are within each venue, receive the same handicapped speed.
  2. The differences in competitive ability between the different venues is preserved in the handicapped speeds

Since all of the methods equate the handicapped speed and the actual speed at some point between the ages of 25 and 35, the second of the above requirements will be fulfilled provided the first is met. Consequently, I will focus on the first requirement and grade each method on its ability to achieve consistency among the handicapped speeds within each venue.

One measure of within venue consistency is the average deviation (i.e. average absolute error) among the handicapped speeds between the ages of 35 and 85.  Thus, if the handicapped speeds within a venue are all the same, the handicapping method performs ideally and the average error is zero.  On the other hand, as the differences among the handicapped speeds becomes larger, the average error grows proportionally.

For example, using the YALE_AG method, the average deviation among the male 5K state record handicapped speeds is 0.471 mph (miles per hour).

To make the interpretation of the average deviation in handicapped speeds somewhat more intuitive, miles per hour can be converted to the same scale as age (i.e. years) as follows:

Between the ages of 35 and 85, the male 5K actual state record speeds declined at an average rate of 0.125 mph/year.  Thus the average deviation of 0.471 mph in handicapped speeds is equivalent to (0.471 mph) ÷ (0.125 mph/year) = 3.77 years.

So for this venue, the average error in the handicapped speeds is 3.77 years. Is this good or bad and what letter grade should be assigned to the performance of the YALE_AG method in the male 5K state record venue?  To answer this question, consider the following two observations:

  1. Differences in age among individuals who have the same integer age are generally regarded as inconsequential by all race venues. Although we may record the exact age in days, state and world records are generally not maintained for intervals shorter than a year. Thus we do not see separate records maintained for ages 50.0, 50.1, 50.2, etc. Local race results generally show finishers ages in whole numbers. Consequently, for a particular venue, a handicapping system having an average absolute error rate of less than one year can be considered to have very good performance.
  2. Differences in age of 5 years or more are usually regarded as quite significant. Larger local races will most commonly separate individuals differing in age by 5 or more years into separate age groups. Even though ARRS reports single age records, above 40 they highlight the best performance in each 5-year interval. USATF maintains American individual masters records in 5 year intervals. If a handicapping system has an average absolute error rate greater than 5 years for a particular venue, we can conclude that it fails for that venue.

Having established that an average error of less than one year is an “A” and an average error of five or more years is an “F”, the intervening spread can be partitioned uniformly to provide the following grade scale:

  •    “A”: Error less than 1.00 years
  •    “B”: Error between 1.00 and 2.33 years
  •    “C”: Error between 2.33 and 3.67 years
  •    “D”: Error between 3.67 and 5.00 years
  •    “F”: Error equal to 5.00 years or more

Using this scale, report cards with the grades achieved are shown for each handicapping method in tables 2a and 2b.

Summary of Results

For the local races, only the BDR_AH method showed a reasonable performance, with grades of B and Coefficients of Determination at 97% or above for both males and females in each local venue. On the other hand, the other three methods all had a Coefficients of Determination at 90% or below on all local venues and received straight “F’s” on the male local venues.

The UD_5KH failed on all eleven venues for both genders. Nonetheless, at the world level, the other three methods all performed reasonably well with BDR_AH getting straight “A’s”, WMA_AG getting “A’s” and “B’s”, and YALE_AG receiving “C’s” and one “B”.  However, this might be expected since all three of these methods are directly derived from the same single age world records that were used in the evaluation process.  Possibly a useful question is “Why didn’t all three of these methods get straight “A’s” at the world level?”

The following discussion provides more detail on each of the methods.

UD_5KH Discussion

Even though the UD_5KH method performed poorly, with some methodological adjustments, it does have the potential to provide a significant insight into the aging process itself: Can a unified approach explain of the effect of age on individuals ranging in ability from the general population (measuring VO2max) to world class athletes achieving a single age record?

Among the methodological issues with the UD_5KH, as published by Vanderburgh and Laubach in 2007, is that it relies on two studies on the “Changes in Aerobic Power” of Men and Women by Jackson et.al, 1995 and 1996. The studies by Jackson et.al. assume a linear effect of age; however, a 2005 study by Fleg et.al. showed “Accelerated longitudinal decline of aerobic capacity in healthy older adults.”  (See also Ades and Toth, 2005).  Also, the Jackson age coefficients, as used by Vanderburgh, are “corrected” for physical activity and body composition, which are themselves highly correlated with age.  This causes the decline with age to be substantially underestimated, as is graphically illustrated in Figure 2.

It should additionally be noted that the Jackson studies, especially for women, underrepresent older adults in that the oldest woman was only 64.

Vanderburgh and Laubach also published “Validation of a 5K Age and Weight Run Handicap Model” in 2006.  Unfortunately this study is of limited value due to questionable statistical methodology.  Two assumptions should have been challenged during peer review.  First, the assumption is made that the absence of a linear correlation between age and handicapped run times implies there is no relation between age and handicapped run times.  Second, individuals whose times were outliers to the model were successively excluded for (supposed) lack of sufficient effort (i.e. when data that did not fit the model very well is excluded, the model fits the remaining data better!)

Nevertheless, this approach has significant potential, I hope it is revisited with the appropriate methodological corrections.

YALE_AG and WMA_AG Discussion

A fundamental assumption of both the YALE_AG and the WMA_AG methods is that for each age a unique frontier or upper biological limit to human performance exists, and that individuals of different ages who have performances at the upper biological limit for their age can be regarded as having equal performances.

A second assumption is that individuals of differing ages who perform below the biological limit but at the same percentage of the biological limit for their age can be regarded as having equal performance.

As far as I can tell, these assumptions and the resulting models have not been tested until this article. The WMA_AG performs somewhat better than the YALE_AG, primarily because it has more parameters which allow a closer fit to the data.

The second assumption is particularly problematic because not only has it not been tested, but the alternatives do not appear to have been considered. For example, with males, the YALE_AG 5K “biological limit” at 80  takes 44% more time and is about 4.36 mph slower than at 40 years. Now suppose a less than world class 40 year old can run a 5K in 18 minutes, what is the equivalent time for an 80 year old?  44% more time suggests about 26 minutes, whereas 4.36 mph slower suggests about 31 minutes.

As in the above example, I reworked the YALE_AG handicapped performances for the 3 male Local venues using absolute changes in speed rather than % change in time. This simple adjustment to the YALE_AG model resulted in a dramatic improvement, reducing the average deviation by more than 50% and changing the Local venue grades from three “F’s” to three “C’s”!

BDR_AH Discussion

Although BDR_AH did well for all of the venues considered here, even the least competitive venue (Age Group Winners in midsized 5K races) represents athletes who are well above average.  Moreover, even though the BDR_AH model fits the local venues reasonably well, the fit for the state and world venues is consistently better.  Consequently, the trend is somewhat worrisome in that most of the applications of age handicapping/grading probably occur at performance levels equal to local age group winners and below.

This begs the question, how should the effect of aging on maximal athletic performance be modeled across the full spectrum of human ability, ranging from ordinary individuals up to the most elite world class athletes? (Since, a large percentage of older individuals may not be able to complete an endurance race at a running speed, their maximal performance may need to be measured on a treadmill or some other more controlled venue.)

At the other end of the spectrum, what is the best way to estimate the frontier or upper limit of human performance at each age? Should it be directly estimated as with YALE_AG and WMA_AG using only “non-dominated” single age running records?  Or should it be extrapolated from models, such as BDR_AH, which are initially derived to fit the entire complement of single age running records?

The evaluations in this study were based on the age range 35-85. This was done in consistency with common conventions related to “masters” runners.  However, many local races start the masters category at age 40, and Racing Among the Ages suggests that the last major inflection points for both participation rates and straggling rates occur between the ages of 40 and 50 in local 5K races.  So, what is the best age range to use in evaluating the performance of the various age handicapping methods for masters runners?  And this of course begs another question:  how should we evaluate the performance of age handicapping methods at other life stages?

Future research into all of these questions promises to deliver important insights into the aging process and human performance.

WMA Age-Grade Standards for Winners of the 30-34, 35-39, and 40-44 Age Groups

Current WMA Age-Grading Standards are extremely aggressive. In fact, in the entire modern era of sports statistics, only about a half dozen isolated performances have met the Standard for 5K road races.  The great majority of single age world record holders have never had a performance that met the Standard.  Consequently, ordinary athletes may have difficulty connecting with these Standards.  The purpose of this article is to re-express the WMA standards in terms that almost every 5K participant can relate to:  the age-group winner.

For road and track racing as well as other sports, World Masters Athletics (WMA) has developed and maintains performance standards for each single year of age. Athletes of various ages are evaluated in terms of how well they stack up against the event standard for their age.  For example, a runner in a 5K road race might be 70% as fast as the standard for his or her age.  Based on how well an athlete compares to the standard for his or her age, WMA has created labels for various levels of performance as follows:

  • Above 90%     World Class Level
  • Above 80%     National Class Level
  • Above 70%     Regional Class Level
  • Above 60%     Local Class Level

This begs the question of how we might relate these various levels, especially the Local, Regional, and National classes, to concrete performances that are familiar to the athletes that are somewhat below World class. For example, if you are a “Regional” class athlete, how often might you win your age group in a 5K road race?

One of the most complete sets of WMA age standards and probably the best known is the WMA Age-Grading Calculator.  This calculator is also the basis for many (almost all?) of the other on-line age-grading calculators.  And although the methodology produces significant biases for the youngest and oldest race participants (see Racing Among the Ages), it appears to be reasonably consistent within gender across a wide spectrum of abilities in the heart of the age range, i.e. between the ages of 30 and 44.  Consequently, in this post we will look at the following age groups:  30-34, 35-39, and 40-44.

Computational Example

[If you are not interested in the computational details, skip to the results section below.]

As an example, consider “Joe”, a 32 year old male who can run a 5K in 21:33. The WMA 2015 standard for a 5 km road race is 13:05.  With a time of 21:33, Joe will perform at 60.7% of the standard, and hence is just above the threshold for a “local class” athlete.  We will also note that a time of 21:33 also corresponds to the 87.29th percentile among 32 year old males.

Among all male and female 5K finishers of all ages, 4.9% are males between 30 and 34[ref].  Suppose “Joe” decides to participate in a very small race expected to have just 40 total runners in addition to himself.  He would then expect to compete against an average of 40 x 4.9% = 1.96 other runners in the M30-34 age group.  If the expected 40 runners are a representative sample of all runners, then the actual number of runners in the M30-34 age group will follow a Poisson distribution with mean 1.96.

Using Poisson distribution with a mean of 1.96 suggests there is a reasonable probability (0.141) that no other competitor shows up for Joe’s age group and he will then have a 100% chance of winning his age group. There is a probability of 0.276 that exactly one other competitor shows up in the M30-34 age group.  Since Joe is calculated to be at the 87.29th percentile among his peers, he will have a 0.8729 probability of defeating this competitor.  Thus the probability that exactly one other age group competitor shows up and that Joe beats him is 0.276 x 0.8729 = 0.241.  The Poisson probability that Joe has exactly 2 competitors is 0.271 and the probability that he beats both is 0.8729 x 0.8729 = 0.7620.  Thus, the combined probability that exactly 2 competitors show up and that Joe beats them both is 0.271 x 0.7620 = 0.206.

It is possible to make similar calculations for every possible number of competitors for Joe in the M30-34 age group, i.e. for 0,1,2,3,4,5,6 . . . etc. When we add up the probability that Joe wins his age group across all possible numbers of competitors, we can calculate that Joe’s overall probability of winning is 0.141+0.241+0.206+0.118+0.050+0.017+0.005+. . . = 0.779.  Thus, a male between 30 and 34 who competes at 60.7% of the WMA standard will have approximately 0.779 chance of winning his 5-yr age group in a race expected to have a total of only 40 other participants.  Since Joe, competing at 60.7% of the WMA standard, will usually win his age group in these very small races, we can correctly state that a 60.7% age-grade is significantly superior to the typical age-group winner in a race with just 40 participants.

Now suppose that Joe participates in another race expected to have 110 total participants in addition to himself. In this case, if we go thru the above calculations for the larger race, we find that Joe will have “only” a 0.501 probability of winning the M30-34 age group.  Thus, half of the time Joe will win his age group and half the time he will not.  Consequently, we can conclude that, among males 30-34, a 60.7% age-grade is equivalent to the median or typical age-group winner in races having a total of 110 participants.

Results_:_WMA Age Grade for Age-Group Winners in 5K Races

The graph below is based on the age group winners for six age groups: F30-34, F35-39, F40-44, M30-34, M35-39, and M40-44.  For each of these age groups the median WMA Age Grade was calculated for each of three race sizes: 110, 500, and 3,000.  The averages are as follows:

Total Race Participants                  Avg. WMA AG for Age Group Winners

  •            110                                                  60% (Local Class Level)
  •           500                                                  70% (Regional Class Level)
  •        3,000                                                  80% (National Class Level)

In the graph you will note that, although the combined average of males and females match the WMA classes very well, the WMA Age Grade assigned to female age group winners is consistently below that given to males. For example, for races with 110 total participants, the average AG assigned to males is 3.5 percentage points below that assigned to females.  For races with 500 participants the difference is 4.9% and for races with 3,000 participants the difference is 3.4%.

Consequently, within the range of abilities and the range of ages considered here, there is an average bias of about 4% against the females. If the WMA AG of males and females are directly compared, 4% should first be added to the female AG.  For example, a female with a 66% AG is performing at a level equivalent to a 70% male AG.