WINE JUDGING MORE ART THAN SCIENCE

There's a lot more art - and artfulness - than science in the matter of wine judging, particularly in big wine shows ... photo Grant Nowell, with apologies to Frans Hals

.
Robert Hodgson's frank science
Cruel stats of swirl, sniff & spit
Facts swab wine race jockeys 
by PHILIP WHITE


It was hardly news to long-time followers of Robert Hodgson’s painstaking science, but this week’s viral winers’ shock horror nyah-nyah I-told-you-so flash-in-the-attachments spitbucket revealed again some of his findings relative to wine judges. 

Like, fair dinkum, wine judging is not science.

Hodgson is a retired oceanographer with a background in statistics and a small winery, Fieldbrook, in California.  His curiosity led him to maintain a study of the vagaries of wine show results. Commencing with an analysis of the scores his own products won in various competitions, he went on to mount larger-scale testing of the judging results over entire wine shows.

Hodgson’s eight years of methodical analysis reveals that even the most respected wine judges vary widely in their scoring of wines, not just from week-to-week and day-to-day, but from minute-to-minute.

"The results are disturbing," Hodgson told David Darbyshire of  The Observer.  "Only about 10% of judges are consistent and those judges who were consistent one year were ordinary the next year.”

Unsurprisingly, this variation increases significantly as the number of wines judged in each sitting multiplies, which is why most wine shows are judged by panels of judges working together. Typically, their scores for each wine are averaged to achieve a final ranking.  Under the instruction of the chief judge, they keep a check on each other.

My most recent experience in this arcane world of Wine Races occurred last week when I assisted in the selection of McLaren Vale wines to be poured at the next Gorgeous Food, Wine and Music Festival.  The wines were all served blind.  After a large mixed class of what were loosely called Reds – Anything But Shiraz, I was disappointed to the point of depression.  When we six judges retreated to discuss our findings, this feeling was common.  

Regardless of the wide range of “new” and “alternative” grape varieties on display, and the stalwarts like Cabernet sauvignon and Grenache, it had been very difficult to find anything of exceptional quality, and we all shared the feeling that we doubted our palates, and confused and exhausted ourselves re-examining and scouring the lines of glasses in search of something outstanding which we all believed should be there but wasn’t.

Admittedly, many of my personal favourites from the region were not entered: some winemakers didn’t have the volumes required, preferred to keep their stock for sales to regular buyers, like at cellar-door, or simply forgot to enter.  Nevertheless, there were dozens of wines there from reputed large producers, and many from the smaller wildcats, cult heroes, and respected pioneers.

If the same selection had been poured to the judges at the annual McLaren Vale Wine Show, half the entries would have been awarded medals.  While these would be mainly bronze, simply indicating the wine had no overt faults and was technically sound, there would be several silvers on average, a gold or two, and a trophy.

If our team of tasters was composed of local winemakers rather than experts who don’t make wine, I have no doubt the results would have been more sympathetic to the competitors.

Whoreses for causes, see?

My mantra for the selection of wine show judges is simple.  Any winemaker who doesn’t recognize their own wines in a blind tasting should not be judging – their organoleptic skills are inadequate.  On the other hand, any winemaking judge who fails to award their own wines well, or wines of similar styles, should not be expecting us to buy them.

There’s been an increasing attempt around the burgeoning wine-racing circuit to select expert judges who are not winemakers, or at least to import winemaking judges from other regions, but it’s still too easy to influence the results by inviting judges who the selection panel know to be sympathetic to certain styles of wine.

Then, of course, as Hodgson’s work reveals, you risk the variable induced by organoleptic exhaustion in the bigger classes, a problem which can be partly diminished by selecting bigger tasting teams or better ones, but even this ends up leveling the results because the scores are averaged.
 
Derbyshire also reports the works of French academic Frédéric Brochet and American economist Robin Goldstein, who independently proved wine judges award higher scores to wines they perceive to be more expensive.  And he mentions British psychologist  Professor Richard Wiseman, whose testing showed that of 578 people invited “to comment on a range of red and white wines, varying from £3.49 for a claret to £30 for champagne, and tasted blind ... People could tell the difference between wines under £5 and those above £10 only 53% of the time for whites and only 47% of the time for reds,” meaning that “overall they would have been just as a successful flipping a coin to guess.”

My research is slightly different.  I was once invited to present a tasting to a large conference of hardrock mine managers and mining industry executives and directors from all over the world.  We did this over a four-course dinner.  I selected two good wines to harmoniously accompany each course; one under $10 and one over $30.  Nobody knew their type or identity.  In the vast majority of cases, these red-loving, high-earning men thought the cheaper wine to be the over-$30 job.

Knowing the parsimony, if not the sheer meanness of mining executives, and I’m only half-joking, this indicated a presumption that wine of the price they commonly drank seemed the best, and therefore most valuable, to them.

This test was hardly scientific in the clinical sense, but it gave me a fairly good idea of how many enthusiasts simply get things wrong.  While all the wines were Australian, there was no discernable difference between the opinions of the Australian drinkers and those of the internationals.

All this led me long ago to withdraw from the giant wine-racing circuit, and sit at home with my glass and the endless string of bottles that arrive unsolicited, and simply get on with explaining which ones I prefer and why.

And yes, without tasting blind, I award them scores.  It’s a one-man continuous field wine race, and it’s a long course.  Unlike the quick glimpse of each entry the judges have time to take in the big national sprints, at least I get to savour each wine over many days, to best grasp their nature and potential.  In fact, I watch them run until they die. 

So trust me if you dare.

The Gorgeous Festival, by the way, will take place beneath the red gums at Serafino Winery on Friday 22nd and Saturday 23rd of November.  I can guarantee the wines served there will all be of superior quality.  If you doubt me, enjoy your water, and see how you go scoring the music.  To my palate, the organisers are lining up some damn fine acts. 

Keep an eye on the website:  http://www.gorgeousfestival.com.au/

FOOTNOTE:

As regular readers are aware, I award wines a score out of a maximum one hundred points.  While many accuse me of scoring consistently high, I should explain that it generally takes me dozens of wines to find one worth more than ninety points.  Tasting almost daily over a normal month, my average score is usually around seventy points.  Fifty pointers are all too common.  I have never awarded a perfect hundred.
.

To read of the mystery tongue-in-cheek parasite that infests the mouths of some wine judges click here. For a detailed discussion of the vagaries of the judging of Australia's most famous wine trophy, the Jimmy Watson, click here ... The George Grainger Aldridge cartoons are from our forthcoming funny book; working title:  Evidence of Vineyards on Mars (out soon - almost ready for the printer).