One last small detail must be taken into account before fully understanding what this figure reveals, it concerns the different sized “moustaches” that appear on each side of the small black boxes. The smaller these moustaches are, the weaker the variability is between the opinions of the judges. If they are bigger, the judges have not answered in the same way. This is an important piece of data because the more the judges are in agreement with each other, the more this signifies that their opinion is linked to clear and objective evaluation criteria, or in any case conditioned by the same kinds of learning. As the boxes represent averages, it is logical that there would be no moustaches for the group of 18 judges given that there is only one answer. Conversely, these moustaches are bigger when they combine the separate opinions of the 18 judges on the far left of the X axis.

Very competent non-experts

A lot of information can be obtained from this figure. The correlation coefficients of the experts and non-experts are similar overall. This signifies that the non-experts are also “objective” when they judge singing accuracy. The results are also well below the red line which means that they are significant. “In any event, the researcher points out, “The moustaches for the non-experts are bigger than those for the experts. This means that the experts supply more conditioned answers than the non-experts which are more variable in their evaluations. We can see that a common sensitivity appears here. In fact, if the non-experts were not fully in agreement with each other, the moustaches would cover the entire box. After a certain number of judges, the non-experts show performances that are almost equivalent to those of the experts”.

A second observation concerns the two tests offered to the non-experts. The figure shows that they react in a similar way both times. “This is encouraging. When the same experiment is carried out twice in a relatively short space of time, we want to find out if there has been a change in strategy or an effect of learning. That is not the case in this instance”. This result bears testimony to the fact that the judgment of the non-experts is stable from one time to another and that their level of inherent knowledge is high enough to rule out a significant learning effect during the experiment. The figure shows more slightly smaller moustaches in the third column which signifies less variability between the evaluations of the different judges.

This figure illustrates the strong relationship between the accuracy criteria measured and the evaluation by groups of listeners but it does not make it possible to say what predict or explain the evaluations. In other words, what are the objective criteria for accuracy that our ears are sensitive to? “By means of the statistics, we can observe that both the experts and non-experts do not attach a lot of importance to melodic contour to judge the level of accuracy of a song”. Conversely, errors linked to the intervals were taken into account by the two panels. On the other hand, only the experts seemed to have taken account of changes in tonality during the same piece of singing. “This fact may appear to be very surprising. From another viewpoint, this notion appears later in the development, both in terms of perception and musical production. This does not however mean that the non-experts do not hear this type of error. It means that this is not a sufficiently important criterion for them and does not appear as a predictive variable in our statistical model”.

Statistics to transform the subjective into objective

The study shows that the faculty to distinguish accurate from inaccurate singing results, in part, from implicit musical learning and this prior learning applies to a majority of the population. The number of years of music practice results in competence but does not specifically make for better judges. More broadly, what interests Pauline Larrouy-Maestri, is to do justice laypeople and go beyond popular belief in order to understand where differences come from. “In many studies, the differences between the two groups are overestimated simply because one of the two groups does not understand what is expected of it. In the present case, I ask a simple question. Is a particular song in tune or not? If I had asked for a list of inaccuracies related to intervals or to judge whether the piece was sung in the correct tonality, I would have had much better results among the experts than among the non-experts. These results would probably have been linked to the fact that the non-experts had not understood the question. They would have demonstrated nothing in terms of their real perceptual abilities. The method developed here allows us to ask a simple question, accessible to all, and then, by means of the computer programs and statistical tools, to examine the objective reasons that can influence their answers. We then outline the relationship between clear criteria we have identified and the evaluation by the judges. We therefore seek to understand the reasons for their judgements. This type of method seems to me to be particularly interesting and promising for the study of the sensitivities or judgements which initially seem very subjective, whether in the area of music or in other fields of research”.