Testing ≠ Reviewing: A Blind Coffee Test Full of Flaws

On September 4, the Hangzhou Consumer Protection Committee organized a "Takeaway Coffee Consumer Representative Tasting Meeting" and selected 9 citizens to conduct a blind test on the coffee of Starbucks, COSTA, Luckin Coffee, KFC, McDonald's, Ksamiq, and FamilyMart (Pai Ke Coffee).

The test results show that Pai Ke Coffee topped the list, with Starbucks and Luckin Coffee at the bottom. This set of "reversal" rankings was widely circulated by many media and attracted the attention of many consumers. However, can this blind test result represent the market taste and is the rating really accurate?

First of all, the test itself is full of loopholes and inconsistent standards. According to the "Evaluation Registration Form", the "overall value" of the seventh dimension is not the average score of the previous six dimensions (cleanliness, sweetness, mouth feel, fragrance, aftertaste, and balance), nor is it the comprehensive score of the previous six dimensions. It is the same as the evaluation of the previous six dimensions, which is the subjective "overall impression score" of nine people. The data cited by most media is the "overall evaluation", which is neither comprehensive nor objective.

According to media reports, the Hangzhou Consumer Protection Committee selected seven brands of large cups of latte for evaluation, and specifically marked "sugar-free, cream by default". Through the Luckin Coffee APP, it was found that Luckin Coffee did not have "sugar-free, cream by default" latte, which means that the seven brands of coffee tested this time are not the same type of coffee.

In addition, according to the sweetness option in the "Evaluation Registration Form", there are three dimensions in the evaluation of sweetness: moderate sweetness, strong sweetness, and too sweet. In fact, latte itself does not contain sugar, and the sweetness of coffee and milk itself cannot reach the level of strong or too sweet, which makes this "blind test" seem to be out of common sense and too ridiculous.

In addition to the "randomness" of the selection criteria and products, the professionalism of the nine citizens is also difficult to verify.

On August 29, the Hangzhou Consumer Protection Committee published an article titled "Invite you to do a blind coffee test! Judge which coffee is the most "conscientious"!" on the "Hangzhou 315" WeChat official account, which stated that 10 ordinary coffee lovers and 10 media representatives would be invited to serve as judges for the blind test.

The tweet was read more than 400 times. The exact number of people who signed up for the test is unknown, but for some unknown reason, only 9 citizens participated in the test in the end.

Although they are ordinary coffee lovers, some citizens who participated in the test have been drinking coffee for more than 20 years, but they did not taste the Starbucks they have been following for many years. The person in charge of the Hangzhou Consumer Protection Committee believes that "a thousand people have a thousand tastes, and their coffee scores are also based on personal preferences. Therefore, this is a social, pure consumer test."

Therefore, this is not a "coffee evaluation" in the strict sense, but more like an entertaining "test". The data is not of reference value, but the impact it brings is not small. The results of such gamification are quickly used and promoted by commercial interests, causing adverse effects on other coffee brands and companies, and are irresponsible to consumers.