May | June 2020
Free Article Preview from this issue of Roast magazine
Story and photos by Anne Cooper
WHETHER WE WORK AT SMALL, MIDSIZE OR LARGE-SCALE ROASTING COMPANIES, we all need to evaluate the success (or failure) of our production roasting in some form. How else will we know if the coffee we have roasted is as intended and the flavors of the end product are “true to taste”?
Members of the coffee industry often say that quantifying roast quality in general is a huge weakness in our industry, and I have to agree.
The historically low barrier to entry into coffee roasting brings with it a level of inconsistency in skills, technique and overall base understanding of the processes and procedures required for production roasting.
Also, to an extent, most roasters are self-taught (or were taught by another roaster who was likely self-taught) and, as a result, have developed their own methods. This also contributes to a lack of consistency and understanding of some key production roasting processes and procedures.
Production roast evaluation cuppings.
Coffee roasting encompasses production and manufacturing, and as such, it needs to be considered more seriously. As a result of not being clearly defined and monitored, the roasting industry lacks strict procedures and unified documents for essential practices like production roast evaluation (PRE).
When I polled many roasters on the topic of PRE, it became clear that within the global coffee roasting community, roasters use varying techniques and documents to evaluate production roasts.
So, what should the coffee industry be considering to help roasters improve their PRE skills? What factors are involved, and how can we further improve PRE processes and procedures to ensure that we are all consistently evaluating our production coffees as being true to taste?
Production roast evaluation cuppings.
Why Do We Need to Perform Production Roast Evaluations?
PRE is a critical activity for coffee roasting companies to promote consistency, reinforce roasting skills, gain product knowledge, and stay true to the intended flavors of the final roasted product. As roasters, we are in the business of flavor.
Regularly performing PRE is an ideal opportunity to continually improve roast profiles, as PRE is the ultimate connection between the roast profile and desired end flavor of a specific product.
Along with assessing intended flavors, PRE also helps to assess and uphold a brand or individual’s roasting style or ethos. For example, a business owner not involved in the roasting process might have a “rogue roaster” who is doing their own thing, being influenced by current trends and ignoring specific brand flavor requirements; this sort of individuality can destroy a business. And if you are not the owner of the roasting business, you need clear guidance from management on how they want the coffees to taste so you can then use your roasting skill and knowledge to develop an appropriate roast profile that will deliver the specific product flavors to be reflected in the PRE.
Doing PRE also helps roasters to monitor, over time, the properties and flavor characteristics of the green beans they’ve purchased (which may be affected by roasting room and storage conditions), as more than ever we are in an era of experimental processing and unique flavors.
Additionally, PRE is beneficial for promoting better understanding of the roasting machine (its controls, burner, drum and probe types and more), as well as troubleshooting any roasting machine issues, such as general set-up and limitations in power and controls, which can cause tasting and roast defects from not being able to deliver the roast profile or “cooking strategy” you want to achieve.
PRE is also useful for building consistency in roasting teams. The process presents valuable opportunities to learn from other team members with different levels of experience, who calibrate and compare with each other. The team explores and learns how to properly evaluate what they are meant to be tasting and achieving in a product, brand and market with a specific roast profile and end flavor.
When Is the Best Time to Assess?
Ideally, as part of a roasting company’s standard quality control procedures, roasters should be doing their PRE before the final, roasted product reaches the customer.
When polling roasters on when the best time is to carry out PRE, the majority said the day after roasting. Others would also assess over a range of time (anywhere from 24 to 48 to 72 hours to 7 days after roasting), as the coffee rests and degasses.
But still there is no industry standard for when to perform PRE, possibly due to the wide range of experience, strong individual opinions and influences, and number of samples to be evaluated, as well as scheduling and managing business costs among the many different small and large-size roasting companies in our industry.
Not ever tasting or evaluating your roasts is poor practice but was surprisingly prevalent among roasters I polled. No roaster should be roasting coffee without taking the time to taste/evaluate their own coffee. Some roasters say they use customer feedback as their evaluation gauge—but waiting for customer feedback to roll in weeks after roasting the coffee is too late, especially if the feedback is not good.
When possible, tasting in-store with your customers (after you have done your own PRE) is good practice and a wonderful way to get to know your customers and build a good, trusted business relationship with them as well.
Often, roasters also claim that they are too busy or do not have enough time to do PRE.
If you are short on time, be more strategic and do your PRE during the pre-heat cycle of your roaster—this is a great time to be doing your tasting. I certainly understand and relate to the pressure for time, the need to multi-task and “get things done”—a sentiment that was strongly echoed by many roasters I polled. However, not supporting your PRE with a reliable sample and data collection routine will affect your ability to make any good evaluations about your roasting.
Throughout your production day, you can also prep your samples to be tasted while roasting (that first 5 minutes of the roast is a great time to do this—social media and other distractions can wait!) or use other strategies, like having your packing team collect samples based on your instructions with the sampling code/number so that it matches your roast data, and you can keep multi-tasking and smashing out your production roasting.
While the timing of when to do PRE may vary between roasters based on their company’s specific operating procedures, one should always head into a roasting session with an idea of what the coffee should (or will) taste like—as every roasting session is an opportunity to improve or fix a previous mistake.
How else will you know if your roast profile is on point and the end product flavors are true to taste?
FIGURE 1. Production Roast Evaluation
Knowing the Purpose
The No. 1 rule of cupping, tasting or evaluating roasted coffee is to always know the purpose. Why are you evaluating this coffee—is it for purchase or production? If you are determining whether to buy a specific green coffee, then you are looking for characteristics inherent to the green, thus requiring a scoring scale relevant to a level of quality that reflects specific green characteristics.
But for production, in my experience, it is more about evaluating the intensity of a specific quality (in terms of low, medium or high) that contributes to an overall flavor expectation (see Figure 1 above).
We can then assess whether those flavors have met product flavor expectations (are true to taste) on a scale of 1 to 3 (1 = not ideal; 2 = O.K.; 3 = ideal), because now we are also thinking about suitability for an intended brew method or recipe, such as filter or espresso.
So, with PRE, we are essentially assessing the success of our roast profile—how have we as roasters influenced the intensity of individual qualities and desired flavors with our roast profile?—regardless if it is a commercial or specialty grade coffee/product. Furthermore, we are now looking at the question of whether the coffee we are evaluating suits an intended brew method—as each brew method has its own specific roast profile approach suited to individual markets or palates.
This is where the “weakness” theory and PRE starts to get a little bit more layered and complicated. However, a clear quality control process will help to give better insight to your PRE, roast profile and everything else involved in creating your desired end flavors.
Quality control processes in action.
Establish a Clear Quality Control Process
Currently with PRE, there are many different quality control (QC) approaches and techniques being employed by roasters worldwide and traditionally always done in a cupping format. When doing Q-grading or evaluating green coffee for purchase, there are strict QC industry guidelines to follow that could (and should) be the case for evaluating production roasts as well.
In my experience, and among polled roasters, a good QC process for PRE involves:
FIGURE 2. Flavor tongue map
There are great tools out there to help with practicing flavor identification, but nothing beats the real thing—especially from a cultural palate perspective, as you quickly learn that tasting a strawberry in Australia (quite acidic) is very different from tasting a strawberry in Korea (incredibly lush and sweet), and this does have an effect on perceived cultural palate flavor descriptors in coffee.
Some roasters do go over the top with their QC process and get a case of analysis paralysis by evaluating far too many cups at once. This technique can hinder your ability to make good decisions about your roasting, as you’ll be greatly fatigued from the large number of samples. You won’t have enough time to evaluate properly and provide appropriate feedback or comparison among team members, thus defeating the purpose of the cupping/tasting in the first place. Cupping and tasting should not be done simply for the sake of tasting, but rather with a purpose to evaluate and learn from your roasting successes and mistakes.
Practice tasting and describing different flavors.
Another QC issue to highlight is determining the best method of brewing/extraction for PRE. Should production roasts be evaluated using the intended brew method, such as espresso, rather than in the cupping format?
Many roasters are comfortable enough to QC their production roasts as espresso the day after roasting and determine whether it is on track or not. This may challenge the theories around freshness for espresso extraction that exist in our industry, but if a roaster is consistent with this evaluation technique and can successfully move on to their next roast session confident that everything is as it should be, then they are achieving the main goals of their PRE.
So, a good, clear QC process is essential to the success of your PRE, but it also needs support from your roast data.
Data is Knowledge, Knowledge is Power
Data collection is an area of weakness for some roasters, who can fall victim to the “I’m too busy” excuse. On the other hand, it can also be overdone to the point that the wrong kind of data is being collected, which then bears no value in terms of being able to use it to make good roasting evaluations/decisions.
In my experience, whenever I am tasting or evaluating my production roasts, I always have these key data documents available for reference during PRE discussions on the success or failure of a roast profile (and I encourage others to do so):
FIGURE 3. Daily Roast Log
FIGURE 4. Master Production Evaluation Sensory Form
FIGURE 5. Parameters Chart
It is essential to keep some form of daily roast log with a record of key milestones and checkpoints achieved throughout the roast. In my production roasting, I always record the timing of physical changes—from green to yellow, yellow to first crack, and first crack to end—because this is the key to consistency in the roast profile and end flavors. (For more on this, see “Can You Taste the Roasting System?” in Roast’s May/June 2019 issue.) Some may say it is “old school” to follow the color/physical change of the beans or that it doesn’t matter—don’t let them convince you of this. Color change, or more specifically the timing of color/physical changes and milestone events throughout the roast, is imperative to consistently creating specific flavors (see Figure 6 below).
Specific products will then each have their own desired timing for these milestones, as well as parameters for defining the overall end roasting style. I specifically developed a parameters chart through my own extensive roasting and training experience to help support roasters, and show them how to finish their roasts for a particular product in line with their specific machines’ probes, because knowing and deciding how to finish a roast is one of the most terrifying and daunting parts of learning how to roast!
If you have these documents next to you when evaluating your production roasts and identify a product’s flavor as being “out of spec” in comparison to the benchmark Master PRE, you’ll be able to confirm this by checking the roast profile data on the daily roast log against your unique, desired product parameters chart, see where it’s not quite right, and then know how to fix or adjust your roast profile accordingly. It’s a great feedback loop and can make doing PRE much easier.
FIGURE 6. Color Change Throughout the Roast
How Best to Communicate
Finally, alongside the clear QC process and supporting data documents, we need to be able to consistently communicate and record our PRE in an appropriate format. Communication is the last piece in the “weakness theory” puzzle because PRE requires a more specific production evaluation form, which our industry doesn’t seem to have—yet.
A PRE form I created for myself (and others) to use is focused on being able to properly record and communicate levels of intensity of a particular quality (as low, medium or high) and then decide whether those levels assessed are true to taste for the specific product flavors, on a scale of 1 to 3 (1 = not ideal; 2 = O.K.; 3 = ideal). This format can be used for different brew methods as well (see Figure 7 below).
FIGURE 7. Production Cupping Form
Figure 8 (below) is an example of an evaluation form where the Brazil is in spec and the Colombia is not quite in spec but, because the supporting roast and product data is available for reference and comparison, we can determine how to start fixing the roast profile for the Colombia and get it back into spec.
The majority of roasters I polled do generate their own PRE forms, which confirms the need for a more consistent, industry-wide PRE document. Many other roasters simply make general notes on a blank piece of paper, noting what they like and don’t like without any supporting documentation, which clearly doesn’t help them to benchmark, progress or adjust their roasting accordingly.
FIGURE 8. Evaluation Form Example
But a lot of roasters still make the mistake of using the wrong form of assessment for evaluating their production roasts. Unfortunately, that means using the SCA arabica quality/purchasing evaluation/cupping form (see Figure 9 below). In PRE, this form can really send a roaster on the wrong path because its evaluation structure of generating a “quality score” doesn’t match or reflect the purposes of evaluating and focusing on the “intensity” required in production roasting.
FIGURE 9. SCA Cupping Form (Not ideal for production roasting evaluation)
I can understand that roasters—especially those who are new to the industry—want to find something with a base of consistency to use for their production evaluations (which is possibly how the use of the SCA form gained traction), but does it really allow us as roasters to see how to connect to our roast profile and identify anything that needs fixing? Unfortunately, the answer is no.
Hence, this is where a major weakness lies in PRE.
I also believe that misusing the SCA form for PRE is contributing to the degradation of the true value and purpose of the Q/SCA 80-point specialty coffee evaluation cupping form. Thus, I appeal to roasters to stop using this evaluation form for their production roast evaluations.
There are fabulous cupping apps out there, but I have chosen not to delve into these as not everyone uses them or likes using them. Using these apps is not currently an industry standard, although perhaps it could or should be. Even though they are still score-based, some cupping apps are certainly leading the way by adapting the PRE documents that roasters have created for themselves into an easy format to help them carry out their PRE more consistently and efficiently.
All this being said, though, one can understand why many have commented that quantifying roast quality in general is considered a weakness in our industry.
Coffee is essentially loved by everyone, and everyone has a strong opinion about how it should taste, and how it is best roasted and evaluated across different brewing methods and cultural markets.
A first solution could be as simple as replacing “balance” with “true to taste” as the more clearly defined end purpose in our production evaluations. In fact, true to taste must become a universal part of our PRE vocabulary, as it brings with it so much clarity, consistency and confirmation of a decision on what an end product flavor should be, and it clearly captures the essence of what PRE is all about—we either nailed our roast profile and intended flavors, or we didn’t.
And regularly doing production roast evaluations with supporting data, a clear QC process and a purposeful PRE form will always ensure success in your roasting and an end product flavor that is always true to taste.
The reality is, due to the range of roasting companies (commercial and specialty) in our industry, we’re not all operating in a “protocol evaluation world” as much as we would like to. So, as an industry, we need to find some consistent ways to help satisfy this variation in companies and PRE techniques—just how, and through whom, remains the conundrum.
Overall, as roasters, we are in the business of flavor, and we need to know what we have done to influence end flavors, how we did it and how to fix it if needed. And a unified, standard, purposeful, overall-industry-accepted production roast evaluation document will definitely help to set our dynamic industry on the right track.
ANNE COOPER is an Australian roaster who has gained extensive experience in her 25 years in the coffee industry. A former member of the Roasters Guild Executive Council and Education Committee, Anne now works with her training and consulting company, Equilibrium Master Roasters. There, she continues to inspire flame keepers and develop her extensive roasting skills and knowledge while working on a wide range of roasting machines with small- and large-scale roasting companies, employing various roasting processes and techniques.