It takes plenty of practice to become a skilled science communicator—and “skilled communication” is more subjective than researchers have assumed, a team of researchers found out when they tried measuring the success of their own graduate science communication course.

Despite the fact that most of us learn to talk around age three, effectively conveying information to another person so they act on it can be a lifelong challenge. George Bernard Shaw memorably said the single biggest problem in communication is the illusion that it has taken place. The playwright had a point.

If merely communicating is widely recognized to be difficult, communicating about science can be doubly so. Particularly in the United States, clashes over values and culture and the politicization of certain topics can make it hard to find a common place to even start a conversation. But as a society we face challenges from climate change, viruses, decaying infrastructure, agricultural failures, and many other issues that could be profitably informed by scientific data—if we could only find a way to communicate with each other.

A UConn science communication course jointly offered by the departments of Journalism and Ecology and Environmental Biology (EEB) has tried for 15 years to help students do exactly that. The class brought journalism and graduate science students together to learn from each other. The journalists learned how to question scientists and write about arcane concepts for regular readers, and the scientists practiced explaining difficult ideas, with the resulting news stories for feedback.

But after more than a decade of teaching the class, journalism professor Bob Wyss, former journalist and EEB staff scientist Bob Capers, and EEB professor Margaret Rubega decided they needed a way to evaluate how well the class was actually working. Did students really improve their communication skills? Self-evaluations by the students always gave the class high marks, but self-evaluations are notoriously unreliable. As Rubega points out, everyone who takes a driver’s education course feels more confident about driving—but confidence does not a good driver make.

Instead of self-reports, Capers, Rubega and Wyss ran a three-year study comparing their science graduate students’ spoken communication skills before and after the class with those of equivalent grad students who did not take the course. They teamed up with Andrew M. MacDonald at the Université de Montréal, and UConn Communication professor Anne Oeldorf Hirsch and EEB post doctoral researcher Kevin Burgio to conduct the study. They had the hundreds of undergraduate students who take Communication 100 class each year be the judges of communication quality.

The results, published in Science Communication in December, were not what the researchers expected. Not only did the graduate students who took the science communication course not improve, but the audience themselves did not agree about who was, and was not, a good communicator.

“How do you measure effective communication? It’s hard, so context specific,” says Oeldorf-Hirsch. She points out that there are many goals you can have for a specific message. Do you want the listener to remember it? Like it? Understand it? Do you want to persuade them? For the purposes of the study, the team decided to have the undergrad judges focus on just three things: the speaker’s clarity, trustworthiness and engagement.

But even for those simple goals, the audience did not agree. Even the graduate student with the highest average rating was scored low by some reviewers, and the graduate student with the lowest average rating still received some high scores.

“We went into this thinking we’d get an answer,” to whether the class was working, but “instead, we found a whole mound of complexities,” says Wyss.

They have lots of questions about the meaning of the results. One guess is that a one-semester course just doesn’t provide enough time, or practice, to improve working communication skills.

“Nobody masters a complex skill in six months. It takes time and practice; you have to do it over and over again,” says Capers.

Oeldorf-Hirsch says it may also be that there are unique challenges to communicating about science. Or that we aren’t measuring the right things. One thing is sure; if the semester-long class with multiple opportunities to practice science communication is not effective, the many two-hour or day-long seminars on science communication are probably not any better, at least as standalone classes.

But that’s not to say they are useless, either.

“We don’t believe for a minute that students in our classes learn nothing,” Rubega says. “It is clear from their performance in class that they know a lot about the pitfalls of communicating to public audiences, and the theory of what successful communication should look like. What our study shows is that knowing about it, and being able to actually communicate successfully with an audience, are different.” The other researchers agree. And they’re also investigating now whether written communication is different than spoken; they’re still evaluating results from a different class on writing about science.

And if the effectiveness of a class is measured by the outcomes, the science communication class has had some pretty good ones. Take Kevin Burgio, who worked as a post doc on this study. He took the science communication class as an undergrad in 2009, and then several times as a graduate student. He subsequently published a science story he began writing in class through a syndication service and participated in a science storytelling workshop in 2018. And now he’s the Director of Conservation Science at the New York City Audubon, a role that requires significant outreach to the public about science.