Monday, July 4, 2016

Chocolate milk and concussions

Can chocolate milk actually help athletes recover cognitive function faster after concussion? A press release from the University of Maryland says that it can. In fact, the press release says that a study by a researcher at the University of Maryland School of Public Health found that drinking Fifth Quarter Fresh® fortified chocolate milk, with nothing particularly special added to it, did help athletes maintain cognitive function or recover it more quickly throughout the football season. According to the manufacturer, this milk comes from special cows that produce an especially healthy kind of milk.

At least one school district in Western Maryland has said that it will buy that milk and make sure that their kids in that district get it. No doubt, other people around the country, excited by the discovery that drinking chocolate milk from special cows can counteract the impact of concussions, will be enthusiastic about it too.

But they shouldn't be. It turns out that when journalists began to probe the facts behind the claims, they found that no study had been published—or even submitted—to prove the points that were in the press release. The researcher has not even presented these findings at a public meeting. 

It also turned out that Fifth Quarter Fresh milk is partly subsidized by a state program in Maryland and partly by the dairies that make it. Thus, the press release is talking about a product that the school and state industries are trying to promote—Maryland cows' milk. They are trying to promote the state's dairy industry, which sheds doubt on whether anyone can credit these claims.

I do not know how this press release managed to be distributed without a paper to back it up, but I believe that it has some lessons about what to do when patients come in and say, for example: "I heard that coffee is good for me"; "I heard that I should drink a lot of red wine"; "I heard that I should eat kale every day—and doing this will bring good health." 

All kinds of claims are floating around on the Internet about everything from bee pollen to who knows what, saying that they will help you. What you need to talk about with patients is whether any actual research exists to back up these claims. In the case of the Maryland study, there were no verified data, only an opinion put forward by a particular person at the university. You need to ask whether the information has been confirmed. Determine whether there is a second study that says, "Yes, we can verify what the first study found about red wine or kale or bee pollen or honey or coffee; we have replicated the results."

I do not believe that people should be changing their lifestyles or spending money on magical food remedies or preventatives unless more than one paper published in some reputable journals back it up. I do not expect every doctor to know whether there's a study out there to back up every particular claim that is made, but if patients are going to bother to find this stuff on the Internet, they can bother to find out whether there is a journal or a scientific article—or what the foundation is for the claim. I believe that you can ask the patient to do that…

The lesson of chocolate milk is: Let the buyer beware. You cannot always trust what makes it into news reports. You cannot always trust what makes it onto the Internet. Conflicts of interest can sometimes fuel premature announcements about the benefits of things. Before they change their lifestyles or empty their pocketbooks, you and your patients must fully understand that they need evidence to show that it is worth their while.

Art Caplan at the Division of Medical Ethics at the NYU Langone Medical Center in New York.


  1. The original press release. An update added later will follow.

    COLLEGE PARK, Md. — Fifth Quarter Fresh, a new, high-protein chocolate milk, helped high school football players improve their cognitive and motor function over the course of a season, even after experiencing concussions, a new preliminary University of Maryland study shows.

    The study, funded through the Maryland Industrial Partnerships program and conducted by Jae Kun Shim, a professor of kinesiology in the School of Public Health, followed 474 football players from seven high schools in Western Maryland throughout the fall 2014 season.

    “High school football players, regardless of concussions, who drank Fifth Quarter Fresh chocolate milk during the season, showed positive results overall,” said Shim. “Athletes who drank the milk, compared to those who did not, scored higher after the season than before it started, specifically in the areas of verbal and visual memory.”

    Football players were tested before the season, after concussions and post-season using Immediate Post-concussion Assessment and Cognitive Testing, also called ImPACT™, a widely used computer-based evaluation for concussions. Overall, 36 variables for attention span, working memory, sustained and selective attention time, response variability, non-verbal problem solving and reaction time were measured in the study.

    Experimental groups drank Fifth Quarter Fresh after each practice and game, sometimes six days a week, while control groups did not consume the chocolate milk. Analysis was performed on two separate groups: athletes who experienced concussions during the season and those who did not. Both non-concussed and concussed groups showed positive effects from the chocolate milk.

    Non-concussed athletes who drank Maryland-produced Fifth Quarter Fresh showed better cognitive and motor scores over nine test measures after the season as compared to the control group.
    Concussed athletes drinking the milk improved cognitive and motor scores in four measures after the season as compared to those who did not.(continued)

  2. (continued)The remaining test scores did not show a statistically significant difference between the experimental and control groups over the season, according to Shim.

    He suggested that the naturally occurring high levels of specific nutrients in Fifth Quarter Fresh likely contributed to the results. “Branched chain amino acids (BCAAs) are important for energy metabolism and neurotransmitter synthesis in the brain,” said Shim. “Previous studies have shown that BCAA supplementation has resulted in improved cognition in mice with brain injuries.”

    Shim also cited carbohydrates, calcium and electrolytes, all of which he says are likely to be critical for the recovery process after brain injuries.

    While the study’s results indicate a strong link between milk and the reduction of concussion-related symptoms, researchers caution that more in-depth studies are necessary to be conclusive.

    Fifth Quarter Fresh is a fat-free chocolate milk made by combining nutrient-rich milk (yielding 40 percent more protein, calcium and electrolytes than conventional milk) with the benefits of a pasteurization process that preserves proteins and makes them easier for the body to absorb, according to the company.

    Fifth Quarter Fresh has a balance of fast-absorbed whey and sustained-release casein proteins that provide a quick burst of amino acids followed by a continuous supply over several more hours, according to Richard Doak, co-founder of Fifth Quarter Fresh.

    The company maintains that protecting student athletes and helping them perform at a higher level was the reason they created Fifth Quarter Fresh in the first place.

    “We believe there is a real need to improve nutrition for young athletes. Fifth Quarter Fresh may help them prevent injuries by providing their bodies with the nutrients they need to heal and repair. This study suggests that,” said Doak. “Our milk provides 20 grams of protein and five grams of undamaged BCAAs per 14-ounce serving—naturally. We use no supplements and no preservatives—it is fresh chocolate milk.”

    Officials in Washington County, Md., home to all seven high schools participating in the study, are now considering the broad adoption of Fifth Quarter Fresh in sports programs throughout its school system.

    “There is nothing more important than protecting our student-athletes,” said Clayton Wilcox, superintendent of Washington County Public Schools. “Now that we understand the findings of this study, we are determined to provide Fifth Quarter Fresh to all of our athletes.”

    Earlier this year, UMD released the preliminary results of a study showing that Fifth Quarter Fresh outperformed leading commercial workout recovery drinks for endurance recovery by 13-17 percent.

    Fifth Quarter Fresh is produced through the Hagerstown-based Lanco-Pennland Quality Milk Producers, a farmer-owned, farmer-run cooperative with nearly 650 members that spans the U.S. East Coast. Frederick-based Dairy Maid Dairy bottles it.

    The University of Maryland study was made possible by the Maryland Industrial Partnerships (MIPS) program, which jointly funds commercial product development projects teaming Maryland companies with University of Maryland faculty.

  3. The update to the press release added later:

    UPDATE: This press release refers to study results that are preliminary and have not been subjected to the peer review scientific process.

  4. The press release from the University of Maryland claimed that Fifth Quarter Fresh has helped young football players improve cognitive and motor function during the football season even if they suffered concussion injuries.

    Professor Jae Kun Shim, who conducted the study, suggested the levels of nutrients in the milk likely contributed to the results. He explained that the chocolate milk contains branched chain amino acids, which are important for energy metabolism and neurotransmitter synthesis in the brain, along with high levels of carbohydrates, calcium and electrolytes.

    The study also coincides with the release of the movie ‘Concussion’ which has got people thinking more about the damaging effects of collisions in sport. Concussion is the extreme end of these effects, but constant milder impacts appear to still have an incremental damaging effect on the brain.

    It’s exciting to think that there might be a way of reversing damage to the brain that can be caused by the constant jolting of the brain in sports like football.

    However the study has been heavily criticised for these reasons:

    It has not been published in any scientific journal for peer review. It’s currently only been published as a press release.

    There are no comparative reports with other chocolate milk drinks – or even plain milk.

    The study was part-funded (10%) by the Maryland Industrial Partnership of which Fifth Quarter Fresh is a member.

    Although the beverage is fat-free, it contains 42 grams of sugar: over half of this is added sugar.

    The University of Maryland has now issued a statement saying the results were ‘promising’ and ‘preliminary’ but that their researchers advised more in-depth studies were required to be conclusive.

  5. The University of Maryland, College Park, approved a study that linked a specialty chocolate milk to faster recovery from sports and even concussions despite serious concerns raised by reviewers that the proposal lacked an understanding of how to conduct nutritional research among humans and that the lead researcher lacked experience in the study area, according to a university review committee.

    Chief Research Officer Patrick O’Shea in January initiated an institutional review of the study after media reports questioned the procedure, findings and promotion of the study before it had been peer-reviewed, published or even written in full. In a report published Friday, the committee concluded that the study’s lead researcher, associate professor of kinesiology Jae Kun Shim, violated university regulations for failing to report $200,000 given to his research lab by the Allied Milk Foundation as a conflict of interest and that his conduct during and after the study represented “significant deviations from accepted practices in the conduct of research."

    In all, the committee’s 23 findings and 15 recommendations reveal a lack of scientific review for a project that involved human subjects, including minors; inappropriate product endorsement; and a troubling lack of comprehension of conflict of interest ethics among faculty and administrators. While these findings could indicate systemic problems, committee chair and former University of Maryland provost Ann G. Wylie emphasized in an April 1 conference call that the report speaks only to the single study. O'Shea said he believes the failings were an isolated incident, but that the university would be "vigilant."

  6. News releases from the University of Maryland July and December, 2015

  7. The bulletin atop a University of Maryland news release was provocative: “Concussion-related measures improved in high school football players who drank new chocolate milk, U-Md. study shows.”

    But an update posted below that finding in late December added a backpedaling caveat rarely seen from a major research university: “This press release refers to study results that are preliminary and have not been subjected to the peer review scientific process.”

    The December news release touting a beverage called Fifth Quarter Fresh has become a significant embarrassment in College Park as officials scramble to learn how and why it was published prematurely. The beverage is produced by a small western Maryland company that helped fund the study, through a program based at U-Md. that connects businesses with universities for product-development research.

    On Friday, U-Md. convened a high-level panel to review the episode. Chaired by former U-Md. provost Ann Wylie, the panel also includes the vice provost for research at Johns Hopkins University, Denis Wirtz, as well as College Park’s dean of behavioral and social sciences and two senior faculty members who monitor conflict-of-interest issues.

    Patrick O’Shea, the university’s vice president and chief research officer, said he assembled the panel because he wants to ensure that College Park’s communications about scientific research are trustworthy at a university with a global reputation for research prowess. Hundreds of millions of dollars a year flow through U-Md. for research and development.

    “I don’t want this kind of thing to happen again because I value the information we give to the public. The information should be reliable,” O’Shea said. “I also value the reputation of the university. We have the public interest at heart, and the reputation of the university is connected to that. The public should be able to rely on what we say.”…(continued)

  8. (continued)MIPS teamed Fluid Motion LLC of Keedysville, Md., with Jae Kun Shim, an associate professor of kinesiology at College Park.

    Fluid Motion began producing Fifth Quarter Fresh in 2013, said co-founder Richard Doak. The chocolate milk is marketed as a high-protein, post-workout recovery drink. It is sold in Hagerstown area stores and directly to some colleges and high schools, Doak said. Bucknell University is a customer.

    Doak and his colleagues wanted to know what effects their beverage would have on consumers. Shim’s research interest was in brain and motor performance control.

    In an email, Shim said he won two $100,000 grants through MIPS. One, in August 2013, was to study the effects of post-exercise recovery drinks on strength and endurance. The second, a year later, was to study the effect of the drinks on post-exercise recovery and cumulative minor brain trauma.

    In each case, Fluid Motion paid 10 percent, and MIPS paid the rest. Shim said he had no financial interest or other connections to the company…

    The second news release drew far more attention because it focused on concussions and football, a subject of huge national interest. It was released at about the same time as the movie “Concussion,” starring Will Smith, a drama about efforts to force the National Football League to take brain-damage issues seriously.

    The December release said the second study tracked 474 football players at seven western Maryland high schools in fall 2014. Experimental groups drank Fifth Quarter Fresh after practice and games, and control groups did not. The players were tested before the season, after concussions and after the season to examine their attention span, working memory, nonverbal problem-solving and other cognitive indicators.

    “High school football players, regardless of concussions, who drank Fifth Quarter Fresh chocolate milk during the season, showed positive results overall,” Shim said in the release. “Athletes who drank the milk, compared to those who did not, scored higher after the season than before it started, specifically in the areas of verbal and visual memory.”

    The claims in the release quickly drew scrutiny from journalists who wanted to see the full study, a routine question. But in this case, there was no formal report. The most that U-Md. could provide to those who asked was a brief PowerPoint slide presentation that charted and graphed some of Shim’s data.

    The lack of supporting detail for a university-issued news release boosting a company product was astounding to critics. “The University of Maryland has a burgeoning chocolate-milk concussion scandal on its hands,” wrote Jesse Singal, a senior editor at New York magazine...

    “I believe that any release of information related to public health should be handled carefully,” he wrote to The Washington Post. “In regards to this study, more care could have been taken in releasing preliminary study findings before [the] peer review process.” The two MIPS studies, he said, were the first in his career shared publicly before undergoing peer review...

    “I have devoted my life to research and education, and my work has been guided by integrity and ethics,” Shim said in the email. “Contrary to some of the media reports, researchers have no reason to ‘help’ a company. That is certainly true in this case. My intentions with these studies were to pursue discovery of new knowledge through scientific investigation.”

    Shim said he plans to submit manuscripts for peer-review journal publication, a process that can take several months. He said the PowerPoint slides were not meant to be shown to the public as a final product, but were developed as an internal update on the trend of the findings.

  9. On December 22, the University of Maryland published a remarkable press release about some research it had conducted. According to the release, a study conducted by a professor at the UMD School of Public Health had shown that a product called Fifth Quarter Fresh — basically, a fancy, fortified chocolate milk — “helped high school football players improve their cognitive and motor function over the course of a season, even after experiencing concussions.”

    Given the current focus on youth concussions, it’s no surprise that this news traveled fast and that the claim appears to have benefited the company in question. Motivated by what appeared to be sturdy scientific evidence, a rural school superintendent in Maryland said he’s “planning to buy $25,000 worth” of the stuff next year to help protect his kids, reported STAT News. The claim also caught the eye of an editor at, who asked Andrew Holtz, a writer there, to dig into the details. So he did something science journalists do every day: He reached out to Maryland to get a copy of the study the press release was written off of.

    The release in question, published December 22, said it was pegged to a study by Dr. Jae Kun Shim, “a professor of kinesiology in the School of Public Health.” It had to do with cognitive performance, concussions, and FQF. Shim was very enthusiastic about what he had discovered in his study: “High school football players, regardless of concussions, who drank Fifth Quarter Fresh chocolate milk during the season, showed positive results overall,” he was quoted as saying. “Athletes who drank the milk, compared to those who did not, scored higher after the season than before it started, specifically in the areas of verbal and visual memory.” The details were sparse, but the implication seemed to be that the drink could help with concussion recovery. (Shim is out of the country and hasn’t returned two emails I sent him.)(continued)

  10. (continued)Holtz wanted to know if the study matched the hype, but quickly found himself stymied. As he wrote on January 11, Eric Schurr, a communications staffer at the University’s Maryland Industrial Partnerships (MIPS) program, told him that, whoops, actually there was no study. “Not only wasn’t this study published, it might never be submitted for publication,” Holtz wrote. “There wasn’t even an unpublished report they could send me.” The next day, Schurr sent Holtz an email changing his tune somewhat — it read, in part, “Since this is a preliminary study, we have learned it will make more sense to speak with you once there are more conclusive research results.”

    This is exceptionally unusual, and things have only gotten weirder as the details of both this study — really, “study” — and its consequences have emerged. Holtz’s back-and-forth with the university kicked off what is quickly becoming a genuine scandal in College Park — a scandal that touches on vital issues of scientific ethics, the collision of money and research, and the lightning-quick pace at which pseudoscience can lead vulnerable people astray. And it all boils down to a simple question: How the hell could the University of Maryland have allowed this to happen?...(continued)

  11. (continued)A situation in which a glowing study of a product is partially funded by that product’s manufacturers is never a good look, but in this case things are particularly shady. The press release’s summation of the study’s findings point to deal-breaking, get-you-flunked-out-of-stats-101 methodological problems:..

    Comparing those who drank the product to those who did not is virtually meaningless. Maybe something about drinking milk, any milk, helps explain the scores. Maybe drinking anything does. Maybe the kids who were instructed to drink milk at a specific time took their overall health seriously in a way that produced these outcomes without the milk doing anything at all. When you conduct a comparison this simplistic, a million maybes — or potentially confounding variables, as researchers would call them — are released into the atmosphere. Under normal circumstances, it’s the researcher’s job to convince his or her peers, and the public, that there was a maybe-containment mechanism in place.

    My attempts to get a copy of whatever it was that led to this report originally met the same fate as Holtz’s. In a phone conversation with Brown, first she told me she didn’t want to send it because explaining it required expertise she didn’t have. No problem, I replied. Send it over, and I’ll find my own expert. Then she changed tack: Since Maryland was conducting an investigation — more on this soon — she couldn’t send it. This didn’t really make sense either, because either way the study exists and journalists have a right to access it given that UMD touted its findings. After we hung up, I wrote her an email explaining that I had never, in my time working at Science of Us, ever encountered a situation in which a university refused to hand over the study underlying a press release. It was only after that final bit of nudging that she sent over the PowerPoint presentation — apparently, the closest thing to a “study” the university has. I believe we’re the first to publish it, and it’s here if you’d like to read it…

    So that’s where we are: Well after this scandal broke, and well after UMD acknowledged it was serious enough to trigger an internal review, someone could still easily go to that web page and come away believing Fifth Quarter Fresh has magical medicinal properties. If the release is going to stay up at all, rather than be taken down and replaced with a note about what had happened (ideally with, for transparency’s sake, a link to the original with a prominent note indicating none of the claims in it have been verified), the first paragraph of the press release needs to contain disclosures that the research had been partially funded by the company, that none of the claims that followed had been peer-reviewed, and that the entire press release was now the subject of an internal review. None of these things are clear; there’s no reason to believe people aren’t still getting fooled. This is a scandal.

  12. There are serious statistical red flags as well. In particular, Moser highlighted the fact that a number of so-called p-values higher than .05 appeared to have been counted as statistically significant indications of FQF’s powers. A p-value basically just indicates how likely it is that an event occurred by chance. In statistics, it’s standard for the cutoff for “significance” to be set at p < .05, which just means there’s a less-than-1-in-20 chance that a result occurred as the result of random noise rather than some meaningful effect. P-values are far from the be–all and end–all of a finding’s strength, but they’re viewed as a vital first step in establishing whether a given relationship is meaningful. “In research, a significant or positive research finding would typically result in p-values at the .05 level or BELOW, to indicate that the finding observed was at greater-than-chance level,” Moser wrote. “So I am confused as to how the authors of this study can claim such positive results, unless there is something I am missing here.”

    Looking at the slides, it appears Shim simply decided that even though the normal scientific standard for a meaningful p-value is less than .05, he’d go with a looser definition. On the ninth slide, which covers “Participants & Experimental Design,” a nondescript note indicates as such: …Significance level: p<.10…

    That last line indicates that he is adopting a different, less common, and less strict standard for significance. And when you read on, something interesting jumps out: Every single “positive result” associated with the chocolate milk has a p-value that would not be considered significant by the normal, scientifically accepted definition of the term. In other words, it appears Shim cut a hole in the border fence of significance so he could slip his results in under cover of darkness. Dr. Brian Nosek, a University of Virginia psychologist who is also the co-founder and executive director of the Center for Open Science, agreed. He explained that while there are circumstances in which adopting a lower threshold of significance is acceptable, one way to evaluate a decision to do so is to ask whether it suddenly, conveniently gives the researcher far more significant results than they otherwise would have — clearly the case here.

    What’s amazing is that there were actually two “studies” of this sort conducted by Shim. The other one was about the postexercise recovery benefits of FQF, and, wouldn’t you know it, FQF measures up quite nicely there as well. It was published back in July and had all of the same problems as the concussions study. Neither Brown nor Schurr had a copy of that study they could provide me with — they said Shim could, but I haven’t heard back. I asked Schurr if he had seen an actual study or at least a slide deck, and he said via email that the source was “Conversations and debriefs with Dr. Shim” — meaning he hadn’t. I asked Brown point-blank via email if she was sure the study existed, as opposed to the press release just having been written off of some raw data. “The findings report, presumably, would be similar in format to what I’ve previously shared with you,” she said, referring to the concussion slides she had sent over. “Dr. Shim is the right source to confirm that.” Doesn’t sound promising.

  13. See for a link to the PowerPoint presentation. The link is "here" in the 9:10 pm comment above.

  14. Shim said the news release was initiated and developed by the communications office of the Maryland Technology Enterprise Institute, which oversees MIPS.

    “I believe that any release of information related to public health should be handled carefully,” he wrote to The Washington Post. “In regards to this study, more care could have been taken in releasing preliminary study findings before [the] peer review process.” The two MIPS studies, he said, were the first in his career shared publicly before undergoing peer review.

    Shim, 43, who holds a doctorate in kinesiology from Penn State, has been on the College Park faculty since 2005, and he has published more than 60 peer-reviewed journal articles. He said he regularly reviews manuscripts for scientific journals and is an associate editor of the journal Frontiers in Human Neuroscience.