Misrepresentation? Not even once!

The developers of a US drug prevention programme who invested heavily in media advertising – and have used it to trumpet their own success – have been accused of misrepresenting evaluation data.

The non-profit behind the Montana Meth Project has been publicising the impressive results of their campaign against meth-amphetamine use for several years. But researchers and the press have become increasingly suspicious of the extravagance of their claims and of their backers’ media connections.

Launched in 2005 by the billionaire philanthropist Thomas Siebel, the Montana Meth Project has centred on a graphic social advertising campaign designed by the San Francisco partnership Venables, Bell and Partners and using Hollywood filmmakers.

Siebel told a US Senate committee in 2007 that the results had been 'more significant than any drug prevention programme in history'.

With the slogan 'not even once,' the adverts have portrayed the young people who use meth as dangerous, untrustworthy and exploitative and shown them threatening to kill their parents and prostituting themselves for their next hit.

In the first two years of the campaign, children in Montana were bombarded with 45,000 television screenings backed up with radio ads and billboard posters.

Thanks to its widely publicised impact on drug use, the program has since attracted public funding in excess of $2 million and has been expanded to six other US states.

MMP’s developers say it has caused 'dramatic shifts in the perception of risks associated with meth use, more frequent parent-child communications, greater social disapproval, and significant declines in meth use and associated crime'.

They can produce evaluation documents to back up their claims. But the only research available they conducted themselves and none of it has been published in peer reviewed journals.

Stern academic criticism of the lack of independent verification by David Erceg-Hurn from the University of Western Australia has been published in the journal <em>Prevention Science</em>.

He concludes that the studies were based on poor research designs and that they misrepresented or obscured negative outcomes. He found that some of the claims made were even contradicted by MMP’s own data.

The developers collected information about attitudes and use of meth-amphetamine before the ad campaign began in 2005, and in each subsequent year. The first two waves of data in 2005 and 2006 were flawed from the outset, says Erceg-Hurn.

The first problem was that the evaluation recruited participants and collected information online. Research elsewhere shows that internet users are generally more affluent and educated. Three quarters of the sample they ended up with were girls; the gender split in Montana is 50-50.

Erceg-Hurn maintains that not only was the sample unrepresentative of young people in Montana, but it may also have been skewed in terms of sex, education and employment – three factors associated with using meth-amphetamines.

He goes on to argue that the published results were manicured and the authors tried to obscure evidence of negative impact, and to overstate the positive.

For example, the main findings included in the evaluation report summary stated that the proportion of children who thought using meth constituted a 'great or moderate risk' declined by several percent. However, buried in the detail of the appendices was the fact that the number of children who thought there was 'no risk' increased by a similar rate.

The report also implied, says Erceg-Hurn, that  disapproval of meth use in Montana by over 90% of the population was a result of MMP, when its own baseline results from 2005 show that it was already as high before the campaign began.

He concedes that later evaluations in 2007 and 2008 were based on more representative samples and better data collection methods but argues that the evaluators were even vaguer about how they came to their conclusions.

The later reports did not include any information on statistical significance nor a breakdown of their analyses, so they could not be corroborated.

Nor did they compare the data with previous years - which might have told them about how things had changed. Erceg-Hurn’s own comparisons showed no reduction.  

Throughout their reports the developers seem convinced that any change in meth use in Montana is the result of the MMP campaign. However, parallel data indicates a downward trend in meth use, beginning five years before the inception of the program.

Police say changes to the law in July 2005, which restricted the sale of the chemicals used to make the drug, halved the number of seizures and pushed up prices.