Evidence Soup

We've got to stop drooling at shiny data visualizations and keep searching for non-obvious evidence.

Yes, we want people to provide hard evidence supporting their claims. But aren't some things just too obvious to bother with? Shouldn't we use our talents to discover things we don't already know?

I like a shiny data visualization as much as the next guy, but recent details about Facebook analysis made me wonder why some great minds (with great resources) are focusing on some pretty unimportant stuff.

Here's what happened. Stephen Wolfram - who is certainly much smarter than me - released findings from his Wolfram|Alpha Personal Analytics for Facebook project. Among the revelations is that we have a peak number of friends around age 20, with that number slowly dwindling as we reach our 70s.

Sort of like the actual world, wouldn't you say? Doesn't this just confirm what we already know, only with a cool, crowd-sourced, data-donation methodology and attractive data visualization?

Doesn't pass the "six smart people" test. This reminds me of something Rita Gunther McGrath has talked about. She, too, is critical of obvious / we-already-know-that research, saying that if "six smart people in a room" can immediately see it, then it isn't something that needs formalized research. (Rita's a smart one, and I suggest following her at @rgmgrath).

The Daily Mail covered this with the headline "Growing old on Facebook: Search data reveals we talk more about the weather and  politics as we age." (Sadly, reading their story, it's tough to tell if they're being sarcastic or not.) Wolfram found we accumulate clusters of friends as we grow older - with the average 35-year-old having four clusters (again, no kidding: neighbors, work friends, friends from back home, etc.).

Wolfram has done some great stuff, such as A New Kind of Science. I know social network analysis can be important, and reveal useful things about how new knowledge develops. Let's see more of that.

Happy Fun-with-Evidence Friday. Have a great weekend, everybody.

Skeptical about the evidence in business books? The evidence on sales volumes is iffy, too.

I quit business books a long time ago, and I'm not the only one. Yet we hear endlessly about things like cheese moving and power posing.

Turns out, we should question not only the evidence being claimed by the authors, but also some of their claims about popularity.

A recent Wall Street Journal investigation reveals evidence that some 'bestsellers' don't belong on those lists at all. In The Mystery of the Book Sales Spike [How Are Some Authors Landing On Best-Seller Lists? They're Buying Their Way], Jeffery Trachtenberg explains how authors can hire a marketing firm to purchase books ahead of their publication date, creating artificial - and short-lived - sales spikes.

"Last August, a book titled 'Leapfrogging' hit The Wall Street Journal's list of best-selling business titles upon its debut. The following week, sales of the book, written by first-time author Soren Kaplan, plunged 99% and it fell off the list.

Something similar happened when the hardcover edition of 'Networking is Dead,' was published in mid-December. A week after selling enough copies to make it onto the Journal's business best-seller list, more hardcover copies of the book were returned than sold, says book-sales tracker Nielsen BookScan."

Buyer beware. Even if the evidence doesn't actually confirm that an author is "best-selling", his/her speaking and consulting fees will likely get a nice bump. (Creating more opportunities for hapless folks to pay for bad evidence.)

Of course, the keepers of the bestseller lists are nervous about this type of thing, and have controls in place to prevent such trickery. Yet some books are getting through anyway.

Happy Fun-with-Evidence Friday.

 

Evidence doesn't solve problems, people do. Richard Florida forgets that not everything is a Toyota (or Cheesecake) factory.

You know I love all things evidence. But while evidence helps us understand past events, it can get in our way when we're trying bold, new things. And just because we understand important evidence doesn't mean we know how to solve important problems.

Was reminded of that today. Richard Florida was on NPR Morning Edition (I think most of NPR's business journalism is simplistic and insulting, by the way). Florida is known for evidence showing that a city "is much better off if it has a large share of knowledge workers, of innovators, entrepreneurs, artists, professionals that make up the creative class". (He has a nice overview of innovation evidence in the Feb 1 Atlantic. And this evidence on Chicago's class divisions.)

But tech incubators don't fix everything. Florida is finding that places with "creative class concentrations have greater levels of inequality". Here's anecdotal evidence from a friend in San Francisco's tech-dominated SOMA neighborhood: Rent $3,900/month for 830 sq ft. Now that's an inequality stimulus if ever there was one.

Evidence doesn't solve problems, people do. So what does Mr. Florida suggest we do about the "non-creative" classes? Vast economic evidence doesn't lead directly to reasonable solutions. When asked on NPR for one concrete thing a mayor could do "that would improve the problem you're talking about", Richard Florida fails to impress: He says "boost the wages and salaries of the more than the 60 million workers who deliver our services, who prepare our food, take care of our homes, wait on us in stores.... The first thing you have to do is figure out a way to engage the service companies, to call them together in a summit and say we're going to work together to boost the wages and productivity of service workers. At the same time, begin to work on more affordable housing options, begin to work on how to increase density. If you're in a city like Chicago or New York, where there is transit, encourage the development of more knowledge-intensive and creative work around transit nodes and connections."

Bzzzt! Thank you for playing. Re: housing density: Lots of people are already trying that. And w/r/t a wage summit, I can't think of a time when employers put their heads together to increase pay.

Florida says: "The way to do that is not to falsely do it by just imposing minimum standards, although that may be worth doing in some big cities, by the way. The way to do that is to make their work more productive, to engage them in innovation, just like we did in our factories. When we began to organize workers in team, organize them in quality circles, engage them in continuous improvement, the factories got more productive and the wages of the workers went up. Listening to the workers and making them part of the solution, not just in factories, not just in Silicon Valley high-tech work, but in all sorts of work - that's the path to prosperity in the future."

Not everything is a Toyota factory. Or a Cheesecake Factory. Yes, the Cheesecake Factory is famously well-managed and profitable (hopefully this is reflected in people's paychecks). But I'd like to hear a creative proposal to solve America's in-home healthcare problem: Evidence tells us the elderly population is exploding, that employees are underpaid and unprotected by regulations. Meanwhile, agencies providing these services are constrained on many fronts. But can management and technology create prosperity out of car trips to care for old people in their homes? 

Fun-with-Evidence Friday: Do you suffer from evidence euphoria? Or research rapture?

Well, hell. Another malady to worry about. Sean Pidgeon described research rapture in a recent New York Times column. Here I am, searching for evidence to support my every move, only to learn it's an addiction. I'll call it evidence euphoria.

I thought I was just being thorough. As Pidgeon explains: "If I may paraphrase various informal definitions that are to be found online, research rapture is something like this: A state of enthusiasm or exaltation arising from the exhaustive study of a topic or period of history; the delightful but dangerous condition of becoming repeatedly sidetracked in following intriguing threads of information, or constantly searching for one more elusive fact."

Maybe this is why my dissertation took so long. Happy Fun-with-Evidence Friday, everybody.

A must-read: "Does the Language Fit the Evidence? Association Versus Causation."

Science is easy; explaining it is hard. Back in 2009, Evidence Soup recommended the excellent Health News Review, whose mission is to "hold health and medical journalism accountable" (more about them at the end of this post). They've published Tips for Understanding Studies (available for purchase here). One of their free online writeups is a must-read for anyone working with evidence. Yes, it's basic -- but depending on your level of experience, will be a valuable refresher, an intro to research methods, or a guide to science writing.

Does The Language Fit The Evidence? – Association Versus Causation was put together by Mark Zweig, MD, and Emily DeVoto, PhD, "two people who have thought a lot about how reporters cover medical research". (I've been acquainted with Emily DeVoto for several years; I like that her catchphrase is "The plural of anecdote is not data.").

Passive language vs. active voice. The authors describe how an 'association' can be inadvertently misconstrued as a cause/effect relationship:

"A subtle trap occurs in the transition from the cautious, nondirectional, noncausal, passive language that scientists use in reporting the results of observational studies to the active language favored in mass media.... For example, a description of an association (e.g., associated with reduced risk) can become, via a change to the active voice (reduces risk), an unwarranted description of cause and effect. There is a world of difference in meaning between saying 'A was associated with increased B' and saying 'A increased B.'" [emphasis is mine]

These are subtle things with tremendous importance. When I was in school, I often heard "Correlation doesn't mean causation." Evidently, the difference is still be a big hurdle both for experts and non-experts. Zweig and DeVoto illustrate how things can go awry in this helpful example:

Study design	Prospective cohort study of dietary fat and age-related maculopathy (observational).
Researchers’ version of results	A 40% reduction of incident early age-related maculopathy was associated with fish consumption at least once a week.
Journalist’s version of results	Eating fish may help preserve eyesight in older people.
Problem	Preserve and help are both active and causal; may help sounds like a caveat designed to convey uncertainty, but causality is still implied.
Suggested language	“People who ate fish at least once a week were observed to have fewer cases of a certain type of eye problem. However, a true experimental randomized trial would be required in order to attribute this to their fish consumption, rather than to some other factor in their lives. This was an observational study – not a trial.”

 

Abstracts stick to the facts. The authors note that the "language in a scientific publication is carefully chosen for the conclusion in the abstract or in the text, but not used so strictly in the discussion section. Thus, borrowing language from scientific papers warrants caution."

You can follow @HealthNewsRevu on Twitter. The project is led by Gary Schwitzer and funded by the Foundation for Informed Medical Decision Making.

Show me the evidence. Education policy debate needs a major overhaul.

U.S. education is in crisis. Wherever you stand on reform/ vouchers/ charter etc., I think you'll agree our schools could be better. Big dollars go into training, innovation, and research, but there's a distinct lack of agreement on how to improve outcomes. It gets worse: Stakeholders don't even seem to agree on what the outcomes have been.

I've taken a look at contributions to this debate (not academic stuff, but instead reports by advocates, foundations, think tanks). I don't want to pick on any particular group, but will use a few examples to show why the evidence needs to be presented more effectively. Traditional formats aren't adequate.

Last year, the Friedman Foundation for Educational Choice released The School Staffing Surge: Decades of Employment Growth in America’s Public Schools, by Benjamin Scafidi [pdf here]. Recently, Joydeep Roy of the National Education Policy Center (University of Colorado) released a critique ("Review of The School Staffing Surge") as part of the Think Tank Review project [pdf here].

Show me the evidence, you two. These publications reflect a widespread problem: Evidence buried in unwieldy documents. It's too difficult to find, interpret, or apply.

What educational outcomes have been observed? Where are straightforward visuals linking school staffing or school choice with performance?

Where are the connections between likely causes and effects -- or at the very least, "associations"? We're seeing endless, descriptive statistics. The discussions are long-form paragraphs, supported by traditional footnotes and charts quantifying one variable over time (e.g., staff:pupil ratio, test scores). Supposed relationships between key variables are  mentioned, not illustrated.

The Friedman report talks about outcomes, including NAEP scores (shown here), and hypothesizes about contributing factors, such as parental education and # of siblings. But it only loosely connects them to the much-discussed administrative/non-teaching headcounts or to the "free choice" (charter schools etc.) being advocated. They should be showing us specific influential factors, and associating them with specific outcomes.

In making its counterargument, NEPC follows a similar traditional path. For example, on page 9: "The report puts significant faith on school reform efforts that are either unproven or have not sufficiently demonstrated their potential to raise student achievement....  However, the empirical literature suggests that these policies have not generated consistent and substantial benefits.¹³ "

Show, don't tell. NEPC simply refers to footnote 13, rather than illustrating their conclusions. (And I'm taking off points for their claim that Friedman's school reform efforts are "unproven". Here at Evidence Soup, we know not to say something is 'proven' or 'unproven' -- don't get me started.)

In NEPC's defense, the onus was on Friedman to clearly identify causes / consequences of the school staffing surge, and concrete outcomes of school choice. However, NEPC should have synthesized their evidence to illustrate points more effectively. That said, they draw some solid conclusions (p. 3):

"...the report would have been helpful had it explored the causes and consequences of faster growth of teachers, staff and administrators, examining the variation across states and over time. However, the report’s approach is instead to document employment growth, label the growth as problematic, and then recommend certain favored policies (i.e., vouchers and school choice)....

"Further, the report’s recommendations section is problematic in its uncritical presentation of school choice as a solution to financial and staffing increases. The report presents no evidence that school choice - whose record on improving educational outcomes and efficacy is mixed - will resolve this 'problem.'"

Our children deserve better. Evidence could be put to better use if it were more accessible to educators and policy makers.

What counts as good evidence? Alliance for Useful Evidence offers food for thought.

"What counts as good evidence?" is a great conversation starter. The UK-based Alliance for Useful Evidence / Nesta are hosting a seminar Friday morning to "explore what is realistic in terms of standards of evidence for social policy, programmes and practice." Details: What is Good Evidence? Standards, Kitemarks, and Forms of Evidence, 9 November 2012, 9:30-11:30 (GMT), London. The event is chaired by Geoff Mulgan, CEO of Nesta ; speakers include Dr. Gillian Leng, Deputy Chief Executive for Health and Social Care, NICE; and Dr. Louise Morpeth, Co-Director Dartington Social Research Unit.

Prompting that discussion is a 'provocation paper', What Counts as Good Evidence?, by Sandra Nutley, Alison Powell, and Huw Davies. They're with the University of St Andrews Research Unit for Research Utilisation (RURU). Let me know if you want me to send you a copy (I'm tracy AT evidencesoup DOT com).

The evidence journey. This paper doesn't break lots of new ground, but it's a useful recap of the state of evidence-seeking from a policy / program standpoint. While the authors do touch on bottom-up evidence schemes, the focus here isn't on crowdsourced evidence (such as recent health tech efforts). I love how they describe the effort to establish a basis for public policy as the evidence journey. Some highlights:

Hierarchies are too simple. We know the simple Level I/II labeling schemes, identifying how evidence is collected, are useful but insufficient. The authors explain that "study design has long been used as a key marker for evidence quality, but such ‘hierarchies of evidence’ raise many issues and have remained contested. Extending the hierarchies so that they also consider the quality of study conduct or the use of underpinning theory have... exposed new fault-lines of debate.... [S]everal agencies and authors have developed more complex matrix approaches for identifying evidence quality in ways that are more closely linked to the wider range of policy or practice questions being addressed."

Research evidence is good stuff. But the authors remind us "there are other ways of knowing things.  One schema (Brechin & Siddell, 2000) highlights three different ways of knowing": empirical, theoretical, and experiential.

Do standards help? The authors provide a very nice list of evidence standards & rating schemes (GRADE, Top Tier Evidence, etc.) - that is reason enough to get your hands on a copy of the paper. And they note the scarcity of evidence on effectiveness of these rating schemes.

Incidentally, Davies and Nutley contributed to the 2000 book What Works? Evidence-Based Policy and Practice that I've always admired (Davies, Nutley and Smith, Policy Press).

Patient-centered outcomes? Evidently we don't have those yet, and we don't agree on how to get there.

The Patient-Center Outcomes Research Institute is a big deal these days, "authorized by Congress to conduct research to provide information about the best available evidence to help patients and their health care providers make more informed decisions. PCORI’s research is intended to give patients a better understanding of the prevention, treatment and care options available, and the science that supports those options".  (Holy smokerstacks, shouldn't we be doing that already?)

Nobody agrees on anything these days. Today, David Introcaso criticized the institute in PCORI’s Problem: Reasons Are Not Causes. PCORI Executive Director, Joe Selby, responded in Enhancing Communication To Advance Research And Practice Is Among PCORI’s Key Priorities.

Introcaso argues against a focus on measurement and systems thinking, and claims PCORI fails to emphasize patient-provider interaction. He raises some interesting points, but I think he wanders off into the weeds and fails to propose concrete paths to quality improvement. For instance, Introcaso correctly notes that despite overwhelming evidence about the benefits of hand washing, adherence rates are low. He asks "What does this suggest about PCORI’s chances of filling the quality gap by producing and then disseminating ever more evidence?" Yet he suggests "the fundamental way to improve health care quality is by designing delivery that improves the interaction between provider and patient.... People are not planes. They cannot be engineered."  Agreed, but how would provider-patient engagement resolve the hand-washing dilemma? Plus, PCORI is producing new flavors of evidence in the form of comparative effectiveness research (more about that below).

So there. PCORI's Selby responds by saying "'Communication and Dissemination Research' is one of our five priorities, and our funding announcement for this priority calls for studies of communication between patients and clinicians, of shared decision making, of considering patients values and preferences in the communication, and of training clinicians in the practice of shared decision making. Our emphasis on better understanding patient-provider communication as a means of improving practice and outcomes also is evident in our Pilot Projects. One of the eight specific areas of interest listed in the solicitation for our $30 million Pilot Projects program was the study of the interactions of patients with their clinicians and healthcare systems."

I don't see anyone pretending that CER and patient-centered outcomes are straightforward. Among other things, PCORI is soliciting research questions (you can submit one here). And this is all intertwined with comparative effectiveness research; see the October special CER issue of Health Affairs. One of the discussions in the October Health Affairs is a RAND piece on Five Reasons That Many Comparative Effectiveness Studies Fail To Change Patient Care And Clinical Practice. These include financial incentives, ambiguity of study results, and cognitive biases.

 

Evidence shows natural gas can grow the U.S. economy. If people will look at the evidence.

Bitter political conflicts threaten growth of the U.S. economy, and energy issues are no exception. I believe a better presentation of certain key evidence could improve people's understanding of natural gas. First I'll highlight some evidence about production. Then I'll provide some hard evidence showing why we need more transparent, efficient regulation.

Natural gas is your friend. Yes, the Gulf oil spill was atrocious. So is overreliance on Saudi oil. But we'll need fossil fuels for the rest of our lives. It's easy to demonize the petroleum industry (I once heard a Prius-driving neighbor say she didn't want to 'give $ to big oil', but evidently 'big automotive' a/k/a Toyota is okay).

Natural gas is a relatively low-cost, clean-burning option. New technologies have grown the U.S. supply, which can keep prices low. Media love doing stories about hydraulic fracturing (fracking), and interviewing ranchers who claim to have been injured. There will be some costs, but the benefits of a reliable, affordable, domestic natural gas supply far outweigh those costs. And we can export the stuff: Apache is being joined by Shell and others with their eyes on the LNG (liquefied natural gas) export prize. (Apache is an innovative company that drilled for natural gas before it was cool. BTW, I met my dear husband when we both worked at Apache years ago.)

Evidence of benefits. A new IHS study, America's New Energy Future, illustrates how natural gas can contribute substantially to the U.S. economy. Among other things, domestic production has cut imports, reduced the balance of payments by $75B, and created 1.5+ million jobs.  Daniel Yergin, who knows about such things, wrote a recent Wall Street Journal op/ed piece recapping the report [paywall].

Better presentation, please: The IHS authors follow the traditional practice of listing sources and models used, so there's transparency. What they don't do is provide evidence that's interactive, tangible, and easily digestible; for instance, providing specific findings linked to supporting evidence.

Evidence about environmental effects is unclear. Misinformation abounds, and it's hard to know the truth about natural gas. Stereotypical stakeholders include 1) Energy producers who sponsor extensive PR campaigns without producing specific evidence. 2) Granola-munching environmental regulators who cynically assume everyone in industry is a lying, polluting cheat - and have no grasp of economic costs/benefits. 3) Regulators who cozy up to industry in hopes of landing lucrative jobs. 4) Producers who nonchalantly drill, spill, twirl their pencil-thin mustaches, and flee with fistfuls of dollars.

Regulators aren't such good friends after all. Drilling for gas and constructing LNG facilities requires billion-dollar investments and years of planning. An opaque, inefficient regulatory environment benefits no one (except empire-building bureaucrats and attorneys).

Evidence about requirements should be more clear. Do we want to protect the environment, improve efficiency, and produce energy responsibly? If so, why not have a one-stop system where regulators identify the rules in plain English, and producers respond with transparent evidence of compliance? (There are some plain-language requirements for the federal government, but this goes beyond that.) People should collaborate on compliance plans proactively, rather than through enforcement actions in an atmosphere of mistrust.

Compliance costs are hidden evidence. Here's what it was like for me, working in environmental compliance for a natural gas producer/processor. The management teams I worked with invested heavily in following the rules, permitting everything properly, etc. But it was very difficult to know what the regulations required (just ask anyone trying to open a restaurant). Federal, state, and local agencies overlap, and sometimes protect their turf more than the environment. Some play "gotcha!" when a rule is violated, even if you were making a good-faith effort to comply. (See specific examples at the end of this post.*)

Environmental evidence should be balanced against evidence of benefits. Producers must be held to strict standards that protect groundwater and human health. But rules should balance the interests of landowners, producers, and the general public, protecting our economy and national security while also protecting the environment. Too often, the big picture is blocked by a NIMBY chorus. The U.S. sometimes requires agencies to weigh costs against benefits, though it's nearly impossible to depoliticize rulemaking (my PhD dissertation studied the role of cost-benefit analysis and evidence in federal regulations under Executive Order 12291; my findings were not encouraging).

*Evidence of compliance costs is hidden, but real. I should know: I've been there. These are small examples, but illustrate the unproductive way we regulate energy producers. True story: My team was once told by a state regulatory staffer we had all the air quality permits we needed, only to be confronted by the federal Environmental Protection Agency (EPA) and threatened with multi-million dollar fines.

Another true story: Many regulators won't put anything in writing until they want to punish you for something. I once spoke to a state agency about a specific reporting requirement, and they said (over the telephone) it didn't apply. Shortly thereafter, the same agency told a colleague he was required to prepare a report for a separate property; he did so at substantial expense. Later he was told it wasn't necessary, but no one would own up to having previously required it. How does this protect the environment?

Coalition for Evidence-Based Policy seeks RCTs for social programs.

Randomized, controlled trials (RCTs) are typically viewed as the gold standard in developing evidence and separating what works from what doesn't. (I'll leave a discussion of RCTs for another day.)

The Macarthur-funded, Washington, DC-based Coalition for Evidence seeks to "increase government effectiveness through the use of rigorous evidence" - and by rigorous, they mean derived from RCTs. The group says that: "[i]n most areas of social policy – such as education, poverty reduction, and crime prevention – government programs often are implemented with little regard to evidence, costing billions of dollars yet failing to address critical social problems. However, rigorous studies have identified a few highly-effective program models and strategies ('interventions'), suggesting that a concerted government effort to build the number of these proven interventions, and spur their widespread use, could bring rapid progress to social policy similar to that which transformed medicine..."

These guys are drinking the Kool-Aid. The Coalition continues: "[A] central theme of our advocacy, consistent with a recent National Academies recommendation, is that evidence of effectiveness generally cannot be considered definitive without ultimate confirmation in well-conducted randomized controlled trials."

Their Top Tier Evidence initiative is intended as "a validated resource used by federal officials to assist policy officials in identifying interventions meeting the Congressional Top Tier evidence standard, defined in recent legislative provisions as 'well-designed randomized controlled trials [showing] sizeable, sustained effects on important… outcomes' (e.g., Public Laws 110-161 and 111-8)."

Why can't more organizations do this? With that in mind, the group recently identified two programs as promising, describing them as Near Top Tier (satisfying most, but not all, their criteria for rigorous and evidence-based). I like that the Coalition provides an evidence summary explaining its reasoning and the underlying evidence that supports its rating of subject programs. For the Child FIRST program, key findings include "40-70% reduction in serious levels of (i) child conduct and language development problems, and (ii) mothers’ psychological distress, one year after random assignment." For the PMTO program, findings include: "Sons of women in the program group had substantially fewer arrests over nine years (an average of 0.76 arrests per boy in the PMTO group versus 1.34 per boy in the control group)."

But wait, there's more. Each evidence summary provides a brief cost-benefit analysis pinpointing the potential taxpayer costs or savings from the subject program.This is complicated, I know, but we can't be promoting widespread programs that are unable to document some type of benefit.

High hurdle. The 2009 National Academies report (mentioned earlier) says:

“Federal and state agencies should prioritize the use of evidence-based programs and promote the rigorous evaluation of prevention and promotion programs in a variety of settings in order to increase the knowledge base of what works, for whom, and under what conditions. The definition of evidence-based should be determined by applying established scientific criteria. In applying scientific criteria, the agencies should consider the following standards:

• Evidence for efficacy or effectiveness of prevention and promotion programs should be based on designs that provide significant confidence in the results....

• When evaluations with such experimental designs are not available, evidence for efficacy or effectiveness cannot be considered definitive, even if based on the next strongest designs, including those with at least one matched comparison."

I like these guys.