Avoiding the Research Rabbit Hole

Most beginning students start poking around in the research literature when they start writing their own monographs and they really have no idea what they are doing.  You can get sucked so far down the research rabbit hole that your productivity plummets,  if you don’t learn how to narrow a search.  * You can do this even pretty easily even if you DO know how to narrow a search. 🙂 *

The best way to do this is to use specific search terms and to limit yourself to the most recent research.  I focus on not using much research that is older than five years and actively avoid using anything older than ten.

I recommend SciCurve to beginners.  It teaches some common research terms while you are poking around, but  the visuals also show connections that will give you some direction in your research. I am not going to fib the aesthetics appeal to me, as well. Below is what shows up if you ask for a report on kava.

To the left is what it looks like when you ask it to show you a map.  Maybe its because I am a visual learner, but I really like this.  Also if you have your Lazy Scholar add-in, every time you click on one of the bubbles, it will look for the full text of the article on the web.

Now-omics is not as pretty but it is specific to the life sciences and it lets you generate a news feed built with only recent research.  I like it for beginners because they can plug in a very broad search term like a particular plant and then use the topics down the side to filter the larger search.

TRIP has a really handy free search function that allows you to frame your search using the PICO mnemonic I mentioned last week.  The following search returned 98 results defined by type, such as ongoing clinical trials, primary research or guidelines.  

 

 

 

 

Spharro is probably my favorite for keeping up with the latest research, though.  You can make “channels” about every plant if you want.  The platform really allows you to narrow down what you see based on  information about authors, journals and MeSH terms you may have seen on SciCurve or TRIP.

This is the point when people ask why I just don’t stick with Pub Med? There are so many reasons I don’t like Pub Med-most of them having to do with its limitations. Those just getting into research aren’t likely to know common medical subject headings (MeSH) or research terms that will produce good search results.  Pub Med also doesn’t give you the ability to search for a single type of literature or ongoing clinical trials, the way TRIP does or only search through the most up-to-date literature the way Nowomics does.  By that I don’t mean the date of publication so much as I mean that often times the versions of the papers offered on  PubMed are not the final version of the publication.

Specific Paper

In my last few posts on research, one thing that should have become clear is that it is really hard to critically assess a study, if you don’t have the whole thing. Using abstracts limits your understanding of the study design and you don’t get to see the concluding discussion. The sites above are going to send you to a lot of papers where everything but the abstract is hidden by a paywall.   Don’t give up though because there are ways around that, sometimes.

The first thing  I like to suggest is installing the Chrome add-on Lazy Scholar. It is useful for looking for full text versions of a  paper and it will suggest alternative titles that might be related to your research. If you hit Cite, it sends the citation to your clipboard for pasting into a document. My favorite thing about it is the nifty little option that lets me block non-scholarly sites on the Internet for one hour. If Lazy Scholar has not worked its magic and located a copy, the next thing to try is an old-fashioned Google Scholar  search using  the title of the article followed by pdf.

If you can’t find it that way, try to determine if you can get some access to online databases through your local library.  Depending on the size of your library,  availability may be limited.  If that is the case, move on to looking to see what your state library offers.  Many of you might not know this is a thing, but your state library will send out a card number to anyone who lives in the state which grants access to their online resources.  What is available varies by state, but here are some examples.

Iowa
South Dakota
Kansas

Finally, check around some of the places where researchers are able to post their own research for sharing.

Researchgate
Academia
Scribd
PubPeer
Peer Evaluation
ScienceOpen

I think this will wrap up what I have to say about research for right now unless readers have questions.  I am very much involved in my own research projects right now as I am updating my class outlines for the MIdwest Herb Fest and Traditions in Western Herbalism Conference with the most recent information I can find.

Often students in medical and nursing programs are asked are asked to write up a clinical appraisal of a research study. The first step to doing this is to summarize the study using the PICO mnemonic.  This is a method which biomedical researchers are taught to formulate good clinical questions which can also be used to evaluate research papers.  Using this method is going to be most useful if you have solid knowledge of biological sciences and some sort of clinical experience.

Keep in mind here that  while you are trying to assess how well put together the study is, ultimately what you want to figure out is how this study is relative to your work.

P—Patient
What are the characteristics of the subject or population?
N=number of subjects.
What is the condition or disease being examined?

I—Intervention or exposure
How did the researchers interact with this subject or population?  Were they treating the condition or observing risk factors?

C—Comparison
Was there is the alternative to the primary intervention, for example a placebo or were two interventions being compared?  Were two different populations or being observed?

O—Outcome
What is the relevant outcome- improvement, recovery, remission, death? Did any complications develop? Did the intervention have any side effects.

Now you can move on to critically assessing the study design and outcomes.

What you are looking for here is bias. Bias is a term that is used to used to refer to some sort of partiality on the part of the researchers or the subjects. There are many types of bias. For example, selection bias occurs when potentially eligible subjects are excluded from the study based on interviewer bias.  Ascertainment bias can happening when blinding or masking isn’t done properly and researchers or participants have a certain expectation of the intervention.

Sometimes bias is hard to determine. Publication bias occurs internally when editors of  journals favor studies that come to their preferred conclusions.

Are the Methods Valid?

Was the initial research question clear and impartial?  Do the researchers have a hidden agenda- possibly motivated by marketing motives or keeping their funding.  This is sometimes called the vested interest bias.

Was the sample of subjects clearly defined and representative of what you might encounter in clinical practice?

Were the subjects’ prognosis similar?  Their clinical characteristics need to be similar enough to draw valid conclusions. Were they all from a similar demographic, at a similar stage in a disease? If there was co-morbidity were the coinciding conditions similar?   What are possible confounding variables which means could other factors have contributed to the outcome?

Is the intervention reasonable?   When looking at studies about herbs, we absolutely have to determine if the dosage is one a human can even attain without eating a whole tree.

Was the follow-up sufficiently complete?  Did the study follow at least 80% of the patients through to recovery or an alternate disease outcome such as remission or death?

Were objective and unbiased outcome criteria used?  This means was there some sort of mechanism incorporated into the study such as a standardized pain scale which eliminate the use of subjective conclusions on the part of the researchers as to subjects’ improvement.

What Are the Results?

Does there appear to be any selective reporting on the part of the researchers?

If this is a cohort study were the outcomes and exposures to risk factors assessed similarly.

How likely are the outcomes over time? You might see a  relative risk ratio mentioned in RCT’s and prospective studies.   In retrospective studies, they will mention the odd-ratio.

How logical and accurate are the researcher’s conclusions?  What is the confidence interval- that is the measure of how likely you would be to achieve the same results if you repeated the study?

How Can I Apply the Results to Care of Clients?

Were the study subjects  and methods of management similar to those in my practice?  Can I apply these results to interventions I use with clients in my practice?

You can also look to see who funded the study and whether or not the researchers declared any conflicts of interest, but honestly there are a lot of ways to hide that information.

After you ask yourself all these questions you can summarize the study by listing its strengths and limitations.

As I mentioned yesterday, I am going to take a moment to make the case against reducing scientific research to the in vitro is invalid– in vivo is valid binary.  I feel like the biggest issue with this is that it significantly limits our understanding of how research is being done and how to evaluate it.  I also think that hanging on to old-fashioned ideas about in vitro studies perpetuates the unnecessary use of animals in studies. So I want to take some time to break that up a bit.

There is a lot more to be determined than whether we are looking at a human clinical trial or a petri dish experiment using a cell line purchased from a biotech lab. Furthermore, there is no reason to discount that in vitro research out-of-hand.  If we want to evaluate the efficacy of a study we need to look at the design of the experiment which I will tackle later in the week.  Before I dive into the methodology.  I should probably explain a few basic terms in the context of biological research.

Medium– a substance that various cultures are grown in.  The most commonly used mediums are probably nutrient broths and agar.

Bacterial culture – this type of culture involves growing a microbial colony in lab equipment for experimentation.   Biofilm models (single or mixed-species) can be cultured in the lab, also.  They are useful in determining the effect that the biofilms have on host tissue or the effects that an agent might have on a biofilm, but it has been a tedious process. The following journal article breaks down that history,  a little:

Lebeaux, David, Ashwini Chauhan, Olaya Rendueles, and Christophe Beloin. “From in Vitro to in Vivo Models of Bacterial Biofilm-Related Infections.” Pathogens 2, no. 2 (May 13, 2013): 288–356.

Cell cultures–  a cell culture is made by removing a cell from its original environment and incubating it in favorable medium (often agar) in lab equipment.  Additives are needed for the growth of the culture.  For example, viruses need living host cells to live.  Animal cells need some type of blood serum which contains growth factors and hormones they would be receiving through perfusion in the body.

Sometimes these cells are modified in some way and stored cell repositories in the form of cell lines or “immortalized” cells to be distributed for future research. That’s why sometimes you will see the authors mention the biobank where they got their material.  Here you can read about the biobank founded by one of the pioneers in this field- The Coriell Institute.

Researchers are also able to culture tissue and organs.  This is an interesting article that explains some of the history of that process. Use of human tissue explants to study human infectious agents.   Not too long ago researchers were able to culture a new windpipe for a man using his own cells.[1]

Now let’s investigate the different types of research methodology you will run across in research studies.  I have them listed in the order they are often considered to be most reliable, but keep in mind there are reasons researchers choose a particular methodology and they are all necessary, and useful, components of scientific research.

In Vivo Experiments

In Vivo experiments are those experiments done on a living organism.  They generally take the form of animal studies in labs or human clinical trials. Microbiologists (who are doing many of the studies herbal researchers use) may call an experiment done on isolated cells in vivo as long as they are living and haven’t been modified- for example they might use cultured cells from a biopsy.  This is more accurately described as an ex vivo experiment, but not all researchers use the same terminology. So you have to read the fine print-not just the abstract.

In Situ Experiments

Sometimes you might see the term in situ used instead of in vivo.  In Situ experiments are performed in exactly the place which the issue being investigated occurs without any sort of interference from the researchers.  An in situ experiment might take the form of examining cells or tissues of a functioning organ with blood is being delivered (perfusion) or examining a plant while growing in its natural environment, as opposed to bringing it into the lab.  I think they are quite relevant to studying plants as it has been proven conclusively that plants grown in the wild produce more secondary metabolites than lab plants.

Ex Vivo Experiments

These are experiments that work with cells or tissue from an organism in an external environment but do not alter the cells or tissue, as opposed to in vitro tests which might manipulate the cells to immortalize them, or for some other reason.  You can read more about the differences here, if you are interested.

An ex vivo project allows researchers to do things they couldn’t ethically do, otherwise.   Working with cultured tissues and organs is one of the answers to reducing the use of animals in scientific research.[2]

In Vitro Experiments

In vitro studies are experiments are carried out using microorganisms, cells or biological molecules which have in some way been disrupted.  The individual parts are tested or analyzed outside their normal biological environment-generally in some sort of glass lab equipment.

One of the most amazing in vitro advances in recent years is the discovery of induced pluripotent cells. Researchers managed to re-program skin cells into a cell with stem cell like capabilities that could then be used to generate any cell in the body-even diseased ones.  So what you end up with in the dish is a specific model of a disease. This technology is eliminating the need for in vivo research which involved infecting animals with diseases.

Also experiments done on whole, living organisms which have been cloned and never left a lab are sometimes still referred to as in vitro because the subject has been scientifically tampered with. Again this is specific to herbalists because it happens a lot with plants.

In vitro experiments are often dismissed with the caveat that how something works in a petri dish, is not how it works in the human body.  This can absolutely be true at times, but is not always necessarily the case. In vitro technology is being refined and improved all the time.  This method can absolutely be relevant when looking at an agent’s efficacy. They have also used cultures to replace some of the more obnoxious animal tests like the Draize eye irritancy test.[3]

In Silico Experiments

These experiments are conducted by using computer models for research. You will sometimes see them referred to as lab-on-a-chip technologies.  They are already using these computer models to replace primates used for brain research.

Animal Research

I have very strong concerns about using animals for research and think that every measure should be used to “restrict, refine and replace.”[4] This is one of the reasons, I feel so strongly about helping people to understanding the validity of ex vivo and in vitro research, is that as long as we are spreading the misconception that this research is always less valid than in vivo research we will never end animal experimentation.

This idea that in vitro research is not valid has been amplified by people defending the use of animals in scientific research.  In 2011, Norvo Nordisk proved that using cell cultures in their quality control process was as effective as animal tests, in order to eliminate animal testing in their facility.  The ridiculous part is that it  took them over a decade to do so convincingly- not because the testing wasn’t accurate but because of the deeply entrenched thinking on in vitro methodology.

On the other hand, it is absolutely not true that animal research is never applicable to humans, and can be eliminated entirely-not yet.  Due to the chronic disease epidemic and the advances in epigenetics, we need to be able to track these illnesses through multiple generations.  Mice share 98% of their DNA with humans. For example, the two human tachykinin genes TAC1 and TAC3 correspond to Tac1 and Tac2 genes in mice.[5]  They also have relatively short life spans.  It was new technology enabling researchers to use mice in genetic research that stopped the decline in animal research in the 80’s.

References:
[1] Yong, Ed. “Will We Ever... Grow Synthetic Organs in the Lab?” Future. Accessed April 12, 2016. http://www.bbc.com/future/story/20120223-will-we-ever-create-organs.

[2]  “Human Tissue for in Vitro Research as an Alternative to Animal Experiments: A Charitable ‘Honest Broker’ Model to Fulfil Ethical and Legal Regulations and to Protect Research Participants.” Alternatives to Laboratory Animals. Accessed June 21, 2016. http://www.atla.org.uk/human-tissue-for-in-vitro-research-as-an-alternative-to-animal-experiments-a-charitable-honest-broker-model-to-fulfil-ethical-and-legal-regulations-and-to-protect-research-participa/

[3] Doke, Sonali K., and Shashikant C. Dhawale. “Alternatives to Animal Testing: A Review.” Saudi Pharmaceutical Journal 23, no. 3 (July 2015): 223–29.

[4] National Academy of Sciences. Guide for the Care and Use of Laboratory Animals. Washington, D.C.: National Academies Press, 1996. http://www.nap.edu/catalog/5140.

[5] Preedy, Victor R., ed. Neuropathology of Drug Addictions and Substance Misuse Volume 1: Foundations of Understanding, Tobacco, Alcohol, Cannabinoids and Opioids. London, UK: Academic Press, 2016. p. 190

When I was 34 years old, I decided to go back to college and then promptly got pregnant with my fourth child. No one was more shocked than I was when suddenly I found my science class was more appealing to me than my humanities classes.  I think it was hormonal.

Many biology credits later, I am a science gal. I like to back myself up with valid sources of research, whether I am writing about the history of skullcap or neurotransmitters.  I love playing in my lab. I adore my microscope.

Some herb people aren’t interested in scientific research.  In fact, in some circles there is pushback against scientific research and having been around in the days when science was being used to refute the usefulness of herbs, I understand where it is coming from.  But the boomers haven’t noticed that with the advent of integrative medicine, (and what some consider an impending attempt to co-opt herbal medicine) there is more biomedical research supporting the use of herbal therapeutics being published all the time.

 So while I get it, I question the wisdom of falling too far behind the times. I decided to combine my love of the traditional and science by using modern research to support really old ideas which I pull from primary documents whenever possible.   It makes me happy, but I still can’t help but be bothered by the amount of misinformation I am seeing out there.

I am definitely not alone in this concern.  A while back, I read an article which reiterated many concerns I have about junk herbalism on the web. “Instead of trying to translate what the best-available research evidence tells us about how to live, we report on the latest studies out of context, with little focus on how they were designed, whether they were unduly conflicted by study funders, and whether they agree or disagree with the rest of the research.”[ii]

Many of us know that authors on sites like Natural News and Green Med Info are guilty of regurgitating research articles, without really understanding the contents, but many “more reputable” sites like Science Daily are also guilty of this approach. That article that is circulating this week about intermittent fasting is a prime example. The study is not great science but it seems to be great click-bait.  It generates hits and often whether or not a health journalist gets a follow up assignment is based on the number of hits an article gets.

Writers relaying health research to the public seem unable to recognize statistical nightmare when they see it.  It’s not entirely their fault. Biomedical researchers are cooking the books, a bit.  When I was in college, I took a statistics class specific to biology majors, rather than calculus. (I am one of those mythical neurodivergents who does not love math.)  When I began this class, the professor mentioned we could use statistics to make our results support our hypothesis and went on to teach us how.

Consequently, I was not at all surprised last year when Richard Horton, editor of the prestigious medical journal The Lancet, published his scathing commentary on research in biomedicine saying “Our love of “significance” pollutes the literature with many a statistical fairy-tale” [i] while lamenting the fact that none of the parties involved have any incentive to address the issue of bad scientific practices in biomedicine.

(I am interested in having a discussion about who passes as a health journalist?  I spent just enough time soliciting writing assignments on Elance that I am pretty much appalled by the qualifications of those hired to write blog posts and eBooks on the subject.  I absolutely disagree with the author of the Vox article who says health writers don’t need a science background, but that is the recent battle cry of online publications, because they hire unqualified people.   My mother was an award winning education journalist, hired because she was an education minor.  Successful political journalists also tend to have a background in political science. Sports journalists tend to be former athletes and so on and so forth.  But I digress.)

This article’s rather limited solution to the problem was that health journalists should only use the systemic reviews that inform evidence based medicine, which is going to understandably put off some people.  The research used by evidence based medicine has been critiqued heavily in recent years[iii].

I grudgingly tend to agree with this advice even though it runs counter to what you might hear from other herbal researchers who flip flop this pyramid.  I maintain that  without training, you aren’t going to be able to make head nor tails of the data presented in a study, as it is often designed to mislead and full of confounding variables. Telling a lay person to critique methodology is absolutely pointless if they don’t have any knowledge on which to base their critique.  So I want to talk about research.

 I thought I would begin “Let’s Talk About Research Week” by defining and discussing the research pyramid I use to assess the reliability of scientific literature I come across.  Almost every nursing program and medical program has one of these.  This is just the one my school used which I am rather attached to.  It doesn’t include expert opinion at the base, but you certainly could include sources like expert interviews and textbooks on the bottom.   There are advantages, disadvantages and design traps that researchers fall into when using any of these study types, but that may be too much to cover on a blog.

The pyramid also doesn’t include the various types of research methodology.  Most filtered sources include literature which uses a variety of these methods.  We will get to that tomorrow when I break up the whole in vitro–in vivo binary, a little.

The tip of the pyramid is populated by what are called filtered information sources. That means that someone else has already looked at the body of research included in the references and drawn conclusions based on that research.  You need to use your experience and critical thinking when looking at filtered information. They can also be based on “statistical fairytales” or more frequently exhibit researcher bias against complementary medicine.

Meta-Analysis and Systemic Review

When compiling a systemic review, researchers look at a many, many studies on a subject and draw conclusions based on that body of research.  The reviews should tell you what search terms they used to acquire this literature, so you can perform a similar search.  A meta-analysis is the most reliable type of systemic review as it has more statistical significance, but see my aforementioned concerns about statistics and biomedical research. Some sites where you can obtain reviews or their abstracts include the NIHR CRD Database,  NHS EED,  Cochrane Database, DARE, Campbell Collaboration Library and TRIP. (I could write a whole blog entry on how much I love TRIP, but suffice it to say it is absolutely worth $40 a year and I do not use PubMed. Ever.)

Practice Guidelines:

People rarely mention these types of literature to those studying herbal but there are more and more tailored specifically to integrative medicine interventions, all the time.  They can also useful if you want to find up-to-date information on conventional clinical practice.  Go to the National Guideline Clearinghouse or TRIP and do a search for “complementary”. You can also use Up-to-Date and Clinical Key.  These are behind paywalls, but some information is available for free.

Now we will move on to unfiltered sources of information which basically means no one has applied their critical thinking skills to this literature for you.  That doesn’t mean you can’t. As far as where to find them and how to assess them…more about that on Wednesday.

Randomized Controlled Trials 
These studies randomly sort subjects into experimental or control groups.  Blinding (or masking) occurs when the participants don’t know which group they are in. Double blinding occurs when the researchers don’t either.  Consequently, double blind RCT’s are the most reliable, however there are ethical concerns that come with double blinding. The key here is that the researchers are actively experimenting on the subjects.

Cohort (Prospective) Study

A cohort is simply a population that shares similar traits.  A cohort study is generally a longitudinal observational study. Researchers formulate a hypothesis and then watch the cohort for a pre-established amount of time.  The Nurses’ Health Study is one example of a cohort study. Sometimes researchers compare one cohort to another. An example would be determining sexual function of 35 year-old males who smoke versus 35 year-old males who don’t smoke.

Case Control (Retrospective) Study

This study compares cases (subjects with an illness) with controls (healthy subjects) and looks at their past exposure to risk factors to try to determine a relationship.  It relies on the accurate memories of all participants.  One type of memory issue is called recall bias.  People with a condition are looking for answers and therefor more likely to remember exposure to risk factors.

Case Report

This is literature that describes the symptoms of a specific subject and the subject’s response to a particular intervention protocol. Case studies are awesome.  In my opinion, they should be the foundational unit of research because they are where we generate new ideas about interventions, but they can’t be taken too seriously as far as evidence. If I can only cite one case report, that is at best an interesting anecdote.  If I had ten cases with similar initial conditions, similar intervention protocols, and a similar outcome, that’s called a case series and conclusions based on a case series carry a little more weight. They also establish the need for further research on a subject.  Unfortunately, you rarely see them.

Tomorrow, I will post something that breaks down some of the methodology you will see in this literature and next week I plan to talk about about how to get at the research and other tips for writers such as evaluating clinical questions.

References:

[i] Horton, Richard. “Offline: What Is Medicine’s 5 Sigma.” The Lancet 385, no. 9976 (2015): 1380.

[ii] Belluz, Julia. “Health Journalism Has a Serious Evidence Problem. Here’s a Plan to Save It.” Vox, June 21, 2016. http://www.vox.com/2016/6/21/11962568/health-journalism-evidence-based-medicine.

[iii] Every-Palmer, Susanna, and Jeremy Howick. “How Evidence-Based Medicine Is Failing due to Biased Trials and Selective Publication: EBM Fails due to Biased Trials and Selective Publication.” Journal of Evaluation in Clinical Practice 20, no. 6 (December 2014): 908–14.