Additional weak evidence about chiropractic treatment

Over the holiday weekend, Liz Ditz sent me the reference to a study that I have now downloaded and read. I’m reporting my notes here. I see that she has a related post over on her blog, I Speak of Dreams.

In “Developmental Delay Syndromes: Psychometric Testing Before and After Chiropractic Treatment of 157 Children,” Scott Cuthbert and Michel Barras present the results of an analysis of pretest-posttest scores for children who received chiropractic treatment at a clinic in Lausanne (CH). They reported that the children had higher scores after treatment, leading them to conclude that “This report suggests that a multimodal chiropractic method that assesses and treats motor dysfunction reduced symptoms and enhanced the cognitive performance in this group of children.”

Here is the abstract for this report. After it, I’ll explain why I find this study provides uncompelling evidence in support of chiropractic treatment for Learning Disabilities.

Objective: This study presents a case series of 157 children with developmental delay syndromes, including the conditions such as dyspraxia, dyslexia, attention-deficit hyperactivity disorder, and learning disabilities who received chiropractic care.
Clinical Features: A consecutive sample of 157 children aged 6 to 13 years (86 boys and 71 girls) with difficulties in reading, learning, social interaction, and school performance who met these inclusion criteria were included. Intervention and Outcomes: Each patient received a multimodal chiropractic treatment protocol, applied kinesiology chiropractic technique. The outcome measures were a series of 8 standardized psychometric tests given to the children by a certified speech therapist pre- and posttreatment, which evaluate 20 separate areas of cognitive function, including patient- or parent-reported improvements in school performance, social interaction, and sporting activities. Individual and group data showed that at the end of treatment, the 157 children showed improvements in the 8 psychometric tests and 20 areas of cognitive function compared with their values before treatment. Their ability to concentrate, maintain focus and attention, and control impulsivity and their performance at home and school improved.
Conclusions: This report suggests that a multimodal chiropractic method that assesses and treats motor dysfunction reduced symptoms and enhanced the cognitive performance in this group of children.

Cuthbert, S. C., & Barras, M. (2009). Developmental delay syndromes: Psychometric testing before and after chiropractic treatment of 157 children. Journal of Manipulative Physiological Therapy, 32, 660-669.

According to Dr. Yannick Pauli, a chiropractor from Lausanne (CH), this “supports multimodal chiropractic as an effective ADHD treatment.” I’m not so sure. To be considered strong evidence for the effectiveness of a practice, a study must meet at least three criteria (Gersten et al., 2000). First, let me simply list the criteria and, following that list provide an explanation of each criterion and how the present study fails to meet them.

  1. A powerful study should include a representative sample of the population;
  2. A powerful study should measure outcomes with trustworthy, objective instruments that assess performance on relevant variables; and
  3. A powerful study should employ a research design that allows comparison of the practice to other practices or no practice at all (e.g., a “control group”).

So, what do those standards mean? And how do they apply to this particular study?

  1. Representative sample: Because it is usually impossible to study an entire population, researchers usually employ only a sample of individuals from the population. Ideally, for the sample of individuals to represent the entire population, they should be drawn randomly from it. This is very rare in intervention research, however. This sample for this study is a consecutive case series, all the cases at a clinic during a given time period. Such samples are sometimes used in medical research, but their limitations are substantial, as the authors acknowledge in their statement of limitations (p. 666). Essentially, we can only safely generalize the results to individuals who find their way to that clinic. We do not have any way of knowing whether those children are or are not on average very much like all the other children who have dyspraxia, dyslexia, learning disabilities, and attention-deficit hyperactivity disorder (ADHD). Maybe they are quite similar, or maybe they differ in some important ways (e.g., parents with resources to pay for chiropractic examinations and treatments, if payment is required). The US National Cancer Institute ranks a consecutive case series study as a 3 on a 1-4 scale where 1 is the strongest type of design. Some good news about this sample: It included over 150 cases.
  2. Trustworthy measures: Measures of children’s outcomes are especially important in clinical research. They should assess the factors that are the most relevant to the problems the children experience (e.g., reading for those with dyslexia), should be trustworthy (i.e., reliable and valid), and objective. Drs. Cuthbert and Barras reported that the participants in this study were assessed on eight measures (I give the full name of the instruments in parentheses; the names used by the authors are listed first): complex figure of Rey test (Rey-Osterrieth Complex Figure); Borel-Maisonny test; Porteus maze test; oriented signs test; auditory memory test of Rey (Rey Auditory Verbal Learning Test); Piaget-Head tests; rhythm reproduction test of Stamback; and the facial motricity test of Stamback. Most of these tests assess perceptual-motor or memory skill. For example, on the Rey-Osterrieth Complex Figure children would be shown a complicated line drawing and copy it then have to redraw it from memory. On the Rey Auditory Verbal Learning Test, the children would hear a list of 15 unrelated words repeated five times and they would be asked to repeat the list; this is done for multiple lists and then the children are asked to recall one of those lists ˜30-35 minutes after the last practice trial on it. They reported that both pre- and post-testing was performed by “a qualified logopedist (a speech therapist)… employed only for conducting the psychometric testing,” which is better than having the same person conduct both the therapy and the measurement. However, they provide no evidence about the reliability and validity of the instruments.
  3. Research design: To blunt alternative explanations, the most powerful design (n simple terms) usually involves a comparison between a group of children who receive the experimental treatment and another group that does not receive it; it is from this “true experimental design” that we have the language “experimental group” and “control group.” Ideally, a representative sample of children would be randomly assigned to either the experimental or control group. Such a design protects against other factors (pre-existing differences between children in one group or the other; other changes that may have occurred during the same time period as the intervention; maturation; etc.) that might reasonably have caused changes in individuals. In this study, there is no control group (as the authors acknowledged), let alone a randomly constituted one. This is what Campbell and Stanley (1963) call a “one-group, pretest-posttest design” (p. 7). As a result, we simply have no way of knowing whether the higher scores for the children receiving chiropracty at post-testing would exceed, equal, and even be lower than scores on the same measures obtained for the children in a control condition. Maybe the differences in the scores from pretest to posttest are simply a result of the children (a) having taken the test once before and getting higher scores the second time; (b) growing older and, hence, being able to answer more items correctly because of greater experiences with similar tasks; or (c) learning some of the skills assessed by the tests from some other source (e.g., reading instruction). Indeed, there are other possible threats to the internal validity of a pretest-posttest study I’ve not described here, but these three all seem like reasonably possible explanations for higher scores on posttest that Drs. Buthbert and Barras reported.

Thus, in summary, in this study (a) there is a sample from which we cannot generalize, (b) measures that fit with the view of these problems being undergirded by a perceptual-motor problem but do not examine the main presenting problems of the children (e.g., hyperactive behavior), and (c) a design that does not permit us to make conclusions about the whether any differences found at posttest are the result of the treatment. To go with that stew of problems, the authors give minimal statistical analyses of the resulting data; they report average improvements for the entire group and percentage of children who had higher, the same, or lower levels of performance on the posttest than they had on the pretest. Therefore, it makes very good sense that the authors wrote that “[t]his report suggests” that the treatment was effective.

By the way, as best I can tell, “developmental delay syndromes” (DDS) is a higher-order classification used by chiropractors to refer to children with dyslexia, dyspraxia, Tourette’s Syndrome, attention-deficit hyperactivity disorder (ADHD, ADD), and obsessive-compulsive disoder (OCD). In the article, Drs. Cuthbert and Barras refer to four of these more specific disorders: dyspraxia, dyslexia, learning disabilities, and ADHD. I did not find a resource from which I could extract a definitive diagnostic schedule for DDS. They cite a book or pamphlet by A. L. Rosner entitled Infant and Child Care: An Assessment of Research from the Foundation for Chiropractic Education and Research.

I searched for the Foundation for Chiropractic Education and Research and found a FaceBook page that pointed to two Web sites: http://www.FCER.org and http://www.DCConsult.com (the former is in page-holder state by a Web hosting company, but the latter functions).

There is plenty more about this study that could be discussed, but I’m out of time. I hope others will dig into it and contribute to the conversation.

Link to Dr. Pauli’s article on “UnRitilin Solution.” Link to the National Cancer Institute page about levels of evidence in research.

References

Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Boston: Houghton-Mifflin.

Gersten, R., Baker, S., & Lloyd, J. (2000). Designing high-quality research in special education: Group experimental design. Journal of Special Education, 34, 2-18.

12 Responses to “Additional weak evidence about chiropractic treatment”


  • As always, John, a thoughtful and careful analysis of a piece of research. I’m always glad to know you’re still on the case.

  • St. Louis Chiropractor

    .., still based on the consumer on whether they will consider chiropractic practices… I have tried it myself.. there is no harm in it..

  • seattle chiropractor

    Your article was full of information that is very useful. I think your a very intelligent person. About chiropractic, It’s new that is why I still don’t believe on that. I need to try it before I believe on it.

  • central alberta chiropractor

    Based on my research there is no harm using chiropractic practices but it up to the patient if there were going to used it or not..

  • Just think, it will not called “chiropractic treatment” if it will cause a harm to a patients, treatment is a cure and i also believe in this kind of treatment, maybe the intention of the people who said that to other peoples that chiropractic treatment cause harm is to advertise their own treatment ^_^ so bad for him that many knows that this kind of treatment is really effective.

  • For those who are reading these comments, please understand that the comments from “Central Alberta Chiropractor,” “Seattle Chiropractor,” and “St. Louis Chiropractor” all came from the same computer, even though the person or people using that computer listed different e-mail addresses and pointed to different Web sites advertising chiropractic services.

  • Now that’s very odd, John! Skepticat got much the same today on her blog post Chiropractic is crap (she doesn’t mince her words!).

    Her comment today came from one ‘downtown chicago chiropractor’. Not downtown Chicago, but downtown Manilla! I wonder if this is your unimaginative and not very internet-savvy poster?

  • chiropractor specialist woodstock ga

    Thanks for writing this. It was interesting. You seem very knowledgeable in your field, and conscious of health issues.

  • Zeno, thanks for the observation. For interest, I consulted APIC.net and learned that, indeed, the posts pointing at “Central Alberta Chiropractor,” “Seattle Chiropractor,” and “St. Louis Chiropractor” were originating from a computer that connects to the Internet via the Philippine Long Distance Telephone Company. So does the one pointing to “Boise Chiropractor,” though it is a probably a different computer.

    As you and other savvy readers probably know, Zeno, one can purchase the services of blog commenting services. These are companies that promise to promote clients’ Web sites. I don’t know whether they work from lists or use an alert system to find target sites for posting, but the service at least boarders on spamming, it seems to me.

    A very interesting question, as I see it, is who is paying for this service. I could buy one personally to help promote LD Blog, but then the workers would be surfing about posting only about it. In this case, the apparent blog commenting service is posting references to multiple (related) sites. Perhaps the chiropractors at these sites have banded together to purchase a commenting service. Perhaps a chiropractic association provides the service for its members. Perhaps…perhaps…perhaps.

    I spent a few minutes poking through the Web site of one organization. (I note with admiration that you and your colleagues have investigated a lot of bogus claims.) I didn’t locate any direct evidence of the American Chiropractic Association funding such an effort; such evidence might show up in the minutes of meetings of the executive board or in tax returns that are supposed to be made pubic (“990 Returns”). However, there were references to “PR Tools & Tips” and to a members-only mailing list which might, potential sources of evidence about advertising strategies.

    For grins, I’ll check with Skepticat. We can compare the IP addresses for the commenters.

    However, I don’t want to get away from the focus of this message and others on LD Blog: Those who advocate chiropractic treatment for Learning Disabilities and other education problems do no have evidence that those treatments are effective.

  • houston chiropractor

    Some don’t believe on chiropractic treatment. But there many proof to prove that chiropractic treatment treats. I do believe that chiropractic can treat us from any back pain.

  • Well, folks, here’s another comment that looks for all the world like an advertising drop. My best surmise is that some chiropractors or chiropractic group is paying for a service that finds Internet references to chiropracty and then has people drop comments on those sites. The one for “houston chiropractor” is from 112.200.198.169, which is a computer that connects to the Internet from the a Philippines. Previous examples:

    ==> 112.200.197.24 advertised for chiros in Alberta (CA), St. Louis (MO, US), and Seattle (WA, US).
    ==> 112.200.212.52 advertised a Boise (ID, US) chiro.

    Another blogger who goes by “Skepticat” has a post on her site that has received comments from the same computers. Check her post entitled “Chiropractic is crap.”

    The advantage to hiring someone to drop comments is that the referred-to Web sites (e.g., “Earth Services”) will appear at or near the top of a list of sites one gets when one searches for “earth services” and location (e.g., Alberta, St. Louis, etc.). Paying for such a service increases the chances that one’s “service” (e.g., chiropractic) will harvest customers (AKA ‘patients’) from Internet searches.

  • Thank you for this well done and well documented commentary on chiropractic research.

    I do find myself in agreement with you that single studies of small populations without control groups are weak arguments for efficacy of treatment.

    In my own practice we make very few claims for chiropractic efficacy apart from pain relief and improved mobility for certain musculo-skeletal complaints – well documented and readily apparent phenomenon.

    It would be interesting to examine other established medical sub-specialities with the same skeptical rigor, the claims and practices of podiatry for an example.

    Grant Stowell, D.C.

Comments are currently closed.