In “Developmental Delay Syndromes: Psychometric Testing Before and After Chiropractic Treatment of 157 Children,” Scott Cuthbert and Michel Barras present the results of an analysis of pretest-posttest scores for children who received chiropractic treatment at a clinic in Lausanne (CH). They reported that the children had higher scores after treatment, leading them to conclude that “This report suggests that a multimodal chiropractic method that assesses and treats motor dysfunction reduced symptoms and enhanced the cognitive performance in this group of children.”
Here is the abstract for this report. After it, I’ll explain why I find this study provides uncompelling evidence in support of chiropractic treatment for Learning Disabilities.
Objective: This study presents a case series of 157 children with developmental delay syndromes, including the conditions such as dyspraxia, dyslexia, attention-deficit hyperactivity disorder, and learning disabilities who received chiropractic care.
Clinical Features: A consecutive sample of 157 children aged 6 to 13 years (86 boys and 71 girls) with difficulties in reading, learning, social interaction, and school performance who met these inclusion criteria were included. Intervention and Outcomes: Each patient received a multimodal chiropractic treatment protocol, applied kinesiology chiropractic technique. The outcome measures were a series of 8 standardized psychometric tests given to the children by a certified speech therapist pre- and posttreatment, which evaluate 20 separate areas of cognitive function, including patient- or parent-reported improvements in school performance, social interaction, and sporting activities. Individual and group data showed that at the end of treatment, the 157 children showed improvements in the 8 psychometric tests and 20 areas of cognitive function compared with their values before treatment. Their ability to concentrate, maintain focus and attention, and control impulsivity and their performance at home and school improved.
Conclusions: This report suggests that a multimodal chiropractic method that assesses and treats motor dysfunction reduced symptoms and enhanced the cognitive performance in this group of children.
Cuthbert, S. C., & Barras, M. (2009). Developmental delay syndromes: Psychometric testing before and after chiropractic treatment of 157 children. Journal of Manipulative Physiological Therapy, 32, 660-669.
According to Dr. Yannick Pauli, a chiropractor from Lausanne (CH), this “supports multimodal chiropractic as an effective ADHD treatment.” I’m not so sure. To be considered strong evidence for the effectiveness of a practice, a study must meet at least three criteria (Gersten et al., 2000). First, let me simply list the criteria and, following that list provide an explanation of each criterion and how the present study fails to meet them.
- A powerful study should include a representative sample of the population;
- A powerful study should measure outcomes with trustworthy, objective instruments that assess performance on relevant variables; and
- A powerful study should employ a research design that allows comparison of the practice to other practices or no practice at all (e.g., a “control group”).
So, what do those standards mean? And how do they apply to this particular study?
- Representative sample: Because it is usually impossible to study an entire population, researchers usually employ only a sample of individuals from the population. Ideally, for the sample of individuals to represent the entire population, they should be drawn randomly from it. This is very rare in intervention research, however. This sample for this study is a consecutive case series, all the cases at a clinic during a given time period. Such samples are sometimes used in medical research, but their limitations are substantial, as the authors acknowledge in their statement of limitations (p. 666). Essentially, we can only safely generalize the results to individuals who find their way to that clinic. We do not have any way of knowing whether those children are or are not on average very much like all the other children who have dyspraxia, dyslexia, learning disabilities, and attention-deficit hyperactivity disorder (ADHD). Maybe they are quite similar, or maybe they differ in some important ways (e.g., parents with resources to pay for chiropractic examinations and treatments, if payment is required). The US National Cancer Institute ranks a consecutive case series study as a 3 on a 1-4 scale where 1 is the strongest type of design. Some good news about this sample: It included over 150 cases.
- Trustworthy measures: Measures of children’s outcomes are especially important in clinical research. They should assess the factors that are the most relevant to the problems the children experience (e.g., reading for those with dyslexia), should be trustworthy (i.e., reliable and valid), and objective. Drs. Cuthbert and Barras reported that the participants in this study were assessed on eight measures (I give the full name of the instruments in parentheses; the names used by the authors are listed first): complex figure of Rey test (Rey-Osterrieth Complex Figure); Borel-Maisonny test; Porteus maze test; oriented signs test; auditory memory test of Rey (Rey Auditory Verbal Learning Test); Piaget-Head tests; rhythm reproduction test of Stamback; and the facial motricity test of Stamback. Most of these tests assess perceptual-motor or memory skill. For example, on the Rey-Osterrieth Complex Figure children would be shown a complicated line drawing and copy it then have to redraw it from memory. On the Rey Auditory Verbal Learning Test, the children would hear a list of 15 unrelated words repeated five times and they would be asked to repeat the list; this is done for multiple lists and then the children are asked to recall one of those lists ˜30-35 minutes after the last practice trial on it. They reported that both pre- and post-testing was performed by “a qualified logopedist (a speech therapist)… employed only for conducting the psychometric testing,” which is better than having the same person conduct both the therapy and the measurement. However, they provide no evidence about the reliability and validity of the instruments.
- Research design: To blunt alternative explanations, the most powerful design (n simple terms) usually involves a comparison between a group of children who receive the experimental treatment and another group that does not receive it; it is from this “true experimental design” that we have the language “experimental group” and “control group.” Ideally, a representative sample of children would be randomly assigned to either the experimental or control group. Such a design protects against other factors (pre-existing differences between children in one group or the other; other changes that may have occurred during the same time period as the intervention; maturation; etc.) that might reasonably have caused changes in individuals. In this study, there is no control group (as the authors acknowledged), let alone a randomly constituted one. This is what Campbell and Stanley (1963) call a “one-group, pretest-posttest design” (p. 7). As a result, we simply have no way of knowing whether the higher scores for the children receiving chiropracty at post-testing would exceed, equal, and even be lower than scores on the same measures obtained for the children in a control condition. Maybe the differences in the scores from pretest to posttest are simply a result of the children (a) having taken the test once before and getting higher scores the second time; (b) growing older and, hence, being able to answer more items correctly because of greater experiences with similar tasks; or (c) learning some of the skills assessed by the tests from some other source (e.g., reading instruction). Indeed, there are other possible threats to the internal validity of a pretest-posttest study I’ve not described here, but these three all seem like reasonably possible explanations for higher scores on posttest that Drs. Buthbert and Barras reported.
Thus, in summary, in this study (a) there is a sample from which we cannot generalize, (b) measures that fit with the view of these problems being undergirded by a perceptual-motor problem but do not examine the main presenting problems of the children (e.g., hyperactive behavior), and (c) a design that does not permit us to make conclusions about the whether any differences found at posttest are the result of the treatment. To go with that stew of problems, the authors give minimal statistical analyses of the resulting data; they report average improvements for the entire group and percentage of children who had higher, the same, or lower levels of performance on the posttest than they had on the pretest. Therefore, it makes very good sense that the authors wrote that “[t]his report suggests” that the treatment was effective.
By the way, as best I can tell, “developmental delay syndromes” (DDS) is a higher-order classification used by chiropractors to refer to children with dyslexia, dyspraxia, Tourette’s Syndrome, attention-deficit hyperactivity disorder (ADHD, ADD), and obsessive-compulsive disoder (OCD). In the article, Drs. Cuthbert and Barras refer to four of these more specific disorders: dyspraxia, dyslexia, learning disabilities, and ADHD. I did not find a resource from which I could extract a definitive diagnostic schedule for DDS. They cite a book or pamphlet by A. L. Rosner entitled Infant and Child Care: An Assessment of Research from the Foundation for Chiropractic Education and Research.
I searched for the Foundation for Chiropractic Education and Research and found a FaceBook page that pointed to two Web sites: http://www.FCER.org and http://www.DCConsult.com (the former is in page-holder state by a Web hosting company, but the latter functions).
There is plenty more about this study that could be discussed, but I’m out of time. I hope others will dig into it and contribute to the conversation.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Boston: Houghton-Mifflin.
Gersten, R., Baker, S., & Lloyd, J. (2000). Designing high-quality research in special education: Group experimental design. Journal of Special Education, 34, 2-18.