Editorial
Randomized Controlled Trials and Psychotherapy Research
Arthur Rifkin, M.D.
This issue of The American Journal of Psychiatry contains three randomized controlled trials of psychotherapy. I would like first to discuss general methodological issues of psychotherapy research and then apply them to the three studies in this issue.
Does effective psychotherapy share a general characteristic of providing comfort and hope, or by the therapeutic value of a specific technique? If we approach the issue scientifically, we quickly reach the central problem of psychotherapy: What is the proper comparison group? Most randomized controlled trials of psychotherapy used a waiting list comparison group. This does not seem adequate. Imagine that you agreed to participate in a study comparing psychotherapy with a waiting list and were assigned to the waiting list. Would not you feel disappointed? Then, if the psychotherapy group did better, would that mean it was effective or that it did not influence the natural course of the illness and those disappointed subjects on the waiting list did worse than the natural course? Clearly we need a better comparison group than a waiting list.
The pragmatic question for most axis I disorders is not whether psychotherapy works, but how it compares with drug treatment, since we have drugs that are proven effective for those disorders. If we compare drug treatment with psychotherapy, we face the problem of interpretation when there are no significant differences between treatments. Does this mean that both treatments work, or neither? If one treatment is superior to the other, does that mean the superior one is effective or that one treatment impairs the natural course while the other did not affect it?
We have much more evidence for the effectiveness of drug treatment than psychotherapy. We do not need further placebo-controlled studies to demonstrate that the drugs given the indication for their use by the Food and Drug Administration are effective. Yet, while the drugs clearly are better than placebo, many patients do not respond. We have no doubt that imipramine is an effective antidepressant, yet as of the mid-1960s, one-third of placebo-controlled trials were negative (1). Depression must be a heterogeneous disorder, and only a proportion of patients are drug responders. It is quite possible that a particular cohort contains a low proportion of drug responders and will not show a difference from placebo. If we compare psychotherapy with an active drug, we may find no difference because both are ineffective. It seems very important to include a pill placebo group to detect such instances. This also controls, at least partially, for the role of expectation influencing the effect of psychotherapy. It does not seem useful to test the value of psychotherapy versus drug treatment in a cohort insensitive to drug treatment (2).
In drug studies, we have no need to worry about the identity of the drug. This becomes a large problem in psychotherapy studies. One therapist may do psychotherapy differently from another and differently among different subjects, even if the therapist agrees to use the same technique. Psychotherapy studies deal with this problem by careful training, by using manuals, and by taping sessions for later evaluation. This can improve the consistency of psychotherapy, but at the cost of deviating from usual practice. Can we generalize results of manualized psychotherapy studies to the population of therapists?
The three studies published in this issue illustrate different methodologies. Weiss et al. used two treatments they expected to be effective, although hypothesizing that one would be superior to the other. In subjects with substance use disorder and bipolar disorder, they compared a form of group therapy that focused on both disorders with one that focused on substance abuse only. Surprisingly, the group therapy that focused on both disorders affected substance abuse more than the group therapy that focused only on substance abuse.
All studies have limitations, this one included. The subjects not only received drug treatment during the trial, but also, some had changes in drug treatment during the study, and most received individual psychotherapy and one-half attended self-help groups. This confounds and perhaps weakens the effect of the group psychotherapy.
The days of alcohol use per month differed between groups by 4.4–6.5, which is statistically significant, but leaves us wondering weather the difference is clinically significant as measured by effect size. According to my calculations, the effect size is only 0.32, a small effect. As previously discussed, when comparing two active treatments without a pill placebo, we cannot say whether one treatment helped or the other interfered with recovery.
The study by Sijbrandij et al. used a waiting list comparison group to test the treatment of brief cognitive behavior therapy (CBT) for posttraumatic stress disorder (PTSD). They found a significant advantage for CBT after 1 week and not after 4 months. Does this mean that CBT had a small effect or that assignment to the waiting list impaired recovery? Use of a waiting list also raises ethical questions, in this case keeping someone with PTSD without treatment for 4 months.
The study by Greeven et al. is the most methodologically clear. It compares CBT with paroxetine for hypochondriasis, using a pill placebo comparison group. Hypochondriasis is a neglected disorder. It has prevalence in primary care settings of 4.2%–8.5%, and we lack a proven treatment. This study is welcome. The authors report that both treatments are superior to placebo and equal to each other. The presence of a placebo group establishes that not only are the treatments equal, but effective (superior to placebo). They help the reader to assess the degree of efficacy by reporting effect sizes. For their main outcome measure, the effect sizes were 0.44 for CBT versus placebo, 0.40 for paroxetine versus placebo, and 0.09 for CBT versus paroxetine. We consider an effect size of 0.3 a small effect, while 0.5 is a modest one, and >0.8 is a large effect. The results presented by Greeven et al. are somewhat better for completers. The treatments not only reduced hypochondriacal symptoms but concomitant symptoms of anxiety, depression, and somatoform disorder.
Psychotherapy research is not for the faint of heart. There is a thicket of problems, yet for how long should psychotherapy constitute a large part of the treatments offered by mental health clinicians without proof of efficacy in these days of evidence-based medicine?
Footnotes
Address correspondence and reprint requests to Dr. Rifkin, Department of Psychiatry, Zucker Hillside Hospital, 263rd Street, Glen Oaks, NY 11004; rifkin@lij.edu (e-mail).
Dr. Rifkin reports no competing interests.
References
Klein DF, Davis JM: Diagnosis and drug treatment of psychiatric disorders. Baltimore, Williams and Wilkins, 1969
Klein DF: Control groups in pharmacotherapy and psychotherapy evaluations, 1997. (http://journals.apa.org/treatment/vol1/97_a1.html [accessed 10/18/06])
© 2007 American Psychiatric Association
Randomized Controlled Trials and Psychotherapy Research
Arthur Rifkin, M.D.
This issue of The American Journal of Psychiatry contains three randomized controlled trials of psychotherapy. I would like first to discuss general methodological issues of psychotherapy research and then apply them to the three studies in this issue.
Does effective psychotherapy share a general characteristic of providing comfort and hope, or by the therapeutic value of a specific technique? If we approach the issue scientifically, we quickly reach the central problem of psychotherapy: What is the proper comparison group? Most randomized controlled trials of psychotherapy used a waiting list comparison group. This does not seem adequate. Imagine that you agreed to participate in a study comparing psychotherapy with a waiting list and were assigned to the waiting list. Would not you feel disappointed? Then, if the psychotherapy group did better, would that mean it was effective or that it did not influence the natural course of the illness and those disappointed subjects on the waiting list did worse than the natural course? Clearly we need a better comparison group than a waiting list.
The pragmatic question for most axis I disorders is not whether psychotherapy works, but how it compares with drug treatment, since we have drugs that are proven effective for those disorders. If we compare drug treatment with psychotherapy, we face the problem of interpretation when there are no significant differences between treatments. Does this mean that both treatments work, or neither? If one treatment is superior to the other, does that mean the superior one is effective or that one treatment impairs the natural course while the other did not affect it?
We have much more evidence for the effectiveness of drug treatment than psychotherapy. We do not need further placebo-controlled studies to demonstrate that the drugs given the indication for their use by the Food and Drug Administration are effective. Yet, while the drugs clearly are better than placebo, many patients do not respond. We have no doubt that imipramine is an effective antidepressant, yet as of the mid-1960s, one-third of placebo-controlled trials were negative (1). Depression must be a heterogeneous disorder, and only a proportion of patients are drug responders. It is quite possible that a particular cohort contains a low proportion of drug responders and will not show a difference from placebo. If we compare psychotherapy with an active drug, we may find no difference because both are ineffective. It seems very important to include a pill placebo group to detect such instances. This also controls, at least partially, for the role of expectation influencing the effect of psychotherapy. It does not seem useful to test the value of psychotherapy versus drug treatment in a cohort insensitive to drug treatment (2).
In drug studies, we have no need to worry about the identity of the drug. This becomes a large problem in psychotherapy studies. One therapist may do psychotherapy differently from another and differently among different subjects, even if the therapist agrees to use the same technique. Psychotherapy studies deal with this problem by careful training, by using manuals, and by taping sessions for later evaluation. This can improve the consistency of psychotherapy, but at the cost of deviating from usual practice. Can we generalize results of manualized psychotherapy studies to the population of therapists?
The three studies published in this issue illustrate different methodologies. Weiss et al. used two treatments they expected to be effective, although hypothesizing that one would be superior to the other. In subjects with substance use disorder and bipolar disorder, they compared a form of group therapy that focused on both disorders with one that focused on substance abuse only. Surprisingly, the group therapy that focused on both disorders affected substance abuse more than the group therapy that focused only on substance abuse.
All studies have limitations, this one included. The subjects not only received drug treatment during the trial, but also, some had changes in drug treatment during the study, and most received individual psychotherapy and one-half attended self-help groups. This confounds and perhaps weakens the effect of the group psychotherapy.
The days of alcohol use per month differed between groups by 4.4–6.5, which is statistically significant, but leaves us wondering weather the difference is clinically significant as measured by effect size. According to my calculations, the effect size is only 0.32, a small effect. As previously discussed, when comparing two active treatments without a pill placebo, we cannot say whether one treatment helped or the other interfered with recovery.
The study by Sijbrandij et al. used a waiting list comparison group to test the treatment of brief cognitive behavior therapy (CBT) for posttraumatic stress disorder (PTSD). They found a significant advantage for CBT after 1 week and not after 4 months. Does this mean that CBT had a small effect or that assignment to the waiting list impaired recovery? Use of a waiting list also raises ethical questions, in this case keeping someone with PTSD without treatment for 4 months.
The study by Greeven et al. is the most methodologically clear. It compares CBT with paroxetine for hypochondriasis, using a pill placebo comparison group. Hypochondriasis is a neglected disorder. It has prevalence in primary care settings of 4.2%–8.5%, and we lack a proven treatment. This study is welcome. The authors report that both treatments are superior to placebo and equal to each other. The presence of a placebo group establishes that not only are the treatments equal, but effective (superior to placebo). They help the reader to assess the degree of efficacy by reporting effect sizes. For their main outcome measure, the effect sizes were 0.44 for CBT versus placebo, 0.40 for paroxetine versus placebo, and 0.09 for CBT versus paroxetine. We consider an effect size of 0.3 a small effect, while 0.5 is a modest one, and >0.8 is a large effect. The results presented by Greeven et al. are somewhat better for completers. The treatments not only reduced hypochondriacal symptoms but concomitant symptoms of anxiety, depression, and somatoform disorder.
Psychotherapy research is not for the faint of heart. There is a thicket of problems, yet for how long should psychotherapy constitute a large part of the treatments offered by mental health clinicians without proof of efficacy in these days of evidence-based medicine?
Footnotes
Address correspondence and reprint requests to Dr. Rifkin, Department of Psychiatry, Zucker Hillside Hospital, 263rd Street, Glen Oaks, NY 11004; rifkin@lij.edu (e-mail).
Dr. Rifkin reports no competing interests.
References
Klein DF, Davis JM: Diagnosis and drug treatment of psychiatric disorders. Baltimore, Williams and Wilkins, 1969
Klein DF: Control groups in pharmacotherapy and psychotherapy evaluations, 1997. (http://journals.apa.org/treatment/vol1/97_a1.html [accessed 10/18/06])
© 2007 American Psychiatric Association
