Welcome to the analysis of my outcome statistics. This may be something you have never seen done before, at least in this much detail. I’m not going to go into a big intro on this one, because it was a monster effort just putting this together. With that being said, I am no expert when it comes to statistical analysis, so please feel free to let me know if you see any potential mistakes, have any questions or feedback on what I’m doing.

I will say this though, doing this has felt like more support as to why I believe we should spend less time researching new techniques and therapies and more time analysing our own outcome data. In doing this exercise I have learnt things about my caseload that I never would have learnt had I decided to put the bulk of my professional development into learning more ACT therapy, for example.

It’s amazing when you have outcome data, not only how many questions you can ask and investigate, something that is well worth the energy – while immense at first. It’s all worth it when I learn about my clients and how I work, in ways I never could have achieved before.

Finally, it will be useful to know that my data has come from the two outcome measures I have predominantly used. The Outcome Rating Scale and Session Rating Scale. Though I ceased using the SRS through 2023, something I’ll likely share more about in the future.

First some context

To paint a clearer picture, it would probably be useful if I gave you a brief idea of the people I see. I work in a private practice in Sydney, Australia. The suburb I work in consists predominantly of people who are of an Anglo-Saxon background and earn above average wages when compared to the rest of Sydney. They are across the lifespan (from the age of 12 and upwards) and their presenting challenges tend to be in the mild to moderate intensity range. I have a fairly broad scope of the kinds of mental health challenges I work with as well, in that I’ll work with just about anything except for chronic pain, severe eating disorders and psychosis – because I don’t have as much experience in these areas.

I know some people like a niche, who knows maybe one day I’ll find one. At this point I enjoy the wide variety, it keeps me on my toes.

Next, I thought I would provide some explanations on why I have conducted this statistical analysis in the way I have. To do so I’ll answer a couple of questions I was asked by some people on Facebook:

1. Why did you only present data on client’s who had an initial/base-line ORS score that was below clinical cut off (<25 points)?

This was based on advice I had from other therapists that are more confident in understanding statistics than I am. Which was that there is research suggesting when client’s have an initial ORS score >25, they subsequent ORS scores are less likely to significantly change. It’s a situation that can act as a confounding variable on the data overall when you mix these in with clients who scored <25 in their initial ORS. One of the main theories about why this can be an issue is because clients who score >25 with their initial ORS have ‘less room to grow’ with ORS scores over time, as the maximum score is 40.

I did notice an impact on my data as well, while it was small, including initial ORS scores of >25 could impact effect size ratings by approximately 0.2 at least. These clients are also far more likely to end therapy above clinical cut off as well. Since I started tracking ORS scores with FIT Outcomes in 2021, I have had 64 clients with an initial ORS of >25, with 71.7% reaching target – where as 48.85% have reached target for client’s with an initial ORS <25 (n = 174). It’s essentially a case of comparing apples and oranges.

• 2. The changes you saw in outcome data year to year – were they expected changes in line with research?

Again, this is based on feedback from others who are more familiar and involved in the research of Deliberate Practice as used by therapists. Apparently once an effect size of 0.6 is achieved, subsequent gains in effect size is very slow. Unfortunately I don’t have specific knowledge to speak to that in more depth.

My interpretation is that the change 2021 to 2022 was an expected increase. I say this because while the effect size numbers have gone up and down over the 3 years, the numbers have stayed with the ‘medium/moderate’ effect ranges. In short, I have stayed largely moderately effective. Though I’m also aware that effect size has its flaws as a tool for statistical analysis – it’s the best I have access to for now.

In saying that, a drop in 2023 perhaps wasn’t expected, but more on that as we read on…

Now into the good stuff

My outcome statistics

Year	Closed Cases	Closed Cases sessions Ave	Active/ongoing Cases sessions Ave	Intake ORS score Ave	ORS raw change Ave	Drop out %	% reaching target*	% reaching reliable change	% reaching clinical change	Effect size	Relative effect size
2021	55	4.21	6.71	17.76	7.44	26.8	58.7	3.63	43.63	0.62	-0.15
2022	59	8.34	14.38	17.28	10.15	6.8	71.2	3.39	61.01	0.76	0.00
2023	58	8.00	17.09	16.70	7.77	17.2	54.3	5.17	41.37	0.53	-0.23
Total Now	174	6.92	–	17.30	8.53	16.3	61.6	4.02	48.85	0.63	-0.13

Above you will find the table that presents my outcome statistics across three years of Deliberate Practice. It’s worth knowing as well that I won’t be analysing 2021 a whole lot in this post, the reason being is that I now work so very differently to how I did back then. It just felt more fruitful to focus on the observed differences between 2022 versus 2023.

If you have seen me post some outcome statistics before, please that I have gone into more depth this time around. Now that I have three years to compare, I wanted to go deeper.

In the past I would have first sought to analyse my caseload by reviewing each client and whether I kept well to the ‘foundations’ of therapy with them – for example, whether there was effective goal-setting, how well I listened or if I responded to feedback effectively.

This time around though I didn’t want to assume that the changes in my outcome stats year to year were solely down to alignment with such foundational principles/skills.

I made this decision because, perhaps against my better judgement, I just wasn’t convinced that I was foundationally worse in 2023 compared to 2022 as the outcome data above may suggest. It’s because of that feeling that I decided to start my analysis in another place, by exploring statistics that I had either ignored or not paid much attention to in the past. I wanted to see if there was more to the story. In the end, I had had conducted my analysis at three levels:

• Level 1: Looking at Initial SRS scores.

As soon as I decided I wanted to start by looking deeper, I looked for statistics in the FIT Outcomes output that I had not paid much attention to, if any, in the past. The first one that stood out to me was the client’s initial SRS scores, as presented in the table below.

Side note: In case you end up wondering, my 2023 SRS count was so low because that was the first year I transitioned from using the SRS to other ways of eliciting feedback, essentially using my own scales I had created based on client feedback.

Year	Count Intake SRS	Intake SRS score ave	Count Intake SRS above 36 (more than 1 session)
2021	49	29.94	7
2022	48	31.67	9
2023	7	32.80	N/A

So why did I decide to look at clients who had an initial SRS score of >36?

I remembered that high initial SRS scores can be an issue for therapy in some cases. In that a high initial score can mean that the client could be engaging in a form of positive impression management or social desirability. They might not want to hurt the therapists feelings by giving a lower score. If true, the downside is that areas where the therapy could improve might not be raised, but then contributes to premature drop out.

A theory I had was whether high initial SRS scores were part of why I had a ‘better’ year in 2022.

As you can see in the above table, while the average initial SRS score for client’s was higher in 2022 on average, the number of clients with an initial ORS score >36 was very similar for 2022 versus 2023. I then came to the conclusion that it’s unlikely that initial SRS scores accounted for a significant difference in 2022 results.

• Level 2: Looking at long term/high volume clients

At the 2nd level, I had a reasonably strong hunch that I had gained more long-term/high volume clients over the years. On one hand, this made sense because I had been working at one clinic for 3 years, longer than I had been working at one clinic previously.

Even so, I hoped this was also a sign that I was improving as a therapist over time, I wanted to have a closer look.

Year	Number of short term/low volume clients (No more than 10 sessions)	Number of long term/high volume clients (10 sessions or more)	% short term/low volume clients reaching clinical change	% of long term/high volume clients reaching clinical change	Number of long term clients that finished therapy in a different year to when they started	Number of current/ongoing long term/high volume clients from each year
2021	54	1	(N=27) – 68.51%	(N=1) – 100%	0	8
2022	44	15	(N=25) – 56.81%	(N=12) – 80%	13	10
2023	46	12	(N = 15) – 32.06%	(N=6) – 50%	10	17
Total						35

Please bear with me with this one, as the table above might look a bit complicated. Firstly, I decided that it wasn’t just about having long term clients, but how did their outcomes compare to my short term/lower volume clients?

For 2022 versus 2023, I had a similar number of short and longer term clients, which hopefully means a fairer comparison. It seems that my longer term/higher volume clients fair batter when it comes to their outcomes. This may be because they have found therapy useful, they have wanted to keep going. I often tell my clients after we resolve their first presenting challenge, that they are welcome to continue so long as we can identify something that is worth working on. I don’t force that however, if a client wants to end therapy, I let them.

There is another interesting observation, while I had double in long term/high volume clients reach clinical change in 2022 (n = 12) versus 2023 (n = 6), I have more current/ongoing clients from 2023. In fact, at the time of writing, I have 35 current/ongoing long term/high volume clients now, which is not only more than I have ever had, but higher than the maximum possible in 2022, which would have been 18 at most (8+10 from above table)

If the above findings are accurate, it could have an important implication for my outcome statistics and overall effectiveness. If long-term/high volume clients tend to have better outcomes, I’d be willing to bet that my outcome stats will improve this year versus 2023.

It will also come down to how many long term/high volume clients decide to end therapy this year.

• Level 3: Review each client case

At this final level, I decided to look at what I normally would in analysing my cases. Which is how many closed cases I had per year, how many were premature drop outs, were a mutually agreed ending and how many reached clinical change on the ORS.

Looking at the quantitative information in the table below, the main observations that jump out at me are:

• 1. I seem to be better at initial sessions than in 2021.
• 2. I had no yellow zone drop outs in 2022 and more clients reach clinical change than in the other two years (not forgetting from the previous table I had more long term/high volume clients reach clinical change in 2022)

Year	Closed Cases	# initial only/red-zone/drop out	# initial only/mutual agree end	# mutual agree end/red-zone or yellow zone (2+ sessions)	# drop out/red-zone /deterioration	# intermediate/yellow zone drop out	# reached clinical change	# reached reliable change	# reached clinical/reliable change
2021	55	6	3	11	2	7	24	2	26
2022	59	0	5	12	4	0	36	2	38
2023	58	3	6	15	2	5	24	3	27
Total	174	9	14	39	8	12	85	7	92

The numbers don’t tell me much about what could have explained the differences, such as why I had no yellow zone endings and more clients reaching clinical change in 2022.

In the interest of answering that I decided to have a closer look at each client across the three years. Including:

• 1. Age range
• 2. Gender
• 3. Therapy mode (face to face, telehealth, phone)
• 4. Initial ORS score
• 5. Final ORS score
• 6. ORS raw score change
• 7. Initial SRS score
• 8. Number of sessions
• 9. Year started therapy
• 10. Year ended therapy
• 11. Therapy duration (months)
• 12. Qualitative observations – whether there was goal setting, quality of the feedback relationship and/or why they client ended therapy

As far as I could tell, I didn’t seem to form any bad habits in 2023 to explain the difference in outcome stats compared to 2022. In 2023, I kept goal-setting with clients often (while listening to their goals) and had good feedback relationships. There were few occasions were these foundations were not in place, but they occurred mostly in the start of 2023. In short, I couldn’t see any obvious signs that I had gone backwards in 2023 in terms of the positive habits I had formed in deliberate practice through the previous two years.

In conclusion

At the end of the day, it’s possible that 2022 was a standout year for me and that I could have formed some bad habits since then, that I’m not yet aware of. Though after my analysis I’m not convinced. It could also be said that it seems that I have become more engaging as a therapist. My average number of sessions have increased and continues to do so among active/current clients. While on the other hand I don’t seem to have yet become more effective, as my effect sizes have largely remained in the moderate range across the three years and my percentage of clients reaching clinical change has shifted up and down.

Where my mind ends up however, is that it seems possible that long term/high volume clients could be the key to understanding my true effectiveness. Until my average number of sessions and number of long term/high volume clients reach a ceiling, my effectiveness may remain somewhat unclear.

So where do I go from here?

In essence, I’m going to wait until my average number of sessions with active clients stops climbing. In the meantime I don’t plan on making any major changes to how I operate, I’m going to stay the course. I’ll focus this year mostly on consolidating and strengthening my engagement with my current Deliberate Practice Ecosystem as it stands, especially focusing on deep listening.

I am glad that I have done this, while it was a huge effort to analysis my data so thoroughly (and then write about it) it was immensely rewarding. My understanding of my caseload is deeper than ever, more than it ever could be did I not have this information. There’s no way I could have guessed the findings I’ve found, without outcome data.

I’m also glad it has given me a future system of levels to consider when analysing my data again in future years. I’m also going to log and analyse each client this year as they end therapy, so that the level 3 analysis does not take me as long in future years.

I hope you found this interesting and as always that it may have inspired you to try down a similar path for yourself.

---

Go to previous or next post

If you want to stay in the loop for future posts, then please feel free to sign up for our mailing list: