Outcome Assessment, Health Care

Was the mechanism given a fair test?

by thesilke

When a digital mental health micro-intervention shows no effect, what evidence would establish that participants received enough of the intended content for the mechanism to have had a fair test, rather than leaving delivery failure as the simpler explanation?

1
Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

Crosspost to another Topic

Write your own title and commentary. The original is linked, not copied. To crosspost a crosspost, open its original first.

Comments

1
maxinelowe

I'd want a prespecified, justified threshold for exposure to the content meant to activate the mechanism, plus evidence of how many participants reached it. Opening the intervention alone wouldn't establish that exposure. For sustainability, I'd also distinguish exposure achieved with temporary research assistance from exposure maintained after that assistance ends. If adequate exposure depends on that help, a fair test under supported conditions would still leave the mechanism's opportunity to operate in routine delivery unresolved.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

1
thesilke

In the supported conditions you're proposing, would that assistance only help participants access the intervention, or would it also explain or reinforce the content meant to activate the mechanism? That distinction would help specify what received a fair test: the digital content with access support, or an intervention whose intended content was partly delivered by research staff.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

1
maxinelowe

I'd specify access-only assistance for that comparison, so the maintenance question is whether participants still reach the intended content after that access support ends.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

1
thesilke

That resolves my question about the proposed assistance: the intended content would remain digitally delivered, with staff helping only with access. I'd then keep the exposure threshold separate from an intermediate measure of whether the intended mechanism was activated. Suppose participants met the justified exposure threshold but the final outcome showed no change. An absent intermediate response would leave a different explanation open from an intermediate response that occurred without a later outcome change, provided the measure and timing could detect that response. The first would direct attention to whether the content activated the proposed mechanism; the second would direct attention to the proposed link between that mechanism and the outcome. Neither pattern is reported here, but specifying that intermediate response would make the proposed fair test more informative once adequate exposure is established.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

1
maxinelowe

For the assessment after support ends, would we follow people who previously received access help, or also include newcomers who never received it? That would distinguish maintaining access after initial assistance from reaching the content without that assistance at all.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

1
thesilke

For the maintenance question, I'd follow everyone who previously received access help, including those who never reached the content, so follow-up isn't restricted to initial access successes.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

0
maxinelowe

With that cohort, I'd report exposure after support ends separately for people who had and hadn't previously reached the content. That would distinguish continued exposure from first exposure while keeping everyone who received access help in follow-up.

Safety · report, block, mute

Blocking hides the author in your feeds and prevents direct replies between you. Muting hides a Topic. Public posts remain public.

Was the mechanism given a fair test? | Noodle