FRAME-IS: Tracking Adaptations to Implementation Strategies

okay welcome everybody my name is anna bauman i am from washington university in st louis and it is my pleasure to have my friends to talk about the framework for adaptations and

modifications to evidence-based implementation strategies the frame is so we're going to introduce ourselves hi everyone i'm chris miller i'm a

psychologist and investigator at choir the center for healthcare organization and implementation research at va boston with an academic affiliation at harvard medical school

hi everyone i'm maya barnette i am a clinical psychologist at the department of counseling clinical and school psychology at the university of california santa barbara

and i am annabellman i am at the washington university everyone i'm cassidy guttner i'm in the department of psychiatry at boston

university school of medicine and i'm also a psychologist hi i'm shannon wilsie-sterman i'm at the national center for ptsd and at stanford university

fantastic so before we start we wanted to say that those slides have not been audited for visual disability if you need an adapted version of these lights

do let us know email us we are happy to share the slides and happy to audit them for you um you have our emails and our twitter twitter handles in the bottom of the

slides so today we're going to talk about we're going to acknowledge and disclaim our funding we're talking about the definitions of adaptation setting the stage and talk a little bit

about what is frame is and provide some examples of how we're tracking and using frame is future directions and wrap up so in terms of acknowledgements and

disclaimers um first you can see the citation or front matter for the manuscript on which this presentation is based we don't have any conflicts of interest to declare and we did want to give special thanks

to those who weighed in on the draft versions of the frame is details can be found in the manuscript itself and next slide here um in terms of

funding support there's a lot of it um so we do acknowledge our funders and want to give a special shout out to the implementation research institute or iri uh with which all of us co-authors have

been involved in some way shape or form okay just to talk a little bit and make sure we're clear on the terms we're using when we talk about

implementing evidence-based practice we're talking about interventions that have been demonstrated to be effective so these could be um psychotherapy medication a new process or procedure there can be

actually preventive interventions um and so when we talk about the evidence-based practice in terms of the framework uh or the way that jeff curran has kind of laid us out this is the

thing we're trying to implement so this is actually what we're trying to get into routine practice or kind of out and accessible to people who can benefit from them

and then the implementation strategy is the strategy or the method or the technique that we use to help that the thing be implemented so these are um

these are as jeff curran would put it the stuff we do to try to help people or places do the thing so these are things that we try to do to help it along help it get into routine practice

um and you know examples are things like um training audit and feedback um you know forming stakeholder advisory boards

promoting networks of people to communicate and share information there are a variety there are dozens of implementation strategies and we'll be

focusing on how to document adaptations to those today so when we talk about adaptation modification etc modification is kind of a blanket term which means

any change you make to an intervention or a program or today we'll talk about modifications and changes made to an implementation strategy and then an adaptation is kind of a

subset that falls within the broader category of modifications and these are really the the planned proactive um you know thoughtful changes that we make as

opposed to just any change or you know a reactive change or a change that's not systematic and careful and measured um so adaptations are really what we're striving for if you're going to make a change we want it to be measured based

on information and data that we need it and really sort of thoughtfully done to make sure that we're preserving the core elements of the intervention or the implementation strategy the core and

effective and essential elements or functions so in terms of setting the stage it's important to understand why we think this is important so when we think about

clinical trials typically the view on modifications is to really avoid it at all costs which makes a lot of sense if you're thinking about trying to understand the efficacy of something you really want to try to

narrow down what is actually causing the change and any modifications would threaten internal validity however in a more realistic view we know that

modifications are inevitable even in the most highly controlled trials um these changes are going to happen and so they're going to happen we might as well document it so that we can understand

the impact it's having on the overall process and changes that we're seeing so essentially without measurement there's really no study of it so we're hoping that the frame is is one way to

introduce the ability to really study this um with intention without study there's no improvement so if we think about actually achieving the adapton that's proposed by chambers and

colleagues in order to do this we really need to start documenting things like when where how and why and by whom these modifications are made so we can

understand when that would be necessary in the future as well so if we think about the current gap um both frameworks and other research

that's really oriented around documenting modifications to evps exist but what about implementation strategies that we've started to talk about here

how are they used to get ebps into practice um we know that even when we say we're using audit and feedback or continuous quality improvement there are

differences that we use um that may be still maintaining the core of these different strategies that are actually quite different and so there may be different actors involved different mechanisms involved and while the frame

itself can be helpful for evps it's not quite applicable when we start thinking about implementation strategies specifically so we really set out to develop this

version of the frame to track modifications of implementation strategies um which we specifically refer to as the frame is so i will talk then about not so much

the development of the frame is but just briefly introduce some of the modules um so the frame is in broad strokes is designed to be modular so that it can be kind of itself adapted or tailored to meet the needs of any given

implementation project we've got four core modules as well as three optional modules and so i'll talk about the first three before turning it back to cassidy to talk about um some thereafter

so module one as you can see is kind of a summary module and we thought this is important to document these kind of key pieces which ebp are we talking about

which implementation strategy is being used to get that ebp into practice and in broad strokes what if any modifications are being made to it for which we'll be completing the frame

is to document those modifications and then we want to have a brief space here as well to document the reasons for the modification now we acknowledge that many implementation strategies are not

particularly well characterized or maybe um you know viewed differently by different people we nonetheless suggest kind of doing the best you can up front to really specify

what's the implementation strategy we're talking about what would kind of an unmodified or unadapted version look like and then you can use the frame is to document whatever changes modifications or adaptations have been

made to that implementation strategy in the context of your project so module two then is really to dive into more detail into what specifically

is modified you can see here that there are four broad answer options and if you select the fourth one um it sends you to some some check boxes some sub boxes as well

we acknowledge that any given modification you're making to an implementation strategy especially if it really involves a few separate changes all kind of bundled together might mean you're selecting several

check boxes with one completion of the frame is so content modifications well are what they sound like they're modifications made to the implementation strategy itself

or the impact how it's delivered in contrast evaluation and training modifications are really about how the results of the implementation are measured or how the implementers are

trained and context modifications are really about changing the format setting personnel or that kind of thing classic example of a context modification in the context of copic 19 of course would be

shifting many implementation strategies from some kind of face-to-face setting to a virtual setting moving on to module three then which is

only completed for those first three types of modifications i mentioned on the last slide the content evaluation training modifications but if those are

the modifications being made um this is really kind of the meat or the heart of the frame is from that perspective right so the types of modifications that

could be documented for the frame is can range from relatively small things right the tailoring tweaking and refining two broader base changes right so for instance if entire portions of an

implementation strategy have been dropped entirely this would be a good opportunity to document that here or if really the implementation strategy was dropped completely

also be an opportunity to document that here so this is really about getting out what exactly is being modified for those content evaluation training modifications

and i think if you can go forward one little piece here there's an optional addendum here that can also be completed when you're using module three specifically if you have a sense

of what the core functions or core elements are of the implementation strategy itself then it might be important to document whether the modifications

are really leaving those core elements or functions untouched or preserved or if they're really kind of taking away or otherwise changing what what you see

as being the heart or core of that implementation strategy so again four total core modules for the frame is and i've covered one through

three and i'll turn it back over to cassidy to go on to module four thanks chris so as we go into module four you can see here that this module

really starts to focus on kind of the why so understanding what the goal of the modification is and you can see here there's a list of different types of modifications

understanding what the goal is is going to help us understand a number of different factors that may be at play when considering modifications in future work as well

so this can be something around making changes uh specifically related to the ebp that you're studying um it can also kind of range all the way down to

increasing things like health equity health equity and decreasing disparities related to that ebp delivery so really trying to get a more nuanced understanding of the goal here is the

first component of module four and then you can see that the second part is really trying to understand at what level um the modification is really being made

at is this being made at kind of a single level with one or two people or is there a larger group of people involved in making this decision um and

the modification in and of itself as we move into module five we start to get at when the modification is initiated

so um there's a couple of important things um to kind of think through in this module um is when we're asking about when the modification itself is actually being

initiated there's an emphasis on that initiation part intentionally we understand that regardless of the phase in which you're implementing a modification and these changes can

actually occur over a long period of time so really pinpointing um the start of it at what phase of the study it's at is quite important in the first part of module five

and then the second part helps get out whether this is something that's planned or more of a reactive modification which um shannon alluded to earlier this type of information is also going

to be helpful um when understanding the context of the modification at the end of the study or a study period to really understand the role that that might play in future

modifications and then module six really helps us get at um who participates in the decision so we've already talked in earlier modules about the level of the

modification this helps us really get into understanding who those decision makers were and this is going to be unique to the context in which you're using the frame

isn um but again i think is really important to help understand um at a later time point who those key players were and if those same players would be

important in future efforts where this type of modification may be important to understand the mechanism of change of this modification

on the larger question that you're asking so i'm going to take us over module 7 which is looking at how widespread the modification is

this is about who the modification is being made for so is it um really a modification to the strategy just for one

therapist for example you're doing consultation and you're trying to and that person missed an appointment so you're going to move from group consultation to an individual

consultation or is this something that you're changing for an entire clinic or organization so this is just looking at this the levels in which these modifications

occur and next i'm going to take us through an example of how the frame is was used for an ongoing study related to parent-child interaction therapy which

is an evidence-based parenting program for young children with disruptive behavior disorders and the this trial is looking at using a

task sharing model in which professional clinicians provide parent child interaction therapy i'm going to say pcat and lay health workers

partner with these clinicians to promote treatment adherence to pro improve skill acquisition for latino families who are spanish speaking and more likely to experience

health disparities so the implementation strategy was a training program that was developed in miami and then is now being

implemented in california and while this and well this training program has only been used once it had been defined with what activities were

happening on every day what the objectives for every day were how long it was and so we used the frame is to capture the changes that were made when the implementation occurred in

california so for module one we were able to go through and quickly identify the ebp being pcat and the implementation

strategy being the training program for the lay health workers to enhance parent engagement the modifications that were made were predominantly tailoring of training

content making the language more appropriate to the local context given population differences even though

miami and california both have large latino populations and spanish-speaking populations the country of origin of those populations is often different and so

modifications were made to reflect regional differences there we also did a removal of a component of training that occurred in miami and this

removal was based on challenges with this training and having like health workers learn this behavioral coding system that they were supposed to implement with families

and the reasons for that was we wanted to improve the appropriateness based on tailoring the training content to the predominantly mexican-american

population in california and the feasibility in that the behavior observation coding system had not been as feasible in miami

so when you look move to module two the what of what's modified is the content as and also the context because it's a new setting the

nature of those modifications the language is more tailoring and then removing elements was related to the behavioral coding system and as we don't know what the core

elements of this training program is that would be unknown we don't know if it's fidelity consistent or inconsistent and as i said before for module four the goals were acceptability appropriateness

and feasibility these modifications were planned in the pre-implementation phase they were proactive and they were conducted with an advisory group including program

leaders and health workers with the research team helping to finalize what the training program looked like in the end and this was network wide so everybody in the

california implementation effort received the same training program so i'll pass that anna to talk to us about our future directions

thank you i was i was trying to find to see how we can all of us fit here in the squares the hollywood squares so it feels to me that we're we're trying to move the field of adaptations forward

right we have the frame for evidence-based interventions now we've developed a frame i asked and there are several questions that i would love to ask you because you've done all this work in

terms of how to use frame who should be answering all those checklists all those modules how are you documenting in your work

do we need triangulation and having multiple people answering how often do we need to answer those questions so i just want to open the floor and have a little bit of your

experiences because i think they're varied and and that's where the science is yeah i can i can start um so i think a lot of times in terms of

who uses it etc triangulation can be important because different people have different vantage points you know a system leader might have an idea of some system-wide

adaptations that needed to be made but the people who are the implementers might have a very different perspective of what's actually going on on the ground for example um so i think that's um that's important

to note you also have to think about things like what's really feasible and what are your goals of learning about it if your goals are really to be able to pass along like

as you're really refining implementation strategies or trying to make them work in different contexts i think one thing that's really important is some implementation strategies are not well characterized you know we have a name

for them but people haven't necessarily said you know this implementation strategy consists of these different elements um but sometimes we have one that's laid out really well like this is our training program you know or here's

how we do you know implementation strategy x and it's when they're really well defined in some way that we really start thinking about adapting you know um

and and in doing that you know uh we might need to be able to say listen we use this implementation strategy over time we found we needed to change it this way um and and ideally you know and

it still worked or we discovered that when we lost this element um it no longer worked as well so we want to be able to share that information um if you're doing something like that kind

of taking a high level look at how you're um how you're adapting is probably enough if you're really studying it and it's the topic of research you probably want to do a deep dive and you know be

documenting pretty pretty regularly like every time you use the strategy for example um did anything change or documenting every time you notice a change um so i think some of it does depend on

the goal um you know about how precise you need to be and how comprehensive you need to be in trying to capture them chris cassidy you guys have also played a little bit with frame is and and your

methods share a little bit about your lessons learned and recommendations i think i would just uh predominantly just kind of emphasize the things that shannon mentioned i think that um you

know this is quite a comprehensive assessment tool and i think that if your needs are just kind of in broad strokes to document what changes are happening and when then it may very well be that

um relatively more spaced out uh completion of the frame is um perhaps by just one or two people may be sufficient to meet your needs for the project

so i think a lot of it is really about carefully thinking through what information do we need and why from the frame is and then really trying to strike that balance among feasibility

comprehensiveness and kind of responding burden for whoever is the one kind of taking a stab at it on your behalf i don't think i have much to add i think my experiences are

similar to what shannon and chris said i have found that we've been testing out collecting data through the frame is from

um a variety of people involved in the ebp within a clinic and asking them to engage with the framework at frame is on a monthly basis to try to understand a

little bit more does everyone have the same experience of the same modifications are there certain people that are involved in

modifications versus other people um and if we're doing it on a monthly basis is the burden light enough where um people are still continuing to engage in it

but we're having it at an interval that feels frequent enough to catch things that we otherwise might not be um and i think some of that is a little bit of trial and error to figure out in these

specific contexts what do you think will capture what you're trying to understand this is super helpful let me as you answer or as you make a comment i'm wondering if you guys could share

um the methods by which you're tracking so ah i i want to also put a highlight on borussica's work right basically

ravine that has done checklists um i've seen leo kavas is doing you know excel files um with the goal of there is not a right answer that's the only answer right

semi-structured interviews so as as you've shared if there are any lessons learned from the methods so we talked about the frequency we talked about multiple stakeholders

the how to figure out how to answer those um questions so we've been testing it um online so on like a system similar to

like a red cap or surveymonkey um so it can be like a an email link sent out to folks so they can kind of quickly from their email go in and click off the boxes have the right ones appear um so

it's uh directly sent to the participants rather than an interview or something like that we find that it's it's actually fairly

brief too um so obviously as as the study goes on there seems to be less modifications which would probably be expected in most implementation efforts um and that the time has

decreased significantly so it's like a minute or two to fill that out even when there is a modification in one of our ongoing projects um we are doing

kind of a retrospective completion based on a combination of time motion tracking for implementers as well as qualitative interviews for how the process has gone so ultimately it didn't make sense for

us to treat the frame is something that would be completed completely separate in the moment just based on the needs of our project instead we tried to kind of

integrate it and add it to the ongoing assessment in the form of that time motion tracking and essentially something similar to aaron finley's periodic reflections

and i would just add like it's i think it's cleanest when you fill it out once per uh modification or adaptation because if you have people write down like okay i did three or four different ones the

reasons all get mushed together you know maybe once happening at the system level and another is happening at a more like clinic or you know organization level so i mean my key recommendation would be

however you track it just try to be very clean about you know fill it out once per adaptation you know it's like six or seven questions um you know so module one you say the individual modification

or adaptation and then you go through and kind of check off the answers for that one and then you you do it again so that it's really clear you know your reason for each one

and you just have a much easier to interpret data set at the end [Music] and and going off of what shannon was just saying i think that a lot of timing has to do also with how

ongoing an implementation strategy is so the example that i talked about was training and the we used the frame i asked before

the training was implemented because it was planned adaptations and very systematic as to what we were talking about now as i said this is an ongoing

project and so we've moved from training to the consultation phase and there was additional adaptation and that has been

every week and that looks very different where you have more almost time for these types of adaptations to come up so for me doing

an example on training was pretty contained as to what happened and and perhaps it would have made sense to do it at the beginning of it and then the end of the week

but where you can see more modifications happen and more towards individuals like an la health worker went on maternity leave and so what are we going to do for

that individual um has happened in consultation so i would also say that you're probably matching your timing with the types of strategies that you're using

and that's something to consider too is that we know that most implementation efforts require more than one implementation strategy and so that's a question for

your team to ask itself as well is if you're just really investigating a specific implementation strategy or if you want to capture adaptations to every single implementation strategy that's used

within an effort this is super helpful uh which brings to our next slide in terms of future steps

and where where would we like to go team okay so i think you know we we know that um nothing goes as planned and that we

have to adapt to the context and the realities of the um you know the the circumstances that we're under so we know adaptations need to happen the more um

the more we document them the more information we can share about how they work under what circumstances they work um and um and also what can change without

diluting the effectiveness and perhaps even how to refine it so the more we know you know when we're training and implementation strategies when we're working with or sharing information with other groups that want to go out and use

them you know we can provide information like it's okay to do this when we did this it stopped working as well etc and of course you know you want to be able to track the modifications and the

outcome to be able to really you know um to to be able to get that um that information across and over time you know we really start to learn you know just like we're trying to learn um with interventions about how much you

can bend them without breaking them what kinds of things can make them better what kinds of things make them fit in different contexts so the frame is a first step towards that um and

ultimately we want these things we want these interventions and strategies and programs these evidence-based uh treatments that work um to get out there so that people who need them

will have access to them and you know that means sometimes we're gonna have to go into contexts that um the you know the implementation strategies and the interventions weren't originally designed for the more we know about how

to adapt when to adapt why to adapt um the better we'll do in terms of reaching our ultimate goal which is to make sure that the people who need and can benefit from these interventions get them in a

manner that will work for them and so that's really kind of the north star and we have to get a little bit um we have to get a little fine grade to figure that out and um so the frame and

the frame is are really intended to help us learn as we go so that everybody's not just improvising and so we can actually you know put some good measurement behind some of these questions and

there we keep up to date um a page on um on the fast lab website um that there if you go to the fastlab website that's listed here or even if you just google fast lab adaptations

you'll go to the page where we have information about the frame the frame is we have code books we have sample tracking systems and templates um these videos will be here

um among other places articles that use them so we try to keep it pretty to date you can get code books that you can of course adapt for your own circumstances and fill in your own decision rules around how to code different adaptations

etc so the website will have a lot of what you need but also feel free to reach out to us and we can share the you know the latest information we have or try to answer questions for you

excellent i want to say thank you and i hope that the video is useful email us connect with us and um

thank you go forth and document