Approved Assessments » Our Pilot Study

Our Pilot Study

Pilot Project for Comprehensive Mental Health Assessment - Cheyenne River

Sioux Tribe 2021-2022

Annette Bosworth, MD – Principal Investigator

Mental health screening is a requirement prior to sentencing for substance-related crimes in many jurisdictions.

Covid 19 compounded systemic delays in mental health and drug and alcohol screening in the judiciary and court systems.

The Pilot Project for Comprehensive Mental Health Assessment (Project) seeks to better understand and enhance the quality and efficiency of screening for comprehensive behavioral health and substance use disorders at the local jails in South Dakota by comparing an electronic assessment currently used in the civilian sector, outside of jail, to the standard in-person assessment. Evaluating this tool could strengthen the confidence of those using the tool outside the incarcerated setting and offer a valuable resource to incarcerated populations. After a substance-related arrest, care delivery can stretch from 4 weeks to 13 months. Fewer resources within a community often link to the longest wait times. This tool may offer South Dakota communities opportunities for an innovative tool to lift the standard of care in rural areas to those with denser populations. If the Approved Assessment electronic tool performs as well as or better than an in-person assessment, rural communities throughout the region will benefit.

Project research question:

Is the Approved Assessment screener comparable to the current method of in-person screening for mental health issues and substance-related problems?

This pilot project will evaluate the electronic comprehensive behavioral health screening tool through two objecti

Objective 1: Timely, thorough, complete Mental Health Assessments for incarcerated individuals while navigating restricted access due to COVID-19.

Objective 2: Validate the usefulness of Approved Assessments with incarcerated individuals.

Methodology:
A methodology for the Project began with a mindful approach to this vulnerable population. The methodology sought to enhance the quality and timeliness of behavioral health screening for incarcerated individuals. This included both mental health and substance use-related screenings. Approved Assessment’s online, comprehensive behavioral health assessment was the tool selected for the Project.

Disclosures:
Annette Bosworth, MD disclosed her Financial Conflict of Interest (FCOI.) Dr. Bosworth complied with the University of South Dakota IRB FCOI rules. She was involved in the committees and evaluation of the study results but was not the sole interpreter. Several contractors were enlisted through the professional relationships of MBRI to ensure expertise and collaboration for the successful completion of the Project.

Methodology:
Fifty jailed participants completed the initial assessment using the tool. They were then randomly assigned to:
1. Reliability -Retook online assessments using the tool. (10 participants)
2. Validation –Mental health provider assessed the client in person. (30 participants)
3. Cognitive Interview –Met with the Research coordinator about their experience with the online assessment. (10 participants)

Two assessments were done. Counselor 1 used Approved Assessments. Counselor 2 performed the standard in-person assessment.

Counselor 1: No interaction with participants. No follow-up phone calls for clarification. No information from electronic medical records. Only the information from the tool.

Counselor 2: Psychologist working in the Wakpá Wašté Mental Wellness & Addiction Counseling program. Lives in the community. Has access to electronic medical records and conducts the live interview.

Counselor 3: Compared the in-person assessment to Approved Assessments

The following scales were used for numerical comparison:

The Substance Use Levels of Care (LOC) Scale:
A. Non-LOC 0: Concern/Advice
B. LOC 0.5: Early Intervention
C. LOC 1: Outpatient Services
D. LOC 2: Intensive Outpatient (IOP) Treatment / Partial Hospitalization Services
E. LOC 3: Residential / Inpatient Services
F. LOC 4: Medically Managed Intensive Inpatient Services

The Mental Health Levels of Care (LOC) scale:
Non-LOC 0: Concern / Advice for Mental Health
LOC 0.5: Early Mental Health Intervention
LOC 1: Local Mental Health Outpatient Services
LOC 2: Local Advanced Mental Health Services
LOC 3: Inpatient Mental Health Services
LOC 4: Medically Managed Inpatient Mental Health Services

57 participants recruited

50 completed projects

5 unavailable to complete due to early release

2 voluntarily terminate participation

Results:

Comparison of In-Person vs. Approved Assessment

Validation: The mental health kappa value was as anticipated, with 27 out of 30 participants showing agreement between the two methods for their mental health recommendations. Three participants had different recommendations, with slight differences in the level of care. The simple kappa for mental health treatment recommendations was 0.72, indicating substantial agreement between Approved Assessment and in-person evaluations. For substance use treatment recommendations, the agreement was not as strong, with 18 out of 30 recommendations showing complete agreement. The simple kappa for substance use was 0.22, indicating fair agreement between the two screening methods.

Reliability: The reliability kappa statistic was high for mental health at 0.783 and almost perfect for substance use at 0.848, suggesting substantial agreement and high reliability.

Qualitative: Ten participants provided feedback on their experience with the online screening instruments. Overall, they found Approved Assessment easy to complete and of the right length. They appreciated the use of electronic tablets, particularly the touch screen. Some mentioned the need to reread questions for clarification, but overall, they found the reading easy. Most participants used the audio version, which helped them focus and understand the questions better. Participants felt that using Approved Assessment made it easier to answer questions honestly and expressed a sense of security in having their voice heard.

Challenge	Description	Impact to Study
Scheduling/ Access to Care	Designated times for the in-person assessments often conflicted with prior court dates and the counselor’s schedule.	The difficulty in scheduling timely in-person assessments threatened the retention of participants. The study mandated that only 7 days could separate the AA and In-person methods. We limited the intake of participants based on Counselor #2’s schedule. Originally the times blocked for these appointments conflicted with the weekly court schedule for participants to present in front of the Judge.
Gender Ratio	The goal was to recruit equal numbers of males and females; however, the overall sample inmate population was not equal to 50:50.	The study sample represented the inmate population of the facility which reflects a male-to-female ratio of 60:40.
Temporary Releases	Temporary jail releases were granted to many participants by the Judge for various unforeseeable circumstances.	Unexpected Temporary Releases resulted in 5 of the 7 participants being dropped from the study. Their early jail release removed our access to the participants, losing them to follow-up.

Challenge	Description	Impact to Study
Retention	The study lost seven participants.	5 participants were unexpectedly released from jail, and 2 voluntarily opted out of the study. The 2 voluntary drop-outs occurred in the reliability participants. Both stated they did not want to repeat the AA method due to its length.
COVID Regulations	Surges in COVID-19 cases closed the Social Detox facility and used the space to quarantine inmates.	Reassignment of the designated space, stopped the interviews for potential participants, and removed access to the Kiosks where the AA method was administered.
Limited Literacy & Technology Skills	Several participants lacked literacy and technology skills and required assistance with interpretation.	Progress was slowed due to the required assistance with setting up emails, teaching technology skills to use the tablet, and ensuring the participants with low literacy used the auditory assistance where the text was read to them.
Tribal Organization	Administrative Leave was granted to Tribal employees by the Tribal Government.	Administrative leave due to Tribal Holidays, Inclement Weather, and high COVID census on the reservation resulted in the mandatory Administrative leave of tribal employees.
Language	The study anticipated low-literacy levels with the assessment tool and the informed consent.	Limited literacy was addressed with the automated text-to-speech functionality within the AA tool. Some participants struggled with the terminology and required assistance translating this information into understandable language. This could have resulted in misinterpretation of parts of the study.

Conclusion:

The project findings indicate that the Approved Assessment online tool is comparable to in-person screening for mental health and substance use. Despite COVID-19 limitations, Approved Assessments provided a comparable and cost-effective method for rural Native American participants to receive the desired care without imposing financial strain on clients and judicial systems. The statistical analysis, despite the small sample size, supports the tool’s similarity to in-person screening (kappa of 0.72 at a 95% confidence interval, suggesting substantial agreement). The lower kappa in the substance use arm is attributed to the complexity of formulating recommendations across ASAM dimensions and treatment levels.

The Approved Assessment method captured more comprehensive participant details compared to in-person screenings, benefitting from the advantages of online information collection, particularly in a jail setting. The study employed a friendly Research Coordinator who used an iPad to administer the online assessment, allowing ample time for participants to answer sensitive questions privately with options to pause and resume as needed. The systematic electronic algorithm ensured that no questions were skipped and offered reflection time for participants. The online format fostered anonymity, reducing pressure and potentially promoting candid answers and greater willingness to share.

In contrast, most in-person interviews required participants to be transported from jail to a local clinic, exposing them to familiar community members, which likely influenced their responses. The in-person approach involved multiple paper screenings in the waiting room and a provider interview in the exam room. An in-person counselor also traveled to the jail for five client assessments. In both scenarios, participants answered questions swiftly to accommodate the clinic’s busy schedule.

An easily accessible assessment tool for incarcerated individuals on the Cheyenne River Sioux Reservation offers numerous benefits. Approved Assessments delivered timely and thorough screenings for behavioral health and substance use comparable to traditional approaches. Furthermore, participants reported a sense of ownership and empowerment after completing the Approved Assessment, which may enhance treatment success. The immediate and ongoing access to written recommendations further contributes to positive outcomes.

Considering the significant underfunding and understaffing of substance use and mental health services, delayed care creates economic hardships for patients and their families, which has a ripple effect on the community. Tribal leaders have recognized the need for culturally appropriate resources to address these challenges. An efficient and valuable online assessment tool can increase participants’ access to quality care by reducing wait times while keeping them connected to their community, families, and local culture.