Table 1 Representative participant quotes by engagement theme. NB: This table should be located in the Results section of this paper after: “Please see Table 1 for representative participant quotes for each section.”

Need for Patient Engagement

“In surveys, I also think that it's not uncommon for AI specifically, not to have a lot of patient engagement, because of the specific software and its use. So if it's diagnosing cancer and whatnot, patients wouldn't be using it, it would just be the doctor. So it's not surprising that they're not involved in that process.”

“I think it's really important to have to expand the [AI] space and expand the people that are participating in it. Because with AI, as in a lot of things, it's the question of garbage in, garbage out. So if you have such a small sample [size], the information you’re basing your policies on will not be consistent with reality.”

“Like, I personally, I'm not surprised, but it's very upsetting to see, you know, the majority of the participants were identified as white, because realistically, we cannot generalize that to the entire population. And it just shows that we need more race-based research focusing on black individuals, brown individuals, indigenous populations, and so on. Because at the end of the day, also, racial and ethnic minorities receive a lower quality of health care than white people. Like that's what all the studies say, right? And then that kind of ties back in with the patient engagement rollout, who are we actually reaching out to?”

“And I'm just speaking out of the experience, as a patient advisor with a variety of health care settings, both in education and in hospitals, is that most of the patients participating, the patients really participating are usually very homogeneous. They have post-secondary education, they either have or have had work and are retired, they have a home, they have a certain level of income, they speak English, usually as their first language. And they're mostly white, mostly women, too. So mostly from a very, very homogeneous set of care. And it's been a struggle, because even if the organizations want to do more outreach to patients, the way the whole patient participation is set up, is sometimes not conducive to bring, for example, people who work all day and are only available in the evening, or on the weekends. Right. So there's the setup for participation also has a lot of barriers.”

Patient Recruitment

“I think that it should be a part of the clinics [where] every patient that comes in and is asked ‘hey, we need your input on this AI tech’. Because you need a diverse sample and people who are willingly going to want to participate in studies and focus groups they're definitely going to be similar in some way.”

“And consider who is the population in Toronto, for example, there's a large majority of people whose language is not first language is not English. They we celebrate our diversity in the fact that we are in an immigrant city, that people learn English to work, they cannot read a paper. This is from my Latin American community, that's my experience, they don't know how to read or write. They don't want to rely on other people to actually take care of their health. So how are we making it fully accessible? Not only in terms of accessibility for persons with disability, which is the law, but also for those who are not health literate, who don't have the capacity to speak either official language?”

Timing of Engagement

“So to me, patient engagement in AI should invite the patient right at the beginning to even identify the problem: ‘what is the problem that we want to research?’ Because health care is a two sided coin, the priorities of the investigators and health care providers will be on how they can do the work best and more efficiently and more effectively for delivery. From the other side of the coin, from the patient perspective, I will be more interested in what are the barriers for me to actually access that health care, they give me the best medication prescription? Do I have the money for it? So I would like the question for research or the problem identified, to have input from people who are going to be the users.”

Patient Education/Training

“…having a round table with experts that are doing the actual AI, like a programmer, who does the algorithm…. Like having those high-end professions at the table as well as researchers. So if participants have a question, we get the answer right away.”

“It’s up to the provider to educate, you should educate them on those things that they're not aware of, and it can take time. But that's what quality entails, you need to take more time to give good results, basically.”

Methods of Engagement

“There's not a lot of like qualitative perceptions. It's mostly through surveys of satisfaction and acceptability and AI interventions. And I think that if you want to engage the patient more, you need to talk to them.”

“So I would think that a modular approach would be the best. And by modular I mean, invite a group for a three stage or three meetings, it could be one meeting in person, another meeting, online, etc. Just event, there's a variety of ways to participate. So you addressed a variety of communication styles and learning styles, you also have the opportunity to see familiar faces to feel that this is a safer space, I have not been harmed, I feel like impressed this group, in make it purposefully diverse.”

“You could have small groups in libraries…libraries are a really good way to reach people and have computers so people who can't afford to have a phone or computer can also access it that way.”

Process of Engagement

“Also, I think that having funding for to say to the person, you know, if it is in person, we're going to give you lunch or dinner, we're going to pay for your transportation, we're gonna recognize your one hour commitment with this much money. So then persons may say, ‘yeah, I can skip my gig, because I can go to this place and still not lose income.”

“So how are we making it fully accessible? Not only in terms of accessibility for persons with disability, which is the law, but also for those who are not health literate, who don't have the capacity to speak either official language? And so are we allowing ourselves a budget in timeframe to prepare the patient and inform the patient so then they can feel more appropriate? Because otherwise they just feel like oh, this is too educated for me this is too English for me and they self you know, self-eliminate before even participating”

“But I also want to bring the concept of upstream. So, the researchers need to educate themselves, they need to also advocate with their schools, their faculties, because right now, today, all the researchers are graduating, are still being taught, from the perspective that health research is ‘how can we fix the problem?’ And when I show up at my doctor's office, I am the problem. And they want to fix me, not the disease, they want to fix me. And it's like, no, I have a chronic illness. It's never going to be fixed. So how about we both have a chat, and we learn how to manage the disease in a way that benefits my day to day living? So, they are still looking at patient equals disease equals deficit, and we need to flip that conversation. Because patients are empowered, I'm in charge of my life, not my doctor. So then how can my doctor help me with that? Right, so we need to go upstream, educate the researchers, but the researchers also need to look at their schools or an inclusive type of education.”

“The second part to that, for that to be effective would be to recognize the patient's knowledge and experience as a skill. I said, my experience of disability is a skill. It took me 30 years to know everything I knew about my disease, I didn't choose this, but I have to have that knowledge. That's my skill. So when I participate in I bring that knowledge, I think it should be recognized because when I invite another professional that is going to bring their hard earned knowledge, it's usually recognized.”

“I was invited to create a workshop for PhD and master students who were actually doing some research about diseases and treatments, and they have never, ever talked to a patient. So they were looking at the patient as an object, of ‘how can I fix this disease’, but they didn't know how that disease impacts the life of the person. Will the person be able to adopt and embrace your research solution? So I think the culture of inclusion of seeing the patient as a true partner in their health care, it's a learning curve.”

“First, contact the general population, educate them, and then ask for the participation. So a few of those might want to participate when the research is done. So you go back to the patients who were engaged and say, ‘this is the publication, do you have anything else to add?’, if they have helped you write the paper, acknowledge them as co authors. Go back to the community and tell them listen, we did this research, this is how it's going to be published. And the next time around the community will embrace and say, yes, we want to participate, because we're seeing, they see us not just at the token, there's not an extractive process of our knowledge, but it's actually enriching it back, when they're finished, they bring back that knowledge.”

“For everyone, the researchers, but also the patients that you engage, to educate them in an anti-oppression framework. So then, with my participation, I'm not just saying Ladies and gentlemen and maybe harming a person who was not binary in their identity. And that's extremely important that as a patient, my participation with all the love that I bring is not harming others in the process.”

Evaluation of Engagement

“I think it's a very difficult thing to do, because there's not a final product. So the engagement process is a continuum. So I would say that having the goal in mind and say, for this particular research, treated as a quality improvement project.”

“I feel like it's almost impossible to know that up front, you'd have to like, incorporate the policies that were developed from the results, and then see if it improves health after a few years of actually implementing those policies.”

“So different, like multiple kinds of opportunities to say and share what's happening with me as a patient. I think that's important.”

“Other than an evaluation at the end of the project, to allow the patients for very honest and open feedback? Did they feel engaged? Did they feel that the knowledge of the subject in that particular project transformed? Like they have gained or learned something from it? And have they seen that the input, not just my input, but the input that the group was focusing more towards was actually considered?”

“I guess a short answer: good quality patient engagement is: a) include the patient right from the start and b) make it fully accessible including time/funding to properly do it”