Amazing Chatbots For Workflow Learning with David T. Livingston, Ph.D. [134]

This is the eLearn podcast. If you’re passionate about the future of learning, you’re in the right place. The expert guests on this show provide insights into the latest strategies, practices,

and technologies for creating killer online learning outcomes. My name’s Ladek, and I’m your host from Open LMS. The eLearn podcast is sponsored by eLearn Magazine,

your go -to resource for all things online learning. Click -by -click how -to articles, the latest in EdTech, spotlight on successful outcomes and trends in the marketplace. Subscribe today and never miss a post at eLearnMagazine .com and Open LMS,

a company leveraging open -source software to deliver effective, customized, and engaging learning experiences for schools, universities, companies, and governments around the world since 2005.

Learn more at Open LMS .net. Hello everyone, my name’s Ladek, and my guest for this episode is David T. Livingston, who is the Executive Director of Health Plan Workforce Development at Kaiser Permanente.

In this automated conversation, David and I talk about why are chatbots a good tool for workforce development, especially when it comes to reinforcing learning.

And then I ask David, what were the five ways he was seeking to improve this sales performance through the use of chatbots? Next, David and I talk about why Kaiser chose the chatbot platform at all,

and then how challenging the implementation was. And then we, you know, hint, we talk about why is human intervention still super important even if you’re using chatbots? We then talk about what was David’s strategy to create success with the chatbot program,

and, you know, the particular challenges that he faced in terms of interactions, in terms of feedback, and how they improved the process overall. I then ask David, you know,

to talk about how does he ensure that his entire day doesn’t just become texting with others because now that he’s using chatbots for this workforce development. And then finally, as we always do, we ask David to talk about how the chatbot program will continue to expand,

or not, at Kaiser in the future and what they’re looking for from this platform to continue to improve their business. And remember, we record this podcast live so that we can interact with you,

our listeners, in real time. If you’d like to join the fund every week on LinkedIn or on Facebook or on YouTube, just come over to elearnmagazine .com and subscribe. Now,

I give you David. Welcome to everyone. Again, my name is Ladik. I’m with a company called Open LMS, as you can see. And I’m very, very excited to have my guest here today, David Livingston of Kaiser Permanente.

David, how are you today? Fine, Mike. How are you? I am wonderful. Tell us, you know, I know that you probably don’t have a grand piano in your office or living room with you. What part of the universe do you actually sit in?

I actually live in Palm Springs, California. Oh, okay. Fantastic. Right. Well, I hope that as we’re talking now that the weather is good for you and not so, not so horrid. It’s getting better.

We’ve had a really rough winter for us to be in the 50 degree weather for months is odd, but we’re approaching 70 right now. So it’s pretty good. Fantastic.

David, you are the Executive Director of Health Plan Workforce Development at Kaiser. I’d like to give everybody the opportunity rather than, you know, reading a bio to sort of tell us about yourself.

Where’d you come from? What do you do right now? Like, you know, how would you like to position yourself? That’s great. So my background, I’m originally from the East Coast. I moved out to California about 10 years ago to join Kaiser.

My background is in marketing, sales, sales management, and learning and development. Prior to coming to Kaiser, I had many jobs at the Xerox Corporation.

And my last 10 years at Xerox was leading learning and development for sales for Xerox. And so in 2012, I graduated from Capella University with a PhD in education in the specific area of training and performance improvement,

and was recruited by Kaiser to come out and lead their learning and development organization for sales in California. – Okay, fantastic. Wow, this is so cool, Xerox to Kaiser.

We’re here specifically today to talk about chatbots, which obviously is a little more prescient today than when you and I scheduled this conversation a little while ago with the advent of chatbt and GPT and whatnot.

But set the stage for us. Why were chatbots a thing that, as you took over your role there at Kaiser that you wanted to consider, and specifically,

we had talked about in workflow, sorry, workforce development, so set the stage for us. – So what we were looking at was how do we improve the,

gather or create a reinforcement strategy? We all know that when we’re in a learning space, we’re teaching, we’re having experiences around a topic and really trying to get people to grasp and retain critical information.

And typically in this space, people have knowledge gaps. They have skill gaps and innovation gaps, environmental gaps and communication gaps. And we were trying to figure out in all those areas,

how do we actually build a strategy? Is it gonna close those gaps? And so we were looking for a technology and a tool and a consistent methodology would help us to reduce the gaps that we identified around reinforcing knowledge,

skills and behavior. – Hmm, and can you like give me a, I’m trying to sense the scale of the problem. Like how many people are you trying to engage with?

Like are we talking thousands, hundreds of thousands? – Is it across all aspects of the company from, you know, from pharmacies to doctors to what not?

Or like, what’s your wheelhouse? – Just a really good to narrow it down because Pfizer is over 225 ,000 employees. So I do not have responsibility for the 225 ,000.

– Come on, David, you can do it, you got a PhD. – I have a smaller chunk of the population, it’s about 2 ,000. And so I specifically target the sales and account management underwriting populations,

which sell and market our services to our clients, brokers, and customers. So that population is much smaller. So what we’re gonna talk about today is a reinforcement strategy to support a specific program that we deploy,

supporting our managers around coaching. So the overall idea was to develop a reinforcement strategy and how to do that was to actually use a particular software tool to help us close that gap around reinforcement.

You know, and everyone on this call will know that the biggest issue we have around any knowledge acquisition is that 87%, and this is research done by the Gartner Group, 87 % which learned,

you actually lose it within the first 30 days. So you only retain 13 % of what you actually learn if it’s not reinforced. So that’s a huge loss gap.

Think about you investing in something, and after 30 days, someone will retain 13 % of it. – Believe me, I’ve experienced this recently myself, where I had a colleague show me a process here at OpenLMOS,

I literally wrote to him the next day, I said, “Look, I am sorry, but none of that’s stuck. None of it at all.” I’m like, “Please make me a lumely so that I can figure it out.” – Well, it’s even worse when you go to a presentation,

that’s a whole different conversation, ’cause we actually have done some research around retention of presentations, ’cause it sells people. The most important thing is for them to get their message across,

and so that people can take action in their presentation. The problem is typically when you give a presentation, people are not focused on what’s really critical. And so when you leave that presentation,

if I’m not specific on what I tell you, what I want you to remember and retain, you’re gonna remember random things, and have nothing to do with it. – Yeah, you’re gonna remember the temperature of the room or what you had for lunch,

or hey, how come Cheryl didn’t show up? – You might remember a picture on the screen that you saw that was really interesting, but that’s not what I want you to remember. – Interesting. – I need to physically focus on the critical things and repeat them several times to that presentation.

So at the end, and then I need to ask you at the end, you know, when you ever seen at the end of a presentation, you see someone says thank you Guess what you’re gonna remember when you left out the presentation. Thank you. Absolutely hundred percent That’s not the message you want to leave with so so this is all going to the point of reinforcements Frequently important and how we how can we how can we do that in

a way? That’s going to help people retain to be able to perform better on the job You know, I this is not my converse it like it’s not you’re not interviewing me But you just struck a huge memory in me and that when I was in fourth grade that the band the band teachers said he said look There’s two things that people remember at your conference Concert the first note and the last note Those are the two things

that they remember everything else in between is it’s beautiful and you’re right But those are the two things that they remember. So there you go. Interesting What your teacher what he gave you was really if you think about the neuroscience around it It’s really what he was ahead of his curve back then.

Yeah, absolutely So so talk to me about so reinforcing so it sounds to me like you’re trying to train your Salesforce about how to Reinforce their message to the people they’re they’re communicating to is it’s almost like it’s almost like That your strategy is then something that they then need to implement as well.

Do I have that correct? Yeah, so the strategy was really around five areas number one We wanted to be able to reinforce the content We were actually delivering to them because we enforce the content means they’re going to have access that information when they need it When they’re out in front of the sales Their sales audience we also wanted to create a culture of continuous learning So that people realize that you

don’t just come to an event and that’s it There’s a constant certain cycle of learning Then we also wanted to ensure that in our strategy we designed something that was going to be engaging and personalized Because the more personalized something is the more it’s towards me versus general great.

I’m gonna have more allegiance to that We want to also automate the process so that something we couldn’t do– we’re all working with less people and less resources,

so we want to have something automated. And lastly, we want to be able to collect data about this process. And so when we built the strategy, that’s what we’re looking for, and that’s what we were trying to drive towards in our process.

Hi there. I’m sorry to break into the show right now. But if you’re enjoying this show, if you are challenged, if you’re inspired, if you’re learning something, if you think that you’re going to be able to get something out of this to put into your practice, do me a quick favor.

Pause right now and just hit Subscribe on your podcast player right now. It doesn’t matter which one. Just hit Subscribe, because that way it’ll make sure that you never miss an episode in the future. Thanks. Now, back to the show.

Once you’ve created the strategy, what’s the next step? Like, did you immediately go out and find a tool? Was this something that you built bespoke in -house? Did you call a meeting of 1 ,000 sales professionals and be like,

hey, guess what? Things are changing at Kaiser. Like, what did you do? It’s so funny, because I attend many conferences throughout the year, and I’m always looking for, how can I improve the workflow,

the work process, the stickiness of whatever we do in learning. And I happen to attend a conference. My organization is a member of the International Society for Performance Improvement.

They host a yearly conference, ISPI. And at this particular conference, I went through a series of workshops. And one of the workshops was on the use of chatbots and learning.

And if– Light bulb went off or what? Yes, a light bulb went off. And literally at this meeting, they were talking about using chatbots as a reinforcement strategy.

I’m like, problem, answer. [LAUGHTER] Now, so I’m going to call you out here a little bit. Like, so you got a PhD.

It’s fairly recent. You’ve got a decent amount of experience. Does the term chatbot in terms of technology and implementation, does that cause you anxiety? Or you’re just like,

look, I got a team. They can take care of this. Or you’re like, look, I can throw this together. No problem. – No, so the one thing that, we all have experienced chatbots, and so the one thing that we all have experienced with, is negative reaction to chatbots,

because most of us in our lives have interacted with chatbots. Oftentimes, when you’re a consumer, and you’re engaged with a company, and you’re online, oftentimes you’re not talking to a live person,

you’re talking to an automated chatbot. And so that engagement, for some people, has been positive, for many it’s been negative. So, that was a concern that I had when I first heard about the technology.

– Can I ask you to do a little bit of research, which maybe this is your next paper that you write in, that how many people experience chatbots, either for the first or second time, or their experience with a company, is in one of those times when they need a critical problem solved?

And then that ends up becoming a super negative experience, right? ‘Cause you go to a company to ask for help, when it’s like, hey, I need to change my credit card, or I got billed wrong, or something’s not working, and the first thing that you run into is that?

Oh my Lord, somebody needs to publish that paper immediately. – Yeah, and so what you just talked about was one of the things that we had to make sure of, that when people were engaged with our chatbot, they didn’t have that reaction.

So what that means is the company’s not really maintaining and monitoring the chats, they’re coming back and forth. So in that situation, in our process, when someone sends, let’s say you ask a question that has nothing to do with the chat conversation,

we flag it, it gets sent to us automatically, and we know we can respond back to that individual. It might not be as quickly as we’d like to, but we will respond back. So the person’s question or inquiry doesn’t get lost in the process,

but most of the time, if you think about, if you’ve been engaged with an online chatbot, when you ask that random off -kill question, the system doesn’t know how to react, and it will start giving you miscellaneous information as nothing to do with what you’re talking about,

and then that’s what frustrates you as a user. So the process we have will enable that person to still send that question through, they will still assume, in fact, when I tell you about how people interact with our chatbots,

they often think it’s me, ’cause I’m actually the voice of the chatbot. – Oh, I understand. – And so you’ll see, as I talk further, we’ll talk about how people respond to the chatbots. – So just to make sure that we’re all on the same page going forward here.

Who are the people who are interacting? Is this an internal system? So your sales team that you’re working with, this is an internal system that they’re chatting with, yeah? – Yeah, so it’s all internal to our employees at Kaiser.

And so the example I’m gonna share with you today is really specifically around managers who are involved in a specific program that we deliver. But I also use these chatbots, and I’ll just give you a little bit of a preview.

We also use a communication tool in my department. And so I’ll talk about that as well. – Okay, so take us there. So what is the, what’s it look like? What’s the technology? I mean,

if you can reveal that, if it’s your spoke or if it’s an actual tool and– – Yeah, so it’s a tool and I don’t wanna reinforce that particular company.

There are many chatbots available to everyone out there. But we use one that is fairly simple to use, fairly simple as a user as well as us programming.

My staff went to a like a day class online with the vendor to come up to speed. And we’ve had very little intervention from the vendor since that on start of this process.

So it’s fairly simple to program. The key that we found in the process is the logical conversation around when, it’s almost like a learning tree.

So when you send out a chatbot, you have to think through logically, what potential responses would there be based on the question you’re asking? So that’s the most intricacy of that process is to think through the logics behind the questions you ask.

Once you get that down, it’s fairly simple. I’ll ask you today what color is the sky and most likely the response should be blue, but it could be cloudy, it could be sunny,

right? So you just have to think through that process, but we have gotten down to a science of how to do that when we prompt questions, and normally they’re around a specific topic, and so they’re not yes or no,

but very seldom do we leave open -ended. We sometimes do open -ended questions, but typically they’re focused on a specific topic. We’re trying to gather, we’re trying to reinforce something and something they’ve learned,

so it’s a little bit more specific. So when you started with this tool, did you say, okay, look, here’s one outcome that we want to achieve with our sales force,

and then let’s build out the branch and build out those scenarios and whatnot, and then you moved on, or was it more of a strategy like here’s five outcomes and you did it all at once, or like what was your method of madness to create some success there?

– So the method of madness was, I mentioned that the strategy had those five elements, which were very important. So the tool itself was, well, first off, we wanted to test it. We wanted to see what kind of engagement.

So we did a pilot with a small group of individuals in this particular program. So the program we selected is our coaching program. We use the GROW model. And the GROW model allows us to teach a process to our leaders around how to conduct a coaching session.

It’s goals, reality, options, what’s next. So they use that step, that process to get through the, but in teaching them coaching skills and a coaching methodology,

we wanted to reinforce once they leave the program. Remember I talked earlier about that 87 % loss? – Oh yeah, 100%. – Yeah, we wanted to reinforce. So the chatbot would allow us to send out reinforcement tips,

videos, PowerPoint slides, anything we thought that could trigger them in response to reinforce some of the knowledge, skills, and behaviors that we taught in the program.

So that was the reason why the chatbots were there to help follow up with additional reinforcement resources and problem for knowledge checks post the learning intervention.

– And so it’s interesting ’cause most people think chatbot, hey, I have an issue, I’m gonna go to the, wherever that chatbot lives and I’ll start to, I will investigate and then hopefully arrive at the outcome of solving my issue or getting the knowledge I need.

But it sounds to me like you have a two -way street where people can do that, but then also you’re doing the retrieval practice and whatnot that you’re prompting people as well, is that correct? – Absolutely. So the chatbot,

as you think about it, when we talk about it in the consumer realm, it’s kind of like you’re inquiring a question and you’re getting an answer. In this particular situation, we’re actually pushing information out as a resource and then oftentimes prompting them with a question around that topic or area of interest so that we can get a response from them.

So it’s a good way because number one, it’s a soft way to introduce a question to someone. Everyone has, the population we’re working with all have Apple devices,

all Apple phones. So they iPhone, so they all have access to using that network. And so it’s easy, we all have ability to chat, we all chat at work through various means.

So it’s not a foreign process. So to receive a chat from the learning and development group and what we did to make the chatbot more personal,

’cause remember I talked about personalization, we created an avatar. So the chatbot, so we talked about the program was called Rowe, Rowe Coaching. We created an avatar called Coach Grover.

So it’s a– – Please tell me he’s blue, please tell me he’s blue. – He’s not blue. He’s a bit emoji that is attached to every chatbot we send out around this program.

And so when you think about the program, you think about Coach Grover, he shows up as this person, is this, you know, avatar and people can relate to the avatar.

If I say you a blank text, could I still get across the message, but it wouldn’t feel personalized. So the Coach Grover emoji changes. He’s, every time he’s being delivered out to you,

he’s in a different, you know, physical shape, color, right? Emotion. And so that personalizes the message, not only the static message I’m sending, but also the emoji makes it different.

A hundred percent. And so just to get a little more clarity on the reception of messages is, you know, again, we don’t have to name label or anything like that, but is this being delivered on mobile devices?

Is this only like when I’m sitting at my desk, you know, in front of my laptop, like, you know, I can get a text anytime. And, you know, whatever your corporate chat system is, like you can show up there. Yes. So the beauty of the program is that it works.

And I don’t, I can’t articulate all the different avenues we have at Kaiser use teams. So we can communicate via team. We can communicate via Slack, Yammer.

We can also communicate via our Apple phones through text messaging. So those are the vehicles that we use at Kaiser. So you have this bullet,

you know, sorry, light bulb moment at a conference. You come back, you engage a company, you get a test case up and running. Yes. What were the, what were the hiccups? What were the challenges?

What were the things, you know, how did you fall on your face to kind of need to reiterate or retool or figure out? Yeah. So the most important thing was the amount of messaging being sent out.

So we initially started with, it was a four week test. And so we went with approximately in that five day work week,

we would send out maybe five to seven text messages a week. And guess what we found? It was a little too much. People didn’t respond. We were like,

I grow over, get out of here, get out of here. So we actually took that from five to seven down to three. Three was sufficient enough to engage with people over the course of that four weeks,

and to remind them and give them resources. So that was one of the things we did find out right away. We and we actually followed up with the test audience to understand, you know, what did you like?

What do you like about the content? What do you like about the number of times we deliver the content to you? And so they were able to give us feedback about that. And so that’s where we landed. They love the use of the tool. They love the use of the chat.

They loved Coach Grover. They were engaging with Coach Grover. What they didn’t like was the number of times we were sending out the chat messages. Is there any way that you are able to like,

did you ever get that comment where it’s like, wow, Grover showed up, you know, right before I had a one on one. And you know, I had this coaching moment. So I knew what it’d do. Like, is there a way that you could be predictive about that?

Or know when these types of like when coaching moments are about to happen in this particular scenario or no, there’s no way for us to predict that situation. What we try to do is provide them with access to resources.

So if they’re getting in a coaching situation, I need to gain either access to a video of just remembering in the particular, how did they use the growth process, they can have access to that.

So we do provide through the coach Grover the chat bot, access to those resources, but I couldn’t be in the moment. Because I would know how to respond to them. Unless I guess if I guess we could set it up if they wanted to send a note to coach Grover in the moment,

but that would take a lot of planning. Yeah, sure, of course. Yes. Um, or I mean, I’m just, of course, I’m just thinking that loud word, they prompt you saying,

Hey, look, I’m about to do you know, I’m about to have one of these moments, you know, can you give me a bullet point ABC or something? I don’t know. Yeah, so we talked earlier about the autumn autumn autumn,

autumn, again, the word, automation of this process. So what we do in essence is we set these up well in advance and then they extend out, they’re generated automatically.

So there’s not a person that’s involved. Once we set it up, it’s all done through automation. So what you’re asking for is human intervention and that would take resources and sometimes we don’t have the resources to do that.

So if we had resources that could actually man that, that could be possible. – Interesting, I believe those resources are called chat GPT, just so you know. Anyway, it’s coming for you.

So as you went through your four week test and then you got back some feedback and it seemed like you got kind of a green light, how do you scale this?

Is this just like, wow, okay, a few tweaks, fewer messages and then boom, it’s rather than five managers or go into a hundred managers, what’s up with that? – Yeah, so really nice question. So what we did is actually we’ve scaled Coach Grover.

So now there are programs that have a similar reinforcement strategy with different avatars associated with those programs, but we’ve been able to use our foundational knowledge from the Coach Grover chat bot to expand it to other areas.

And in fact, one of the areas we expanded during COVID, it was really important for me to drive better collaboration among my staff.

And because we used to be in a building working together, now we’re all virtual, we had to figure out, well, how do we connect in a better way? So how do we make sure that people are getting the right resources around self -care?

So I created a David Livingston chat bot that is a weekly chat bot out to my staff. We did it weekly throughout COVID, we’re now only twice a month, but what it was was a way for me to connect with my staff in a meaningful way to check in on them,

to make sure they had access to the right resources around self -care and motivation, all the things we need over the last two years. And that chatbot is still being used today because I still need to communicate and check in with them.

We just sent one out the other day. It had to do with the fact that we’re almost to spring break and people are going to start taking vacation. And I’m trying to prompt people to take time off to spend time with their family and friends to take a break because the first three months of the year are normally very chaotic and hectic.

So you said a word in there. You created this chatbot so that you can be more present, right? They basically are scaling yourself. Can you be a little more specific?

So you’re prompting them either to do something or are you actually getting these responses? For instance, a couple of weeks ago,

we wanted to know what people were doing for self -care. What was their number one self -care process? What were they doing? What did they read? Did they take walks? Because we were just trying to assess the organization.

Believe it or not, which is just hard to believe, we’ve been doing this for almost three years. When that chatbot comes out from me, they think it’s actually me and they respond to me as though they’re speaking to me,

David Livingston. I’m thinking to a tool. I’m not the tool. Now, I do see the output of the tool, but people respond to me and sometimes I personalize. I’ll send them a picture of what I did over the weekend and they’ll respond back.

Here’s David. Here’s what I did over the weekend. But again, it’s not me personally. It’s an automated tool that’s responding to them in a fashion, but it does create collaboration and connection. Oh,

yeah, 100%. So as someone who may be in a similar position as you or who’s thinking about using a tool like this, I could see in the back of their mind that like, wow, this could get kind of overwhelming, especially as you’re talking about 1 ,000 staff,

people maybe have more. How do you manage this process so that it doesn’t eat up your entire day every day and you’re not just sort of caught in chat rooms and those kinds of things? Yeah.

So depending on the program, there’s a team of folks within my department who actually manage those programs and their chatbots. And remember, they are planned well in advance. We know when the class is starting, we know when it ends,

we know. So that’s a little bit easier to do. The ones that I actually deploy, we meet on like every couple of weeks to kind of plan out ’cause these have to be timely.

We can’t do them like the class. We can actually set them up well in advance, months in advance because it’s the same class, the same material, but with my topic, it’s gotta be relevant because I can’t,

can you imagine getting some message from me that has nothing to do with this month? I’m gonna delete that chat, right? I’m not gonna engage with it.

So might have to be much more timely. So there is a little bit of planning and execution that goes on with my staff as we work on this, but it’s maybe, we might spend 30 or 40 minutes twice a month so it’s not a huge investment.

Now my staff has to go out and do some research around the topic area, pulling those things together, but for me personally, it’s maybe an hour a month. – Okay, fantastic. – Yes. – So as you have,

you know, we’ve realized success around this, it sounds like that you’re getting good feedback and also achieving the outcomes you wanna achieve. What’s the, what’s the future hold? What’s the, you know, the six to 12 months?

Is there more chatbots coming or is there, are you doubling down on this or what? – Well, the thing that you mentioned earlier and I didn’t share with you the results. We talked a little bit about how the pilot felt about the program,

but you know, we also do something. We use a net promoter, and so net promoter actually speaks to, you know, what someone referred this tool to someone else. And so anything over 50 % is amazing.

So for coach Grover, the numbers were 70%. So outstanding. It means that people felt it was a really good use of their time and engagement. It also received a 4 .9 out of five in terms of star rating.

So really good response from the population engage about it. So what that prompted us to do is number one, we’re continue looking for avenues to reinforce. So we have many tools we use for reinforcement.

Chatbots are one of them. So as I mentioned earlier, we started with the one for Coach Grover, but now there are five additional. So we’re always seeking for opportunities to expand the use of the chatbots within the learning space.

Fantastic. David Livingston, you are, oh boy, I almost completely screwed up your trailer. You’re the Executive Director of Health Plan Workforce Development at Kaiser Foundation Health Plan,

which is Kaiser Permanente to most people. I can’t thank you enough for taking time out of what I’m sure is an incredibly busy day to speak with me today about chatbots and how you’ve done it. If somebody wanted to reach out,

ask you a question or pick your brain about how you did this and how they could do it, how should they reach out to you? So they can email me. My email is david .t .livingston @kpkezerpermanente .org.

Fantastic. Thank you again, David. I really appreciate your time and I wish you a wonderful day. You too. Thanks again. Thank you again for listening to the E learn podcast here from Open LMS.

I just wanted to ask one more time, if you enjoyed this show, if you learned something, if you were inspired, if you were challenged, if you feel like, you know, this is something you can take into your practice, please do me a favor and right now on your podcast player,

hit subscribe. That way, you’re never going to miss a future episode. Also, come over to elearnmagazine .com and subscribe there as well because we have tons of great information about how to create killer online learning outcomes.

Thanks.