WEBVTT 00:01:59.000 --> 00:02:01.000 >>: Hello everybody. 00:02:01.000 --> 00:02:09.000 And welcome. 00:02:09.000 --> 00:02:11.000 I'm Aerius Franklin, a board member and facilitator of the workshop today. 00:02:11.000 --> 00:02:14.000 Thank you for joining us. 00:02:14.000 --> 00:02:16.000 I want to mention a few housekeeping 00:02:16.000 --> 00:02:22.000 items before we get started to they. 00:02:22.000 --> 00:02:28.000 Fist when you scroll over the screen, menu block pops up and depending on 00:02:28.000 --> 00:02:29.000 what device you are on, you can find that menu bar at the top or the at the 00:02:29.000 --> 00:02:31.000 bottom of the screen. 00:02:31.000 --> 00:02:33.000 That is where you will find the close 00:02:33.000 --> 00:02:36.000 captioning for today's session. 00:02:36.000 --> 00:02:41.000 You can true the captioning by 00:02:41.000 --> 00:02:44.000 selecting the close captioning tab on the menu bar. 00:02:44.000 --> 00:02:48.000 For the sign language interpreting, 00:02:48.000 --> 00:02:52.000 you can find the interpreter on the screen if you would like to change the 00:02:52.000 --> 00:02:56.000 size of the screen, if slides are shown, you can select the line in the 00:02:56.000 --> 00:03:00.000 middle to make the slides larger or smaller. 00:03:00.000 --> 00:03:01.000 If you can technical issues please use 00:03:01.000 --> 00:03:06.000 the chat feature. 00:03:06.000 --> 00:03:10.000 You can find the word bubble in that same menu bar. 00:03:10.000 --> 00:03:15.000 If you would like the turn the chat feature off or if you would are using 00:03:15.000 --> 00:03:19.000 a screen reader, you can do that by pushing alt and H. 00:03:19.000 --> 00:03:22.000 For Q & A times, please note the Q & A 00:03:22.000 --> 00:03:26.000 box is on the menu bar. 00:03:26.000 --> 00:03:32.000 Employees only submit questions there as they may get lost in the chat. 00:03:32.000 --> 00:03:37.000 To ask or make a question or make a comment, you can also select the hand 00:03:37.000 --> 00:03:43.000 option on the menu bar that allows us to know if you have a question. 00:03:43.000 --> 00:03:48.000 For those of you on the phone today are using key strokes, we ask that you 00:03:48.000 --> 00:03:52.000 press star 9 or alt Y on your key pad to raise your hand. 00:03:52.000 --> 00:03:53.000 We will then let you know when you can 00:03:53.000 --> 00:03:58.000 unmute. 00:03:58.000 --> 00:04:04.000 You can select the unmute option or alt A if using key strokes. 00:04:04.000 --> 00:04:09.000 Remember if you join into the conversation, please keep all back 00:04:09.000 --> 00:04:26.000 ground noise down as much as possible to allow us to hear you clearly. 00:04:26.000 --> 00:04:30.000 >>: Block awesome, thanks, Aerius. 00:04:30.000 --> 00:04:32.000 For Mary, would you mind going to the next slide. 00:04:32.000 --> 00:04:40.000 Perfect. 00:04:40.000 --> 00:04:41.000 So this workshop is going to be the part 2 ILRU I Lnet workshop on filling 00:04:41.000 --> 00:04:46.000 the gap. 00:04:46.000 --> 00:04:54.000 And I will introduce our presenters, next slide please. 00:04:54.000 --> 00:04:58.000 First we have daisy, the executive 00:04:58.000 --> 00:05:01.000 vice president of access living Chicago ... 00:05:01.000 --> 00:05:05.000 and has held a variety of management positions. 00:05:05.000 --> 00:05:14.000 We also have Seth who is the director 00:05:14.000 --> 00:05:19.000 of transition services at Lee high valley center for independent living 00:05:19.000 --> 00:05:24.000 LV CIL where he has worked for 9 years and in the social services field for 00:05:24.000 --> 00:05:26.000 over 20 years with the last 9 years directly in the field of transition. 00:05:26.000 --> 00:05:32.000 Also Joe ... 00:05:32.000 --> 00:05:38.000 who is the direct every of employment services at LV CIL where he provides 00:05:38.000 --> 00:05:38.000 operations and fiscal over variety for several of LV CIL's transition and 00:05:38.000 --> 00:05:43.000 employment programs. 00:05:43.000 --> 00:05:44.000 F and then Patricia Yeager who has 00:05:44.000 --> 00:05:50.000 been working in ... 00:05:50.000 --> 00:05:50.000 for over 40 years and is the CEO of the independent center in Colorado 00:05:50.000 --> 00:05:54.000 Spring. 00:05:54.000 --> 00:06:01.000 And also I'm Brooke Curtis the associate director of training and 00:06:01.000 --> 00:06:02.000 publications for the Iowa net and ILRU. 00:06:02.000 --> 00:06:09.000 Perfect. 00:06:09.000 --> 00:06:12.000 So what you will learn in today's session strategies and practices to 00:06:12.000 --> 00:06:16.000 assist your organization in shifting from identifying the gap which was the 00:06:16.000 --> 00:06:22.000 Part 1 workshop to program implementation. 00:06:22.000 --> 00:06:23.000 You'll also learn how to utilize data for program evaluations and to assess 00:06:23.000 --> 00:06:25.000 the success of programs. 00:06:25.000 --> 00:06:26.000 Next slide please. 00:06:26.000 --> 00:06:31.000 Perfect. 00:06:31.000 --> 00:06:36.000 Patricia, I will go ahead and turn things over to you. 00:06:36.000 --> 00:06:39.000 PATRICIA: Too many buttons to push. 00:06:39.000 --> 00:06:42.000 . Well welcome everyone. 00:06:42.000 --> 00:06:47.000 Welcome back P you joined us from the first presentation. 00:06:47.000 --> 00:06:53.000 If you haven't, I think this will still make sense as we talk about 00:06:53.000 --> 00:06:53.000 program design and implementation of at least that's my part of the 00:06:53.000 --> 00:06:59.000 presentation. 00:06:59.000 --> 00:07:05.000 So I'm Patricia Yeager, CEO of 00:07:05.000 --> 00:07:07.000 Independent Center in Colorado Spring my pronouns are she, her, and hers. 00:07:07.000 --> 00:07:12.000 Next slide. 00:07:12.000 --> 00:07:17.000 I don't know about your center but in 00:07:17.000 --> 00:07:22.000 our center sometimes it felt like we were in a popcorn popper and everybody 00:07:22.000 --> 00:07:26.000 had ideas and let's do this and that and they all sound good and you want 00:07:26.000 --> 00:07:29.000 to say yes to everything and then your staff is pulling out their hair 00:07:29.000 --> 00:07:34.000 because everybody's involved in all these different projects. 00:07:34.000 --> 00:07:40.000 So we decided to start working on a 00:07:40.000 --> 00:07:42.000 vetting process as a way to calm the pop important kernels down and have 00:07:42.000 --> 00:07:48.000 them pop up with at a time. 00:07:48.000 --> 00:07:53.000 It takes a bit of time but in the end you will have much better experience 00:07:53.000 --> 00:07:55.000 and know which programs to start and not the start or eve ideas. 00:07:55.000 --> 00:08:01.000 So we start with a form and this is 00:08:01.000 --> 00:08:07.000 copy the way a hand out, all of this is in your packet -- we start with a form. 00:08:07.000 --> 00:08:07.000 The staff person with the idea starts with this form called the 5 Ws and an 00:08:07.000 --> 00:08:10.000 H. 00:08:10.000 --> 00:08:18.000 And they fill it out. 00:08:18.000 --> 00:08:23.000 And maybe it team people, maybe manager program director, what's your considered. 00:08:23.000 --> 00:08:28.000 Who does your idea benefit, who will it ... 00:08:28.000 --> 00:08:31.000 so if you're planning a workshop are you going to take half the staff to do 00:08:31.000 --> 00:08:34.000 and it how will that be the impact. 00:08:34.000 --> 00:08:36.000 When can your idea be implemented? 00:08:36.000 --> 00:08:40.000 Is this a class for the spring? 00:08:40.000 --> 00:08:44.000 Is this a business that's going to take about a year o bring. 00:08:44.000 --> 00:08:47.000 So having an idea, it's answering these questions, getting people to 00:08:47.000 --> 00:08:50.000 think through these kinds of details. 00:08:50.000 --> 00:08:56.000 Where did your idea come from? 00:08:56.000 --> 00:09:03.000 One of my marketing director at some point several years ago had an idea 00:09:03.000 --> 00:09:05.000 that maybe we should do ASL interpreter rereferral as another source of income 00:09:05.000 --> 00:09:08.000 which was pretty interesting and we've been working on that 00:09:08.000 --> 00:09:12.000 But these where that idea came from. 00:09:12.000 --> 00:09:17.000 Why is your idea necessary? 00:09:17.000 --> 00:09:19.000 Want to off a sign language class, the staff, why is it necessary, what's 00:09:19.000 --> 00:09:22.000 you're thinking about this? 00:09:22.000 --> 00:09:27.000 And how will your idea be implement. 00:09:27.000 --> 00:09:32.000 And this is why people get to learn how to do planning, program planning. 00:09:32.000 --> 00:09:41.000 Is it easy, one person job like do a 00:09:41.000 --> 00:09:42.000 class for -- support group around people having trouble with accessing 00:09:42.000 --> 00:09:48.000 healthcare. 00:09:48.000 --> 00:09:55.000 Now that's an easy probably 1-person job, might not require a lot of 00:09:55.000 --> 00:09:56.000 resources so or is it we're going set up a business and it will require 00:09:56.000 --> 00:10:03.000 numerous sources. 00:10:03.000 --> 00:10:03.000 So that helps people get a grip on their idea and get their feet on the 00:10:03.000 --> 00:10:10.000 ground. 00:10:10.000 --> 00:10:15.000 The manager and the director and sometimes the whole leadership team 00:10:15.000 --> 00:10:16.000 will review it and decide ... 00:10:16.000 --> 00:10:20.000 or is it a business plan. 00:10:20.000 --> 00:10:23.000 That way the person doesn't get through all this and then to be told, 00:10:23.000 --> 00:10:32.000 no, we don't have money. 00:10:32.000 --> 00:10:43.000 ... 00:10:43.000 --> 00:10:45.000 we call this our vetting process. 00:10:45.000 --> 00:10:53.000 Once you Dr. 00:10:53.000 --> 00:10:56.000 That 5 Ws and an H, you eater have an idea plan or a business plan or maybe 00:10:56.000 --> 00:11:03.000 you're told, we can't do that now. 00:11:03.000 --> 00:11:08.000 And so in your attachments, I'm 00:11:08.000 --> 00:11:10.000 calling it packet, you have an idea plan workbook and you have a business 00:11:10.000 --> 00:11:14.000 plan workbook. 00:11:14.000 --> 00:11:21.000 And we've done this to make it easier for staff to work through the process 00:11:21.000 --> 00:11:26.000 of answering these questions before we say yes or no. 00:11:26.000 --> 00:11:29.000 So and they're very much the same. 00:11:29.000 --> 00:11:32.000 The difference is how much detail. 00:11:32.000 --> 00:11:38.000 So the first step is to identify the gap. 00:11:38.000 --> 00:11:41.000 Now you talked about community gaps with an Searlier today. 00:11:41.000 --> 00:11:50.000 This is a gap. 00:11:50.000 --> 00:11:55.000 So one of the gaps we worked on was one of our -- our strain community 00:11:55.000 --> 00:12:00.000 transition staff too many people in nursing homes that don't need to be! 00:12:00.000 --> 00:12:07.000 We began to research that, why were they there, how did they get there, we 00:12:07.000 --> 00:12:12.000 identified the gap, we looked at what -- how could -- what would our role be 00:12:12.000 --> 00:12:13.000 and how can we change that dynamic to keep people from going into nursing 00:12:13.000 --> 00:12:17.000 homes. 00:12:17.000 --> 00:12:24.000 And so we Dade lot of writing in identifying the gap. 00:12:24.000 --> 00:12:26.000 Keep people without a nursing homes so they can go only and heal at home. 00:12:26.000 --> 00:12:27.000 Well who would be interested? 00:12:27.000 --> 00:12:34.000 Who would pay for that? 00:12:34.000 --> 00:12:36.000 That's the kind of questions you're going see in the business plan side of 00:12:36.000 --> 00:12:39.000 this process. 00:12:39.000 --> 00:12:43.000 Once we identified the gap, then we 00:12:43.000 --> 00:12:49.000 really go into detail about program design. 00:12:49.000 --> 00:12:56.000 Who's going to do what, are what are the steps, how much time will it take 00:12:56.000 --> 00:12:57.000 each staff person, how much does it cost per hour per each staff person to 00:12:57.000 --> 00:12:58.000 do the work. 00:12:58.000 --> 00:13:04.000 So we develop add plan. 00:13:04.000 --> 00:13:09.000 Going back to identifying the gap, I want to mention when we're doing 00:13:09.000 --> 00:13:15.000 market research here and you're asking your customer who's the customer. 00:13:15.000 --> 00:13:20.000 The customer is a person who pays for it and the beneficiary is the consumer 00:13:20.000 --> 00:13:22.000 so you have two come appropriating, not often come appropriating, but you 00:13:22.000 --> 00:13:27.000 have two different customers that you have to satisfy. 00:13:27.000 --> 00:13:32.000 We went to the hospital and talked to them about their complex cases. 00:13:32.000 --> 00:13:35.000 That's what they call people who have trouble discharging and got lots of 00:13:35.000 --> 00:13:41.000 information that helped in our marketing and our evaluation. 00:13:41.000 --> 00:13:45.000 And helped us in developing ma'am funding. 00:13:45.000 --> 00:13:51.000 Identifying the program gap is very important. 00:13:51.000 --> 00:13:57.000 Then you begin to look at the finances, how much does it cost, how 00:13:57.000 --> 00:14:03.000 many people o you need to serve at a cost per person in order to make the 00:14:03.000 --> 00:14:04.000 program sustainable, break even or even provide revenue to your organization 00:14:04.000 --> 00:14:07.000 for oh projects. 00:14:07.000 --> 00:14:12.000 And then we talk about marketing, how 00:14:12.000 --> 00:14:17.000 will we market this, what information do we need, what data do we need to 00:14:17.000 --> 00:14:21.000 collect, how do we -- what do question do add veer at the same times on 00:14:21.000 --> 00:14:26.000 billboard, what's a way to reach our customers which is why it's so 00:14:26.000 --> 00:14:31.000 important to identify who is going to pay for this because they're the ones 00:14:31.000 --> 00:14:38.000 that are going to enable you to offer the service. 00:14:38.000 --> 00:14:48.000 And then the last piece is evaluation. 00:14:48.000 --> 00:14:57.000 We gnaw they wanted us to help people get out of the hospital on time or 00:14:57.000 --> 00:15:03.000 early and go home and heal correctly, not come back for a readmit. 00:15:03.000 --> 00:15:06.000 So those are the 5 steps of 00:15:06.000 --> 00:15:12.000 identifying or doing a vetting process. 00:15:12.000 --> 00:15:12.000 Whether it's an idea and you still go through the same steps but not that in 00:15:12.000 --> 00:15:17.000 depth. 00:15:17.000 --> 00:15:23.000 Might not have to do much marketing, might have a simpler evaluation plan. 00:15:23.000 --> 00:15:28.000 But if you're setting up a service, you want to know before you set it up 00:15:28.000 --> 00:15:29.000 that what it takes to make it successful before you take the leap 00:15:29.000 --> 00:15:32.000 into doing this. 00:15:32.000 --> 00:15:35.000 Want to look at all your risks, look 00:15:35.000 --> 00:15:41.000 at all the detail. 00:15:41.000 --> 00:15:46.000 This vetting process is what we use to 00:15:46.000 --> 00:15:54.000 look at all new programs, all new ideas before we start anything. 00:15:54.000 --> 00:15:57.000 And it has helped us identify programs we really couldn't sustain. 00:15:57.000 --> 00:16:07.000 We've also done this process with our 00:16:07.000 --> 00:16:12.000 existing program and have closed a program, improved programs, one of 00:16:12.000 --> 00:16:19.000 them we did research that showed we weren't getting enough money P the 00:16:19.000 --> 00:16:27.000 State and the State believe it or not -- we showed them exactly why we 00:16:27.000 --> 00:16:31.000 couldn't do it the reimbursement, they increased it by $5 a month per person 00:16:31.000 --> 00:16:34.000 which made a difference. 00:16:34.000 --> 00:16:36.000 So this information is really powerful 00:16:36.000 --> 00:16:43.000 in help you in setting up a program. 00:16:43.000 --> 00:16:48.000 You have a workbook, two workbook, idea plan and the business plan. 00:16:48.000 --> 00:16:55.000 You can walk through it because it's all the questions you put down to help 00:16:55.000 --> 00:17:01.000 you if you answer these questions, you'll come up with a idea plan or a 00:17:01.000 --> 00:17:03.000 business plan that you can act upon and know why you're accepting or why 00:17:03.000 --> 00:17:13.000 you're not. 00:17:13.000 --> 00:17:18.000 We're very open to helping you with 00:17:18.000 --> 00:17:25.000 this and we've done it now with about 6 different programs in our center, and 00:17:25.000 --> 00:17:29.000 it has truly helped us to streamline so look at what's helpful and what isn't. 00:17:29.000 --> 00:17:35.000 So it's been a great process to go through. 00:17:35.000 --> 00:17:36.000 And that concludes my part of the 00:17:36.000 --> 00:17:41.000 program for now. 00:17:41.000 --> 00:17:44.000 I think we're going to go into how does your organization approach 00:17:44.000 --> 00:17:49.000 program or implementation. 00:17:49.000 --> 00:17:55.000 And I know that the other speakers have stories of their own to tell. 00:17:55.000 --> 00:17:59.000 BROOKE: Thanks Patricia. 00:17:59.000 --> 00:18:03.000 So now Patricia's provided that great over view on how you move from 00:18:03.000 --> 00:18:08.000 identifying the gap to program implementation. 00:18:08.000 --> 00:18:14.000 And we are going to move into our preslated questions for our panelists. 00:18:14.000 --> 00:18:16.000 So the first one is how does your organization approach program 00:18:16.000 --> 00:18:24.000 implementation? 00:18:24.000 --> 00:18:24.000 DAISY: This is daisy from access 00:18:24.000 --> 00:18:27.000 living. 00:18:27.000 --> 00:18:30.000 My pronouns are she, her, is hers. 00:18:30.000 --> 00:18:34.000 So ours is pretty straightforward. 00:18:34.000 --> 00:18:39.000 I don't know how many of you were in Part 1 of the presentation where we 00:18:39.000 --> 00:18:44.000 were talking about identifying the began, but when we get to working on 00:18:44.000 --> 00:18:50.000 implementation, we've already gone through a pretty involved process to 00:18:50.000 --> 00:18:55.000 get information from a wide variety of stakeholders and looking internally at 00:18:55.000 --> 00:18:58.000 information that's helping us figure out what gaps we need to address. 00:18:58.000 --> 00:19:01.000 And then when we're in implementation 00:19:01.000 --> 00:19:08.000 we already then know a specific gap we want to address. 00:19:08.000 --> 00:19:13.000 And the way we approach this because we've already done that involved 00:19:13.000 --> 00:19:13.000 stakeholder process, implementation is very much a staff process at access 00:19:13.000 --> 00:19:18.000 living. 00:19:18.000 --> 00:19:22.000 We make sure that it involves staff at all different levels and often 00:19:22.000 --> 00:19:26.000 across departments depending on what the issue is. 00:19:26.000 --> 00:19:32.000 And we basically get -- it's a group 00:19:32.000 --> 00:19:38.000 process initially to fill out a form, an implementation plan, that we have 00:19:38.000 --> 00:19:45.000 been using for quite a while now, which just outlines what the goal is. 00:19:45.000 --> 00:19:50.000 Outlines key activities, outcomes, tools, and resources that are going to 00:19:50.000 --> 00:19:55.000 be needed in order to implement the program. 00:19:55.000 --> 00:19:56.000 It outline what is organizational functions going to have to be involved 00:19:56.000 --> 00:20:01.000 so what departments. 00:20:01.000 --> 00:20:04.000 We actually get detailed into different milestones usually by 00:20:04.000 --> 00:20:10.000 quarter that we're hoping to see with the program. 00:20:10.000 --> 00:20:14.000 We're talking about who's accountable for the actual implementation both who 00:20:14.000 --> 00:20:18.000 the lead person is and who are the other staff going to be involved. 00:20:18.000 --> 00:20:19.000 And what do we think is going to be the start and end date of the 00:20:19.000 --> 00:20:24.000 implementation plan. 00:20:24.000 --> 00:20:25.000 So that gives us then blueprint for us 00:20:25.000 --> 00:20:31.000 to move forward. 00:20:31.000 --> 00:20:35.000 And of course that's not a static document, we hit barriers along the 00:20:35.000 --> 00:20:40.000 way as always is the case with this kind of work so we need to think of 00:20:40.000 --> 00:20:49.000 that as a living breathing document, but it helps set the course initially 00:20:49.000 --> 00:20:51.000 with how we're going to implement the program 00:20:51.000 --> 00:20:56.000 BROOKE: Thanks, daisy. 00:20:56.000 --> 00:21:05.000 Patricia, Seth, and Joe do you want to also answer this question as well? 00:21:05.000 --> 00:21:11.000 SETH: Yeah I can add in. 00:21:11.000 --> 00:21:12.000 It sounds like we take similar approach to howdy si goes about things 00:21:12.000 --> 00:21:21.000 at her CIL. 00:21:21.000 --> 00:21:28.000 Some of it comes to that whole strategic planning part but there's 00:21:28.000 --> 00:21:33.000 typically things in ... 00:21:33.000 --> 00:21:37.000 and activity. 00:21:37.000 --> 00:21:46.000 So it sounds very similar. 00:21:46.000 --> 00:21:48.000 PATRICIA: We -- I think the process 00:21:48.000 --> 00:21:54.000 I've outlined really speaks all that. 00:21:54.000 --> 00:22:00.000 We do have a -- try to have a team of 4 people if it's a business plan. 00:22:00.000 --> 00:22:07.000 And the financial manager, financial guru in your center is critical to all 00:22:07.000 --> 00:22:11.000 of this to help make sure that we vis they dreamers have our feet on the 00:22:11.000 --> 00:22:16.000 ground in terms of money and what it will actually take to do this. 00:22:16.000 --> 00:22:19.000 But it does take team of people to do this kind of approach. 00:22:19.000 --> 00:22:25.000 And you really got to be careful and 00:22:25.000 --> 00:22:26.000 you can't do four or five of these all at the same time unless you have quite 00:22:26.000 --> 00:22:28.000 a large staff. 00:22:28.000 --> 00:22:32.000 It helps to do this one at a time. 00:22:32.000 --> 00:22:33.000 And let your get used to that process, 00:22:33.000 --> 00:22:37.000 you're teaching them program ... 00:22:37.000 --> 00:22:42.000 which is a great skill to have. 00:22:42.000 --> 00:22:43.000 BROOKE: Awesome. 00:22:43.000 --> 00:22:47.000 Thanks, Patricia. 00:22:47.000 --> 00:22:53.000 Our next question on the next slide. 00:22:53.000 --> 00:22:55.000 So capacity, finance, resource et cetera, how you make sure you have 00:22:55.000 --> 00:22:59.000 these key elements in place? 00:22:59.000 --> 00:23:04.000 Daisy, do you want to start us off again? 00:23:04.000 --> 00:23:09.000 DAISY: Sure I'm happy to. 00:23:09.000 --> 00:23:17.000 So this is a really key question and I would mention a couple things here. 00:23:17.000 --> 00:23:21.000 One, usually when access living is going about our implementation plan, 00:23:21.000 --> 00:23:21.000 part of the activities in the implementation plan actually peak to 00:23:21.000 --> 00:23:28.000 this. 00:23:28.000 --> 00:23:33.000 So we just did a strategic plan refresh like 6 months or so ago and we 00:23:33.000 --> 00:23:33.000 have a whole new area that we're focused on on economic justice and 00:23:33.000 --> 00:23:39.000 opportunity. 00:23:39.000 --> 00:23:43.000 We don't have any direct service of policy staff or community organizing 00:23:43.000 --> 00:23:48.000 staff who are working specifically many that area so one of the first things 00:23:48.000 --> 00:23:51.000 in the implementation plan is find funding to actually have the positions 00:23:51.000 --> 00:23:55.000 on staff we're going to need in order to implement that project. 00:23:55.000 --> 00:24:01.000 The other thing I would point to is 00:24:01.000 --> 00:24:05.000 especially if this is -- this filling a gap is derived out of a strategic 00:24:05.000 --> 00:24:09.000 planning process that's a broad process and we talked about H in the last 00:24:09.000 --> 00:24:11.000 session a little bit, but that really should identify which things you need 00:24:11.000 --> 00:24:16.000 to let go of. 00:24:16.000 --> 00:24:22.000 And so hopefully if you've done that process and really taken that part of 00:24:22.000 --> 00:24:27.000 it seriously, you might have identified some programs that you do need to let 00:24:27.000 --> 00:24:32.000 go of and that might open up some capacity among other staff. 00:24:32.000 --> 00:24:35.000 And well I'll let Patricia added to 00:24:35.000 --> 00:24:40.000 this, but I like what had she was talking about in terms of getting 00:24:40.000 --> 00:24:45.000 really detailed how much programs are going to cost so you have both in 00:24:45.000 --> 00:24:50.000 terms of money and time so you have a really good idea what it is going to 00:24:50.000 --> 00:24:58.000 take in order to get a new program up and running. 00:24:58.000 --> 00:24:59.000 PATRICIA: We're really doing hospital 00:24:59.000 --> 00:25:10.000 to home. 00:25:10.000 --> 00:25:15.000 We had a spreadsheet, no, it was a flowchart that I mean the flowchart to 00:25:15.000 --> 00:25:21.000 end all flowcharts, what happened when the person came in and how they got 00:25:21.000 --> 00:25:27.000 there, how did we get them as a consumer, patient in the hospital, the 00:25:27.000 --> 00:25:36.000 assessment of needs with all the way through going to their home and 00:25:36.000 --> 00:25:41.000 checking it out for access and safety and bringing in home health 00:25:41.000 --> 00:25:45.000 assistance, OT, P T, I mean what he had to set all of that up and make some 00:25:45.000 --> 00:25:50.000 decision about in the first two weeks of this program of being home from the 00:25:50.000 --> 00:25:57.000 hospital, it's probably going to be more medically oriented than 00:25:57.000 --> 00:26:01.000 independent living, so we had to kind of get a range of how many hours of OT 00:26:01.000 --> 00:26:04.000 or P T or any kind of professional help like that be available. 00:26:04.000 --> 00:26:06.000 And then how much did it cost? 00:26:06.000 --> 00:26:07.000 Did it cost week biweek. 00:26:07.000 --> 00:26:12.000 It was exhausting. 00:26:12.000 --> 00:26:13.000 And it was overwhelming to me because I continue deal well with all those 00:26:13.000 --> 00:26:17.000 kinds of details. 00:26:17.000 --> 00:26:20.000 But having program staff, and 00:26:20.000 --> 00:26:26.000 financial staff involved, they were on it like crazy. 00:26:26.000 --> 00:26:30.000 So it's important to have different perspectives on these team to figure 00:26:30.000 --> 00:26:33.000 out can you do it or not. 00:26:33.000 --> 00:26:35.000 And sometimes -- and you also have to 00:26:35.000 --> 00:26:41.000 have a champion for the project. 00:26:41.000 --> 00:26:46.000 We would love to start an ASL referral business, bier right now we 00:26:46.000 --> 00:26:47.000 don't have a channel con for so it we're putting it on hold until we hire 00:26:47.000 --> 00:26:53.000 a new program manager. 00:26:53.000 --> 00:26:59.000 We have a cray can si in the organization so we'll wait until we 00:26:59.000 --> 00:27:00.000 have that champion who's taking this on, this program to make it sure it 00:27:00.000 --> 00:27:04.000 goes forward. 00:27:04.000 --> 00:27:07.000 It's critical to make sure the program actually happens. 00:27:07.000 --> 00:27:13.000 And I will say the planning team needs 00:27:13.000 --> 00:27:19.000 to meet, we met weekly for probably a good part of a year if not more. 00:27:19.000 --> 00:27:23.000 Every Wednesday from 11 to 12 the team came doing and we met weekly with 00:27:23.000 --> 00:27:28.000 what did we find out, what are we doing, that communication, having 00:27:28.000 --> 00:27:32.000 someone who can help you with the communication of all of that tracking 00:27:32.000 --> 00:27:35.000 is pretty important too if it's a big prom. 00:27:35.000 --> 00:27:37.000 program. 00:27:37.000 --> 00:27:38.000 Seth 00:27:38.000 --> 00:27:43.000 SETH: Much agreed on all of that. 00:27:43.000 --> 00:27:48.000 And I think as we've done thing, seeing that having financial folks 00:27:48.000 --> 00:27:52.000 that understand a little bit about programming and programming folks 00:27:52.000 --> 00:27:53.000 having a little bit of understanding about financial aspects is very 00:27:53.000 --> 00:27:58.000 helpful and beneficial. 00:27:58.000 --> 00:28:04.000 Like that has been one of our key things over the years. 00:28:04.000 --> 00:28:09.000 And like Patricia was saying really taking that initial time upfront, lots 00:28:09.000 --> 00:28:13.000 of meetings, lots of discussion, lots of looking at every single point to 00:28:13.000 --> 00:28:20.000 get things especially when you're getting things started is in place and 00:28:20.000 --> 00:28:23.000 costs are covered and really almost over communication with that to just 00:28:23.000 --> 00:28:28.000 make sure things are done and looked at. 00:28:28.000 --> 00:28:29.000 BROOKE: Awesome. 00:28:29.000 --> 00:28:31.000 This is great. 00:28:31.000 --> 00:28:36.000 All excellent points. 00:28:36.000 --> 00:28:41.000 I did also want to mention and Mary put this in the chat, if you have any 00:28:41.000 --> 00:28:43.000 questions at that point, feel free to drop them in the Q & A so that way we 00:28:43.000 --> 00:28:46.000 can have the presenters answer them. 00:28:46.000 --> 00:28:47.000 And now we're going to move to the 00:28:47.000 --> 00:28:51.000 next slide. 00:28:51.000 --> 00:28:55.000 Which is why is program evaluation important? 00:28:55.000 --> 00:28:57.000 And Seth and Joe are going to cover this. 00:28:57.000 --> 00:29:05.000 So Seth I'll turn it over to you. 00:29:05.000 --> 00:29:09.000 JOE: I'm here, Brooke so this is Joe, 00:29:09.000 --> 00:29:13.000 LV CIL, program prove nouns are he, him, and his. 00:29:13.000 --> 00:29:14.000 We wanted to spend a little bit of time talking specifically about 00:29:14.000 --> 00:29:19.000 program evaluation. 00:29:19.000 --> 00:29:24.000 We talked about design, finance, marketing, a all of that stuff and 00:29:24.000 --> 00:29:28.000 that's all so important, but I think we can all fairly say that the most 00:29:28.000 --> 00:29:32.000 neglected aspect of program design is often program evaluation. 00:29:32.000 --> 00:29:37.000 Typically we start programs and services and we get to the end of that 00:29:37.000 --> 00:29:38.000 first year and go okay how are we doing and that's not a good way to do 00:29:38.000 --> 00:29:42.000 program evaluation. 00:29:42.000 --> 00:29:46.000 We know that also many of you have 00:29:46.000 --> 00:29:51.000 experienced with program evaluation, but it's something that is so 00:29:51.000 --> 00:29:57.000 important that you start thinking about right away and it's got to be part of 00:29:57.000 --> 00:30:01.000 your program design from the beginning because even if you're talking about 00:30:01.000 --> 00:30:04.000 something like budget, you have to budget time for the program 00:30:04.000 --> 00:30:09.000 evaluation, not just the staff time to provide service, but you have to have 00:30:09.000 --> 00:30:11.000 that worked in your budget to make sure you have time to do good program 00:30:11.000 --> 00:30:17.000 evaluation. 00:30:17.000 --> 00:30:20.000 Next one please. 00:30:20.000 --> 00:30:23.000 Oh back one (chuckling) this is the fun part. 00:30:23.000 --> 00:30:29.000 Okay so program evaluation. 00:30:29.000 --> 00:30:30.000 We know many of you know the value of it but we can't express how important 00:30:30.000 --> 00:30:34.000 it is. 00:30:34.000 --> 00:30:39.000 Without program evaluation you really have no idea how your program is 00:30:39.000 --> 00:30:40.000 really doing, if you're truly addressing the gap that you identified 00:30:40.000 --> 00:30:43.000 at the beginning of your process. 00:30:43.000 --> 00:30:44.000 And if you're really doing anything to 00:30:44.000 --> 00:30:47.000 close that gap. 00:30:47.000 --> 00:30:52.000 In addition to slide state ... 00:30:52.000 --> 00:31:00.000 help use in a host of way, especially regarding improving your program over 00:31:00.000 --> 00:31:01.000 time and making it even more effective, marketable and even 00:31:01.000 --> 00:31:03.000 profitable. 00:31:03.000 --> 00:31:05.000 As the side says ... 00:31:05.000 --> 00:31:09.000 and they document your success. 00:31:09.000 --> 00:31:09.000 With this information programs are better able to direct (reading 00:31:09.000 --> 00:31:13.000 verbatim) ... 00:31:13.000 --> 00:31:16.000 so where they're most needed and most effective the communities. 00:31:16.000 --> 00:31:17.000 Knowing more about both successes and 00:31:17.000 --> 00:31:20.000 not (reading verbatim) ... 00:31:20.000 --> 00:31:22.000 positive changes within your program. 00:31:22.000 --> 00:31:26.000 Next slide please. 00:31:26.000 --> 00:31:30.000 So there's lots of myths out there 00:31:30.000 --> 00:31:35.000 about program evaluation and there's 3 on the slide for you. 00:31:35.000 --> 00:31:37.000 One says evaluation a useless activity that generates lots of boring 00:31:37.000 --> 00:31:43.000 data with useless conclusion. 00:31:43.000 --> 00:31:47.000 And I would say in some cases this can be true, but if it is, it's the 00:31:47.000 --> 00:31:50.000 result of poorly determined evaluation criteria. 00:31:50.000 --> 00:31:51.000 It becomes a myth when you have the 00:31:51.000 --> 00:31:54.000 right data points and information. 00:31:54.000 --> 00:31:56.000 Second one says evaluation is proving 00:31:56.000 --> 00:31:59.000 the success of failure of a program. 00:31:59.000 --> 00:32:03.000 Evaluation is a lot more than that. 00:32:03.000 --> 00:32:07.000 It's about consistent improvement and avoiding stagnation. 00:32:07.000 --> 00:32:11.000 And then finally evaluation is a 00:32:11.000 --> 00:32:13.000 highly unique and complex process that occurs at a certain (reading 00:32:13.000 --> 00:32:14.000 verbatim) ... 00:32:14.000 --> 00:32:20.000 experts. 00:32:20.000 --> 00:32:26.000 It is true evaluation can be very complex, very difficult, there are 00:32:26.000 --> 00:32:31.000 courses on how to do it well, but if you can simplify and it choose good 00:32:31.000 --> 00:32:33.000 indicaters and questions, you can actually do it on a level that makes 00:32:33.000 --> 00:32:35.000 sense for your organization. 00:32:35.000 --> 00:32:39.000 Next slide. 00:32:39.000 --> 00:32:44.000 Okay. 00:32:44.000 --> 00:32:45.000 Now I'm going to ask Seth if it's okay for him to jump in on this slide 00:32:45.000 --> 00:32:54.000 for me. 00:32:54.000 --> 00:33:00.000 SETH: Yeah sorry about that. 00:33:00.000 --> 00:33:00.000 My mouse wasn't working and my arrow wasn't hitting my stop video might 00:33:00.000 --> 00:33:03.000 thing. 00:33:03.000 --> 00:33:11.000 So my apologizes. 00:33:11.000 --> 00:33:15.000 So program evaluation what it really is is program evaluation is careful 00:33:15.000 --> 00:33:16.000 collecting information about program or some aspect of a program in order to 00:33:16.000 --> 00:33:19.000 make a necessary decision about the program. 00:33:19.000 --> 00:33:25.000 Sof evaluations can include a whole 00:33:25.000 --> 00:33:30.000 bunch of different thing, at least 35 types of evaluations such as things 00:33:30.000 --> 00:33:36.000 like needs assessments, cost benefit analysis (reading verbatim) 00:33:36.000 --> 00:33:38.000 effectiveness, fetch si, formative, summative, goal based process, 00:33:38.000 --> 00:33:41.000 outcomes and a whole lot more. 00:33:41.000 --> 00:33:45.000 The type of evaluation you undertake 00:33:45.000 --> 00:33:45.000 to improve your programs depends on what you want to learn about the 00:33:45.000 --> 00:33:47.000 program. 00:33:47.000 --> 00:33:51.000 This is where we have to preface the 00:33:51.000 --> 00:33:54.000 rest of this presentation, the fact is we can spend days in professional 00:33:54.000 --> 00:33:59.000 training to learn about the ... 00:33:59.000 --> 00:34:02.000 some of you may even have the knowledge and there's wonderful. 00:34:02.000 --> 00:34:02.000 But we're not going to cover that in 00:34:02.000 --> 00:34:07.000 the next hand. 00:34:07.000 --> 00:34:14.000 Of minutes and nor to we expect that level of program evaluation to occur. 00:34:14.000 --> 00:34:19.000 Instead we're going lay the ground work for foundation for you, give you 00:34:19.000 --> 00:34:22.000 some tips and tricks, and then look to you to reach out if you need help or 00:34:22.000 --> 00:34:24.000 more information on this topic. 00:34:24.000 --> 00:34:30.000 Next slide please. 00:34:30.000 --> 00:34:31.000 So with many things in life we want to 00:34:31.000 --> 00:34:36.000 keep it simple. 00:34:36.000 --> 00:34:37.000 So we recommend that you keep the program evaluation piece as simple as 00:34:37.000 --> 00:34:39.000 possible. 00:34:39.000 --> 00:34:43.000 And he's why we think that. 00:34:43.000 --> 00:34:48.000 ... 00:34:48.000 --> 00:34:49.000 which is why it's probably the most neglected part of programming and 00:34:49.000 --> 00:34:52.000 planning and implementation. 00:34:52.000 --> 00:34:57.000 We typically especially in the 00:34:57.000 --> 00:35:00.000 nonprofit world just don't have a lot of time, but even more so we don't 00:35:00.000 --> 00:35:05.000 budget time for it, as Joe said before. 00:35:05.000 --> 00:35:08.000 How are you going to time to determine data, send out surveys, call 00:35:08.000 --> 00:35:13.000 consumers and or customers for feedback. 00:35:13.000 --> 00:35:18.000 Chances are you won't and as long as the program appears to be doing well, 00:35:18.000 --> 00:35:21.000 program evaluation at an efficient level probably won't happen. 00:35:21.000 --> 00:35:25.000 So again our hope is that we get you 00:35:25.000 --> 00:35:31.000 down the path for developing a simple model of program evaluation that you 00:35:31.000 --> 00:35:37.000 really with implement down the road as your program and more importantly your 00:35:37.000 --> 00:35:38.000 consumers will truly benefit from the solid evaluation done on a regular 00:35:38.000 --> 00:35:42.000 basis. 00:35:42.000 --> 00:35:43.000 That said those with ex per tease in 00:35:43.000 --> 00:35:48.000 the area ... 00:35:48.000 --> 00:35:50.000 and you don't have to follow this particular model. 00:35:50.000 --> 00:35:53.000 Next slide please. 00:35:53.000 --> 00:35:55.000 JOE: Yeah. 00:35:55.000 --> 00:35:59.000 All right. 00:35:59.000 --> 00:36:03.000 So I think you can go to the next slide there. 00:36:03.000 --> 00:36:07.000 Goring to talk about those exact things, how goals are measured and how 00:36:07.000 --> 00:36:08.000 you come up with indicators because that's really the key to good program 00:36:08.000 --> 00:36:13.000 evaluation. 00:36:13.000 --> 00:36:16.000 If you come up with good questions and then good ind kiters to measure 00:36:16.000 --> 00:36:23.000 those questions, that's why area the key is. 00:36:23.000 --> 00:36:24.000 So what model so do you use, as the sign says there's millions of them out 00:36:24.000 --> 00:36:28.000 there. 00:36:28.000 --> 00:36:33.000 Perhaps the best one and simplest one 00:36:33.000 --> 00:36:38.000 that I found is directly from the CDC, we don't have the time to get into all 00:36:38.000 --> 00:36:43.000 the areas listed as they talk about engaging stakeholders, describing the 00:36:43.000 --> 00:36:50.000 program, focusing program E sign, gathering credible evidence and 00:36:50.000 --> 00:36:56.000 ensuring use, that's a lot to get into in a very short period so we're going 00:36:56.000 --> 00:36:56.000 to focus specifically on the focusing evaluation design and gathering 00:36:56.000 --> 00:37:00.000 credible evidence. 00:37:00.000 --> 00:37:07.000 Some of them are sell ef explanatory ... 00:37:07.000 --> 00:37:08.000 talking to everybody involved who has a stake in the program or service and 00:37:08.000 --> 00:37:13.000 what they're looking to get out of it. 00:37:13.000 --> 00:37:20.000 But we'll go to the next slide and 00:37:20.000 --> 00:37:23.000 focus on -- so on the part that's called focusing on the evaluation and 00:37:23.000 --> 00:37:26.000 that has to do ... 00:37:26.000 --> 00:37:28.000 in other words what do you want to be able to decide as a result of the 00:37:28.000 --> 00:37:33.000 evaluation? 00:37:33.000 --> 00:37:34.000 For example what determines the program's success for consumer success 00:37:34.000 --> 00:37:39.000 in some ... 00:37:39.000 --> 00:37:40.000 might lead to big picture outcomes such as getting a job or not returning 00:37:40.000 --> 00:37:46.000 to the hospital. 00:37:46.000 --> 00:37:52.000 Or they may be more about nuanced areas such as timely service and 00:37:52.000 --> 00:37:53.000 support, good communication, receiving quality equipment and maybe even the 00:37:53.000 --> 00:37:57.000 cleanliness of the facilities. 00:37:57.000 --> 00:38:01.000 So what about customer, they may also 00:38:01.000 --> 00:38:06.000 have specific outcomes and by customers kind of what we traditionally talk 00:38:06.000 --> 00:38:12.000 about as funder, but Patricia's taught us over the last few months so really 00:38:12.000 --> 00:38:17.000 look at our funders and business as customers so that's why I use that 00:38:17.000 --> 00:38:20.000 language, but they also have nuanced ... 00:38:20.000 --> 00:38:25.000 or also that might have to do with communication goals. 00:38:25.000 --> 00:38:28.000 And then want your agency, what do 00:38:28.000 --> 00:38:32.000 staff think, do they have the training and resources needed? 00:38:32.000 --> 00:38:38.000 Only you and your stakeholders can determine these key questions. 00:38:38.000 --> 00:38:39.000 Make sure you ask what's important to your stakeholders in addition to the 00:38:39.000 --> 00:38:44.000 obvious ones. 00:38:44.000 --> 00:38:46.000 For example if the over all outcome is no to not have consumers turn the 00:38:46.000 --> 00:38:52.000 hospital, that's obvious, the ... 00:38:52.000 --> 00:38:53.000 are we successful in keeping consumers from returning to the hospital after 00:38:53.000 --> 00:38:59.000 discharge? 00:38:59.000 --> 00:39:07.000 So you got to keep these questions 00:39:07.000 --> 00:39:11.000 specific, but you also don't want to get into really specific detail so 00:39:11.000 --> 00:39:15.000 they have to be clear questions but you can't get into the details. 00:39:15.000 --> 00:39:15.000 Because that's what we're going to talk about next, gathering the 00:39:15.000 --> 00:39:18.000 evidence. 00:39:18.000 --> 00:39:23.000 This is where you determine the really 00:39:23.000 --> 00:39:26.000 hard part, what indicaters are you going to use to measure and have the 00:39:26.000 --> 00:39:31.000 data to back and up answer the questions you're asking. 00:39:31.000 --> 00:39:33.000 So as a CDC says the indicator should 00:39:33.000 --> 00:39:44.000 be simple, precise, and measurable. 00:39:44.000 --> 00:39:49.000 Let's focus on that measurable part, indicaters can't be ... 00:39:49.000 --> 00:39:51.000 instead you must come up with relevant measurable data. 00:39:51.000 --> 00:39:53.000 So examples ... 00:39:53.000 --> 00:39:58.000 I know we've short on time. 00:39:58.000 --> 00:40:01.000 If one of our evaluation questions is are we successfully presenting 00:40:01.000 --> 00:40:05.000 customers from returning to the hospital after discharge? 00:40:05.000 --> 00:40:14.000 What indicators could you use to determine this? 00:40:14.000 --> 00:40:16.000 Well as a Sam of that maybe say data 80 percent of consumers ... 00:40:16.000 --> 00:40:18.000 within 3 months of discharge. 00:40:18.000 --> 00:40:21.000 You see how we're going to track data 00:40:21.000 --> 00:40:28.000 that's going help us answer that over all question. 00:40:28.000 --> 00:40:34.000 80 percent of consumers, that's very detailed and specific, will not return 00:40:34.000 --> 00:40:39.000 to the hospital for the same or related diagnosis within 3 months of discharge 00:40:39.000 --> 00:40:41.000 so we're putting a percentage on there and also a timeframe. 00:40:41.000 --> 00:40:44.000 Here's another evaluation question, 00:40:44.000 --> 00:40:47.000 are we successfully helping young adults ... 00:40:47.000 --> 00:40:52.000 if that is their goal? 00:40:52.000 --> 00:40:58.000 An indicator for that question might be 75 percent of young adults have 00:40:58.000 --> 00:41:00.000 obtained competitive employment with or without support within 6 months of 00:41:00.000 --> 00:41:02.000 graduation. 00:41:02.000 --> 00:41:07.000 Then we can talk about nuanced 00:41:07.000 --> 00:41:10.000 questions like are we providing timely and effective customer service to 00:41:10.000 --> 00:41:10.000 consumers? 00:41:10.000 --> 00:41:17.000 ... 00:41:17.000 --> 00:41:19.000 to a survey responded they agreed or strongly agreed that services for ... 00:41:19.000 --> 00:41:22.000 and so on and son. 00:41:22.000 --> 00:41:27.000 You can get into financial questions 00:41:27.000 --> 00:41:29.000 with this, you can do a lot of data analysis, you can be doing surveys to 00:41:29.000 --> 00:41:34.000 gather this information. 00:41:34.000 --> 00:41:35.000 Other thing Kraus want to think about for indicator, don't just make up the 00:41:35.000 --> 00:41:39.000 numbers. 00:41:39.000 --> 00:41:46.000 If you're thinking about creating an indicator like we talked about where 00:41:46.000 --> 00:41:48.000 57 percent of young adults were obtaining employment. 00:41:48.000 --> 00:41:52.000 Why 57 percent? 00:41:52.000 --> 00:41:55.000 Perhap there's a national standard you can use, do one of your customers 00:41:55.000 --> 00:41:58.000 have a benchmark they want you to reach? 00:41:58.000 --> 00:42:02.000 Maybe you can base it on existing evaluation data. 00:42:02.000 --> 00:42:03.000 It's important to determine numbers that will actually tell you something 00:42:03.000 --> 00:42:06.000 about the program. 00:42:06.000 --> 00:42:11.000 Also consider indicaters that are 00:42:11.000 --> 00:42:17.000 going lead you to how services impact the consumers. 00:42:17.000 --> 00:42:21.000 Just the more qualitative things like one person never worked but now loves 00:42:21.000 --> 00:42:25.000 saving her money, that kind of information can help you down the road 00:42:25.000 --> 00:42:27.000 not only to show your program success, but help you with marketing. 00:42:27.000 --> 00:42:30.000 Okay. 00:42:30.000 --> 00:42:32.000 I think I lost my place, sorry. 00:42:32.000 --> 00:42:33.000 All right. 00:42:33.000 --> 00:42:37.000 There we go. 00:42:37.000 --> 00:42:41.000 Then you got to get out there and figure out a way how you're going to 00:42:41.000 --> 00:42:44.000 track and obtain this data and you have to again do this early on. 00:42:44.000 --> 00:42:54.000 It has to be part of your program design. 00:42:54.000 --> 00:43:00.000 Are you going set up tracking through excel or some sort of database ... 00:43:00.000 --> 00:43:05.000 or do or return to the hospital within a certain timeframe. 00:43:05.000 --> 00:43:09.000 If those systems aren't set upright from the get-go ... 00:43:09.000 --> 00:43:09.000 decide if you're going to do a survey and how you're going to get that 00:43:09.000 --> 00:43:14.000 information out there. 00:43:14.000 --> 00:43:17.000 It takes a lot of time to do program evaluation, even at a simple level. 00:43:17.000 --> 00:43:20.000 And then you have to spend the time 00:43:20.000 --> 00:43:21.000 interpreting the data and then presenting it potentially to your 00:43:21.000 --> 00:43:27.000 board and others. 00:43:27.000 --> 00:43:31.000 So in the end, you know, this process 00:43:31.000 --> 00:43:35.000 though is going to help you in so many different ways as we mentioned. 00:43:35.000 --> 00:43:40.000 And again it's something you have to think about right from the start. 00:43:40.000 --> 00:43:45.000 And with that said I'm going turn it back to Seth or Patricia or daisy for 00:43:45.000 --> 00:43:48.000 any comments on program evaluation that I may have raced through that you want 00:43:48.000 --> 00:44:06.000 to stress or initial information. 00:44:06.000 --> 00:44:06.000 DAISY: This is daisy from access 00:44:06.000 --> 00:44:10.000 living. 00:44:10.000 --> 00:44:11.000 I would just stress or kind of amplify, Joe, what you said about 00:44:11.000 --> 00:44:16.000 analysis of the data. 00:44:16.000 --> 00:44:21.000 I think I know where access living has gotten tripped up. 00:44:21.000 --> 00:44:27.000 I feel like for a while we weren't good at evaluation at all. 00:44:27.000 --> 00:44:32.000 Then we got pretty good at actually having the plans in place for what we 00:44:32.000 --> 00:44:37.000 were going to evaluate, tracking the numbers and how we were doing, but 00:44:37.000 --> 00:44:39.000 actually like putting that data to use (chuckling) is like a whole nother 00:44:39.000 --> 00:44:44.000 step that is even harder. 00:44:44.000 --> 00:44:49.000 So you know I think that part of it is 00:44:49.000 --> 00:44:55.000 really important and in terms of having a plan and a structure set up at your 00:44:55.000 --> 00:44:59.000 senter that's going to allow you to analyze the data, have processes in 00:44:59.000 --> 00:45:01.000 place that allow you to analyze the data is really important. 00:45:01.000 --> 00:45:05.000 JOE: Very much agreed. 00:45:05.000 --> 00:45:09.000 And I would love to spend more time talking about that because that is 00:45:09.000 --> 00:45:19.000 really key, daisy, thank you for that. 00:45:19.000 --> 00:45:20.000 PATRICIA: I would add that be curious. 00:45:20.000 --> 00:45:28.000 You know. 00:45:28.000 --> 00:45:32.000 Activate your curiosity, if someone on your staff had an idea that this 00:45:32.000 --> 00:45:32.000 particular service would be good, well let's see, let's ask questions about 00:45:32.000 --> 00:45:35.000 it. 00:45:35.000 --> 00:45:45.000 Who used it, how did they feel about it? 00:45:45.000 --> 00:45:48.000 Ask questions that help you evaluate or help answer the question should we 00:45:48.000 --> 00:46:00.000 continue or to we have an impact, is it going well? 00:46:00.000 --> 00:46:03.000 I mean if we did -- had 50 percent of the people out of nursing homes, 00:46:03.000 --> 00:46:04.000 that's good information to have. 00:46:04.000 --> 00:46:10.000 So be curious. 00:46:10.000 --> 00:46:13.000 Put on your curious t-shirt when you're doing program violation. 00:46:13.000 --> 00:46:16.000 evaluation. 00:46:16.000 --> 00:46:17.000 BROOKE: So we do have another 00:46:17.000 --> 00:46:22.000 question on the next slide. 00:46:22.000 --> 00:46:24.000 And we had some questions coming in the Q and, that we'll get to here in a 00:46:24.000 --> 00:46:31.000 little bit. 00:46:31.000 --> 00:46:34.000 So Patricia and daisy, how have you used the data to prove success or to 00:46:34.000 --> 00:46:36.000 market to potential funders or ... 00:46:36.000 --> 00:46:38.000 (reading verbatim) ... 00:46:38.000 --> 00:46:48.000 partners? 00:46:48.000 --> 00:46:53.000 DAISY: At access living each program 00:46:53.000 --> 00:46:56.000 has its own kind of indicaters and outcomes that they're tracking. 00:46:56.000 --> 00:47:03.000 And then all of that information 00:47:03.000 --> 00:47:10.000 getting turned quarterly and one of the main people it goes to, well there's 00:47:10.000 --> 00:47:14.000 sort of two -- in addition to just oh staff at access living it goes to our 00:47:14.000 --> 00:47:19.000 development staff who are helping doing the fund raising, so they directly 00:47:19.000 --> 00:47:24.000 have the data that they need in order to talk about our programs. 00:47:24.000 --> 00:47:28.000 And then it also goes to we do a 00:47:28.000 --> 00:47:36.000 dashboard that then get shared with our board of directors. 00:47:36.000 --> 00:47:37.000 And those have been good ways to use 00:47:37.000 --> 00:47:42.000 the data to prove success. 00:47:42.000 --> 00:47:47.000 Although I would also say one of things that we make sure staff are 00:47:47.000 --> 00:47:52.000 talking about every quarter are what the barriers they hit so if we didn't 00:47:52.000 --> 00:47:57.000 hit our numbers in one area, we've not saying that's a terrible thing, but we 00:47:57.000 --> 00:48:01.000 need to understand what the barriers is so that we can actually try to go 00:48:01.000 --> 00:48:14.000 around those barriers, get around them. 00:48:14.000 --> 00:48:16.000 PATRICIA: I was just going to say 00:48:16.000 --> 00:48:18.000 data is power. 00:48:18.000 --> 00:48:30.000 Information is power. 00:48:30.000 --> 00:48:32.000 It's like a big flashlight. 00:48:32.000 --> 00:48:35.000 So data is so important. 00:48:35.000 --> 00:48:38.000 And na minutely talk about marketing and using that data. 00:48:38.000 --> 00:48:40.000 So Seth, take it away. 00:48:40.000 --> 00:48:46.000 >>: Set I totally agree, Patricia. 00:48:46.000 --> 00:48:48.000 And I know that we've used data for a whole lot of different things. 00:48:48.000 --> 00:48:49.000 Maybe one thing to show our staff ... 00:48:49.000 --> 00:48:51.000 where they need to be. 00:48:51.000 --> 00:48:56.000 So we've done things with that. 00:48:56.000 --> 00:49:02.000 Maybe we have used data to go to a potential funder and say, hey, we're 00:49:02.000 --> 00:49:07.000 looking at this and this and that and that might help prove where you need 00:49:07.000 --> 00:49:09.000 to be and then evening using some of that at the for 'future funding. 00:49:09.000 --> 00:49:14.000 Maybe grants ... 00:49:14.000 --> 00:49:15.000 that might be able to help supplement some of the other things you're doing 00:49:15.000 --> 00:49:18.000 too. 00:49:18.000 --> 00:49:25.000 So totally agree, data is power. 00:49:25.000 --> 00:49:27.000 PATRICIA: I'm going to reach out and 00:49:27.000 --> 00:49:33.000 go to the next question. 00:49:33.000 --> 00:49:41.000 So I want mare try to handle that. 00:49:41.000 --> 00:49:48.000 But one of the things that we've done that's one of the documents that came 00:49:48.000 --> 00:49:52.000 in your packet (air quotes) so to speak attached to this recording, in our 00:49:52.000 --> 00:49:57.000 hospital to home program we collected quite a bit of data. 00:49:57.000 --> 00:50:03.000 And we went to at one hospital -- and we went to another hospital and said 00:50:03.000 --> 00:50:07.000 and we wrote it all in words and percentage they said hey we want 00:50:07.000 --> 00:50:09.000 pictures and we were like picture and it occurred to that has they were 00:50:09.000 --> 00:50:16.000 asking for an info graph. 00:50:16.000 --> 00:50:21.000 And if your pact you will see the info graphic that we put together for 00:50:21.000 --> 00:50:29.000 Centura hospital in Colorado Springs and it talks about things like how 00:50:29.000 --> 00:50:37.000 much annual savings per patient so the average savings for the initiate we 00:50:37.000 --> 00:50:38.000 worked with was 14,000 there and in over 2 years we projected we would 00:50:38.000 --> 00:50:41.000 save them ... 00:50:41.000 --> 00:50:43.000 We talked about how much it would cost 00:50:43.000 --> 00:50:52.000 per patient and the number of days. 00:50:52.000 --> 00:51:00.000 We talked about payors, who were the payor, we had 56 percent of our payors 00:51:00.000 --> 00:51:02.000 for the this were insurance including no insurance, we could show where the 00:51:02.000 --> 00:51:04.000 insurance or no insurance was coming from. 00:51:04.000 --> 00:51:13.000 And then in our program the biggest 00:51:13.000 --> 00:51:17.000 group was the under or uninsured, 56 percent of the patients we worked with 00:51:17.000 --> 00:51:20.000 with disabilities did not have insurance and we were able to work 00:51:20.000 --> 00:51:25.000 with them to get on Medicaid, whatever we could do. 00:51:25.000 --> 00:51:29.000 So that kind of data really particularly for your customers (air 00:51:29.000 --> 00:51:37.000 quotes) the people who are paying, they want to see the impact, on your bottom 00:51:37.000 --> 00:51:42.000 line, they want to see the impact on the consumer, as well, but really if 00:51:42.000 --> 00:51:48.000 you're going to a business selling a service, they want to know how it's 00:51:48.000 --> 00:51:48.000 going to impact their bottom line you have to have the data to support that 00:51:48.000 --> 00:51:51.000 and show it. 00:51:51.000 --> 00:51:56.000 So that's how we can use data. 00:51:56.000 --> 00:52:03.000 We have a report annually on the hospital to home that talks about the 00:52:03.000 --> 00:52:06.000 demographic, that talks about the types of disabilities and conditions, how 00:52:06.000 --> 00:52:16.000 long they were in our program. 00:52:16.000 --> 00:52:21.000 And how many people were rereadmitted which has been very were low. 00:52:21.000 --> 00:52:26.000 And having that data has helped our current contractor continue the 00:52:26.000 --> 00:52:33.000 contract and I'll have to say when proposed this program, the CEO of the 00:52:33.000 --> 00:52:37.000 hospital hardly even spoke to me, a year later he came up to me at an 00:52:37.000 --> 00:52:40.000 event and gave me a kiss and said this program is great. 00:52:40.000 --> 00:52:41.000 And it's all because of data. 00:52:41.000 --> 00:52:46.000 Our performance for sure. 00:52:46.000 --> 00:52:52.000 So that data you can do a lot with anytime marketing, you can do a lot 00:52:52.000 --> 00:52:56.000 it with in terms of evaluating whether you met your goal, what do you need to tweak. 00:52:56.000 --> 00:53:01.000 As I say data is power. 00:53:01.000 --> 00:53:07.000 >>: Book this is great. 00:53:07.000 --> 00:53:14.000 We're get a lot of questions about data in the Q & A so our first one is 00:53:14.000 --> 00:53:25.000 from she quo yeah which asked has anyone tried to ... 00:53:25.000 --> 00:53:28.000 (reading verbatim) . 00:53:28.000 --> 00:53:36.000 DAISY: We haven't at access living. 00:53:36.000 --> 00:53:39.000 JOE: No, we haven't either. 00:53:39.000 --> 00:53:41.000 PATRICIA: I'm not sure what that is. 00:53:41.000 --> 00:53:45.000 Maybe I'm the last one to know. 00:53:45.000 --> 00:53:55.000 I'm not sure -- a service learning center? 00:53:55.000 --> 00:53:56.000 BROOKE: She quo yeah, if you could 00:53:56.000 --> 00:53:58.000 provide more information. 00:53:58.000 --> 00:54:01.000 I'm also not familiar. 00:54:01.000 --> 00:54:02.000 In the chat. 00:54:02.000 --> 00:54:09.000 JOE: There are some software programs 00:54:09.000 --> 00:54:14.000 out there that help you choose data based on program information and 00:54:14.000 --> 00:54:19.000 things like that and also help you see trends, it proved you with automatic ... 00:54:19.000 --> 00:54:22.000 and one of them is French word that I can't think of right now. 00:54:22.000 --> 00:54:23.000 But there is. 00:54:23.000 --> 00:54:31.000 >>: Hi. 00:54:31.000 --> 00:54:36.000 So a service learning program is actually a program that is in 00:54:36.000 --> 00:54:42.000 partnership between nonprofits of the community and a local community 00:54:42.000 --> 00:54:49.000 college or university and what they do is it provides credits and hands on 00:54:49.000 --> 00:54:54.000 learning for people going into like human services or social work and 00:54:54.000 --> 00:54:57.000 where they come in and have nonprofits for things they don't have enough 00:54:57.000 --> 00:55:00.000 manpower such as program evaluation. 00:55:00.000 --> 00:55:02.000 So it's a win-win for everybody 00:55:02.000 --> 00:55:06.000 essentially. 00:55:06.000 --> 00:55:11.000 JOE: That's great. 00:55:11.000 --> 00:55:15.000 We have taken advantage of lots of programs like that but never for 00:55:15.000 --> 00:55:17.000 program evaluation so I appreciate that thought very much pause that could 00:55:17.000 --> 00:55:19.000 certainly be helpful. 00:55:19.000 --> 00:55:28.000 BROOKE: That's great. 00:55:28.000 --> 00:55:33.000 . And we have Yours truly question. 00:55:33.000 --> 00:55:35.000 we have ... 00:55:35.000 --> 00:55:50.000 are there resources ... 00:55:50.000 --> 00:55:57.000 that's where it seems to break down at my CIL (reading verbatim) ... 00:55:57.000 --> 00:55:59.000 JOE: That's a good questioned. 00:55:59.000 --> 00:56:05.000 These are really good questions. 00:56:05.000 --> 00:56:07.000 I think to me it breaks down because the effort and time it really takes to 00:56:07.000 --> 00:56:12.000 analyze it and use it effectively. 00:56:12.000 --> 00:56:14.000 I don't know if I can think of any resources that help you with that 00:56:14.000 --> 00:56:20.000 evaluation and analysis of your data. 00:56:20.000 --> 00:56:25.000 Daisy, Patricia, anything you can think of or Seth. 00:56:25.000 --> 00:56:26.000 PATRICIA: This goes back to Sequoia 00:56:26.000 --> 00:56:32.000 was suggesting. 00:56:32.000 --> 00:56:39.000 You can go to your local college, I'm pretty sure they have a statistics 00:56:39.000 --> 00:56:45.000 program there, and they will welcome projects that students can work on to 00:56:45.000 --> 00:56:48.000 help you set your data up or if you have your data, help you interpret it. 00:56:48.000 --> 00:56:50.000 And so to me that's a great resource. 00:56:50.000 --> 00:56:52.000 If you're looking to do this without 00:56:52.000 --> 00:57:00.000 paying very much. 00:57:00.000 --> 00:57:09.000 You can also hire a statistician to help you, a program evaluator, they're 00:57:09.000 --> 00:57:12.000 out there at a United States way or your local nonprofit association will 00:57:12.000 --> 00:57:16.000 be able to help you with that as well. 00:57:16.000 --> 00:57:22.000 >>: I would just add if you've 00:57:22.000 --> 00:57:25.000 engaged stakeholders from the beginning in the process, then it becomes more 00:57:25.000 --> 00:57:28.000 natural just happens that you share that data better and use that data 00:57:28.000 --> 00:57:32.000 after the fact because your customers are going to hear about the data they 00:57:32.000 --> 00:57:38.000 were interested in and your board is going want to hear about the ... 00:57:38.000 --> 00:57:42.000 so if your engaging stake holders and coming up with good relevant 00:57:42.000 --> 00:57:44.000 questions, that should happen a little more naturally. 00:57:44.000 --> 00:57:51.000 BROOKE: And I'll just chime in too, a 00:57:51.000 --> 00:57:55.000 lot of survey tools they have like a reporting feature, things a little bit 00:57:55.000 --> 00:58:02.000 et ceteraier F you're just look to get started so they can pull together 00:58:02.000 --> 00:58:08.000 these nice reports that sometimes lets your filter out partial responses and. 00:58:08.000 --> 00:58:15.000 So the one that comes to mind is survey guess mow, equal direction, 00:58:15.000 --> 00:58:17.000 gizmo ... 00:58:17.000 --> 00:58:21.000 awesome. 00:58:21.000 --> 00:58:24.000 And we have I know we've running sort 00:58:24.000 --> 00:58:31.000 on time so we have one other question. 00:58:31.000 --> 00:58:36.000 So for those at CILs who have implemented permanent evaluation, what 00:58:36.000 --> 00:58:41.000 has been most helpful to gain knowledge, guy sans and support for 00:58:41.000 --> 00:58:42.000 getting started with evaluation and troubleshooting along the way. 00:58:42.000 --> 00:58:45.000 So peers, training ... 00:58:45.000 --> 00:58:48.000 and this is from Tracy. 00:58:48.000 --> 00:58:52.000 DAISY: So I would just say go back to 00:58:52.000 --> 00:58:56.000 something that Joe and Seth said which is keep it simple, especially at first 00:58:56.000 --> 00:58:57.000 and try piloting something at one program in uracil. 00:58:57.000 --> 00:59:02.000 You know. 00:59:02.000 --> 00:59:08.000 I don't actually -- I mean if you have the money, you can use an outside 00:59:08.000 --> 00:59:11.000 consultant and it probably would be helpful but I don't see you should see 00:59:11.000 --> 00:59:20.000 that as an impediment getting started ... 00:59:20.000 --> 00:59:22.000 there's good resources that I think Seth and Joe and Patricia have 00:59:22.000 --> 00:59:26.000 provided about program evaluation. 00:59:26.000 --> 00:59:26.000 Start somewhere small for your center. 00:59:26.000 --> 00:59:33.000 JOE: I would agree. 00:59:33.000 --> 00:59:37.000 A lot f what I learned about program evaluation is trial and error. 00:59:37.000 --> 00:59:43.000 And that CDC website is a great tool. 00:59:43.000 --> 00:59:44.000 Very simple, they use very concise language and keep it very simple for 00:59:44.000 --> 00:59:46.000 you to follow. 00:59:46.000 --> 00:59:50.000 And I would also just say like the 00:59:50.000 --> 00:59:52.000 best thing you can to is engage stakeholders in the process. 00:59:52.000 --> 00:59:56.000 Don't try to do it alone. 00:59:56.000 --> 01:00:02.000 I think that's a mistake many people make. 01:00:02.000 --> 01:00:04.000 ... 01:00:04.000 --> 01:00:14.000 and you'll have a better time. 01:00:14.000 --> 01:00:19.000 SETH: Then there might be some 01:00:19.000 --> 01:00:21.000 friendly CILs out there that might be able to help you along the way. 01:00:21.000 --> 01:00:24.000 >>: I think we happen to be looking 01:00:24.000 --> 01:00:28.000 at a few of them here (laughter). 01:00:28.000 --> 01:00:33.000 So in the last minute here, Sandra is 01:00:33.000 --> 01:00:41.000 wondering if anyone is willing to share their strategic plan? 01:00:41.000 --> 01:00:46.000 And if that is so, we would be happy to post it up with the website 01:00:46.000 --> 01:00:49.000 materials or we can hunt Sandra down and send it to her. 01:00:49.000 --> 01:00:52.000 But, yeah, that would be really great if we can offer that. 01:00:52.000 --> 01:01:01.000 And I just wanted to say thank you 01:01:01.000 --> 01:01:06.000 much to all of our panelists and to I ILRU and I Lnet for sponsoring. 01:01:06.000 --> 01:01:11.000 Speak of evaluations we do have one in the chat box for you to fill out 01:01:11.000 --> 01:01:15.000 and there will be one that shows up in your becauser when you close this 01:01:15.000 --> 01:01:23.000 session, it's the same one so you can choose eater way to do it. 01:01:23.000 --> 01:01:24.000 . 01:01:24.000 --> 01:01:26.000 Thank you everybody again.