Institute of Museum and Library Services 2020 Grants to States All States Conference May
12, 2020 Virtual Proceedings by: CASET Associates, Ltd. caset@caset.net
Agenda Item: State of the Nation: Analysis of Grants to States’ SPR Projects
MS.
DEVOE: Hi, this is Teri from IMLS.
Let me just introduce our next speaker, who some of you have met at prior Grants to States
conferences.
Lisa Frehill is not part of our Grants to States team, but she is a wonderful asset
to our program.
She is from our Office of Digital and Information Strategy and is our senior statistician there.
So along with Matt Birnbaum, she is going to be presenting some data analysis from your
wonderful State Program Report projects.
So take it away, Lisa.
MS.
FREHILL: Thank you so much, Teri.
I want to say it is a real privilege to be able to work with the awesome folks in the
Grants to States program and to share this data with you guys that many of you spend
many hours entering into the system.
So I am always very interested in hearing anything that you have to say or any questions
that you might have on it.
This is the fourth turn of the crank.
So basically the way we started, we give you a little bit of an overview of what the SPR
framework.
And thank goodness, the talk that Madison and Michele just gave, gave you a bit of an
introduction to that already.
So I can fly through those slides wicked fast.
And then I am going to talk a little bit about the SLAA Groupings to a high-level summary.
And then go into the description profile which is basically what your basic journalistic
type questions are.
Who, what, when, how, why, and how much?
So there are some optional fields.
I am going to try to make sure that we keep track of some of the caveats.
So one of the other things that we wanted to do was to give you a number of questions
to be keeping in mind as I go through the slide deck.
If you looked at it already, there are a lot of slides.
There is a lot of data.
So the idea is that a whole bunch of trees in this forest.
Please help me understand what the whole forest really needs to look at.
So what are the stories that are in these data that seem to really capture things best
for you in your state or territory.
Are there tools that would help us to make these data more useful to help you inform
your outcomes?
What support do you need from us with data?
And then something we haven't done, and I was hoping to do it this year, was to do a
little bit more data linking in terms of the system has been set up to permit us to do
some linking with the public library survey data, common core data and IPEDS which is
the higher education survey data.
So we haven't done that yet.
We would be interested in hearing what your thoughts are on that.
So those are some questions to keep in mind as I race through the data slides.
So the SPR, it has a number of very interesting analytical capabilities which thrills me to
no end quite obviously.
There are 14 intents, and these can be rolled up to six focal areas.
But at the same time, we can take those intents and drill down into 38 different subjects.
There is also, for no extra charge, three different levels of analysis.
We can analyze the data at the activity level, the project level, and then for states and
territories.
So the intents, I am not going to go through all of these.
I think that this was covered a little bit in the previous talks.
There are 14 intents.
The user chooses an intent.
And then those intents get rolled up to these six focal areas.
And then the top three of these focal areas which later in this presentation you will
see me referred to as the big three really then align very closely with IMLS strategic
goals for 2018 to 2022.
And then there is what I sometimes call topical focal areas relate to human services, employment
and economic development and civic engagement.
And then there are subjects.
So as I said, each of the intent is also related to subject.
There are 37 subjects plus another, and once the user enters an intent, they can also choose
a couple of subjects.
These come in extraordinarily handy.
And my colleague, Matt Birnbaum's talk tomorrow will have used some analysis based on our
analysis on these subject, namely digital literacy and broadband adoption.
In the past, we have done some work looking at summer reading programs, databases, and
a little bit of work looking at some other topics as well.
When we look at the SLAA groupings, we look at the general allotment levels.
Now, a few years back in 2017 and 2018, and those two conference years, we had started
saying, oh, well, let's do large and small by whether they do subgrants, you know, predominantly
subgrants or predominantly SLAA awards.
And we what we heard from the states and territories was you know something, some of us do subgrants
one year, and then the next year, we go to predominantly SLAA.
So that is not a persistent quality.
So we did have a sense that we wanted to kind of bend things and to start to understand
that there were some interesting differences or some differences across these different
groups.
And so what we ended up with was, well, let's look at the allotment levels.
And you can see that we have these three allotment levels, smaller, larger, and what we call
mega, not to be confused with mega Godzilla, just because those are so much larger than
the larger.
They really do differentiate.
And you can see down below, I have got the 2018 population there.
And on the 2018 population, in the 28 states and the five territories outlying areas, they
had a 2018 population of less than five million.
Those that were in the larger, five to 13 million.
And then these four megas, California, Florida, New York, and Texas, all had very large populations.
The allotment sizes are quite a bit different.
And then as you can see, the average number of project activities for each project is
a little bit different.
They are about the same for the larger and the smaller states, but when you get into
the mega states, they have slightly more activities, another half of activity for project than
the others.
So now, as I mentioned, this is going to be our description profile.
We are going to go into how much, why, who the partners are, what kinds of activities
there are, the types of activities, where.
And then a little bit about the subject code.
So the optional fields are partners and locales.
Not all projects report those.
Not all activities report those.
And then as I mentioned before, we are doing a focus this year on broadband and digital
literacy.
So this is what we tend to show as our sort of high-level overview.
I am not going to spend a lot of time on this because I end up having graph after graph
after graph showing a little bit more detail.
But I am going to point out just a couple of things given I can remember how to point
things out on here.
So you can see that what we have seen is that over time, FY15 was a pilot year, and then
14, everybody entered.
But over time, the numbers of projects have gone down.
And as well, the number of activities.
And that is really attributable, to a large extent, sort of learning as people have learned
how to really work with the system, how to enter it.
The excellent advice that Michele and Madison gave you earlier, those kinds of things, people
are gaining a knowledge base on that.
Now, 70 percent of projects had about one or two activities.
So this one, you can't read on the screen and don’t even try.
It is in the deck.
You can then have access to the data, and the next couple of visualizations will show
you some of the trends here.
One of the things, even though you can't see it, what we are seeing is that fewer projects
are specifying more than one intent.
And that has been a real big gain in terms of learning about how to use the system.
So this is a chart that needs a little bit of explanation.
And what you are going to see is that this dark bar is the LSTA funds, median project
budget.
And then the light bit on top is the other funds, but includes your state match, your
in-kind match.
This number up at the top represents the total median budget for the project for that fiscal
year.
And then this percentage down here, this shows you what percentage of that total budget is
from LSTA funds.
So there is a lot going on in this particular chart.
And you can see that the big thing is that we get this kind of interesting finding.
And I am obviously not quite sure why this is.
But the states that were in the 19 large states that were kind of in the middle there, their
projects tend to have lower budgets.
And now, in 2018, they have a proportionally higher spend from LSTA funds.
68 percent as compared to the 62 percent for the megas and the 58 percent for the small
states, the states with the smaller allotment.
The median project budget is higher in the small states.
So it has been kind of interesting.
The percent overall, just to give you the benchmark here, about 68 percent is from LSTA,
but 66 percent in 2018.
So we get that kind of an interesting finding there.
And I would entertain what ideas people have about why this might be the case.
The other way that we have looked at budgets, and our original thought on this back when
we started doing this analysis was, well, maybe there are certain types of projects
that we might be able to identify that states could do quote unquote on a shoestring.
So hey, what are some projects that might be under say $25,000 or under $7,500, like
some very inexpensive projects.
Because the notion in the system, my understanding from the get-go, was that this would provide
a platform by which people could share information about projects.
So we have been binning them by budget category.
And as you can see, what we have got here is that, again, looking at the allotment size
group that consistent with the last chart, the large allotment size group tend to have
smaller budgets on their projects.
They undo the other two.
And the 49 percent of their projects have a total budget of less than $25,0000.
So now, we are going to switch gears.
We are going to talk a little bit about the activities.
So up to now, it was about projects.
Now it is about the activities per project.
And this is really where you see another big difference between the allotment size groups.
And you sort of saw that in that last table where you really couldn't see the numbers
anyway.
But you can see them a little bit better here.
And that is that for the mega group, you can see that 57 percent have one or two activities
per project which is quite a bit less than the small and the large.
And that one-third of them have three to five.
And that 10 percent of them have six or more.
And having six or more is really a characteristic of the very large state.
You don’t see it too much in the small states and in the large states.
So we are done talking about the different size allotment groups and how that affects
things.
Now, we are going to spend a lot more time digging into things like focal areas, activities,
who, how things get done, things like that.
So as I said before, what we have got is two sets of focal areas that -- there are six
focal areas that roll up those 14 intents.
And you can see we have got the big three, information access, institutional capacity,
and lifelong learning.
And those tend to really dominate the numbers of projects that are there.
The big drop that we see is that in FY18 versus FY15, there is a decline by 19 percent in
the number of information access projects.
Kind of hard to see much in the trend down here in the sort of topical categories, human
services, civic engagement and economic development.
But what we see is that because they are so small, they really have not changed in terms
of relative size very much over this time.
So now, we say, okay, well, how many states or territories have projects in each of the
focal areas.
And as you can see, almost everybody has something in institutional capacity, nearly everybody
has something in information access and lifelong learning in the big three.
But then, we have quite a bit fewer states who have projects in the topical areas, in
civic engagement, human services and economic development.
And really, where we have seen the gain over the past couple of years is in civic engagement,
and we have seen a decline in the numbers of projects in human services.
That said, what is kind of interesting is that the median LSTA budget by those focal
area is very different.
So if you recall, I said, well, human services, we have seen a decline in the number of projects.
But we have actually seen the LSTA budget for those, is higher than that in civic engagement
and economic development.
And the highest LSTA spent is in the area of information access, and that is where a
lot of database projects are going to pop up.
So in this chart, this shows you, very similar to the last chart, although I didn’t put
the totals up at the top.
Again, you can get a sense of the relative budget of each of the different focus areas
projects.
And these are the big three.
These here are the big three, information access, institutional capacity and lifelong
learning.
But you can see that they vary greatly in terms of the magnitude of their budget, their
total budget.
Just like the other chart, the dark part shows you the LSTA funds, and the light part shows
you the non-LSTA funds.
And then this percent gives you what percentage of the total budget is from LSTA.
And so even though these are more expensive, they use proportionately less LSTA funds than
the lifelong learning projects.
The same thing we see in human services where about half of the budget is from LSTA funds,
but they are much more expensive projects than are the projects in civic engagement
and economic development.
Economic development are the projects that have the highest percentage of budget from
LSTA funds.
So we have been talking about projects.
Now, we are drilling one step down into the activities.
And if you recall, for many states, 70 percent of the projects have one or two activities.
It is just a handful that have more than that.
So when we talk about activities, we can say, well, what types of activities?
And there are four types of activities in the SPR.
There is content, instruction, planning and evaluation, and procurement.
And this is quite literally how they stack up.
And the most common thing that you see here, the most prevalent thing, is the fact that
the relative mix has not really changed markedly over the past four years in terms of what
kinds of projects or what kinds of activities are engaged in.
The one big difference that we do see is that in FY16, a little bit more slightly higher
percentage and larger number of activities were related to evaluation.
And that of course is when the evaluations for the five-year FY17 year-end projects were
coming due.
So why?
So now, we can do those same stacks, so the types of activities by focal area for the
activity.
And so again, we have got the big three.
And you can see some real dramatic differences in terms of how they allocate across the different
types of activities.
That institutional capacity has a lot more to do with content or with instruction.
And that information access has to do with content.
Getting more information, that is where those databases are showing up.
And that lifelong learning has quite a lot to do with instruction.
And then here to, you can see that light blue for instruction is prevalent in the topical
areas as well.
So now, if we drill into, just look at the FY18 data, and these are another way to look
at those stacks.
And hopefully, your eyes are not glazing over yet.
It is kind of interesting because this gives you a sense of just how different the focal
areas are in terms of the types of activities that end up sort of rolling out.
And here, I have included the procurement which procurement is only allowed under institutional
capacity.
But you can see that it accounts for about 10 percent of the institutional capacity project.
And those are the big three.
If we look at the smaller ones, civic engagement, human services, and economic development,
we can see that again they are overwhelming instructions.
But you can see quite a lot of content associated with the economic development.
So activity locale, in the system it asks is this a statewide or not a statewide project
or activity.
And so what we have seen over time is that the relative percentage of statewide projects
has increased, even though as we see, the numbers of activities that are in the system
have actually decreased.
So now, as you can see, 47 percent of the activities in the STR are statewide.
So locales are one of the optional items.
And so this shows you what types of locales, where the activities are implemented and gives
you the public libraries lead the way in terms of the 42 percent of FY18 activities had as
at least one of its locales, the public library.
And I must note that this is 42 percent of those is set by locales.
They could specify more than one public library.
So this isn't the whole total number of public libraries.
This is the number of activities that have specified that a public library was one of
its locales.
We are seeing a bit increase in terms of the percentage that have specified the SLAA as
a locale.
And that is not a surprise given this previous chart that was showing that there were more
statewide activities.
And the SLAA becomes one of the important locations for that.
And then the proportionate activities as an academic, a school or a special library has
increased by about 3 percent points between FY15 and FY18.
So partner areas are an optional item.
About 56 percent of activities in FY18 reported them.
That is a little bit higher percentage than reported them in FY15.
And FY15, just 52 percent of the activities had a partner area that was reported for it.
And you can see that local states, governments tend to be predominant when a partner area
is specified, non-profit has stayed fairly constant.
I know it is hard to tell because the bars are showing you the numbers of activities
and then the percentages are showing you what percent of the activities that specified it.
And as another point here, like the other chart, an activity can specify more than one
partner area.
And then partner types, so not all of them who specified a partner area specified a partner
type.
Just 38 percent specified a partner type.
But when they did, they overwhelmingly specified libraries as a partner type.
And then a very small percentage did specify museums, cultural heritage organizations and
historical societies as partners.
MS.
GONSALVES: A question popped up.
The question is, would tribal libraries be considered other in most states?
MS.
FREHILL: Actually, that is a very good question.
I would probably want to see what folks would have said in the states because in a sense,
you would have to cross the partner area, which is on this chart, because you can have
a partner area as Native American, American Indian, Native Hawaiian organization with
the partner type.
And I think that that could tell you whether it is a tribal library.
So that is a really good question and that sounds like something that I could have about
20 minutes of fun with sometime.
So I hope somebody writes that one down so I don’t forget it.
But thank you, that is a really good one.
Because sometimes, you could have the same thing like helpful heritage organizations.
Could be tribal organizations as well.
So who are the beneficiaries of the activity?
And you can have the public, and within public, there could be general or targeted.
If there was specific like group that is the focus, and then the library, I say library
workforce, but really this is library more generally.
And you will see that the moment the moment when I get to the next slide.
And so what we see is that over time, there has only been a slight decrease in the percentage
of the activities that are for targeted public audiences.
And a slight increase in the relative proportion of activities that are focused on the library
of an audience.
MS.
GONSALVES: One quick question.
Someone wants to know what actually defines other.
MS.
FREHILL: That is a very good question.
I was actually in the SPR book earlier today, trying to track that down myself.
So I will ask if Matt has some probe of wisdom on that one.
MR.
BIRNBAUM: I don’t have the SPR in front of me, but I am 90 percent certain it is a
type of content activity.
It is the mode under content.
MS.
FREHILL: Okay.
But I think is the question you are asking about this chart, though, where it is like
the partner type says other.
So actually, I think people can fill in other --
MR.
BIRNBAUM: Yes.
This would be partner was not one of the other categories.
MS.
FREHILL: I saw some other things listed as I was looking at the verbal entries there.
And I could make a list.
But it is a variety of things that don’t fit any of these categories.
I will see if I can circle back with you, Joy, on that.
So a moment ago, I mentioned that like it is really not just the library workforce,
but library in general.
And in terms of you look at the activities, about 40 percent in FY18 focus on the library.
And of these, we break this down by the focal areas.
And then within the institutional capacity focal area, which is where most of them are,
almost two-thirds, we further drill down into the intent.
So this is one of the only places within my presentation where I drilled down to the intent
at all.
And you can see that when we look at the 1,182 activities that were focused on the library,
about 69 percent of the 63 percent were specific to the library workforce.
But that quite a number of them really relate to improving library operations, accessing
information, lifelong learning are the topical areas.
So it is important to bear in mind that some of these activities have a lot to do with
providing the support to library staff to be able to implement programming related to
topics or to provide access for patrons to lifelong learning or other information sources.
So if we look at the types of library workforce activities, almost two-thirds of the ones
that are instructions have to do with institutional capacity.
And in this case, the “all others” includes the other focal areas, namely lifelong learning
and the three topical areas.
So you can kind of see the breakdown here of how the institutional capacity continues
to be an important focal area.
And instruction is one of the primary ways in which the activities are implemented.
So when we look at public activities, so now, the public activities account for about 60
percent of the activities that are reported in the SPR.
And as I mentioned earlier, the relative percentage of those that were for targeted groups has
declined slightly, and those for general groups has increased.
But what we can see is that the way in which they are distributed across the different
focal areas really varies.
So that 65 percent of the lifelong learning activities for the general public are targeted
at a specific group, and that is much higher than say the institutional capacity or the
information access funds.
And I see a question, the difference between content and procurement.
So content is a number of ways in which content actually -- it happens.
So procurement is actually purchasing things, whereas content could be like digitizing things.
It could be acquiring like curriculum materials and things like that.
So I think that there is a very subtle difference there.
Procurement is only for institutional capacity.
So if you obtain content that is used for a program for the public, and it is for something
other than institutional capacity, it gets lumped into the content category as an acquisition.
So if we go back, 60 percent are public activities, and what we have is, okay, so among those,
817 of them are targeted.
So 817 represent 28 percent of all activities, 47 percent of the activities for the public
are targeted.
What types of targets are the focus of these?
And age groups is really the number one thing.
And what this is telling you is that 37 percent of the 1,754 public activities are targeted
at some particular age group.
And that is more than one-third of them.
Then if you look down, you can see that the activities that are targeted at say immigrant
groups or specific ethnic groups, are relatively small in terms of the numbers.
So if we drill into the age groups that are targeted by public activities, what we see
is that the youngest age group, 0 to 5, preschool age
group, has actually had the biggest change since FY15.
We have seen an expansion there in terms of the percentage of activities that target early
learners.
As you can see, the general distribution is about the same in FY18 versus FY15.
And then if we look at ethnic groups that are targeted by public activities, what we
see is that Hispanics are the largest group that is targeted.
But that over time, what we have seen is proportionally fewer activities are targeted at various ethnic
groups.
But again, the distribution is about the same in both FY15 and FY18.
I am going to pause there real quick because I have been seeing some of the questions come
up and I am not sure if I have got them all answered.
MS.
GONSALVES: Terry wants to know, didn’t the SPR change in 2016?
Was this included in the assessment of the number of projects, et cetera?
MS.
FREHILL: That is a good question because I am not certain.
Matt, what material change for FY16 would there have been that Terry is referencing
here.
MR.
BIRNBAUM: Yes.
Sure.
That is easy.
The 2014, we started piloting the new SPR with about 15 states.
In 2016, all the states joined the new system.
MS.
FREHILL: So what you were talking about, Terry, was the year 2016 rather than the reporting
of the SPR, reporting year.
Okay.
PARTICIPANT: I believe all the states began to report in FY16.
MS.
FREHILL: Well, they all reported their FY15 data.
It is just that they would have done that in 2016, right?
Keeping our like FYs and calendar years, reporting years squared away.
Any other questions before I like jump into output because this is the fun thing that
I messed with this year that is a little different.
MS.
GONSALVES: I do not see any additional questions.
MS.
FREHILL: I am looking at the Q&A right now and trying to keep trac of that.
The thing I can't see is the clock.
I am sorry.
I just can't see the green thing right now.
I am just going to leap on forward into activity output.
So there is about 40 some odd different output that are available in the SPR.
And these outputs, those of you who have reported them, you see how this rolls out in terms
of, okay, so once you have indicated the activity and said what type it is, what the mode was,
what the format was, all of that kind of stuff, then you have a block where you enter the
activities.
And in fact, in Madison and Michele's presentations, they had a screenshot that showed you that
which I think would be really helpful right about now as a visual.
But what ends up happening then is that you have 40-some odd activities or activity outputs.
So to start doing a little bit of analysis on those outputs that goes beyond just saying
how many of different things came out of this which is what we kind of did for two years
in 2017 and 2018.
This year, I drilled into two types of outputs that seemed like you could sort of well define
them.
Program evaluations, and part of the reason there is that is something that is really
important to do in all of the projects and all of the activities.
And then databases.
The reason is that databases come up all the time.
And in fact, Jaime Bell, I know you are on, has put a bug in my ear that it would be really
great to do what we can to get a handle on, on how much all the databases cost.
So what I did was program evaluations, like I said, it is fairly well defined.
Those outputs are all very well connected to a program evaluation activity.
Databases, on the other hand, I had to do some limiting.
So I am not looking at all the databases that got reported as outputs.
I am only looking at the databases where it was an activity where only databases were
required, and that this was only one activity.
So that way, we were just saying, okay, we are going to really limit ourselves.
And then the metrics that we assembled were how many states reported this, how many activities,
and then how many of them did we get, how many evaluations, how many databases.
What were the costs, the total and the LSTA budgets?
And then we have got sort of what did each thing cost that we got?
So this is a first time of doing this kind of analysis.
And so no visualizations, just two tables of numbers.
And the data are not adjusted for inflation as any of the other financial data.
And basically, I just wanted to kind of point out a couple of things.
And that is that evaluations were reported by 30 to 38 states over the period 2015 to
2018.
A number more in 2016 obviously is the five-year plans were being evaluated.
And there were more activities obviously in 2016.
So 2016, we are going to see kind of a hot bed evaluation activity going on.
And so then we can look at the total number of evaluations.
So this is the number of evaluations that were reported as having been completed in
that year.
And then this is the total cost, total budget, the LSTA budget.
And then this cost per completed evaluation, just pick these numbers in their raw form,
not in these rounded numbers that I have presented here, and by the number of evaluations and
comes up with how much each one costs.
So you can see this kind of trend where the total cost per completed evaluation went from
$7,200 up to $21,000.
But then the median budget per evaluation has actually gone up quite a bit.
It has doubled.
And that is not a surprise as institutions start out with doing evaluations.
It takes a little time for that business practice to take hold.
And it takes a little time for evaluations to start to realize just how much a good evaluation
is going to cost them.
And by good, I mean once that they will be able to actually use and be able to make some
hay with.
So what we are seeing, though, is that kind of interesting trend, too, where the overall
percentage of the evaluation that was covered by the LSTA budget has gone from nearly half
down to less than one-fourth of the evaluation.
So that means that the states are investing in the evaluations themselves.
So that is the evaluation, little outputs analysis on that.
And then the databases which again this had to be limited to just only the outputs, all
of the databases that were acquired based on the number.
These are only activities for which it was a one activity project, the only activity
was related to the databases.
And so we can sort of done a one-to-one matching here.
And so here, you can see that databases are not cheap as you guys all well know.
They are quite expensive in that they have really fluctuated.
The per unit cost has really changed a bit over time.
But whereas the percentage of cost worn by LSTA funds on the evaluations is declined.
It has kind of fluctuated a bit here for the databases.
So that is a first crack at some outputs analysis just looking at databases and evaluations.
Are there any questions about that or any comments?
This is the first time we have presented these analyses to you folks.
MS.
GONSALVES: We don’t have any immediate questions right now, Lisa.
MS.
FREHILL: Okay.
I thought I would give people a chance.
I am glad to see that folks are appreciating that.
I see like a couple of folks saying, interesting, and thank you.
It is good to know my efforts are appreciated.
Thank you, guys.
So this is actually our second look at activity outcomes.
And we did the first one last year.
This is a very -- that is cool.
I am going to mention this before I get into the -- someone said, it would be interesting
to see database cost per use.
Yes, that would be fun.
So yes, if you have ideas about how I might be able to measure that, please let me know.
I think that inquiring minds would like to know that answer.
So I am going to jump into the outcomes now.
This is about questionnaires and Michele and Madison's presentation talked about it.
These outcome questionnaires are aligned with Project Outcome.
And the idea was that this way, it wouldn't, at least in theory, present a huge burden
on libraries to collect it because if they were already participating in Project Outcomes,
they would be collecting these data.
But there are only four instances where we have instructional activities for either the
general public or the library workforce, or we have content or planning and evaluation
for the library workforce.
So there are only four types of things where we collect this.
And you saw this in the previous talks.
I won't spend a lot time on this.
But you can see is the set you enter in the survey findings.
And so that was what the survey looks like.
And then this is what you enter in for the questionnaire.
When you pull these data, there are 108 variables across the four survey types.
The analysis that I end up doing means that there are a few caveats.
So number one, sometimes the survey didn’t really align with the type of activity.
So in those cases, I drop that data because I thought, there is something, I am going
to use a technical term here, funky going on.
And we don’t want to include that data.
The second thing is that there were some activities for which fewer than 10 responses were reported.
And so dropped those from the analysis as well because we know that those are going
to potentially be problematic.
And this is actually kind of a second pass we did.
This year, we did a little drilling into some of the intents with the library workforce.
And the other thing that we could be spending some time doing is looking at formats.
For example, the programs can be administered as either a virtual, an in-person or a combination
of virtual and in-person.
And so, we have an opportunity here to be able to look at whether there might be some
interesting differences and how those different formats are assessed by different audiences.
But that is sitting on the back burner at the present moment.
So the way that we do this is all of these scales that you saw here are what are called
Likert scales.
They are five-point Likert scales.
Strongly disagree, disagree, people sitting right on the fence in the middle, neither
agree or disagree, agree and strongly agree.
And so those five-point scales, you can think of it as having there is a strength of agreement,
and that the scale and the strength of disagreement.
I am going to go ahead and click through and get some more stuff up here.
So you can kind of see the scale here.
And you have got the neutral ones.
These are the people who said neither agree or disagree.
We put half of them on one side of the fence and half on the other side of the fence.
And then we have agree and strongly agree, and we have disagree and strongly disagree.
And we leave the non-respondents in.
I am going to show you why it is we leave the non-respondents in in just a moment.
So mathematically, this is how we represent the Likert scale.
These are the five questions that they are asked.
And these are for the general public activity outcomes.
There were 53,661 responses from 240 activities.
So what we tend to see happen in such instances is what is referred to as regression to the
mean, which means that there is a very high level of agreement, strong agreement on the
item.
Now, why is it that we put the non-respondents over here?
What tends to happen in surveys of this type is a social psychological thing called satis-ficing.
Namely, that people tend to not want to say something negative.
And so it sort of follows that old thing, if you don’t have something nice to say,
don’t say anything at all.
And that is what non-response is, that saying nothing at all.
However, in the particular case of this question, I am more aware of resources and services
provided by the library.
Non-response is actually also kind of a signal that within the context of a Likert scale,
it can be kind of confusing for respondents.
So around 30 percent of people did not respond to this item at all in terms of, like, well,
I sort of knew about this ahead of time.
But now, I can't say I didn’t learn more because, well, that is saying something negative,
and I really liked the project.
So they have this sort of difficult time really processing that, that they don’t want to
say something negative.
But to say something negative isn't necessarily negative, if that makes any sense at all.
So if we look at the library workforce, we had so many responses here on different types
of activities that I was able to say, okay, well, let me look at just in-person programs.
And we have got 8,200 responses for 73 activities.
And this is specifically for instructional programs that were focused on improving the
library workforce.
And you can see that again there is a very high level of agreement here.
But it varies in terms of, yes, they learned something, but they are less likely to have
the same strength of agreement that it will improve services for the general public.
But they still overwhelmingly say this.
So more than 80 percent of people said something positive on all of these four questions.
MS.
GONSALVES: Hi, Lisa.
David wants to know when you get a low response rate for a specific question over time, would
it be the time to re-evaluate that particular question?
MS.
FREHILL: You bet you.
I think last year was our first year, David.
We talked through these at the LSTA coordinators meeting.
And that last one that I was talking about, about the non-response, there were a lot of
options that people said, hey, well, they could not respond because of this or whatever.
So yes, it does signal that you need to kind of go back into the question.
I think what is interesting about these questions, too, is that there is a Likert scale item.
And those are the only ones that get reported in the SPR.
But then there are some open-ended questions.
And I think that those are probably potentially more useful on the ground at the point where
a program or activity is being actually implemented.
And also the open-ended questions can often suggest ideas for other sort of closed-ended
questions that one might wish to ask at a future time.
So that is a really great question.
And I don’t know what the timeline is for this.
I do know that because this is aligned with Project Outcomes, there is a larger context
in which this exists.
And then do we have a process in place to make changes?
I am going to go back to my colleague, Matt, on this.
Do we have a process of that sort, Matt?
MR.
BIRNBAUM: Lisa, I am just trying to digest the question.
Let me give a more complete answer as to why it is not so simple.
So the questions that we have around the surveys, these are aligned to those in Project Outcome
that are run by the Public Library Association.
And that was systematic, so that libraries who are either participating in Project Outcome
or are receiving a Grants to States grant, they wouldn't have to ask two sets of questions
to the same grantees.
So there were lots of thought and care to making sure that the questions that are in
the SPR match those, the Project Outcome.
We have not talked with PLA about what would that process be like for revisiting the questions.
But it is certainly a fair one to be considering in the next couple of years.
MS.
FREHILL: Thank you, Matt.
A lot of moving parts to this particular one, I think.
So now, I also set this up for instructional programs for library operations.
I am just going to kind of click through these.
These are the kinds of programs where it is the library workforce, but the intent was
to improve users, fill in the blank.
So these are those activities where they are doing an instructional program with the library
workforce.
And the intent is so that the library workforce can then do something that will help their
patrons.
And what is interesting is that this is a very, very strong level of agreement that
the library workforce folks did learn something.
And in fact, that they really felt strongly that this was going to hep them improve services
for the public.
And notice how the levels of non-response on these are really quite a bit smaller than
on some of the other questionnaires that we saw.
So this is an example of where we have taken the data, and we have sliced it a couple of
different ways to really get at more specific instances, more specific types of programming
that have been implemented.
This is that content acquisition and creation.
So only the library workforce gets asked these.
There are a small number of responses compared to the others and a number of activities.
And you can again see that the non-response is pretty small, relatively speaking and that
a fairly high level of agreement on it.
But there are only two questions on it.
And then this is the last type of questionnaire there is.
And that is ask the library workforce about the planning and evaluation activities.
And as you can see, it is only 12 activities that had 303 responses.
So it is a very, very small number.
But you see, it is kind of interesting because here, this is the one item on which there
is actually a bit more disagreement than on the others.
On all the others, you saw these tiny little orange and red yellow bars on this side of
the zero.
And here, about planning and evaluation, you see a higher level of disagreement with the
statements.
So on the one hand, you can see, you still have a very high level of agreement that,
at some level, addresses the library needs.
But they are not satisfied.
They are not as highly satisfied with the extent to which the planning and evaluation
activity addresses the library needs.
And then this last item is whether I believe the information from the planning and evaluation
will be applied addressed library needs.
Then you can see there is a little bit going on with all three of these questions together.
So it really is the case.
Based on my sort of gut feel for what we are seeing with these questions, it is really
important to slice them, to try to control as much as possible for the intent.
This one is a much tighter control set of activities, planning and evaluation versus
these others in these other charts where we are looking at instructional programs.
This is a real mixed bag of activities.
This is everything from a citizenship thing to like a healthy diet thing to a yoga, all
kinds of things.
I do see the yellow thing, so I am going to go ahead.
And back to the questions that I started with.
I apologize for not really keeping track of your questions as well as I could have.
But I guess I would throw this back at you and say, are there things in here that you
would like us to be spending more time calling attention to?
Are there things that we could do to best provide these data for you to support your
project?
And I will leave it at that.
And thank you all very much for your attention.
MS.
DEVOE: Thank you, Lisa. this is Teri.
We are getting right to the break time now.
So if you have comments or questions, Lisa put her information on the last slide here.
I will just advance it there, Ifrehill@imls.gov.