on the line at the moment uh we are recording just to let everyone know
and um as we're joining we were kind of dipping into our theme for the
conference which is travel related um all of the grants to states folks
like many of you out in the states miss being in person
this year we were really sad to have to do this conference again a second time
virtually um so instead we're going to have a
virtu a travel theme sort of in our minds and our icebreaker
was if you want to tell us where your you
know a memorable trip was we'd love to hear
about it in the chat box some of you may be joining under my name
we're not sure why that's happening but if you can identify which terry
devoe you are you can click on the three dots
in your picture area and change your name and add your state initials while
you're at it so i want to first introduce the team
and we've got a nice picture of all of us here you'll see some of our
faces throughout the conference dennis if you could just advance the
slide when you have a chance thank you so much um and one
more to the team photo gallery hello there we are um so welcome from
grants to states uh this is our small but mighty team
from left to right we've got myself terry devoe the associate deputy
director our three program officers michelle
farrell dennis nangle and madison bowles and
laura mckenzie who is our newest team member and the program specialist for
our team so we are so delighted that you are here
if we could go to the next slide i have a few quick announcements
um i have noted that our sessions will be recorded and we're going to work on
getting those posted to the website as quick as we can
it may take a week or so so just bear with us
if you haven't done so already you can add your state initials to your name so
that everybody gets to know where we're all coming from
we would ask that you'd mute your microphone when you're not speaking
we can do it for you if we hear that there's some noise
but uh we encourage you to use mute um as needed we're gonna keep the chat
box open between sessions um but and you know i've sort of
prompted for this initial icebreaker about great places you've traveled and
i've seen alaska and um all different parts of the world and
so thank you for sharing your exciting trips with us that make us
nostalgic for getting on planes and going to exciting places
um but as we sort of enter into our session time we would ask
to keep the sort of back and forth cross chat to a minimum and keep the chat
focused on questions around the session or things
that would clarify the content that we're talking about and
simply because we're going to be monitoring the chat
for these questions and it'll help us to sort of scroll through
and see that everything's actually related
if you have questions that end up being specific to your state
it might be better to follow up individually with your program officer
again that would be dennis or michelle or madison
um and we know that this isn't a typical conference
um one of the joys would be if we were together you would have a
chance to talk to each other and network and get
sort of those hallway conversations in and while we don't have as much of that
in our virtual format we do have a time tomorrow well you'll
get you'll get a chance to talk to each other
uh we also know that kosla is planning some member forums
this month so that the chief officers in particular will have a chance to talk to
each other and um as i mls we're going to continue
hosting zoom calls for the lsta coordinators
every other month we haven't sent out the newest schedule yet
but we do anticipate that the next one will be
tuesday april 20th at 3 pm eastern time so we'll be sending out a message with
the new uh lsta coordinators meetup schedule
via zoom and those will be at different times of the day and different days of
the week to sort of accommodate different schedules so that is coming
the next one is tuesday april 20th at 3 pm eastern
for this session and many of our sessions we're going to be inclined to
save questions until the end so if you have
questions you put in the chat box and they don't get answered immediately
don't worry we will come back to them um if there
are questions we don't get to answer during the session time that's allotted
we plan to comb through them and follow up
within a few days after the conference
we're going to use this time and dennis if you would just advance the slide
we have a half hour to kick us off and with that time we thought the best use
would be to to frame and talk about the american rescue plan act funds
which are probably top of mind for you as well
we only have a half hour to do this we only had two weeks
notice that this was happening compared to when our conference was scheduled
so we haven't been able to build in as much time as we probably could have used
on this topic but we're going to do our best
and first to kick us off i'm so pleased that
our director of the institute of museum and library services
crosby kemper could be here to address us
and to really frame this discussion he has
contacts in all parts of the country all levels
of government and stakeholders and so he he knows what these funds um were
written and intended to be used for and and he is
you know happy to be here and and talk to you all
so we are we are so glad that he could join us and kick off this conversation
about arpa which we are um that is the acronym du jour that we are using for
these funds so dennis if you would advance the next
slide we will turn it over to crosby great thanks very much terry and thanks
everybody for uh for being uh here in our virtual
conference uh and uh uh
for uh continuing the great work that you've been doing during the pandemic
uh and uh and we'll be doing post-pandemic which we hope will happen
soon i'll just tell you that my most exotic
travel was to china i taught english in china
in 1984 at the university of sichuan and my
favorite part of that would be between two things climbing
mount of may which is one of the five sacred mountains which is pretty
spectacular you're supposed to be able to see the face of buddha when you
reach the top and later at cocktail hour i will tell
you whether or not i saw the face of buddha
and uh and then i also got to travel down the entire
length of the oxy river uh on a steamboat and and that was
before the the uh the dams uh uh
closed the the gorges um and made it impossible you can't do that anymore
and that was spectacular too but now you want to know about money and so
let me tell you just a little bit about the background
of of the legislation and how we doubled the size of the imls for one year
this will probably not happen again in your lifetime certainly won't
happen in my lifetime uh but it's a spectacular thing
it's spectacular because we're getting so much money
uh and giving it to you but uh it also is a one-time thing and it's
related to one thing obviously it's related to the pandemic
and uh and so i i need i need to say at the
top that that when you think about how to spend
this money be thinking about the response to the
pandemic we hope you use this money well and there are a lot of different ways
you can use it but it is intended uh this is a one time
only intended to be a response to the
pandemic within that one thing that has been very
top of mind uh in the universe of capitol hill
also i believe in the in the white house and the executive branch generally
is the digital divide one thing that in an entirely virtual universe that so
many of us are in uh that is necessary obviously is access
to the internet and the ability to use the internet uh
and that makes what you do uh in the library and what you do for
people outside of the library infinitely more important
and it is a focus of the of the art of money
and i would urge you to to to pay attention to particularly pandemically
related things that you're doing but but the digital divide generally is
an ongoing concern of the united states uh
government uh and i believe we've now finally made
the case that really it's a utility uh but beyond being a utility it has to
be useful utility a utilitarian utility
if you will and and that's where libraries make a huge difference
and you know that better than i do but but i was a librarian once so i know too
uh hot spots are a very important part of that uh senator
manchin uh is a key figure in congress and key
figure in this legislation uh and he's been very heavily focused on
that uh and and i and i think that we should
pay attention to that uh and uh we certainly will be looking
looking for the word and the uh and the delivery of hot spots
uh in uh in what what we're uh what we're focused on
um also telehealth is a huge thing and and that can come in a lot of
different uh guises uh uh basic connectivity allows you to
to to look for community health information etc but
but also the services of librarians is very important to that and again
congress is paying attention to the huge gaps
we have in uh in health care in this country and and probably the
single most important problem right now is vaccines and the uh the the big
divide in terms of who's getting vaccines which has a lot to who
knows about it uh who understands it um and their their
big racial uh and uh income divides on that
your poor and minority you're much less likely to to be near a vaccination spot
uh to know about a vaccination spot to know what a vaccination
is all about to know that it's safe and so
if you can play a role in that um you'll you'll have
uh you'll have rewarded uh the the expectations of the united
states congress and and the biden administration um also
in the in the bill is the phrase workforce development
and i want to outline that a little bit for you i think
everybody in the library world is doing work in that
job applications job resumes uh skill development digital skill
development uh that points you towards uh certification for for jobs uh
and uh even even interviews online that are taking place in libraries
that that sort of thing more advanced versions of that was
uh with uh heavier software partnerships career online high school
and and things like that anything that is educationally related to career
development uh is is really important uh in the view
of uh of folks in in washington the view of
folks who are looking at the various divides
in this country and of course education is
is also really important in that and as a whole
educational resources of any kind uh certainly fit the bill
uh here literally fit the bill and i i would say that with something we
have stressed through the cares act and and through uh the uh uh
rest of american rescue act uh and and will stress going forward in our regular
grant making programs is the notion of community
impact uh partnering with other community
organizations uh at the imls we particularly like it
when you partner with museums of course uh uh our our partners
at the imls but but community partners working on community impact and
obviously particularly focused on the pandemic
though i think what's happened uh last year in the racial protests and our
our growing sense of the inequity and inequality
uh in our country are are very important aspects of this as well
so uh i that's an outline of the intent of the bill as we can tell as best we
can tell from uh from congress uh and from
conversations uh inside the the uh the biden
administration and the executive branch generally
a very large amount of money has been provided
out of the three stimulus bills over the last year
uh we've as i say doubled our size it it is intended uh to to make life
better for for people who've been harmed during the
pandemic in particular but it's also meant to
strengthen libraries response to these divides in our country and so that's an
ongoing focus and terry will tell you a little bit more about the flexibilities
in in our grant programs two things are true
the faster we spend this money the more we'll it will have a direct impact
on on the problems of today which is the point of the bill
but also there is a lot of flexibility and if you're creating something that is
going to benefit uh your community
and pretty good benefit the folks at who are who have been challenged
or left out historically in your community
we will be able to look very favorably on that
so i'd just like to wind up by saying thank you to to all of you for the hard
work under great stress i want to thank my
staff too terry and the staff that that she
introduced they're they're working very very hard uh
working long hours uh under stressful circumstances uh
and and i know you are too and and so i appreciate your um your understanding uh
and appreciate your very hard work and your your
you're continuing to uh uh fulfill the mission
uh of america's libraries thanks thank you crosby i'm going to talk
logistics on arpa for just a little bit and hopefully we'll leave a little time
to answer questions i see a few already coming in
so we'll we'll hold those till the end but on the next slide
we just want to reiterate that we have already sent quite a bit of information
about the american rescue plan act to the group of state library
administrative agencies a select group of contacts these are usually the chief
officers the lsda coordinators and the heads of
library development that are getting these messages
the third message on this list here did just go exclusively to chief officers
but if you're interested in the content and you know that somebody should have
received that message just follow up with them and mostly we
wanted to memorialize it on this slide these slides are available on our
conference agenda on our website so next slide it's going to be very
similar to cares act in the way that this rolls
out for those of you that were around last year for the last
stimulus package that went through imls so
what we had was an existing allotment award for 2020
and then we sort of brought the cares act into the scope of the larger
allotment award with some special provisions
arpa's working very similar you already have your fiscal year 2021 allotment
award and so the arpa grant is going to be
running in parallel with that we have not actually issued these awards
yet we anticipate they'll come in april but uh we will give them the same end
date as your fiscal year 2021 award which will be september
30th of 2022 there is a focus on rapid response
crosby did talk about the optics of getting this money spent quickly
into communities to stimulate the economy to be a stimulus
and to respond to the pandemic which are pressing pressing needs across the
country that said we know that this is a short
amount of time and it is a large amount of money so if
you're feeling like you you're getting to next summer and
there is still money to be spent or things that you anticipated spending
money on aren't coming through because of some supply chain issues or such
uh please let us know and we can work with you on
individual extensions on an as needed basis
the gift of these funds as was the case with cares act is that they come with no
match requirement this is pure federal money for you to spend
with no sort of strings attached in terms of you bringing
state dollars to the table so it is truly that once in a lifetime
kind of funding that uh we don't you know anticipate you seek very much at
all and then we do have some special
spending priorities attached to it um of which crosby kind of outlined the
vision already on the next slide we reiterated what we
sent you in an email so i'm not going to read it all here it's just mostly to
memorialize what we've already sent you and we
anticipate this being the language that goes into your awards in april
um but basically the spending is much like cares
there is first and foremost a digital focus to this
spending digital inclusion secondly uh response and
emergency relief to libraries component of this
that would be kind of along the lines of the personal protective equipment
and helping libraries to safely reopen their physical spaces kind of work that
you did under cares so again very similar
the third part of this we've tried to make a little bit broader because we
know it's a lot of money basically you have a lot of flexibility
here to spend money under allowable cost
provisions that respond to library services
that meet the needs of communities that can include things like personnel
training materials supplies equipment so the full gamut of spending
that you would see in a library is pretty much allowable
under this act we would like it to be sort of the third order spending that
you prioritize but it is there for you as a flexibility
and then we've kept in the provision about reaching tribes and museums
as appropriate and we've also couched at this time
as those tribes and museums that are best positioned to help respond to the
pandemic so next slide
this is all falling again under the umbrella of our existing grants to state
statutory provisions which means it's falling under the lsta
priorities and your five-year plan we believe that your five-year plans are
all broad enough to encompass this work and other provisions within your regular
allotment spending will still apply for example you
can still determine who is eligible for these monies if you decide to sub award
you can still use up to four percent of these funds for administration
that's optional but you have um that latitude
we're still using this um within the purview of the allowable costs for the
program and that means that things like the
children's internet protection act uh when it comes to internet enabled
devices still applies and we are still a
non-construction program so we can't take these funds and start dabbling in
construction but if you imagine our existing grants
to states program and arpa as an overlapping gun diagram
they're very much overlapping and there's just a very small slice of arpa
that has the special provisions around how you should spend it and the fact
that there is no state match next slide
we always have existing overlapping awards
at this point in time shortly as soon as we get into april we anticipate that you
will have four overlapping awards you will have your fiscal 20 award and
cares act which are running in parallel you'll also have your fiscal 21 award
and arpa which are running in parallel and those each of those stimulus funds
will be reported in the state program report with the
other allotment awards and we'll talk a little bit about
the mechanics of that but this is just to to underscore that
you you are administering a lot of federal
money at the moment and having to keep track of a lot of different federal
funding streams and we appreciate the administration
that goes along with this so please do use those four percent administrative
costs appropriately as needed to ensure that you are handling this money
um with the correct stewardship that we need to bring to all this
next slide
in terms of the state program report it's again going to be very similar to
what you what you are doing and will do for the
cares act you will have an interim federal
financial report due this coming december
and that will be one single form in the spr where you report on your allotment
and your arpa funds together then the following year
you will report on the projects and the financial status report for sort of the
end time of the funds they will have ended
the period of performance will have ended and you'll give us kind of the
wrap up report like you always do in the state program report we would
just ask that you title projects that are using arpa funds
with the prefix arpa colon and that will help us differentiate
as program officers start reviewing those projects this is similar to cares
act we said title your projects cares act colon so
that'll be coming up that's the next round of reporting that you'll do
and we'll be sure to have some kind of webinar
some kind of refresher to help you through this when we get to the fall
you still have a maintenance of effort responsibility
however arpa doesn't really change that in any significant way
your three-year average is still your three-year average based on what you've
been reporting in the state report over the years and so that will
just kind of be a parallel process to all of these stimulus funds that are
happening next slide i think we're in the home
stretch we are so grateful to you for
administering this we understand it is a huge
burden on some level that um the states and the communities that are
seeing this funding are looking to you to help administer it
and we we recognize that it's a lot of work
it is a lot of work but you are well positioned to do this
the fact that you did cares act so well speaks to part of why this has come down
the pike again and we are here to help you answer the
questions as best we can we're working through it as well
obviously and we won't always have answers as quick as you might need them
but we will we will do our best and um yeah in the words of the recent
glenn doyle book that i read we can do hard things
we can we are an awesome group of people with a good state federal
collaborative partnership and we know that you are all
very well positioned to get this done and with that
i will leave five minutes for questions hey terry we already have some questions
in the chat so i'll go ahead and get started on them
um mary susie in north dakota asked if arpa funds can be used for equity
diversity and inclusion related efforts such as training for librarians
uh yeah crosby already indicated in the chat that it
may not be that may be a little indirect related to the pandemic in terms of i'd
say it's possible but uh but you we're looking for more direct
immediate help to people outside the library
and you know and that can certainly involve um
funding for staff um training i think is just not high on the list it's you know
it it can be used for it but i i wouldn't put
it that high really good point thank you the next
question we have is around the timeline i believe for
arpa awards um when we can when can folks start
dispersing awards as soon as we make
the awards to you which we anticipate to be sometime in april
you will be able to access the federal funds
through our agms reach system and begin you know asking for payments
uh we don't have an exact date yet so we'll just have to see how quickly we
can move through the sort of administrative
hurdles around making the awards great thank you nicole davies in
colorado asks if they could include talking book
libraries and publicly funded prisons or jail libraries along with school public
and higher ed libraries if your state has the flexibility to
um work with make awards to those types of institutions then we have no problem
with that thank you chris in connecticut asks um
is arba money a part of the moe calculation
it is not the moe calculation is based on the state dollars that you are
spending and since arpa is federal funds it would just be completely outside of
that thank you lena in california
asks would making the library safe for patrons to visit
such as plexiglas installation be okay with the use of arpa funds
we just cautioned with installation that we want it to be like installation light
we don't want it to dabble into construction areas where you would have
to invoke construction trades to be bringing in
big equipment and tearing down walls so the plexiglas installation we've seen
kind of meets this bar of allowable but just proceed with caution
thank you and leah asks to help county law libraries with relief
um
i'm sorry uh minnesota had a similar request similar to nicole's questions to
help county law libraries with relief and but
she's unsure how to do that without supplanting
salaries i'm sorry leah if i don't quite
understand your question yeah and that might be one um we could
just talk with you offline about since it's kind of a specific instance i
mean certainly this is relief funding this is stimulus
funding and so we might have a little bit more latitude
than we normally would to think about
salaries i'm not saying anymore but keep keep the
questions coming
you guys are on it and i will say uh you know a half hour
on the topic of arpa is probably not what we all need collectively to get
through this so please keep sending your program officers
questions and you know look for those forums that we'll have
in late april to connect and talk about it again
now we have a question from nancy in iowa would a state library be able to
hire someone to manage subcontracts for the money
personnel are allowable costs under this so um it would probably fall under the
four percent administration clause if they were managing the grant
monies but yes that would be allowable thank
you and natalie in california asks are
bookmobiles allowable to take services out into the community
yes and um if you can you know couch them as pandemic response
rapid response kind of things all the better
and jamie mott in idaho asks um oh he he missed out on the previous
answer uh the awards uh you can award as soon as we award the
awards which is will be in april um and that's right
yeah it's a little different than the allotment grants which we always sort of
back date to october but because the act wasn't put in place
until very recently we can't we can't do that sort of retroactive
spending thing then someone else asks can we assist
academic and school libraries and again if your state is um
enabled to work with those groups and make sub awards to them then we have no
problem with that
great we did it everybody we we did a half hour
lightning round of arpa and crosby thank you so much for joining us
um we know you're very busy and um and having your vision and your
leadership in this time has been um truly an instrumental piece of this
process so we thank him thanks terry and thanks
everybody
we are moving along in our agenda we don't have a formal break yet we're
going to go right to our next session which um
as we originally conceived it is now a little bit out of order we were going to
do some five-year evaluation things grouped at the end of the day but
because of scheduling issues we're going to bring up matt bernbaum
our supervisory social scientist in
the newly branded office of research and evaluation
or o-r-e matt has seen a number of acronyms
in his time and this is the latest and greatest
i i like it it's simple office of research and evaluation
dennis and engel is going to accompany matt
birnbaum with just a few preferatory remarks about five-year evaluations
and um you know who'd have thought that this would happen right in the year when
you were all administering stimulus funds
um but we are we are prepared for it as a federal agency
we have done our best to make resources available to you so this is a smooth
process a lot of the work will happen on your
end but we're going to do it together and this is our way through this
conference of sort of kicking off the process
so dennis i'm going to turn it over to you and then
matt will be up to talk to us soon
wonderful thank you terry and all the other terries
um it's always nice to have a chorus of terry's
uh joining us um but yes so i plan to spend a very brief
portion at the beginning of this session going over some of the um standard items
related to the evaluation what the guidance says from a process
and deadline perspective and then i will hand it over to matt who will
take more of the meat and potatoes of this presentation to really get into
a more of a deep dive of the best practices some of the ways we define
the data we're looking for and the analysis standards that will be helpful
for both you and your future evaluators to know
so um and as terry mentioned at the beginning of this we thought
well this is uh a travel theme is really sort of appropriate considering the meat
of the conference content because um you know now that we're in this very
weird limbo period of remembering what it was like to
travel thinking back on those wonderful times and then also
kind of seeing the light at the end of the tunnel
where travel plans can be made once again we're in that same position with
our five-year evaluations and a forthcoming five-year plans so before
we talk about the future let us reflect on the very
immediate past and um talk about the five-year
evaluation so before we go any further i'll ask
madison to help me push out an informal poll
we wanted to get a quick glimpse into the level of familiarity
that everyone has or doesn't have with the five-year evaluation process
you should be seeing it soon hopefully
do you see the pool dennis well i'm a co-host so i can't see it
but i think um hopefully yeah matt is give me a thumbs up
great great thanks everyone so we'll take some time i can't see the results
until they're pushed out um madison but um we will very shortly
so i think i will just say i think this might be the later poll
this is the one about evaluators coming on board and that is okay
but just to prep you dennis okay that's great too
it's great everything is great we'll just swap and swap the polls that we
planned to to give everyone so this was on terry
this is related to um uh when they're planning to work with
an evaluator for their five-year evaluation
wonderful right we'll give that a little bit more time madison you can use
your best judgment based on how many um it looks like things are
slowing down right now so we'll go ahead and and yeah that's correct bev um
one response for state or one response for terry whatever um
you prefer all right share results
all right looks like the majority of are looking to start
in the summer okay great it's good to know and and i can't see
those results oh my goodness 34 34 planned to start in the spring of
2021. 44 plan to start in the summer and 22
uh percent plan to start in the fall all right
great thank you all for your flexibility and your engagement
and your feedback um i'm going to move on now
to the another timeline we can't go too far in our presentations
in the conference without sharing yet another timeline we love
timelines we love visual timelines and especially ones that have so many
award layers on it so um for those slaa staff that have worked
with grants to states prior to the pandemic
you might be familiar with a less complex timeline that we typically
present at our annual conferences um this year we've updated the timeline
to reflect the usual two-year awards um as well as the two separate stimulus
funds and where they fall in line with the time frame um and there's obviously
much to digest with the timeline but the the bigger picture here is i want to
direct your attention to the five-year evaluation deadlines um
and the forthcoming five-year plan deadlines and how they intersect with
your quote-unquote day-to-day grant
management deadlines so as you can see the most immediate
evaluation related deadline is march 30th and that's when um the
evaluation is due um and then not too far behind that on
june 30th of 2022 is when your plans are due and uh
again just to kind of get a big picture abroad brush
you see that not too shortly after june 30th
is the october 1st which is the beginning of the new five-year plan
cycle and then much further down the road april 2024
we want to point out that timeline because in the event that you were to
submit a brand new five-year plan and you realize oh i don't like that
goal anymore i want to change it uh there is a process for doing that but
we just will let you know that even if your request to change your goal
one of your goals is approved and that can't be reflected until april of 2024
so we say all of that because obviously planning is key
with this process and because it is a bit of a domino effect with evaluations
due plans do we wanted to give you all a
broader as broad of a perspective as possible for your
planning purposes
and i'd like to now turn our attention to the federal statute that provides
the necessary foundation for much of the information you'll find
in the five-year evaluation guidance while the mention of the evaluation is
relatively brief in u.s code 20 section 913
there are some critical pieces to derive from it
you'll note that the statute says that the state library administrative agency
shall independently evaluate so we emphasize that because that sets the
stage for what many state libraries do with
pursuing a third party evaluator which is why again another piece of the
planning process because if you're going to go third party
that often involves rfps going out for bid
those kinds of things as far as what's considered an independent evaluator
matt is going to touch on that so i just want to point out that the statute says
that um and then additionally the statute
also uh sort of provides the somewhat
uh specific framework for the timeline so you see there in the little excerpt
that the um the evaluation is to be conducted prior
to the end of the five-year plan and this is um so that way there is no
gap between plans and also um i think for the intention of being
able to uh glean the feedback from the
evaluation to inform the forthcoming five year plan and so
that's why it is by design that the timeline is
the for your five-year plan you're evaluating 18
19 and 20.
and here again is i refer to the guidelines for the evaluation this is
all derived from the statute and um
that is a link to the fire evaluation guidelines but that can be found
in our grants estates manual which is basically
the hub for all of our critical resources and guidance
for the grants to states program we want to i want to add here as well
that you are not expected to include cares act funds or activities as part of
the formal evaluation and so the reason
there's several reasons for this and matt will
go into them as well i believe in his section of the presentation
but because your original five-year plan was written
prior to the advent of the cares act funds that we
can really reasonably expect those cares act funded activities to be
evaluated along with your five-year plan
and a quick snapshot of our review process uh
i want to emphasize here that our role at imls related to the evaluations is
that we are accepting them based on them meeting the requirements
in the guidance and we are not grading or evaluating
them um well i just said evaluations but you
know we're not grading them in that sort of substantive way
so we emphasize this and that again will go into the more of the
um statistical components of their motivations but
this is because um we don't want to assume that you knew everything
you can't see the future you can't tell the future you can't anticipate
everything that could potentially deviate your original
intention of your goals so to say that a goal was partially met
does not mean that you failed necessarily
and we would encourage that level of transparency in the evaluation
not only for our intents and purposes so we understand what happened at your
state but also for the hopes that it is
beneficial information and feedback that you can then use for your forthcoming
plan so i just want to emphasize that and
then um from a timeline perspective we have our program officers we have 90
days to finish reviewing allstate's five-year evaluations you all
are on the exact same time table so it's a lot of a lot of reading for us
um in april and june through june of 2022
uh and then at once everything has been accepted and everything is said and done
you'll receive official letters of acceptance
from us that kind of close out the evaluation process
and finally um as far as paying for the evaluations and the plans
uh there's a reference to um some guidance on this
and it's the administrative costs memo that is surprise
also in the grants estates manual um but we want to
emphasize that the um as far as what category of funding they
can go under it can be either an lsda project
like all by itself um it could be under the administrative project or you could
even use state funds to pay for those evaluations so there is
a lot of latitude with what funding category you use but
what we do want to point out here is that
the statute says that as far as how to fund it the determination
will be made by the state library and quote must be handled in a
consistent manner and what that means is if your
evaluation was funded under an admin project for the last cycle
uh the statute expects that it will be consistently funded in the same way
for the future cycle uh but if that if there's uh some extenuating
circumstances or or the other reasons why your evaluation
may be funded differently this cycle than in the past uh please
reach out to your program officer so we can kind of discuss
more of the finer details so that's all of the
perfunctory time with the evaluations uh and so now i'm going to hand it over to
matt birnbaum who's going to dive into the statistical components
thanks to thanks dennis and i'll try to keep the statistics
to a minimum i want to talk a little bit about the five-year evaluation for the
veterans you've heard me give this spiel twice
before in the last decade i'm gonna make some little bit
changes this time i really want to focus more on making
sure that this whole process of evaluation
is as useful to you and your other key stakeholders as possible
next slide
so to do so we're going to focus on five things we're going to just give the
overall framework for doing this evaluation we're going to try
to address some key concerns go over a little bit more of the
guidelines flesh it out a little bit more in detail
and then talk a little bit about next steps with a particular emphasis on ways
that we know mls are going to try to provide
some new types of support to you and your evaluators
and then at the end we'll take some questions and answers and along the way
we'll try to do so as well and i'm already
noting we've got two comments already in the chat to address next
slide please and one more time
okay so for the veterans uh about literally 10 years ago i was
welcomed into the community and at that time the chiefs and all of
you were telling us and other the key partners were telling us hey we've got
some issues in reporting on all the projects that
we're funding it's really impossible to be able to
tell a good story about what difference this is making to the
patrons who are receiving it as well as to other taxpayers as well as
to the taxpayers as well as a whole and so that led to the next
the beginning of measuring success measuring success is mostly as you know
it is in terms of the project reporting in
the state project report we tried our best to capture the promise
made 10 years ago that a summer reading
program in michigan we should be able to compare it using the same
factors as a summer reading program in mississippi
and the same would be true for all the other multitude of
projects and activities that everybody does
so the focus here is providing greater transparency to
how the federal formula grants trans translate
into the hundreds and thousands of projects awarded in all the states and
territories every year to be able to have that data in a form
in which it's shareable in doing so to not only be able to
demonstrate better accountability to federal taxpayers to showing them how
with their taxpayer dollars go but more importantly
to helping all of you to helping public libraries to helping
imless it's helping everybody who's concerned about the ways that libraries
make a difference in their communities that we can use these data to actually
improve the programs in doing so right from the
get-go and it's built into the dna the spr the
project level data is balancing national level needs for
that we need to be able to tell a federal story
with the needs of each of you in your states and territories to be able to
tell the stories that are within your territories and
fundamentally it's cooperative process everything we do is
hopefully building on and strengthening the trust that exists
among all of us next slide please
okay i'm going to give some wonky terms that you're going to see in the
five-year evaluations your evaluators are going to be talking
about this language some of your very savvy stakeholders like some of the
funders the policy makers they're going to be
using this language all right let's start off with the basic what's an
evaluation we know the spr gives us all types of
data we know we can aggregate it to tell certain stories such as the number of
projects that you've awarded the dollar amount
the types of activities that were associated with it
valuation goes a little bit more it goes a little bit more deeper
it takes those those initial trends and it asks some questions about so what
why are we seeing these trends so it's a systematic a process to taking whatever
questions you're asking trying to systematically deliberately
collect answers to provide them and using those answers
asking those questions only for a very practical purpose that we're
trying to strengthen a program so that's the evaluation we do every
five years the implementation well this is the
process in which you get the in this case you get the funds and then
you go about and administering those so we want to better look at you maybe
set some goals maybe you find out the goals were met
they weren't met maybe we're finding some other results
we want to figure out was how much this was because the strategy was right
and how much of it was because of the way things were implemented
we can have a poor result and it could be because we had a great strategy
but the implementation was botched up similarly we could have had a
not so great strategy but seeing that it's succeeding
because the way everything was administered was fantastic
so a lot of evaluation is looking at at the
implementation and we use that using what we call
methodology some of it but not all of it could be
statistical a lot of it is qualitative i'll say it a little
bit later but i'm going to give the punch line again
whenever thinking about your evaluation method let the question that you're
trying to get an answer drive the method sometimes some
questions are really best answered using statistical methods other
times it's qualitative choices and then we've got that process
evaluation that's very much related to the implementation the process
evaluation tells us how well things were being
implemented and we contrast that with the impact
evaluation where we're looking at the results and
those results may be quote unquote outcomes they may be
outputs i'm going to tell everybody here not to get hung up whether it's an
outcome or output i don't think that's helpful what
you really want to be thinking about or what are the results what are the types
of changes that we see associated with the money
that we're funding next slide
okay so i'm gonna just build on some points that dennis just elaborated on
the five-year evaluation objective we're being driven by
the objectives in our statutory legislation
i'll talk about independent evaluation is in a second
for right now we want to make sure that this cooperative process
where the evaluation is useful for you at the state level
is also useful for imos and others who are interested in the national level
stories associated with this program to do that
we're setting up this evaluation a way that maximizes the strengths
of the sbr data so we don't have to reinvent the wheel
we're going to be trying to highlight effective practices
and we'll get to the retrospective questions in a moment
here the i want to highlight effective past practices both for you to make some
changes in your next five-year plan as well as to be able to share it with
others who might be interested in what works and and third to be looking
at that process for implementing the ground the grant making the process
questions and i'm already seeing one chat that i'll respond to now is
crosby's snarky word yes impact impact impact it's our favorite
words too next slide please
all right couple of culvert caveats all right we know
we've had cares that cares grant funds now we've got the arpa funds
what do we do with this opera we don't have to worry because it
doesn't cover this five-year evaluation it's evaluation will cover
fiscal years 2018 19 and 20. we know that cares act funds for doing
it we're going to tell you to exclude the funds
you have that you have a choice to include it but we're telling you to
exclude it if you want to account for
covid 19 related disruptions exclude the cares act funds but if you want to
talk about the disruption that covet 19 has had on you
here are some caveats don't report on either the carriers act or arpa act
if you want to make sure you're separating it
very distinctly don't mix that up with your regular funds for 18 19 and 20.
the most important thing is to think about how the pandemic what experiences
that you've watched over the last 12 months how that might
be influencing the next five years i believe we have already that um
jody jodie is asking ray rose to comment one of the things we're trying to wrap
our heads on is what will the future hold we know
that the pandemic is causing lots of disruptions in the plans and this may be
a really good point where even if you're not looking at the cares act
funds itself you're going to be looking at what do we know about the pandemic in
terms of some contingencies that we want to build into
the next five years that would be fair game in terms of
cares acts specifically in arpa funds specifically
stay tuned we're going to be we're having discussions here
internally about how we want to report on it and we'll come back to you
but right now the biggest message again is you're not obligated to report on
cares act funding in this evaluation it's no mandate if by any chance you
want to treat it separately but most importantly
just think about how the pandemic might be inter interrupt interrupting
where you're going to be going for the five year plan
some of those experiences from the past year
okay let's go on to the next slide
and the next slide
okay fancy slide all right independent off evaluation
the legislation is quite clear that the evaluator has to be independent
the most important thing here is that the evaluator has to be free from
outside influence we basically tell you to use a third party because that almost
guarantees you that they are independent from that
influence so they pass the judgment they're not going to get
fired from their job they may not get a contract with you again
but they're they're going to be able to provide that independence
if you want to be able to do it in-house every now and then we get
we get a state or territory asking us the basic test you got to be able to do
is to be able to show that they're not in the line of fire in terms of
managerial responsibilities for grant making
if they were to provide for some type of negative judgment
they shouldn't be anywhere related to the grant making
conversely they also need to be showing that they've got the skills and
competencies that come with being in the evaluation of evaluation
profession there are certain science social
scientific methods that they should be able to be
quite expert be experts in to be able to ensure that
they can adequately analyze that information
and provide it into you in a way that's credible
so the most important piece now is then saying
that the independent evaluator that you pick
in this evaluation what makes for a successful evaluation it's one that's
going to be useful to you and your key stakeholders we're
going to give you some giving you some places right now where
you can find some more information about it i'm putting a
little heads up now one of the new things that imls is offering is an
evaluative community of practice we'll start it in the summer and we'll
talk about it at the end um but we want to be able to provide
this here now so everyone knows we're going to try to
be with you together and building up everyone's capabilities
next slide please
all right just some just some statistics about the last
five-year evaluation is that last time we saw that there were a total
of 29 evaluators working with the states and territories
one valley only one evaluator was in-house worked
in that slaa the rest were third parties one of the evaluators worked with 20
states um others worked with one two or three
about 22 most of them they evaluate as work with just one single state
leah asks when you say in-house do you mean within our agency or our state
either one we're basically making sure though it
depends how big your agency is and we're making sure
that that person is truly independent of anything dealing with grant making or
oversight of the lsta pro program in your state
it could be that your your agency your state has its own independent research
and evaluation shop that would be the type of person that
might be qualified to provide an independent assessment
next slide please
all right first consumer tip for the day how do we choose that independent
evaluator your cardinal rule is to balance their
ability to offer scientific objectivity with their
with your key your stakeholders able to trust them
so we start off the very top do they have independence
do our stakeholders find them credibility credible
to be able to start diving into that we want to make sure that they can
understand the specific context of the way your
organization functions the environment in which it's
situated can they basically produce some
independent judgments and understand what the reality is for you
in doing so can then they be able to give you findings and
recommendations that are adoptable and transferable
as you move forward to the next five-year plan and you're working with
any of your public libraries and other libraries
that are associated with this program is the cost
is it manageable is the timeline manageable these are all
pieces that's going to make for a good evaluator
uh next slide please so let's talk about that usefulness
what are some good uses of the five-year evaluation we were to talk let's say
two years from now and we would say hey was that a value was that five-year
evaluation really useful here might be some things we might be
thinking about well at the bit at the minimum you basically
had demonstrated accountability with federal law and completing it
we hope all this time and attention does a little bit more than just
accountability we hope we can really provide some good
uses for informing where you're going to head for the next five-year plan
such as trying to figure out what adaptations you want to make
given the disruption from the pandemic we also hope it might help you
and others might identify and share the promising practices
of types of investments that you've made over the last five years
that can be carried forward into the next five years
and we really hope it strengthens your relationship with key stakeholders
in a moment i'm going to talk a little bit more about ways we can think about
ways you can think about making the evaluation more useful for
the key stakeholders but for now let me throw a couple of
ethical caveats out there it's really tempting when you get at
this point and you're starting to identify stakeholders who have different
values different interests different objectives
to try to favor one much more than the other
you're really obligated we're begging you don't misuse the
evaluation to pursue a specific self-interest always maintain
and evaluate objectivity the evaluator is fundamentally going to
be collecting evidence to answer key questions a good question that will
maintain the objectivity is where the answer to yes is as
interesting as the answer is of no and then because you can get an answer
of no be prepared to deal with negative
findings this is when the best learnings come from nobody's going to be
looking to slap you this is where we're almost going to be
seeing with a lesson to learning and adaptation comes from
uh next slide please
okay let's talk about those stakeholders all right
yeah i'm mad i'm hearing you unless i'm hearing you we got to get involved with
the stakeholders how do we do this first thing is as you
start thinking about who your stakeholders are
to think about that they might play different roles in the evaluation
some of them might be respondents such as responding
participating in a survey that you might be doing or an interview or a focus
group some of them might be subject matter
experts who are going to have their their sleeves rolled up with you in
interpreting the draft reports providing some feedback to the
evaluators some of them may be advocates for a specific position
some of them may be the policy makers such as your chief executives
some of them may be the ones who are implementing some of the work that
you're doing such as folks who are sub granting or each of
you in terms of implementing the grants on a year-to-year basis some of
them may be your key partners and some of them may be key
beneficiaries of these money such as some of the
key constituents in the public libraries beyond thinking about the different
roles that stakeholders may play in the evaluations try to think about
critically when do we need to bring them in
some of them may only need to become at the very beginning as you're starting to
formulate what we want that evaluation to do at
the upfront in the planning some may get involved in the middle as
we're starting to implement the evaluation such as our respondents and
subject matter experts some may be coming in more importantly
once we got the final report some of the interim report findings and
we're starting to think about messaging these may be your advocates they may be
your policy makers but these are key people
and then the last piece is going to be the adaptation we're starting to move
into the five-year plan and we want to make sure that we can
that these recommendations are going to be feasible
how do what stakeholders do we need to be thinking about
so this leads me to my most important take home on the third one
right now at the beginning not only start identifying your stakeholders but
plan now for how they can use the evaluation
and not just what what guests must get done in completing the evaluation next
slide please
so now let's talk a little bit about the wonky stuff the evaluation methodology
next slide there's no one size fits all approach
the questions that you ask are going to drive
the types of data you need and the way you're going to analyze that data
we start off with the foundation that everybody's going to start to maximize
what you can get with the sbr data for all your projects
hopefully that will give you some answers and will help drive you where
you need to drill deeper to try to look beyond that you might
have some other administrative records that the evaluator wants to look at like
your last five-year plan some of the annual budgets any key memos
that have been out there some of the legislation
correspondents you've had such as with the rmls program
officer these are all rich sources of information
we might be looking at some published evaluations and other studies like if
your state had conducted an audit on the program
this might be some valuable data for that person any media newspaper
clippings psas you might be having them do interviews
and focus groups you might be having them surveys
you might even have them be working visually in terms of photos
and videos these are all key types of data that we might be looking at
next slide please once you collect the data we got to figure out how to analyze
it there's not going to be a one size fits all
so as we've got different types of data it's going to be amenable to different
types of analysis maybe it might be some statistical
analysis maybe it might be types of qualitative
analysis might be others like case studies social
networks some gis mapping and in general when we do these
evaluations we're seeing a whole blending a mixing of methods
i'll talk about this in a moment but right now just assume that the good
evaluators are going to be blending using different types of methods
to be answering different questions and you should be able to see what they're
doing and have comfort in it that they're that they really have
mastery of the of the methodology next slide please
so last cycle when we looked at the types of methods that we saw
folks do all of them every state used document review because principally they
were relying on the spr almost all the states and territories
did surveys and interviews and focus groups and then we had some
others who were using some additional data um using some other other met
other methods like doing some uh social media analysis
doing some we had one i think it was illinois of my memories correct
where they had her and evaluated a very interesting ethnographic analysis
i'm using a cool piece of software we had about five states who were
looking at the pls data to supplement what they were getting from the sbr
so there's lots of ways you can skin this one
and there's lots of ways to make this very valuable
just as a heads up you know we think then this next year's cycle what will be
different from the last one of course will be
that the evaluators whatever method they'll be using they're going to have
to be working remotely virtually i want you everyone here to know there's
nothing stopping evaluated from doing surveys and interviews
focus groups virtually this is we've been doing this for years decades so
this is not any shouldn't be a major disruption
for any of the qualified credible evaluators
next slide please
this leads me to the second tip and i want to re i said it earlier i want to
set it again don't think i got to pick an evaluator
who knows statistics or an evaluator who knows interviews really what you want to
know is what are the questions that you want answers to
and then see what the evaluator can do the evaluators
will probably they'll probably be a couple of them if they're doing it right
and they'll all have certain methods that they're
particularly competent on and they'll know how to work together in terms of
matching this source of data and the method of analyzing that data to
and to best answer your questions next slide please
more consumer tips do develop your plan before the evaluation
starts ask the important questions figure out
where your stakeholders are landing on it
ensure your value once you do that then you can go and
find your evaluator at that point make sure the evaluators are up front
about their choices they're making on the
method there's a cardinal rule they should be transparent so that one of
their peers can look at what they've done and
basically if they were to have done exactly what they did they would
come to the same exact findings so there is no
risk of bias balance your agency's needs for the evaluation
with those of other stakeholders when you've got conflicts be it
different stakeholders within your agency
or interests of your agency with some of your other stakeholders
always let the pub the larger public interest drive the resolution
don't let self-interest drive it finally always use result based
management principles don't think you're the evaluator
what you really want to do is figure out what they
what your timeline is what the key milestones are
and make sure that they're giving you those key deliverables on those
milestones and you hold them accountable to those results
let the don't don't you take on the headache of how to make the meatloaf
just make sure whatever uh vegeta vegetarian lasagna whatever
meatloaf they're making that it's meeting your needs and
please feel free to throw questions or comments in the chat i'll do my
best to multitask can we in the interim can we go on to the next slide
so some guidelines next slide please
okay in the in the guidance we've got four pieces that we've covered
we're covering the retrospective questions
process questions evaluation methodology independent evaluator requirement in
past two times when i've done this presentation
i spent a lot of time on evaluation methodology and on the independent
evaluator requirement i'm not doing it this time i think what
i've provided you thus far is sufficient and at the end we'll talk about the
community practice for us following up on any of the uncertainties
about methodology and requirement and if you have any more questions about it
we'll we'll we'll handle it but what i'd really like to focus in on
with our remaining time together are on the retrospective questions that are in
the guidelines and the process questions and i'm going to first take a
question from rachel from utah and she asks are we allowed to have additional
state library programs not directly tied to lsta evaluated
while they are completing the rest of the evaluation great question
rachel i'm going to give my answer and i'll let any of my terry and any mike
and program officers answer they this or you're fundamentally required to make an
assessment on the lsta to the extent that you think the lsta is
tied to these other programs included just make sure you can make that own
independent assessment about the lsta program
my colleagues in imls any other thoughts about that question
i just say when we're reviewing it we're gonna want some apples to apples
national comparative data so it should be very clear what the
federal piece is versus anything else well the fed
i should say the federal state piece that is sort of encompassed
in what you're reporting on in the state program report but if you have sideline
programs that you wouldn't typically report in
the state program report just make sure that when they come through the
evaluation they're sort of set aside as separate
somehow yep yeah and rachel we can do it go for
it that makes the evaluation more useful go for it
just you know make sure you're addressing one of the stakeholders on
mls for that for that concern of the evaluation
thanks terry um all right let's move to the next
slide then let's go through the questions
all right so the first set of quiet first questions retrospective and it
says what extent did the sla five-year plans activities
make process towards each goal so we're
asking you to basically assess well for these
fiscal years 18 19 and 20. you funded all these projects these
projects all have their distinct activities
how did this relate to each of the state goals in the five-year
in your five-year plan organize the findings around each goal
categorize your goals is either achieved partly achieved or not achieved
i'm going to flesh this one out in a moment about thinking about how you do
that and identify any key factors associated
with goals that were not achieved
uh next slide please
um the in the last cycle of data when we looked at the last five-year
evaluations we noted that there were 221 goals across all the states and
territories well on the five-year evaluation the
majority 60 the evaluators assessed these as
achieved in 23 states all of their goals were
assessed as achieved in 12 they categorized all the goals as partly
achieved and in the remaining 21 it was a mix
so going forward i wouldn't necessarily think
nothing in my head is i'm an evaluator looking at the results of all these peer
evaluations that thinks that those 23 states that
had all their goals achieved that those states are necessarily
the most successful i'm going to be actually more interested in probably
those other 33 states to start seeing where the learning was
going and how the planning might be going it could be that those 23 states
are just perfect but there's nothing necessarily inherent
in that
uh next slide i'm just jumping in with the time check
uh there's about 15 minutes left in the session just a heads up
awesome thanks laura i think we're doing good but that's awesome thanks for the
heads up sure okay judging the evidence achieved
partly achieved not achieved um here are some ways to think about this
if it's if let's say it wasn't fully achieved i'm not sure
it's partly achieved they're not achieved
here are some things that why it might not be fully achieved
you know we got this we got the work going but we need
more time unexpected things arose like a pandemic
we needed to pivot things they went really great you know went so
great now we're thinking about how we can
scale it up things went so great now we're trying to
think about how to extend it to new audiences
things went really good in hitting that audience but now we're starting to think
about ways we might change the type of delivery to reach that audience
or maybe there were some other issues that came up that we didn't really
factor in like changes in staffing budgeting partners
next slide please
so the second retrospective question asked to what extent
did an slas five-year plans activities achieve results that addressed national
priorities associated with measuring success focal
areas and their corresponding intents quick trivial lesson
measuring success it basically takes each of the projects that you report in
the state project report and says why are
you doing this project we link it to one type of an intent
the attempt itself is mapped to one of six
focal areas could be institutional capacity
information access lifelong learning human services
employment economic development and civic engagement
so by default every project is not only mapped to a state goal
it's also mapped to an intent at the national level which corresponds to a
focal area next slide please
so here's an example of walking through it
maybe you have a state goal which is lifespan learning we can map that to the
imls focal area of lifelong learning maybe we had a project that was stem for
all and that stem for all matched the intent of
improved users general knowledge and skills this is the
focal area of lifelong learning the state goal of lifespan learning if
it was summer reading it matches the intent of formal education and lifelong
learning matches again to the state goal if the
project was senior encore this is general knowledge and
skills also within lifelong learning matches to the state goal we can
continue this through with community connections
trained workforce you can see all the types of examples
what we're just showing you here is the ways in which we can use the evaluation
now from looking at all the hundreds and
thousands of projects that have been taken place over the last several
years we can aggregate it up to a state level
we can go state by state to start looking at some of the themes and
patterns that are happening within the state and
across states and territories next slide please
the third retrospective question asked did any of the following groups
represent a substantial focus of an sla five years plans activities for those
who answer yes to any of the groups please discuss
what extent each group was reached and we're going to
flesh this one out also in terms of the substantial in a moment
library workforce is is one of the groups individuals living
below the poverty line ethnic and minority populations
immigrant refugees individuals with disability and etc we
can map this all within the spr in your project
activity so there's theoretically a way to do that
next slide please
when we looked at the last five-year evaluations we saw what groups were they
were the projects being targeted to we saw that 31 were helping library
workforce we saw 26 helping individuals with
disabilities 18 was school age youth 17 with
with children those five years of age and younger
and eight helping families we also saw some other groups that smaller groups of
states was worse were servicing five states were helping ethnic and
minority populations five states with individuals leaving below
the poverty line four states with limited functional
literacy or information skills and three that were unemployed or
underemployed six states we saw no substantial focus
on specific beneficiary groups but for the vast majority
we could start seeing some targeting on this
uh next slide please the question asked third question asked
what about beneficiaries with a substantial focus how do we figure out
what the substantial focus is what we're saying is for purposes of the
question look at the project activity map it to its project and
figure out if that project's budget had at least 10 percent of the funds
that were committed to it that includes both the total funds both
the lsta and match and that covers 2018 19 and 20
cumulatively next question please next slide please
process questions beyond the results questions we have some process
questions one asks has an sla use data from the spr and elsewhere to
guide activities in its five-year plan specify any changes that you made in the
five-year plan and why these occurred if you had to make some changes
but you change the goals let's say how and whom did you share the data with
and what other types of data did you use and we're asking a very new question
this time around which is how did the last evaluation
inform this one and how have you used this information
throughout the cycle we're trying to basically try to make ways
to make the evaluation more useful if we have this information
it's not only a good question for you to answer to make sure the evaluation's
maximizing its usefulness in your state and territory
but it also helps us at the national level work across
all the states and territories next slide please
all right final final two cents one is good evaluation means it's
addressing meaningful questions those meaningful questions
are questions that you have the answers on you can act on it
you're really good evaluators are those who can take care of the scientific
objectivity that the data is valid reliable their
methods of analysis is credible and that their ways that they're
providing this knowledge is very useful for you and the other
stakeholders buffer your project schedule so you can
anticipate to anything that goes unexpected like the needing more time to
do the survey than necessary and be prepared
from the get-go of how you're going to disseminate the
findings to make sure that others can be using the information that you're
sharing next slide please
some reminders ensure that you select an independent evaluator
talked about that before feel free to follow up with us if you need to
need more advice on this guidance on this
remember that you don't need to include the care funds in this evaluation we
think it's probably more helpful for you just to be thinking
about how the pandemic it has disrupted what was been happening during this
period of time and where you'll be moving as you start thinking about
beneficiary groups think about that 10 threshold that's what we're going to be
using to figure out if your the funds were specifically helping
a particular group calculated across the entire three-year
sp three-year time span i want to add one more thing that i i
needlessly omitted earlier when we talked at all
about making the judgments about achieved partially achieved and not
achieved imless is not setting up any universal
standard for it we're instead asking the evaluators to
be working with you on that and to make sure
they're transparent in terms of the method that they're using
that should be very transparent and up front
and final reminder is to plan for others to be able to use the evaluation
before you doing it before it actually gets going right now during the planning
during the actual process where you got the evaluator on board
and afterwards after that evaluator has dropped off the final report to you
next slide please so next steps of how we're going to try to help next slide
please
okay we will we will provide 2018 2019 data for you and your evaluators we
know the spr can be a little bit complicated
to manage lisa frahill is going to be talking
tomorrow more about this but right now i just want to let everyone know
that we will make the data available for both at for both the project level
data as well as the activity level sometime in this summer we're going to
launch and evaluate a community of practice it's intended for lunch
all of you have your evaluators on place we can bring
the evaluators together we can bring you together as necessary
so that common issues that go across particular
particular states particular territories we can address it together and build the
capacity to make the evaluations most useful
if you've got any questions or comments particularly before we've got this
community of practice up and running don't hesitate to get in touch with the
program officers get in touch with me we'll do our best to
help out and now with our remaining five five minutes let's take any remaining
questions and comments terry i'm going to turn it over to you
for the first question from jean and she asked could imless provide us
with a template for the rfp gene we haven't traditionally done that
but we know that within the states there have been rfps that have been shared
among you and we hope that you'll continue to be
resources to each other
cara cara asks and we'll give dennis's poll results in a second not a question
but a suggestion that it might be helpful to newer coordinators
to have guidance or feedback from this group regarding how much staff time and
money to budget for the evaluation for for
small sla large sla sub grants or newer sub grants
terry it's almost a leading question but i'll have you answer this well we're
going to have a panel this afternoon so you're going to get to hear from your
peers just about those kinds of things
i'm going to ask madison if we had any earlier questions we might have missed
matt did a really great job of answering questions along the way
so i don't believe we have any additional questions
okay great natalie just curious why don't you find it
distinguished useful distinguishing between outcomes and outputs in my mind
outcomes more directly speak to impact am i overthinking the difference
yeah i'm a contrarian natalie most people would agree with you here's what
i would say when we when we when we think about
outcomes and and outputs particularly in the world of library
what we're really trying to think about is what difference did it make
that we spent money on libraries it's very very hard
to find transformative change long-lasting
change that happens in an individual's life
from a library a library is beautiful and wonderful it's small
barriers it reaches the entire population
it's one of the greatest public services we offer in the country
there's it's much more important to me to be able to talk about the results
than that happens from the experience when somebody is touched by a library is
reached by that library than to be able to think about well what
impact did that make in terms of how much more money that person might be
having how much more additional how much more did the test scores might
increase that to me is not such a useful not such a
meaningful question for both those who are trying to build
and improve libraries as well as for policy makers who just want to make sure
that taxpayers dollars are being well spent
what's really important what are the results what are the ways in which these
libraries are reaching different different parts of our
communities across the country natalie i hope i answered your question
so i just want to say one thing about what matt just said i think that
experience that he talks about and he talked about it very eloquently
is itself an outcome and i think it's a narrative outcome we need to tell the
story about it but that that is as important
an outcome as exists in the world and it may be it
probably can't be quantified which i think is is matt's
excellent and completely true point um but it can be described and and uh
and that's part of our job as the imls for sure
and part of your job as librarians and state librarians
uh is to describe to to give the narrative of that
that experience that it can sometimes be transformative and is always worthwhile
it looks like we got a question about um halfway through matt session
that was a little tangential but claudia from florida was asking
my concern is meeting moe next year or the year after given the state
budget cuts resulting from the pandemic i assume many other states are thinking
about this too we don't have a dedicated maintenance of
effort session at this conference but i will say that there is flexibility
in the maintenance of effort provision in that we can give
waivers if you go through sort of a paperwork process with us so if you miss
your moe all is not lost there will be more
guidance that we send out about that later on
all right well we're doing great with the time nice job matt
and dennis thank you for the guidance into the five-year evaluations
we're going to take a little pause on this topic and come back to it this
afternoon as michelle said we're going to take a little layover
so the next session will be after our 10-minute break
we're gonna come back at 10 minutes from now and
um we're going to hear about the two cfr 200 changes which is the uniform
guidance or allowable cost things among others so here we go on our
break and we'll see you shortly to you earlier a few weeks ago
um something we recall in the top 10 the top 10
changes to the uniform guidance and two cfr changes that have come down
just this year our speaker of the day our imls colleague cam trowbridge
is the architect of that document and has
has delved so deeply into the regs so that
you don't have to dive quite as deeply into the regs so we're delighted that
he's here um the handout the top 10 document
is now a handout as part of our conference agenda so if you
don't recall seeing that you can get to it from the agenda that was
linked in some of our invite language to you
and with that i'm going to turn it over to cam our
deputy general counsel
oh you are muted cam is that better yes thank you thank you
i'm just getting the screen up here um
[Music]
i want to welcome everybody for being here um yes i'm just i'm having a little
difficulty just seeing the slides on my
screen
i can see them on my end um are you in maybe if you change the view
up in the upper right part of your corner would that help
oh and you're muted now cam i'm sorry you're muted i'm not hearing you terry
can you hear me now yes i'm just gonna log back in and then
i could i could see you all so pardon me for one second i'm just gonna
go out and then come back in okay
all right i will say as cam's getting set up again we've seen
a lot of interest in this in these topics
um the two cfr 200 if that's not a familiar set of
terminology to you a lot of it covers the allowable cost
provisions that we use in grants to states
um so there are some new and different flavors and nuances
to those allowable costs okay yes we'll get right into it
it's all working now excuse me for for that anyway
um yes as terry said uh there's uh there have been a lot
of changes um in the uniform guidance i'm gonna go over
some of those which i think will impact most of you in your capacities as states
and pass-through entities um and uh you know basically i'm going to give a a
brief overview of the statutory and regular
regulatory framework that imls operates in
um then we're going to talk about the actual changes themselves
and we'll finish with um some found some very good resources
uh for where to go for additional information faqs
and then we'll have a little time at the end for um
for questions so terry if you wouldn't mind going to the
next screen thank you um again i just wanted to give
you a sense of where the uniform guidance and the revisions fit
in the overall universe of statutory and regulatory uh framework
for all of this um imls itself is is authorized uh
under the library and services technology act
um we also have another program the african-american
uh grant program like every federal agency
it's not enough to have just an authorization act you really need to
have appropriations money to make it happen
uh so we have you're familiar with this annual
appropriations what we get every year unusual things we haven't gotten so much
but we do have now are uh karzak funding as well as the
american rescue plan those are those are special appropriations i know terry and
crosby talked a bit about those at the beginning
they would still need to follow when you're implementing them they would
still need to follow the uniform guidance when you're
talking about what types of costs are allowable generally how to oversee
projects financial management internal controls
that type of thing so those are the statutes every agency
also has regulations that implement the statutes
imls has two kinds um there are the imls specific ones
and then and those are very very small and 99 of what we need to follow and you
need to follow are in uniform uh guidance so next slide please
great so so again quickly uh the imls authorization statute
museum and library services act that that sets up the grants the states
program it also has you might have heard of some
of our discretionary grant programs uh national leadership grants
laura bush 21st century grants museum programs
and our native american native hawaiian programs
next slide
um as i said it's not enough to have just an authorization you also need
money to implement it um so apart from the cares act
and the american rescue plan my mls says the bread and butter appropriations uh
this is just a quick snapshot for fy2021 um
the grants to states program has 168 million
uh dollars which if you if you take time and look at this slide
that that's probably the majority of the funds that i melis has
uh goes to you all in the grants to stage program uh
next slide
thank you uh so in terms of the regulations i just wanted to let you
know they're out there they are they most of our mls specific
regulations apply for our museums um there's a definition
of museums there's uh there's a requirement that
museums can't purchase objects um for collections just one i'll make
that very quick that's that's our mls specific regulations what
we and you um should use for the majority
of all your activities with imls are on the next page
are the uniform guidance for grants that's two cfr part 200
it applies to all of our grantees as well as all of our
subgrantees sub-recipients for those of you who are new
to to grants federal grants um if you look at two cfr part 200
probably the most useful provisions for you uh would be found in
subpart a through definitions what's a contract what's a grant what's
federal financial assistance subpart d uh which is really how do you how do you
administer a federal grant what type of internal
controls do you need how do you do reporting uh if you're a
pass-through entity which most of you are what does that mean and
where do we go to find out um what we have to do uh
subpart e is um it's a pretty illustrative list it's not all-inclusive
but it's the types of costs that you can and can't incur so it's
it's a basically a listing of allowable unallowable costs
again when you're carrying out your annual imls
appropriations as well as cares act and american rescue plan
you should be looking at this subparty in particular to see what are generally
allowable cost and finally there's subpart f which is
the audit provisions um that you need to do for your for your
grants the next page please thanks terry
okay so again we're talking about the regulations
the big regulations to cfr um omb published in 2014
basically it was a massive overhaul of of all previous grant regulations
and they put it together they want to make it easy for everyone to find in one
place uh they called it uniform guidance they
promulgated or issued it or it became effective
in uh december 2014. this is the first the uniform guidance revisions we're
talking about today was the first time since 2014
that these uh these regulations have been updated and that's what we're going
to talk about and again i um what's going to follow
are things that pertain to you i think uh most
particularly as states and as pass-through entities
um but it's probably it's it's good if you
could take some time um to look at all the changes and i'll
show you where those can be found at the end
but all of the changes in the unit in the grant revisions
apply uh not just to you as states but also
to to your uh sub-grantees applies to non-states to non-federal entities
and it's this good idea to be familiar with the changes in those provisions
because as as a pass-through entity you really
need to know you need to make sure that um your subgrantees are are following
the rules anyway next slide please
okay so these all became omb published these new revisions on august
13 2020 they didn't just make changes to
the uniform guidance and again when i say uniform guidance i'm talking about
two cfr part 200 which is the basically the
the bulk of grant regulations need to follow
um so omb made big revisions to the uniform guidance
at the same time they also made changes to
two cfr part 200 which is uniform guidance
they also made changes to cfr part 25 which is the sam system and the
universal universal entity identifier section as well as two
cfr parts 70 170 which is is your reporting of
subawards um and they added a new provision a new
part uh to cfr part 183 never contract with the
enemy and we'll we'll talk about those quickly
at the end um next slide please very
okay so these changes omb issued them on august 13th 2020. the majority of
them became effective they went in they are
now to be followed for every award that's made after
november 12th so your new fy 2021
uh grants the states awards would follow um
these uniform guidance grant revisions if you had an existing award maybe say a
2020 award um you would still you would you
wouldn't follow the new revisions you would follow the
the grant uh regulations uniform guidance that was in place at the time
you received your award so you probably need to know both
because uh it sounds like you have currently uh both fy 20 and fy21
awards um so those the bulk of the uniform of the uniform
guidance revisions became effective on november
12th there were two that became effective um on august 13 2020
and and those are uh there's prohibition on certain telecommunications
video surveillance equipment i'll talk about that about that in a minute
and there's a new provision on on termination which you don't need to
worry about so much at this point uh next stage thanks sir
okay so we were trying to figure out what's probably
in the order of what would impact you most immediately um
with these revisions and i think i think reporting will
will be the biggest change for you right off the bat um
your final performance reports under the old guidance had to be
submitted to imls within 90 days after the end of the period of
performance and date omb has now made that push that out a
month to 120 days um the reason om did that was so that
you will have time to collect all the all of your sub sub grantees um
final reports synthesize them and then and then submit your
final report to imls so again you now have until january 30th
i think it's 2023 for um for your new fy 2021 awards
uh sub recipients they don't have that luxury yet they they still
have 90 days to get their final reports to you
so you have 120 days to consolidate all those
we see them consolidate them and send them to imless
your subs still have 90 days to give them to you
those are final reports we're talking about um for the
annual interim reports and i believe this will apply for
uh cares act as well as american rescue plan you
you still need nothing's changed there um they're still due to imls
uh within 90 days at the end of the period of performance for the first
for the first year that makes sense uh next page please
also what's changed and again this is one of the things it hasn't changed much
for you as states but it's changed quite a bit
for your for your subs um so let's let's start with you as states
so you will still you will still continue to follow the
same policies and procedures that you use
for your non-federal funds that hasn't changed what has changed for you
is you now must also comply with two new provisions
in the uniform guidance um the first one is
200.321 and that's contracting with small minority businesses
women's business enterprises and labor surplus areas
and you also need now to comply with uh 200.322
which is domestic preferences for procurements
um i'm going to touch on those in a bit but i just want to let you know so you
you're still going to continue to follow the rules
that you always have your state state laws but now you also need to
you've got two additional ones um in addition to the one you had before
[Music] also to comply with was procurement of
recovery materials but you've got two new ones as well next
page please
okay so again yours your procurement hasn't changed much
but your recipients um that who are not states that you know
your your nonprofits the entities you give sub awards to
omb has re has i think they tried to simplify it
but basically um the procurement methods that your
subs needs to follow now there it's been grouped into three
categories uh informal formal and non-competitive
okay so informal informal you can use um
for procurements for contracts that you're letting up to 250
000 so we talked about informal methods those are all the procurements
you're going to make um or your subs are going to make up to 250
000. informal is divided into two types
of procurement methods there's the micro purchase
method so your sub can purchase anything up to ten thousand
dollars without having to go out and solicit uh
a number of bids for example you can they can use
purchase cards for this um for purchases between ten thousand
dollars and two hundred fifty thousand dollars
the uh the subs need to use
uh small purchase procedures and again these are all
if you want to get not going to take up too much detail time on this but
the details are in 200.320 so small purchase again they can
it's a little bit easier than formal procedures so this is everything
between ten thousand dollars and two hundred fifty thousand
um and all they need to do something to do is
um you need to solicit quotes from an adequate number of qualified sources
um and again that's designed to be less less formal above 250 000
when they procure for 250 000 they need to
follow formal uh procedures and those are sealed bids
uh proposals they need to have uh evaluation criteria
uh it's a lot more formal as you'd expect
um and then for everything above that that that's not
informal or formal is um is a non-competitive sole source so if they
have an emergency they've got to procure something for
they can follow nonprofit non-competitive procedures
if they're buying something that's under 10 000 you know i don't know why they
put this in but that's a non-competitive um procedure as
well and that that also dovetails for the
micro purchase because that's also under ten thousand
there are ways to increase um the micro purchase threshold above
ten thousand dollars uh it's a lot of detail here it can be
done basically your subs can push that threshold up to fifty thousand
dollars by self-certifying that they're low cut
their low uh risk oddity uh and they can go above
fifty thousand if they get imos permission and again
that's to push up the micro purchase threshold so for
purchase cards um without having to go out to a large
number of sources micro purchase is what they
should do that that is a little bit um confusing
there's more detail in there this omb got a lot of comments
on this section and they tried to simplify it all
um but if you look at 200.320 that um that that spells all up what
i've been talking about and if you also look at the federal
register uh you'll see um omb has responded to the comments
they received and responded to those comments and
redesigning this section i think i'll leave that right now uh
terry next slide please
okay again uh it states you're still following
your state procurement procedures uh except you're gonna do
you've got two new procedures to follow one is the domestic preferences
um and that's under 200.322 uh that's new um and again this is when
you're purchasing goods uh or equipment doesn't say
anything about services but if you as a state are are purchasing
equipment for example you need to the greatest extent practical
uh to provide a preference for basically things that are made in the united
states um [Music]
this has to be included in all the sub-awards this this flows down this
requirement um there's no dollar threshold um so
you basically can't say well i'm just buying something for
two thousand dollars do i need to follow the domestic preference um the answer is
yes you do but uh omb recognized that this might
cause people some some issues so they said well just
remember it's you have to follow this to the maximum
extent practicable so if it's not practical possibly super
low dollar value you don't need to um it's not
practical next page please
okay what's new for everybody uh you estates as well as all of your subs
is uh we can't use grant funds you can't use
grant funds um to provide to purchase
telecommunication and video surveillance equipment
from uh or systems that are that are manufactured to come from this
limited number of manufacturers most of them i think all
of them are in china again it's i think it's for national
security reasons this is this requirement is new
it comes to us not just through the grant regulations but through
national defense appropriations and it's basically there's a list right
here but this list can be updated it can be
updated uh by the department of defense so you
may ask where where would i find this list
like of equipment i can't purchase and the next page there
again i'm speaking quickly just because we've got a large amount of material to
get through um yes so and again this is
this is one of those changes that is effective
for all grants after august 13th 2020. so even if you have an existing
grant it's not a new award you've got a 2020 imos award
you can't use funds to purchase this equipment
it's one of the few that became effective august 13th instead of
november 12th but i think what the way to look to make
sure that you're not procuring any equipment that's
prohibited is there's a link here uh system for
award management uh sam and basically on sam there is a
an excluded parties list distillation list and if you type in
if you're thinking of contracting with on entity and you just you can search
to see if their name is on that um that sam exclusion list
that's that's the best way to check uh there's some great
guidance omb has put out great faqs i put a link there
um and that's really helpful but this is this is a new requirement for
for everyone that does grants um us as well as as i'm als
next page please
um okay so cost principles were also changed
again the majority of these um didn't impact you directly
as states uh the one that did is um it's the it's compensation
uh fringe benefits pension plan cost uh i tried to show that the change that
they made there um and i had to look at this several
times myself to to understand it i'm i'm not an
accountant but basically reading the notes from omb they said
that they are revising this the language you see there
that's new to require states and local governments
to be compliant with government accounting standards board
number 68 for pension costs and i think that means that you will be
able to continue to measure pension costs using actual
and funded expenses that's all i'm going to say on that one
next page please
okay so um what are your responsibilities as pass-through
entities when you award subs when you have uh
contracts that you let when you're basically a past strategy
as as most of you are um you need in your award in your sub award
you need to include in that sub-award a um a federally negotiated
negotiated indirect cost rate so this basically means that your your sub
will be able to use an indirect cost rate if they don't if
the sub does not have one does not have a federally
negotiated indirect cost rate um then you and the sub are to work
together uh two one of what two things can happen
um one you can negotiate uh an indirect cost rate with the sub
again this is a sub that doesn't have a federally negotiated indirect cost rate
you can negotiate with that sub to come up with a rate
that rate can be one that a different pass-through entity
and yourself have used before or the sub can say i want to use the 10
de minimis cost rate um again they're using this because they don't
have a federal negotiated cost rate so again in your sub award this is a
number you have to put in the award itself
indirect cost rate if they don't have a federally negotiated one
you can negotiate one with them negotiating a direct cost rate with them
or let them use the ten percent diminished rate uh next page theory
oh i want to say one other thing on that so if you're
where do you look to see if if your sub it does actually does or does not have a
federally negotiating direct cost rate uh omb said those those rates are going
to be all put up on one central central government website um
but they haven't said when and they haven't
said who's going to do that so stay tuned um i just want to say one
more thing about indirect cost rates the de minimis rate and this this is
more for your subs or your um your subsidiaries or your your sub
recipients the 10 rate used to be
if you have never if you're a sub and you've never had a federally negotiated
direct cost rate you could get 10 percent um that's a
cost rate uh the old language it said if you've
never received that uh over time people said well how do i
know i'm a new uh i'm a new authorizing official i'm a
new project director how do i know if my organization has ever had
a federally negotiated indirect cost rate so
omb changed this to basically say if if you don't have a rate right now
so if you're a sub and you don't have a current
federal you don't have a federal current federally negotiated indirect cost rate
you could use the 10 rate and you don't need documentation to justify it
next page
okay the other thing again uh putting in the the indirect cost rate is
one of your new requirements as a pastor entity the other requirement
as the past serenity that's new for you is is audit findings and resolving all
the audit findings uh you you have to look to see that
your subs don't have audit findings if they do you only need to be
concerned about resolving the ones that pertain specifically to your award
not the audit findings that are are general they call them cross-cutting
findings so if your stub recipient has audit
findings and it it covers a number of
awards um you only need to worry about the one that
that you're giving them and that's new uh next page please
um this is sort of new electronic age um you should wherever practical
use open and machine readable formats for for reports for information for
documents you still you still have to accept if
someone only has a paper one you still need to accept paper documentation but
they're really trying to push everybody to the electronic age
where they can they've defined ombs defined
[Music] machine readable format as a format and
standard computer language that can be read automatically by a web
browser or computer system next each week
okay closeout is sort of the other half of the final report
submission that we talked about earlier with you 120 days
um so in the closeout when you're closing out your imls grant
when you're closing out your subs award to you
um basically you still have 120 days to get your final report to imless
your subs still have 90 days that can be extended if there are
reasons to extend it the sub als has 120 days
to liquidate all their final financial obligations
um used to be 90 days now it's 120 days uh next page please oh and cam uh you
have about 15 minutes left just so you know for
timing and a lot of questions yes
all very very quickly yes thank you laura
okay so the um so the they put a little teeth into this omb
they're basically saying um if you have these awards have to be
closed out in a year if um if you don't have all the
information from your subs you just have to close it out with
whatever information you've gotten if imls doesn't receive um
all you know the final reports that are due to it after a year
um imls will close up the award with the information that's received
and imls will then have to report to uh fappis um the fact that
uh the recipient has not complied with the terms and conditions so that's new
um next page please i know we've got a lot of questions
uh other changes uh i've been talking about the uniform guidance changes the
other three changes and i'll go through them
very quickly are uh well let's just start with the next page
okay uh this is for two cfr part uh 25. very quick very quick there's
some terminology changed uh you're probably all familiar with
nuns the duns number data universal numbering system this
comes from dun and bradstreet uh right now everybody has to get a duns
number uh omb and the world are changing over from
duns to a uei which is unique entity
identifier which will be given to everybody to replace guns
and sam sam will give those numbers out sam of course the system for award
management next page please
okay so what's new um for you is that you have to inform
your subs that they need to get a uei unique identity identifier so you can't
make an award a sub award to them unless they have one
so subs have to go in and get get a uei they they have to get a uni uei through
sam but they don't have to complete the full
sam registration process but they do need a uei um
okay next page please
uh so when will this changeover happen um what we're anticipating hearing
is that everyone will still use guns until october 1st this year between
october 1st of this year and early april next year
um sam will accept either duns number or uei
after i think it's going to be april 4th 2022
only uei will be accepted next page please
mls can grant an exception if there's an emergency
that prevents somebody from from completing a uei um
if we do grant that they they still have to get it within 30 days
next page please part two cfr part 170 and again i
apologize so quickly uh what's the only thing
that's really changed for you all is uh you used to have to report sub
awards that were 25 dollars or more uh
that twenty five thousand dollar threshold has been bumped up to thirty
thousand dollars so now you need to report
some awards that are thirty thousand dollars or more
uh next page this last one never contract with the
enemy who would uh this probably won't apply
to most of you i'm not sure when it would it's for
subs that are over 50 000 that are performed outside the
united states and that are are given that you give either as a subgrant or a
contract uh to someone who's actively opposing
united states or coalition forces uh hopefully you won't do that um but
that's a new a new um requirement um just to check
we check before we make awards that um that nobody is uh on this list and again
this is this will be found in the new sam list
um but you probably won't i don't think you're you're doing many
overseas projects to enemy combatants but this is a new a new
provision uh next page please
uh yeah finally so we just tried to put together some resources
there's a lot i've been talking about there's a lot that's changed um
if you need to find pretty much the latest version of
what are these what do these provisions actually look like now
if you go to ecfr.gov that's the electronic
system i think it's done through the general
gpo general publishing office government publishing office
i i go there when i'm just you know someone asking an allowable cost
question i'll go there because it's got all the up-to-date consolidated language
on the regulations if you want to find out more about well why did i
omb change this provision or that provision in
revisions um you can go to the there's a link there for the august 13th
federal register notice and again probably the most important thing is
just knowing what does the regulation say
but if you really want to find out why they make the change this is the place
to look um of course like everything else once
they publish it there were corrections that they had to
make not many and they are then the third link the
ecfr site above has has made the corrections
so again if you go to ecfr.gov you're going to find the latest
consolidated synthesized language and finally there's a cfo.gov
which has a ton of resources that used to be all on performance.gov
is it possible to click and see if we have a second if we click that
uh just the cfo.gov link
oh my god it might be because we're in presence yeah i know that's
there we go okay and then if there's the uh i think it's guidance
policies resources thank you terry
and there's one on uniform guidance it's on the bottom row
and then there should be one that just says redline version yeah
so if if you if if you're trying to get up to speed very quickly in all these
changes and you just want to know what what was
changed i've got a basic understanding but i know what i want to know what's
different this privilege this draft um shows you
way down the bottom of it um you could see what's been
deleted what's been added um it's red line it's actually blue line but
um it's a good it's a good quick reference it's not the official official
official version but it's the best thing i've seen in terms
of uh tracking what's new and what's different
anyway i've gone through this quickly i'm ready for questions if you have them
hi cam thanks so much for that presentation i
have a bunch of questions for you by the way um
debbie muscleman asks do we have to give sub awardees 90 days to submit their
reports can it be less time uh it could be less
time if you negotiate less time with them
if they they get 90 but if you can talk with them and they agree to do less
they can do less great um there was there was a there was a few
questions around around what grant this applies to and i
answered in the in the chat but because the these
changes went to full effect in november your fy 20 award is not affected by your
21 award is is that correct that that's a great question
yes so i guess the easy part of that is your fy21
uh all parts of it now need to follow the the revisions the uniform guidance
changes that we've been talking about if so
that's your fy21 award your fy 20 award
um which was issued to you before november 20th
needs isn't impacted by these changes so these changes would not apply so in
other words um your your report to imls for your fy20
report would still be due to imls in 90 days
your fy21 final report because it's new uh
you would still would need to would now have up to 120 days to be
right um that said the if your fy fy20 although i just said
it needs to follow the pre pre-existing um guidance
uh if the only thing that you if you have an fy20 award and you're
going to procure equipment you need to make sure that um
that you're not uh running a follow the prohibited equipment
restriction which became in which came into effect on august 13th
last year great so again to find that out if you're going to be buying heavy
machinery or just a lot of equipment uh look at to cfr 200.216
which talks about the prohibited equipment most of it's the chinese
equipment just check sam to see that it's on there
um but otherwise your fy20 award would you just follow it
through the way you have been great and that doesn't and these new
changes do apply to arpa i saw nancy's question just come in
because arpa's brand new exactly yes um uh
what is a cognizant agency is it an sla or subawardee um no that no
in that case they're talking about federal agency
so it would be you it would be your your cognitive agency
and it's a very complicated formula to figure out which federal agency it is
but basically uh it comes down to who do you
get most of your federal funds from and if it's i'm a less then we're your
cognizant agency subs should have cognizant agencies generally as well um
because they might be doing other other federal recovery but it but it's
basically the federal agency the federal
government is the cognizant agency
now chris and connecticut asks what does telecommunications include
no okay um i'm gonna see if i could pull up on my
screen telecommunications there's a new um definition i think it's
200.471 um i'll read it for you here
telecommunication and again this is the alaba cost section
um
it basically says telecommunications are allowable costs except for the ones in
216. i think there may be a definition as
well for just telecommunication equipment
yes of course
not able to see it right here but if you check the definition section
2 cfr part 200 um and and just go again i've been going to
ncfr.gov for the latest version great and it
looks like catherine prince put the definition in the chat thank you
catherine thank you um do indirect cost rates need to be an
option for small grants like 5 000 or less
um they haven't made a distinction for that so uh so if you've got a sub
grant that you're giving um they haven't said below a certain
dollar you don't need to give um indirect cost rates so i would say
yes all several words um should have an
indirect cost rate um i think that answered is there a
dollar amount threshold for indirect cost rate i think that's essentially the
same question um so it's all sub grants or i guess
there is a dollar amount threshold for indirect cost rates in
general um not that i'm aware of again they're
not they're not for contracts but they are for
for sub awards if that makes a difference so
you know if you're giving if you're giving it to a non-profit
um they probably should have one if they don't have one
then they're entitled to um to get to the ten percent the minimum rate
or another rate that you negotiate with them but but i haven't seen it say
um you don't need to include nine direct cost rates
on your sub awards and chris asks how was this different
from the previous regulations um for the indirect cost rate what i
understand it's different in that the that you and the sub
can negotiate that rate um so but right now i think the change is
well first of all the changes you have to
have in your sub award you have to have an indirect cost rate
put in there um and they go into more detail about
it's it should be that if the sub has a federally negotiated indirect cost rate
that should be the rate but they seem to put more emphasis now
on if they don't have a rate you can you can work with them um
either to get a rate that you negotiate with them or
or to give them the ten percent rate okay um
lee asks can we just continue to use the de minimis instead of negotiating rates
um
if well first of all if they have a federally negotiated rate already
you have to use that they're entitled to use that if they don't
and you're negotiating with them for an eight percent rate
they could say no no i want i want the 10 rate so
um they need a rate they need to use the indirect the federally negotiated rate
if they don't have that you can negotiate a rate with them but
but they can always say i want the ten percent rate
all right now does e er erica mccormick is asking does each
subword must does it must have does it have to
have an indirect cost rate or can they decline a rate
well um i think if you're negotiating with them and they and they say i don't
i don't want to use the 10 rate um then you could negotiate it to zero if
that's okay with them um with the sub but yes
you could have a zero rate if if that's acceptable to them
um marian in florida asks okay i think that's the same question
are we saying we must give them an indirect cost rate
um you already answered that um karen asks
can a sub-grantee with an indirect cost rate offer to take a lower percentage
uh i think the answer is yes if they have a they're always entitled to their
if they've got one their federally negotiated indirect cost rate
but if for some reason they don't want to use that um
and and you negotiate with them for something less uh
then that that would be acceptable all right there are a lot more questions
and unfortunately i think we're at time and we want to be respectful of the
breaks so what we will do is uh we will you know
round up the questions that weren't answered and we can either have a
lightning round at some other juncture in the conference
or we can follow up individually with folks if it
seems like these are more individual questions
but i want to thank him for presenting a lot of content
in a really short amount of time and for helping clear up some of the nuances
of what this means for us thank you cam thank you terry and i just
want to thank you all dude because i know
it's not there's it's almost 300 pages of text that they put out some change
not some not change but um i i feel your pain it is hard to get
through all this but again if you look if you look at the redline version um
which is that last week link we looked at at cfo.gov that
that'll show you what's changed what's show you what's taken out and what's new
um so that may be the easiest fastest way to talk about speedo
but thank you terry all right very good uh we are headed
into a break now and we'll come back for a panel that is again related to the
five year plans so we'll see you in 10 minutes thanks
everyone
so we are moving now into a panel which dennis is leading and
i am going to sort of turn it over to dennis to get us started
and get our minds back in the five-year evaluation space
all right thank you so much terry um i was so tempted during the break to
change the slide to put terry to vote three times in the panels just to keep
the keep the joke running i don't know why
the whole thing tickles me so much but i can't seem to get over it
anyway uh um as the title um refers to
we have the great pleasure of uh bringing on
three experienced um state library folks slash lsj coordinators who have done the
evaluation process before and they um three of them bring very
unique perspectives as well and so i'm excited to get started um
i'll introduce them and uh i'll also stop sharing a
powerpoint so it's more one-on-one faces and conversation with
the panelists uh and so i'll introduce them i'll ask
them a couple of questions and then there should be ample time for
some questions from everyone here so we can really
talk about as much evaluations from a practical logistical perspective as much
as possible so um what i'm going to do now is
bring our panelists to the spotlight if you will so i'm going to
spotlight myself and karen and ellen
and janet hopefully this is working the way i am hoping it will work
and i'm gonna stop sharing my screen it should just
be us okay this should work just fine um let me know if
there are any or let somebody else who's paying
attention to the chat um if there's any issues and
we can um go from there so um without any
further ado i'll go ahead and introduce our three
wonderful panelists uh karen egan from the illinois state library um
she has been at the state library for 25 years
but her first job in libraries was when she was 16.
she was hired by the local public library primarily to shelf books during
their summer reading program and then she started a professional
career as a school librarian and went straight to wonderful state
library work she was originally hired to work as the
youth services consultant until the director called her in one day
and said have i got a deal for you lsta coordinator was calling her name
and she was doing that for uh the next 18 years and then two years
ago approximately she was promoted to the
associate director of grants and programs which
is the equivalent to library development roles
and this is the fourth evaluation that she's been directly
involved in uh next up on our panelists list
is uh janet mckenney from the maine state library
she's the director of library development and the lst coordinator
uh and the library development division at the maine state library supports all
library types in maine public academic k-12 and special
by providing statewide services via our via their staff and with contracted
partners um which includes technology initiatives
statewide ils the statewide catalog uh ill band
delivery service professional development
consulting service and the state network's internet connections
so uh just you know a modest list of your average state library work and
janet herself has worked in special academic and public libraries for 34
years and has been with the maine state library since
2006 and this will be her third evaluation
and last but not least kellen ralya from the alabama public library service
she is the assistant director slash lsta coordinator at the alabama public
library service i think you're seeing a trend with these
esteemed panelists that they're slash lsj coordinator uh because they've been
there long enough to um absorb and to do many things at the
state library and she has more than 30 years of
experience with libraries 18 of those at the state library in
particular and she has three evaluations under her
belt and is gearing up for the fourth so to put
all of that together did some fast math and collectively on our panel we have 89
years of library experience and eight
evaluations have been completed between the three of them
so i don't think there's much more that i need to say to
justify our helping them speak to their experience about the evaluation process
and we're so grateful that they're here to share their experience
so i am going to start by opening up the panel questions with a simple question
uh why each of them chose the evaluator they chose for their
2013-2017 plans um and chosen at random but deliberately
um and organized way uh we'll start with janet from maine to kick off the
discussion thanks dennis so um the maine state
library is part of causeline which is the council of state
library agencies in the northeast and traditionally there's about 12 slas
in there and traditionally we group together
to do a joint rfp and um not every sla signs on but
most of them do and so michael york our esteemed state
librarian from new hampshire is also the treasurer
of causeline so he spearheads the um the rfp for us um i think in the last
evaluation um we had six libraries on the rfp but ended up
with 10 libraries joining in and the
um we had about four responses and then each state reviews the rfp responses
and then sends their rankings to michael and
so then it's basically judged by you know in that way
um so the last evaluation quality metrics
uh received the highest rankings from all the states
um it was a new company but had a connection
in history with bill wilson who had performed
the previous evaluations um in the um the last the last round and
so um those us when people were looking at the rfp
responses there were some respondents that weren't really
librarian focused and so that was one of the concerns
when everybody was evaluating the responses so i think quality metrics
because of the link with bill um kind of went up on everybody's
radar and so maine does mainly main does statewide
programs and we've never done sub-grants so the
evaluation is a little simpler in many ways than those
of the other states that had sub-grants um you know through the contract but i
think it was a real challenge um for quality metrics
uh working across the states with not only different approaches
but we seem to have kind of a scrunched timeline because they were a new company
and um i think they had a hard time
kind of coordinating uh what they were doing
so the capacity issue of the rf rfp will be what i'll look for in this
evaluation how many people are there and how many states you know
are are responding to the causeline
rfp and one of the reasons we do do this is the hope that there's not a lot of
evaluate evaluators in new england so travel
costs go in so it's always the evaluators come
and they travel and stay in boston or somewhere centrally
and then you know move out to the different
states to get their work done
for me wait
karen or calling kim sure i'll jump i'll jump in next um in illinois we
contract with the institute for legal and legislative policy studies at the
university of illinois here in springfield
now one of the advantages is that their activities encompass all
branches of government and their primary focus are
research and policy analysis and program evaluation for state government
in illinois we have the ability to contract
with a state agency so we can do an intergovernmental agreement state agency
to state agency without going out for bid and that's
huge those of you involved in the bid process
know what i'm talking about we've used uh this uh
institute for the last two rounds of evaluation and we plan to use it again
this coming round prior to that we have used consultants
but we discovered that when we went with the university
the quality of our evaluation actually improved
because they took a different approach to doing
the evaluation and i'll get into that later and kara
asked earlier about the cost and for us it's under 25 000
usually that's great thanks karen okay
you're ready for me danny i am ready for you all right
well y'all let me first say that working with lsta i find it much easier for my
happy place this is my happy place plus it covers up
the train wreck behind me that is also my office
but before i tell you all about our evaluator i just have this question is
it just me or did these five years go by way too
fast did not we just not just do this like last year
uh as dennis said this i've been here for
three evaluations uh the first two were done by the dean of our library school
at alabama and the one just the five years ago
was kate nevins uh kate was the head of saladnet back in the day
she has since retired from that now some of you new folks may not be
aware of that name but the veterans you guys will probably recognize it now
you'll notice i did not say we were old i said we were veterans there's a big
big difference there i was not involved in the rfp before
but once kate came on board i was extremely involved in
getting her up to speed with the spr as coordinator i have all of the plans from
our subgrantees in our rfp we include that we are
a pass through we do not tell them at least we didn't i have it up on this
screen we did not say how many grants we had which was probably smart
for the rfp out right now because we had over 300 we didn't want to tell them
that right up front so there you go but that's pretty much how we pick them
and like i said i was not involved in the first
uh set now our rfp for this one has gone out we've actually had
two responses do not ask me why people were not disclaimering to do our
evaluation it would have been the social event of the season but that's okay
so we had two response and one of them was quality metrics
and the other one was a firm we had never heard of before
and plus they wanted over a hundred thousand dollars and i'm thinking okay
you must not be real familiar with alabama because we don't really have
that for that so but that's where we are so we'll wait
great thank you um so obviously immediately
you can see um the differences in uh evaluators that were chosen the the
scope of it um whether one the evaluator was used
just for the evaluation or um also for the plan uh and so um
it really is an interesting approach to take it there's varied
it's interesting how varied the approaches are um certainly
so uh the next question i have for the group
is um what were some of the lessons that you learned
the last time throughout the evaluation process
um lessons like how the process of getting the data
for the evaluators working with the evaluator's schedule
um those kinds of um things you might not have anticipated
uh so for this one i will start the answers with uh karen all right well
well first i'm going to share the good bad and the ugly with you
under this question you guys so but first let me start by saying
in illinois we use the lsta funding for a mix of
statewide programs as well as subgrants so keep that in mind as dennis mentioned
earlier the five-year evaluation is really a three-year evaluation
and i just wanted to share with you how this looks in illinois
we use the federal funds the second year that they're available because they're
available for two years and it's usually stick with me on this
math you guys and it's usually on the state fiscal
year of july 1st through the june 30th timeline
for sub grants to have their funds obligated with another 45 days to have
the money paid out so you know it's the july 1st through
september 30th then the second year so it does
give us three years of full activities rather than the five years
and the last round for those three years we had 246
sub grants now that's okay that we're doing three years because
what it does is it gives us enough information to know
if the plan is meeting or not meeting our goals
and there's always that little wiggle room in there are those two years that
don't really get evaluated so just keep that in mind that that
happens with us now our evaluator did use the reports
from the state program reports but he also wanted as much information
as we could provide that included applications sub grants
reports any surveys or analysis that we had
done internally and so that meant that it was too much for us
to email to him so we actually put things on a thumb drive and drove it
over to him and handed it to him and said here you go let us know what
else you need well then we were doing press releases
and as as somebody said earlier the newspaper articles and
whatever we could provide for him and he also
you know followed up with subgrants and followed up with us for more information
as well so once we got the initial initial data
to him he worked with us to figure out what
gaps might be in the in the data that he wanted more
information and and again we either did surveys or
more interviews and you might be surprised to learn
that subgrants misreport
no matter how many times we say pick one focal area there are some
grants who look at or some subgrants who look at the focal
areas like a smorgasbord and say oh i'm going to do that one and
that one on that one for the same project
and they checked that the project will it will
actually impact every one of the focal areas when in fact
we know that's not true so when the state
now the state library enters the reports into the state program report we don't
give the sub grants the authority to do that
because that's where we correct their errors
so that we can fix them when they come into the
spr why i'm saying this is the evaluator recognized our frustration
that subgrants don't always accurately report and it gets uglier
when um they talk about counting attendance but i'll get into
that in a minute so what our evaluator did
and matt mentioned this earlier that he treated all the sub grants
as if they were informants in an ethnography
and he used special software and treated all of their reports
and documents as if each one were an individual interview with that subgrant
and somehow this magic happened and he was able to do this huge analysis of
the results what i especially liked is he unbundled
projects that we bundled in the spr so for example if we took collection
development projects specifically done by public libraries
and bundled them in the spr because michelle really
didn't want to read 100 reports that said the same thing
the evaluator looked at each of those projects individually
and their levels of success in meeting the proposed
outcome as well as the intent of the offering
um some grants also don't always count attendance they count attendance
the same way while we're trying to get to how many
unique individuals did you impact they often report
programming statistics which is totally different
so in many cases the evaluator is looking at apples to oranges statistics
and trying to make some kind of comparison
um it makes it hard to get a grasp on how many
people actually participated in the programming
over the course of whatever period is being evaluated
so that's one of the things we really worked at trying to get
consistent statistics in this current round
now for us the evaluator schedule was not
an issue because we really try to start early and get the information to them as
soon as possible generally by august they finalize
their evaluation data and spend the next five months or so doing their coding and
analysis and follow-up with the target goal to
get a draft evaluation to us by january so that we can work with them on any
deficiencies before the march deadline in or for the deadline to imls
by the end of march and so for us we start as early as
possible and i know we're trying to we we're
actually hoping to start getting them information
as early as next month if we possibly can
great thank you that's excellent uh i will pass it over to kellen
if she wouldn't mind sure okay since we do so much
in subgrants with our allocation uh kate had asked for specific
information regarding the number of grants in
specific categories and the nature of projects so individual
grant reports came in very handy for her as well as
spr info i use the individual grant reports
in my spr info and i certainly feel karen's pain i understand that
in response i also kind of beefed up a tracking spreadsheet
that had been in development to include a little bit more information i've got
it separated by year by quarter by goal
all of that because i want to be able to track all of my figures
and then when it comes time to it i can just say boom here's my numbers and i
don't have to try to find them that is something that we learned
another thing which i found interesting was
we did it'd be interesting how they do it this year because the past ones they
did town hall meetings around the state
don't know what we're going to do this time
when kate was here doing it apls staff did not go
uh she and she i think she had a point she was thinking it may
color what the librarians say it'd be like okay apls is here let's talk about
how fabulous they are we're not there then they might be able
to say you know they really are pretty bad you know they're pretty icky over
there but when she came back she said man she
goes i wish i'd been here she said because they were just singing your
praises so i thought well good at least we're doing something
right so that's a good thing uh something for y'all to think about
uh with the evaluation process also i think is
just making sure that you can do and i feel like i'm parroting karen here
is to get all of that information just as quick as you can because
we include in our rfp we include not only the evaluation of the
current plan but also the development of the future plan
and we mention it in one sentence and here it says using the evaluation apls
also seeks assistance in drafting an lsta five year plan for 23 to 27
based on the parameters of the lsta priority so there you go
evaluate it now write the thing for the next time for us
but it actually worked out pretty well so
that's great thanks kellen and janet oh thanks um so maine doesn't have sub
grants so a lot of the complexity um
that uh kellen and karen talked about we didn't have
but um the last plan was the one that i was
majorly involved with and so first of all the data collection
was difficult because um again we're not looking at the entire five
years i think we were able to look at three years
but there was also the it was two different versions of the spr
data and so downloading the spr data um caused
all kinds of problems and the evaluator was upset that she didn't have
all five years and so that was like one of those learning things like you're
never going to get those five years and and and so we started off on that kind
of panicky foot so you know i'm doing a lot
of pre-evaluation right on my own and trying to do some prep work based on
what i learned in the previous evaluation and so
just more actively um trying to think what the evaluators were
want and look looking back so that we're more prepared
um in both working with them so we're doing
some survey data so that it it's available to the
evaluator and um we also
want to make sure the last time it was very compressed i think they left
maine um last in line because we didn't have
subgrants and so getting the time uh
to do our focus groups you know it just wasn't really
great i think it would go much better now
you know we were trying to use uh video conferencing
and you know was pre-zoom so anyways uh
we didn't get the uh the amount of participation in the focus groups
because it was too fast and was too narrow and we didn't
have time to plan so um you know that's something that i'm
really going to pay attention to and you know really you know um
talking to the evaluator i think about the state live
the state's library landscape and how you do grants because
if they come from a state that does sub-grants and now they're working with
the state that doesn't do sub-grants there's kind of a
realignment um that has to go on about you know what
they think about and what they do and how they're looking at your
statewide programs so um i think that's
about it according it's much it took much more of my time than i had
anticipated and um i think overall
like spread over the year it i think it was about ten percent of my time
but that ten percent of my time was concentrated in about six months
that's great that was and i'm glad you kind of uh segued into that because
um even though we do have one more question i think we have
enough time to um ask this question that jamie brought up in the
chat there i think there's a collective interest in getting a sense
of how much time the this evaluation process takes um
uh and i'd be curious too is is it more time on the front end or is it kind
of consistently time consuming as you're doing the
data collection and and that kind of thing i mean you don't have to go into
too much detail but i think they people would be interested to get a
some picture of what to expect
uh can i speak to that yeah please i would say that the more you can get
together beforehand the better off you're going to be and
i think karen would agree carries over there nodding in illinois
because it kind of saves time a little bit later
like this time this the people who come in oh i'm going to be handing them a
whole bunch of stuff and because i want them to have what i
think will help them do a proper evaluation of of the plan
i think that we've we've done i think pretty well um
when we wrote the plan five years ago unfortunately our
crystal ball was um at the cleaners and therefore our the words covet and
pandemic do not appear anywhere at all so that kind of put a little hinky in
the thing so i'd be curious to see what the partially achieved and achieved
comes back as and i would like to make a a point that
uh matt said it earlier um finding out that you did not achieve something is
just as important as finding out that you achieved something
because it's all a learning thing and it's either something you need to expand
or it's not going to work in your state or whatever and it's just all a big
learning experience so hearing sorry you didn't
do that well it's not such a bad thing
i agree with caitlin is that our primary time is up front
because we want to gather the information to get it ready to give to
the evaluator and that takes time up front and so
while they are doing their analysis it's kind of a quiet time for us and
then once they get to a certain point they will have questions
that they're coming back to us and they might want to do focus groups or
interviews and they want us to make those
introductions first for example to a subgrant before they
they don't want to just cold call them and say hey guess what i'm an evaluator
that's like getting those phone calls of you know you need to renew your vehicle
warranty or whatever it was but um um so yeah up front is where the bulk of
the time is just give getting everything ready and then once
we get the draft is going over with them again and
sitting down with them and looking at their recommendations
right yeah and i'll just add on to that we had
to spend a lot of time with the editing
process um and finding that nice balance between
the narrative and the the data in there so that was a
little bit intense so i think you know just because
of the timing and how ours ran i would say i had
a significant upfront and then a bit of a lull but then it
was you know pretty uh constant during and so you know a busy beginning of the
summer and then you know um into october
it just really you know got a lot of review
of the draft plan in the data it's great thank you so the um
the final uh official question prepared question i have but we still have plenty
of time for others to um pose any additional questions or have
additional conversation um the question i have is what was the
impact of the last evaluation um did it inform
your 2018-2022 plan and if so um why or why not
okay uh i'm gonna i had written some notes here
and so i'm just gonna read this y'all just forgive me but i'm afraid i'll
forget if i don't the evaluation report verified what the
slaw staff thought regarding the usage of lsta
funds for example the most common grant
programs were collection development and technology and the i.t and
consulting services were heavily used by the public
libraries there were no major changes added to the
new plan for the 18 to 22 based on user survey and focus group
data having the plan development aspect
included in the rfp also helps streamline the process in
essence the evaluation and development procedures were actually done
concurrently so we could do them side by side
so afraid i was going to forget something write it down sure
no that's helpful information so um for maine i think the
the biggest takeaway from the evaluation and we're
actually looking at it already based on um uh response from federal program
officer that one of our programs was really too big
and it had too many activities so we looked at those
activities in-house when we were writing the plan
based on what was written in the evaluation and we created
two new programs out of that one big program
so the next plan plan absorbed many of the same activities and all the
same kind of things but you know did you know separate
separate focus um we had ce combined with our
consulting um activities and we split those
up and that was probably the the largest thing
but then we're really able to kind of focus on those
two programs and look at them separately so i think that was
the best thing we learned as we were reading through the evaluation
was the evaluator's outside look at you know the program we had
already changed the program of how we were working with libraries
from three regions working towards nine regions
or working that way and so it was a logical time to
make that change in the plan so um you know and it was also a time when um
you know lsta funding was you know under question and whatever
and so we really struggled with writing the plan that year um
of how aspirational to be because you know the the the federal funding
you know in my mind was questionable and we had people on staff that were
really worried about using losing the federal funding
so um but um i think the you know the the best thing was is
that we really were able to take stuff from the evaluation and work it into the
new five-year plan
great
um well for for us the last round was kind of like
an anomaly for illinois the last evaluation and plan
were due in the mid midst of the budget crisis that we were going through in
illinois uh for those of you that didn't know for
two years the state of illinois was without a
state budget there was absolutely no spending
authority which meant we couldn't make our moe for two
years because we didn't have the authority to
spend money and organizations dependent on state
funding were shuttering their doors we drafted the 2018-22 plan while we
were in survival mode we were carefully watching our draw down
of the lsta funds so that the limited matching funds we
had could cover lsta funds being drawn down forget the
moe we were worried about the match we weren't comfort comfortable proposing
to do anything perceived as innovative and
creative during that time when survival was the
key word and status quo was a good thing so there
were no major changes at that time that we made
to our five-year plan however the um evaluation
gave us some really good information and you know
matt talked about partially meeting some of your goals
we looked at that as opportunities that maybe those were goals that should
continue because the library still needed time to
address those issues and so that's how we looked at it
as well and we used we asked for recommendations
from the evaluator and used those in our plan with him also realizing
that it was survival time at that point
it's really really insightful um yeah well i i appreciate all of your
perspectives and we still have plenty of time for questions i
what you were mentioning karen um also just reminds me of how
the five-year planning process really does help
empower each state to take the broader purposes and priorities of lsta and
tailor it to what you all know is specifically going
on in your state and i think that's a really
what you just described karen as a great um reflection of of its
value in that way um we have a question from sulein but at the same time
i neglected at the beginning of this panel to push out another poll
it's a very quick one and if madison can do that while i ask swollen's question
it just gives us a sense of um how many folks who have or in attendance
today have done an evaluation before or if
this is their first time we're very curious to know kind of what
the what the forecast is for all for everybody across the country
this uh this time around so sullen's question was do you think
the evaluator's location will matter this time
she was thinking much of it could be remote with
with little to no travel compared to the last cycle
i i think it will help because we tried to do some
remote stuff but you know just trying to gather people
together you know when the evaluators had
you know like here are the days i can come to maine now
we have to find the people that can come to the focus group on that day
at that specific time so there was you know you know i think because our
evaluator um had actually too much to do that was
a problem so i think everybody's really used to zoom now and so
i'm really kind of looking forward to it because um
i i think that will be super helpful and you know i'm looking forward to it i
think there'll be actually more communication
yeah i think oh go ahead i'm sorry i think that's what we expect as well
the the previous three that i was here for
like i said the uh one of the evaluations
was in state and then kate of course is just over in georgia so she could just
hop of course across the border over here so these
the ones that we're i believe that we will be using and i
forget what state they're from but they're not from here and
but i think that janet's right it's going to be zoom meeting time
i really do something different absolutely yeah i think zoom is going to
be really popular for for um answering questions about
the evaluation and the use of the funds well y'all don't know it but see i've
got my jamma pants on down here i'm just
well we have the results of the poll oh and 53 of you
have previously been involved in the five-year valuation
um and 47 have not so right just give you 47 english
and 8 percent is terry developed i will have beaten that to the ground
and i'm still i'm still there with it uh but that's that's great and you know
it also speaks to um how well of a community of practice
all the lst coordinators are as well so um hopefully with our continuing um
informal meetups i'm sure that these kinds of more specific questions will
come up i think michael from wisconsin already
uh posted uh the alabama
rfp that's on i forget it's on a website somewhere
libraryconsultants.org i think um and so you know there's there's
plenty of um there's plenty of knowledge to uh
absorb among our peers so i am i'm thrilled to hear from
from everyone's perspective i am exceptionally grateful for karen kellen
and janet for be willing to share um their experience
uh and their expertise they are all very um
evaluations are all very you know special snowflakes
every every cycle so uh it's great to hear
the different approaches that that you can take
yeah i i think i answered in the chat somebody was talking about
how much how much money and i think karen you did
as as well so you know for both of us i think it was under
twenty five thousand dollars but i did see that question
in the yeah so that's a good one i think our
i think our last one was probably at it was 35 36 something
like that sure sick wasn't over 100 like those other people i thought oh my god
right i was like wow okay
excellent well if there aren't any more questions i think we can
wrap up a teensy bit early um but thank you again to our panelists i'll
clap on behalf of all of our all of our co-attendees
um yes and so i will bring us back to our regularly scheduled powerpoint slide
thank you all thanks dennis
and while that's getting switched out thank you panelists
um for that state perspective that we frankly can't give
as imls that is so valuable to everybody on the call today
um we're coming to the time in our program
that we we hope you've stuck around for because we always enjoy this so much
it's a chance for us to reflect on all the great things that you have done
over the course of this year dennis already mentioned
michael dennison's great work to sort of gather up these evaluation
details a special shout out to michael in wisconsin
and we have more and then as soon as we're done with this
i have gotten confirmation that cam will come back
and he we will do the lightning round of the two cfr 200
uh questions um and they may be you know a little partial but
we'll do our best and then we'll officially wrap up for the day so please
stick around um and at this point to handle our
recognition portion i want to turn it over to our program
specialist laura mckenzie who many of you are
probably going to hear from and see for the first time so please
uh yay welcome laura thanks so much terry um i am very
happy to present uh some great awards for all of you
and in keeping with our travel theme here our first award category is
priority boarding and these will go to earliest spr
submissions so you know whenever you travel
it's always a good idea to get to the airport or train station
early just to be sure that everything is going as planned
this year we are pleased to recognize states that submitted their spr
early as our early birds and the following states had the
earliest submission the first is from maine on december 3rd
thank you maine nebraska was the second earliest submission
sending it in on december 4th
and our third award goes to idaho they submitted the report on december
7th thank you so much it makes it so much easier
for the program officers to begin the flight
that is starting the spr report next up is the pre-check category so
this is for fastest certs and assurances
as you know another good tip is to use pre-check when you're traveling
and when we send out the award document we asked
that you return the certs and assurances within 10 days so we would love to
recognize the following states that quickly got their paperwork
completed it was as if they'd signed up for the pre-check service
thank you alaska maryland michigan and west virginia
all of them submitted their certs on january 21st
and now we'll recognize our tour guides or mentors
of course when you're traveling and going someplace new
it's always good to have a tour guide someone who makes the trip go smoother
so we would like to recognize the following people who have acted as tour
guides sharing with new coordinators their
administrative knowledge of the grants to states program the
first mentor we are recognizing is cindy borden from ohio
thank you cindy next rachel cook from utah karen riesch from michigan
jennifer peacock from mississippi karen egan
from illinois gene healey from colorado janet mckinney from maine nancy medema
from iowa and kathleen moyer pfeiffer from new jersey and we would
also like to especially thank three people
who are stepping down from the mentor role a few of our very
own rick steves if you will judy tyree from
oklahoma debbie hall from arkansas and jamie ball
from arizona we would like to acknowledge the
contributions these women have made over the years in sharing their
experience an extensive knowledge of the grant
program
now lastly fiscal frequent flyers that's all of you all of
the terries all of the states everyone knows that
when you're a frequent flyer you win rewards
this has been a really tough year for all of you because you have had to
manage multiple grants so imls is acknowledging
you with a frequent grant recipient award
we truly appreciate all of your efforts in grant management
this award entitles you to contact imls if you have any trouble with your
tickets can't find your seat or your luggage imls
program staff is here to help you and now i'll turn it back to terry
oh thank you and well done state recipients of the the travel pun awards
and thank you for the great chat dialogue that is improving upon our
ideas we will look for the spr platinum edition at some point in the future
um so at this point we we are finishing like nice and early on our scheduled
program so let's take some time and do the cam
strawbridge round two we've got our questions
teed up and then also as um cam's sort of unmuting and getting ready to
to answer this barrage of questions i'll just note that tomorrow
um we the the terry devoe plethora of identities was fine today
tomorrow we actually need to somehow address this because we've got to get
you guys into breakout groups so if you come in using the same link
that you did today please be sure to change your name by as
mary susie said you can turn on your camera real quick to figure out which
terry to vote you are you can go to the three dots and you can
rename yourself and we would appreciate that
um even better if you did register for the conference
you should be able to somewhere in your email find that registration
confirmation and come in under the link that is specific to you
um we we did not intend to give you my credential we really thought it was a
generic one so uh clearly we messed up but um tomorrow you can you know look
for that identifier registration email that you
have or you can register again and i think dennis is going to put great
it's in the chat box right now doesn't matter if you already registered
you can register again get another confirmation
and then come in as yourself tomorrow because that'll make
dennis's job a lot easier and we all want to make dennis's job a lot easier
so um at this point let us go back to some questions that we combed
through the chat and thought were probably unanswered
um so we're having all kinds of whiplash today we're going back and forth on the
five-year evaluations now we're back to two cfr 200 regulatory stuff
um so cam are you up and with us up up and with you and i didn't i didn't
make the screen go down as soon as i started talking so i'm
i'm feeling 100 awesome awesome uh we are up and away so the
the question um that i had written down first as unanswered
is there a threshold for duns and now the ue and uh we were interpreting that
as maybe a dollar threshold where you would or would not have to use a duns or
a ue with your sub recipient so i hope that's the correct
interpretation yeah i i look uh very quickly at this and i i
forgot to say at the beginning of this isn't necessarily legal advice i'm
just trying to uh just be as helpful generally as i can i
probably the place to look for that um and i i just checked quickly is is
two cfr part 25 uh which talks about
registering in sam i checked that and the only uh dollar
threshold i saw was for foreign organizations
and if they're less than 25 000 or if the agency of the imls grants an
emergency exception for someone so they wouldn't have to get
a registration number uei or or uh duns um
but if even if finalists granted that they would still have to get
um the registration within 30 days so i didn't see a
low value dollar um amount to that i think
i think everybody who's going to be a recipient and soon a sub recipient
will need to register for the uei certainly and don's right now
right still october first yeah on that same uei topic there was a
question can schools use their district's ueis and can public libraries
use their municipalities ueis uh i'm
without knowing more detail i would say that sounds okay
likely yes in practice i'll say we have seen
that um used when we make metals awards for example
and um that is you know a public library using a city's duns number
um is sort of the equivalent of that that we know
happens now are there okay next question are there
exclusions to what may be included in calculating indirect costs
and in parentheses equipment tuition etc yeah so um if you're getting
a negotiate a federally negotiated indirect cost rate
there there probably are various exclusion there's a whole
appendix in tcfr on negotiating rates with the feds and it may depend if
you're a state if you're a hospital if you're a non-profit
um so i don't have a definitive final answer on that
um i would say if uh if your sub recipient is going for the de
minimis 10 rate um which is available under
uh 200.414 f uh which says basically
a sub can apply uh ten percent the minimus rate
to get ready for this to the modified total direct
cost mtdc um and if you if you're into cfr part 200 uh model
or mtdc is defined there's a whole definition for it but it basically means
um you're applying the 10 to all direct salaries and wages applicable
fringe benefits materials and supplies services travel
and up to the first twenty five thousand dollars of each sub award
regardless the period of performance the sub awards
mtdc which is again you're applying the ten percent to mtdc
modified total record cost excludes equipment capital expenditures charges
for patient care rental costs uh tuition remission
scholarships and fellowships um participant support costs and the
portion of each sub-award in excess of twenty-five thousand dollars
uh other items may be excluded may only be excluded when
um necessary to avoid a serious inequity in the distribution of an of indirect
cost and with the approval for the of the cognizant agency for indirect
costs so if you start by going
to 200.414 f talks about the 10 it's applied to model
uh it's a modified total direct cost which is defined
um in the definition section of 200 part 202 cfr 200
on the definition section that that basically tells you
um what can be included or not included in the 10
great thank you cam um there was a question about whether
the fafada um reporting system fsrs and fifata will be switching to the ue
numbers i don't have an answer to that that may be
there's a lot of guidance that's being threatened to be rolled out
um on on the whole use of uei um we haven't we haven't seen it yet um
so i would imagine um if there is guidance it's going to be
forthcoming thank you um okay there was a note
that there was a plethora of concerns around the ue that were discussed on the
listserv this was the lsta coordinators listserv
can we expect more guidance uh
yeah well first of all you're not alone the federal government is still waiting
for agencies are still waiting for more
guidance uh i think omb understands that we've been on some calls where they say
it's it's coming it's coming uh but i i think um
you probably will we'll you'll hear it when we hear it but
they're they do anticipate more guidance on this so
stay tuned just add if today's presentations did not cover your
specific concerns please let your program officer know
it's really helpful to us um to both feed these kinds of questions
to cam and to see you know what it is that we might need
to do some kind of presentation or clarification around
these are all great questions too okay um there was a specific question
about indirect cost rates would selecting a radio button option
count as negotiation for an indirect cost rate um
depends what's playing now i would say i guess we're talking about a radio
button in a that the sub the sub recipient would
press saying yes
i would say i would say probably omb has said um a couple years ago that
that indirect costs are not are not bad things
so i think there's a recognition that they are
legitimate costs so and they've they've spelled out more in terms of the
negotiating negotiating one with the sub or letting
them used to 10 i would say probably a radio button in of itself
isn't isn't prohibited but you just you want to
you want to make it so you're not somehow
pressuring the the sub not to take indirect costs or to take a lower one so
you can use a radio button if that's part of your negotiation it's probably
better to actually have more of a discussion
but i i don't see that that would be per se
prohibited that's great um i did not copy down
all the you know the should and the must questions around um
some of the reporting deadlines because i think that pretty much got fleshed out
in the chat as well as some of the indirect cost rate questions
if you still have questions you can put them in the chat now and i've seen
we have a few more so here's one from our recent batch cam can you
clarify the 90-day rule around the reporting due
date what does negotiate an earlier date mean
if an earlier date is indicated in the contract is that sufficient
negotiation um so i guess we're talking about when
you're when you're working with your subs
and you're they have 90 days under the regs right now to get you
their final reports they they had 90 days before that hasn't
changed they don't have any more days um and i think the only thing that reg
says about a shorter date is um you it could be
shorter than 90 days i think if acceptable to you and the sub
um but i think they still you know if they if they said no i think
it would it would still be they still have 90 days
um i'm looking here's a question uh do subcontractors
have to be reported in fifata and sign all of the sub awards certs
and have a ue we have one contractor run a program
for us um i'm i might have to look more into that
but i think the the ue applies to um
sub-recipients not contractors um but as of now all you can't you have to
let subs know that you you can't give them a
sub award until they um well until they have a ue which will
be after october 1st um but you can't make
the award until they actually have one now i guess a distinction omb thinks is
important is they just need to get a you a ue
they don't necessarily have to be to go through the full sam registration
so i think there's a way in sam you just get a ue
by itself without for the sub without the sub having to go through a full
full blown exam registration
um i think this is in response to that first question we asked you about the
threshold for dunns like a dollar amount um karen egan noted
since duns is necessary for the fodder reporting we could think about the
thirty thousand dollar um amount as a threshold
if that is helpful i don't know cam if you have thoughts on that
yeah i think i think karen's right i think um
that what's changed from before and that um and that on that topic is
it used to be 25 000 for reporting subs and they've moved it up to 30 000. um
i don't know if that answers the question that's really the only change i
saw that they made so uh the usual practice plus the change
to a higher threshold i i'm not sure how many how many fewer
you'd have to to report you know between 25 000 30 but
uh it's 30 now
that's great um there's some questions around you know the
or just some comments maybe around the timing
of distributing arpa funds and what kind of identifiers might be needed
um for the near term dones would be fine it seems
and i think we agree with that uh the ue will come later cam had a
slide on that it had you know a fall date when duns and ue
would sort of be moving together for a while and then
ue all the way next april that that's right so
from what i understand again this is i haven't seen this as a formal full roll
out but from conversations uh with omb on interagency groups right now
through october 1st it's dunn's number um from october 1st
through uh i think it's going to be april 4th
of next year you can use either ue or duns
and then they after april fourth it switches over and they'll only accept
um huey
and then gene in colorado was asking something that another state
kind of already answered but can a sub award be reduced by the amount
of the indirect costs
and michael dennison offered some um examples in wisconsin of how they do
that
yeah are you thinking in terms of a cost share from the sub
using their their indirect cost is their as their contributions their cost share
thereby reducing the amount you would provide
under the sub award itself dean feel free to unmute if it's easier
to just talk through this
thank you yes it's easier to talk let's say that
a library applies for a grant and asks for ten thousand dollars
but then they want to also um charge a indirect cost rate of 10
percent well then can i take that 10 off of the 10 000 and say okay you can
have your 10 000 award but it's just gonna give you
less money to do your actual um purchasing for your project
um yeah let me let me think through this here so
um a couple of things and i i know we just we talked about indirect
cost rates a lot during my regular presentation i sort of
uh went through that quickly i went back and looked uh again these these
requirements on the on the states as pass-through
entities are all are found in 200.332
which is requirements for pass-through entities
and um the part of that which talks about
you need to include in your actual sub-award for the sub-recipient
you need to include a federally approved indirect cost rate if they have one or
if they don't that you negotiate one and or use the ten percent
um i found there's another section another part of this also
in the same section uh three three two a lot of numbers uh a four
b two but you'll see if you go there but it says that sub recipients can
elect to use cost the cost allocation method
uh to account for indirect cost in accordance uh
with the sections given there so i i think if they're saying i don't want
i if you're if the sub is saying i don't want to use my indirect cost
i i would still rather charge it as a direct cost
um they could probably do that i think you'd want to make sure that they're not
kind of double double billing they're not including
direct cost plus an indirect cost rate which would normally include that
have i tangled this up too much
that's fine i have a feeling that the answer is no you really can't do that
because i like we gave out some recipient grants for ten thousand
dollars and i didn't want to have you know give
additional money over that 10 000 for indirect costs because we only
had a finite amount of money to offer
yeah and again i mean they they can elect
not to i mean if you negotiate zero with them
you can do that but it seems you know again going back a couple years ago with
omb they they basically said hey indirect
costs those are those are still costs and um
you know that's why they put so much in here about you know being able to
include them or negotiate them but um yeah i think i think i think
absent um they're saying yeah go ahead don't i
i'm not going to charge indirect costs cost if it's in their budget
they can put it in their budget i guess if you don't like their overall budget
you could reduce it but you may have to negotiate a little bit
on that
but i guess the they're leaning towards there's there's still legitimate cost
um like travel like uh salary costs like anything else
we've got um michael dennison asking maybe more broadly for program officers
hoping we'll get more information about the partial sign up in sam.gov
what does that entail with regard to janet's question if that april 2022 ue
switch happens during our grant period is having a duns at the start of the
grant period acceptable
oh go ahead no please go ahead i i've heard
and again i i mean we're you're waiting for it we're waiting for it more
guidance on this i've heard that for it may be possible
that you if you have um
if the switchover may be automatic then that if you've got a duns number in
there sam.gov might assign you a new uei
i don't don't quote me on that but that i think that's
that's what i've heard because that's this question's been raised a lot
um when it's like yeah yeah that's my understanding as well um
that the key to a seamless changeover from duns to ue is using is
making sure that uh an applicant has takes that dones has the same right of
registration and then it goes seamlessly from duns to ue
without there being a lot of food for all so during that
um fall of 2021 to spring 2022 period states may have to sort of help their
sub recipients you know understand that they should get
that partial sam.gov sign up so that they're ready
when the ueis are sort of given out yeah
not to add more complication to this but there's been recent guidance we're still
looking at it um but it's an omb memorandum
numbered m 21-20 and i i think it we have to look some
more but i think it says in there and this is automatic if you
uh have if you're currently have a grant um and you're up and you're your sam's
number is going to expire between april 1st and i think october
1st uh you may have like it won't expire for
an additional year i think that's one of the emergency sort
of um flexibilities they're giving and from
you know we may have to provide more on this but it uh because we're still
looking at it but at least at least expiring expiring
um numbers won't expire they'll be given more life or i think a
year if if they would normally expire between
april 1st this year and october 1st
so there's um there's a question here is the duns only required right now for
the thirty thousand dollar and above sub awards
or is it for all sub awards
i don't know the answer to that i know uh
for new awards um
they're going to need at least the uei before awards are made
before sub-awards are made whether that's a duns number or
well it maybe it is just a duns number at this point if if they're not going to
make the uei available until october 1st um
but i think you're safe i think you're safe if you at least make sure that
every every sub that you're giving a sub award to now has at least a duns number
um i'm not going to try to
[Music]
parse through the the mora walsh comments i think they're more comments
than questions um i hope i hope they're getting
answered
um
yes um there's some there's some confusion
in general some comments i i am looking for actual questions at this point and
i'm not sure i'm seeing many so i know we're getting to the end
of the time um why don't we tentatively wrap it up
here and just oh okay there's one
one last question i will sneak in cam could you clarify when grantees are
required to obtain bids for large ticket items
yeah so um okay it's here's sort of the answer to this
if you are the grantee right now you are following
whatever state requirements there are for that
what's changed is um you being the state so states still
follow whatever your state rules are for for
large ticket items if it's a um
a non-state you know if it's a nonprofit that's asking the question
their new procurement methods um are broken down again into those
three three categories it's informal
and that's whenever they're buying something up to fifty thousand dollars
and they can for that they could use a micro purchase which means they can use
their their purchase card for that they don't
have to go out and get bids if if it's more than ten thousand
dollars which is the micro purchase threshold but less than 250 000
all they need to do is just um go go solicit an adequate number of sources
and and those two again they're under 250 000 are meant to be
administratively easier not not as formal and everything over 250
000 which is the simplified acquisition threshold
everything over 250 000 would follow the formal purchase
methods and so the formal purchase methods
would be things like sealed bids or formal proposals and and they they flesh
out in that section you know what is a sealed bid and what
is the proposal um and so basically
if it's a if it's not a state that's asking that question
um because they still follow their own rules if it's
uh a nonprofit that's asking that question one of your subs is asking that
question if they're going to be buying something
over 250 000 then they would need to follow the
formal methods which would be either a sealed bid
process or a formal proposal process
and i know this isn't easy the follow-up from what you were just saying
from the same question asker so this doesn't pertain to slaas and
grantees so long as the grantees are libraries
museums tribal museums etc no it it doesn't
the new if if you're a state if you're an slaa
the only thing that's changed for you you still follow your state
um procurement rules because there's sort of federal state sovereignty
and the federal government's like okay state you do
you follow your own rules for this except now we're just adding
you also need to see the contract with uh small minority business women's own
business as well as the domestic preferences but
apart from that nothing's changed for you just follow your state rules
if you are a non-profit then you follow these new procurement
rules i would say for tribes and again for this program lsta program
you're all you're all when i say grantees i'm
talking about state ls slaas um and nobody else but if you're
talking about tribes um you know we have to look into that
more uh but for everybody else other than a state or
a tribe um you follow the new the new
three-prong easy procurement procedures or i think
the government
so i think you touched on a tribal question that just came in because
um question about sovereignty um and and two more i think if the state
procurement rules are more stringent than the federal rules isn't it true
that the state rules apply i think the state rules are always going
to apply whether they're more stringent or not for this
and it's it's 37 200.317 it's very short but it talks about
states follow the state rules um and they may or may not be more
stringent in the federal rules
um and then just to clear up some confusion for federal year
fiscal 21 subgrants which um this particular state will be awarding within
the next month is it done still okay or do i need
to get all the sub grantees to register in sam before they get their award
agreement we we think that's duns we think that's
not i would say it probably it's probably
good to have them all get um at least the duns number
after i think they're all going to be everyone's going to be required to after
october 1st when they're going to be making ueis available
i i think the problem is right now they're not available um
and there's there's this sort of strange window we're in right now
but you you wouldn't go wrong if you had them at least get a duns number i think
nothing more definitive on that and again we hope to get
more guidance from omb and you'll get it as soon as we do
guess what back to procurement so if public libraries adopt state procurement
rules then state procurement applies instead
of federal procurement
um the rules apply let me see i'm just going to look at
this probably it's the last chat um from beth milburn if you can see that
one yeah hold on one i'm just also looking
at the
okay okay um yeah let's if this if this helps to
answer the question i i pulled up again on ecfr.gov
200.317 which is called procurement by state
and it says when procuring property and services
under a federal award like you all would i mls
a state must follow the same policies and procedures it uses
for procurements from from its non-federal funds
so in other words what are you using right now for non-federal funds which i
would imagine are your state funds
and then again it talks about the state will comply with uh 200.321 which is the
new the new one a minority in women's own
businesses and two and three two three which is the
domestic preferences and ensure that every purchase order or other contract
includes any clauses required by
200.327 which hasn't changed all other non-federal all other
non-federal entities including sub-recipients of the state
must follow basically the new the new procurement
rules for them um so the rule here at least is
you when you're using mls funds to procure things
you need to use the same policies and procedures that you use
when you're using that you use for non-federal funds
which i've interpreted as your own state funds
so follow your state rules
um there is another change of topics real quick we'll clear this one up uh we
had a question from a tribe that said we were required to set aside point nine
percent of our state arpa funds for tribal libraries is that accurate
or is that in reference to the imls set aside and not the grants to states arpa
funds that cannot be in reference to the grants to states arpa funds that's not a
requirement of the ones that you guys
administer uh i don't know the answer is that
you're correct terry we don't imls does not have a
a set aside of any percentage for that um of course we'd encourage you to work
with tribes but um there's no specific set aside uh
i don't know i would if i were the state and the tribe asked me
you know said they're required to set aside 0.9
i would i would ask them to show you where that requirement is
it may be there um but it's not an ils requirement
all right well um we i think we need to wrap up
um because we are almost at the top of the hour we
so appreciate cam staying on you can't imagine how hard
it is to just kind of reflexively answer these questions which we
normally ruminate over for you know hours and days at a time
so thank you cam for really indulging us with all your expertise
and uh you know getting us through this morass of
regulations it is so so helpful uh yeah it's an arrest and i yeah
go ahead um i have i have just a fun quick
announcement for the end of the day uh which i have to credit michelle who's
like kind of our pun master at imls so here's my closing announcements for
today as we taxi to the runway and before we exit the conference plane we
have just a few announcements you don't have to check around you
before you deplane because imls will be sending your award certificates to your
email address once we land and the conference is over
within a few days imls will follow up with any questions that weren't answered
in the chat by the flight attendant and imls would like to thank you for
choosing to attend our conference today and we hope that we'll see you all
tomorrow for your next flight so see you tomorrow states you've been
great and we'll see you same time tomorrow adios