ROBERT KARATSU: Okay, good afternoon and we are at our final panel of the day and going
to save the best for last here. And before we get started, I really did want to thank
Maura and the IMLS staff for putting this thing together. You guys did a fantastic job.
(Applause). >> ROBERT KARATSU: And then also to all of
the presenters that shared their inspiration with us today Thank you, all, so much, as
well. And I have actually a little personal thank
you, too, actually again to the IMLS. I mean, John mentioned earlier that going to Washington
D.C. was like the best thing that ever happened to him professionally and I can say the same
thing back in 2013 my library did win the national medal and I did get to go to DC although
John I'm kind of disappointed you're not wearing the medal around your neck.
>> (Away from microphone). >> ROBERT KARATSU: Because I know in rancho
Michelle made a replica for her cat to wear around I think it was a replica, right we
could do a whole program what you can do with a national medal but I digress so anyway our
topic is going to be on evaluation and sustainability. And I thought I would start our part just
to try to frame it maybe with a little story that if you're a library Director or an administrator
or manager and you're in an institution that chases money or in other words like to write
grants well then this may be a little situation that you're going to find yourself in because
this is what I used to do all the time whenever we're in like the visioning or the brainstorming
session, you know, trying to get a grant together there would generally be about four questions
I would ask the person with this great idea, okay, so tell me about your -- what you want
to propose. So the first thing I would ask them, what need is this thing addressing?
Okay. Sounds good. Next one. So tell me about the innovative partnerships. You know, we
all heard about partnerships today. Tell me about these innovative partnerships you've
got for this project. All right, sounds good. Next one is you've got to tell me about measurable
outcomes and outputs. Okay. Sounds good. And then the last one it would always be the
S word, so how are you going to sustain this thing once the grant period is over? And if
they say, I'm going to get another grant, wrong answer And so what we have got today
is our panel, they have kind of figured out that they have unraveled that Gordian Knot
of how to sustained the project and I know they will all be talking about things that
started initially with an LSTA grant or something from the IMLS so really with that I'm going
to turn it over to our panel. We'll introduce them. First we have Dr. Marcia Mardis, she
is a associate Professor at the School of Information, Florida State. Next to her we
have Julie Walker. She is the state librarian from the Georgia Public Library Service. And
then on the end, my former partner in crime, she is the Director of the Rancho Cucamonga
Public Library, Michelle Perera but with that, I give it to you, Marcia, take it away
>> MARCIA MARDIS: I thought we were doing slides today I guess we're not all doing slides
but I'm doing slides so we're going to talk a little bit about the journey from bench
to trench of Web2MARC, which is a spinoff of a 2009 early career researcher grant And
I hope you enjoy it. So let's see. If I can get this go. Web2MARC was born of an idea.
In the last 150 years we've gone from the discovery of germs to organ transplant to
being able to print ears on a 3D printer. We have gone from telegraph to telephone to
more computing power that can put a man on the moon that will put a man on the moon however
you'll notice the classroom hasn't changed all that much in 150 years and one of the
big reasons why it hasn't changed is because content delivery hasn't really changed so
that was really the core of the project that Web2MARC was part of. When we started our
initial research around that project we were out in the field. And talking to teachers
and they said, well, we don't integrate digital content because we don't have the time to
find it and once we find it, frankly, well, I'll translate their personal information
management practices are a little wanting. So my suggestion was why don't you work with
your school librarian and so I refer you back to Renee Franklin's comments just a moment
ago and the school librarian said well I don't know what to do with the stuff either I can
build these hot lists but then I have to keep them up to date I don't have access to the
school Web site yada yada yada I said that destiny system that costs your district $10,000
a year have you thought about cataloging the ones that are currently used that's the other
C word in case you're wondering and they were no longer learning to catalog and the thought
of cataloging something terrified them so I thought this is an interesting problem so
we built a little tool called Web2MARC which you can Google right now Version 1.0 is on
a public Web site and it caught on really quickly so we have tens of thousands of users
overnight so Web2MARC is very simple you paste a URL in a box goes off to the site goes to
content on the site maps it out to a mark record then there are scaffolds to assign
a Common Core standard and next generation science standard and makes educated guesses
through the natural language processing on assigning keywords and you download the mark
records suck it into your online catalog, viola. So I was really surprised that it caught
on this quickly because I would know so through the wonders of graduate students your servers
aren't quite as stable as you would like them to be so I found out because somebody in France
told me before my graduate student even found out that the server was down so it was that
kind of reach that we had really, really quickly. And then the project ended And we thought,
man, how are we going to sustain this thing we've got people who really love it so then
the Office of Commercialization came sniffing around.
And so they talked me into participating in a shark tank which is not -- not the most
pleasant experience you've ever had And if you've ever seen the TV show, it is kind of
like that. Only people aren't as nicely dressed. And so I -- so as you know from watching Shark
Tank there are really two options Ugly Baby or Squatty Potty and if you know Squatty Potty
one of their most successful inventions I was on the ugly baby end of the spectrum but
somehow I got magically back channeled $20,000 to build what we wanted to build to sustain
Web2MARC which is the button on share it toolbar you can pin it on Pinterest tweet it or create
a MARC record for it right from there and download it we get the $15,000 we start to
build the prototype we run into time after time after time we're not getting anywhere.
So we then find that I'll go to this one because -- wow this is not the edited version what
do you know so this is the slide I wanted to show you.
So what I recommend is if your project is something that creates a product and anybody
in your vicinity has had an NSF grant NSF will give you $50,000 to do that market research
through a rather rigorous process called iCore and iCore strips things down to entrepreneurship
I turned to my doctoral student through the first iCore thing I said this is professional
development do you realize what an absolutely beautiful workshop that they are giving us
here about product market fit. About lead with your market. You have to understand what
people want. You don't try to shove your product on them. You have to understand what their
problem really is. And then try to determine whether what you have solves that problem.
Or if what you have could be modified to solve that problem.
So 103 customer interviews later we actually did find out that what we had solved the problem
but not for the end user, for the people who catalog content at repositories like at PBS
at repositories at NSDL and other kinds of content providers, too this is a multi billion
dollar a year industry and bad metadata is costing $958 million alone in the industry
so then the sharks were interested. So I just finished shark tank No. 2 so we're
now in the Squatty Potty realm. So we're getting -- we got into great talks with a company
that I can't name but they used to own a copy machine company and now they own a major education
publisher. That could go well. (Chuckles).
>> MARCIA MARDIS: But to me as I listened to everybody talking today Malcolm Gladwell
was invoked all over the place today and he was invoked when I was writing these slides
as well because really it comes down to the three things everybody is talking about, connection,
making those networks, being expert in being able to I assert your expertise and also being
although salesman is the word that Malcolm had to use but more being the advocate being
out there taking your services to where they need to go then I did a little proof during
the Kansas City event. And so I think what I figured out is what we do is the bench.
Where things need to go in the university realm is into the trench which is the lower
level. So if you want to maintain your project and you're in the trench either you need to
maintain your practice end of things or your research end of things. And you can do that
by scaling out like Project ENABLE did or you can do it by taking it into completely
different direction by building on that thing that's really working like we did in Web2MARC,
it wasn't the public Web site that's working so much as the idea of better metadata and
that's what we're building on so we're exponentially sustaining our project and that's in a different
way so really it's mathematics, everybody. (Chuckles).
>> MARCIA MARDIS: So that's sort of my reflection on that. And if you want to know anything
more about my feelings about bench to trench because I have this whole rant on who is trench
that we can go to. But thanks very much. (Applause).
>> JULIE WALKER: Good afternoon I'm Julie Walker from the Georgia Public Library Service
and I think GPLS has the somewhat deserved reputation of leaping in where angling fear
to trend sometimes because I can remember 12 years ago when we started talking about
building an Open Source ILS and the looks I got like really. But in the aftermath of
that decision we have talked a lot about taking risks and how librarians are usually fairly
risk averse and metaphorically jumping off that cliff but really we did put a tremendous
amount of thought into the sustainability and scalability of the project and really
focused on the long-term results. We really weren't just looking for something cool to
do. We had to -- we of course defined our basic
need and our basic need was staring us right in the face we were absolutely desperate we
were patching together an off the shelf ILS that didn't work for a statewide consortium
and every time they would do an upgrade all of our patches would fail and we would start
all over again and we got to the point where we really had no choice.
We do try to use a lot of the basic tenets of project management when we make decisions
about this and we looked at the buy versus build idea. When we were looking at this.
But there was nothing to buy. We had really done a thorough environmental scan and truly
there was not anything on the marketplace that was designed and built to support a consortium
like the Georgia PINES consortium was built so we absolutely had to do something.
And we decided that we had to build. We had a unique need. It was going to be less expensive
-- well, it wasn't less expensive than to buy -- it was less expensive we could not
buy it and we were the only ones willing to take on that task to accomplish it.
We took a huge amount of time to build the support we needed to sustain this project.
Even though we were kind of going it alone. But our parent agency which is University
of System of Georgia was very skeptical of Open Source software at that time.
And we really had to call in every chip we had just to get them on board but we also
had to work very hard every step of the way to earn their trust and to be really transparent
and have tremendously good communication. So that they would see that we really could
handle this major project And our first goal was just a product that
would work for PINES, work for our member libraries and we were committed to sustaining
it for Georgia and we didn't think so much about planting it around the world at that
time. So what we really needed to do first was secure
the buy-in from our member libraries in PINES Our constituents. So we spent months and months
and months doing focus groups to determine the features they wanted. To gather requirements.
And we wanted mostly to make sure there was a shared ownership before we got to that critical
go-live date. And the Open Source model in that case really
worked in our favor. We were really transparent. There was a huge amount of communication.
The Open Source community is really known for their great communication on any number
of channels. So that helped us. It may have hurt us it
may have been a tactical error in the very first day we went live every Open Source person
in the known world logged in to look at what was happening what's going on in PINES today
and it all crashed. (Chuckles).
>> JULIE WALKER: But the support we received from that Open Source software community was
just invaluable and the proof now is that it's taking on a life of its own. We were
able -- once this worked for PINES and still works for PINES today is a great thing for
us. We're not looking for a different solution. We're just continuing to grow evergreen but
it's way beyond the borders of Georgia and way beyond the borders of the United States
now so that product has really been shown to grow and thrive and we're still very proud
that it was implanted in Georgia for the first time and we still work really hard to continue
to build that evergreen community. We send our entire PINES staff to the evergreen meetings
and they serve on all of the crucial committees for Evergreen so we are absolutely committed
to committing to sustain that product throughout the world and it's really gratifying to see
all of the different libraries that it works for.
But once we caught our breath from Evergreen, a new challenge popped up And the idea started
to build that, hey, we could take the basis of the Evergreen software and build a product
for the libraries for the blind and physically handicapped because we were hearing from a
lot of state libraries that the one product that was available for them was somewhat unsatisfactory
and we said let's take a look at that so thanks to a grant an IMLS grant we were able to pull
together a convening in Atlanta with people from the large majority of the state libraries
to come and talk about that idea and to do a requirements gathering and it was a great
robust useful week-long conference The correct participants were at the table. There was
a lot of enthusiasm about the project. But it was really a case of poor timing because
we had a great White Paper that came out of the project and then we started talking
to the people who had come and who had been the most enthusiastic about participating
about who is in, who is in for funding and helping us and providing resources to build
this. And that was right at the moment the economy went way, way, way south and nobody
had the bandwidth, nobody had the capacity, nobody had the resources to do that project
at that time. We couldn't undertake it alone and nobody
else was able to step up to partner with us. So we had to learn when to let it go and I
think that's a really valuable lesson when we're talking about grant funded projects
or any of these new innovative ideas. Sometimes the time is just not right. Sometimes you
can't afford to let yourself get so attached to an idea that it's going to drain the resources
away from the other things that you have to be doing.
So Loblolly, which was the name of that project Mayet resurface and it's a great example of
knowing when to is step back from an unsustainable idea quickly I'll talk about the project we're
through it's the B4 early Literacy Initiative another IMLS project birth to four or years
before kindergarten focuses on family literacy and reading books before kindergarten so our
libraries are really running with that this summer The great thing about this project
and the thing we're really excited about is that we looked at existing models. We didn't
try to develop a new thing. Just to say, GPLS had developed a new thing we took two nationally
recognized early literacy programs Every Child Ready to ready and Prime Time Preschool very
solid and proven programs we're piloting each of those in six different library systems.
And we're really focusing on evaluation. We know that we are not experts on evaluating
early literacy programs. Most librarians aren't. We don't have that background and training
so we have an expert from the University of Georgia She's been involved every step of
the way. She's going to evaluate the participants in those programs and we're going to come
out at the end with some idea of which one is the most effective for families in Georgia.
And then that will allow us once we look at those results to see what we want to roll
out statewide so those are three big things that we've been working on in Georgia that
our wonderful Federal partners have funded for us and some very different outcomes. Thanks.
(Applause). >> MICHELLE PERERA: Good afternoon everyone
I'm Michelle Perera I'm from the Rancho Cucamonga library and I want to talk to you about our
Play and Learn Islands which is probably my favorite project I've worked on or at least
one of two favorite projects and just in luck I'm going to mention both of them so our play
started with a needs assessment we were trying to figure out with what to do with 14,000
square feet we have on our second floor and it was overwhelmingly a response from our
public that they wanted some kind of interactive experience for their kids. They wanted their
children to learn something through play. They wanted their kids to be able to come
to the library and do something and it was right when the economy tanked so we didn't
have the money to complete the second floor but we thought we can still do something with
this data so we decided to create our Play and Learn Islands which are small scale interactive
exhibits for children we were fortunate to get several grants from the California State
Library to help us fund it as well as some money from our library foundation so we contracted
with a museum designer because those are the experts on creating exhibits. We just got
a couple of pictures that are behind me to show you one was about building kids learning
about design, redesigning another one about digging really very versatile it could be
a sandbox with a dynamo big or Chinese New Year program we put tare cot at that warriors
in there they got to dig those out we found it was something you could use for any type
of event. We also created a Farmer's Market. A little stand. It's a miniature Farmer's
Market that really worked well with our healthy communities initiative in Rancho Cucamonga
and has been popular in other libraries because a lot of cities are doing healthy communities
initiatives and it really conveys the concept of from the field to the market to the table.
Actually there's the one right there with just some pictures of the vegetables and there's
mulch in there they can actually dig and find what they are looking for.
So we piloted these in our libraries. And we found them to be pretty darn popular. Kids
-- we were worried about having to teach kids what to do with them and how to use them.
Kids don't need to be taught anything. They just kind of roll up to something that looks
fun and they play, they learn about sharing and really those good socialization skills
with other kids, as well. So it worked in our library and we thought
well if it can work in our library maybe it can work in other libraries so we went back
to the State Library and applied for some more money to build some more of these that
we could loan for free to other California libraries. And they were super popular in
the other libraries. Well, they work for us, why not work for others. Because the great
thing about them was it gave people something to do when you walked into that library.
There was always an activity out there. It could be the exact same activity every single
time that child walks in because some kids like repetition or it could be completely
different every time you walked in. But the best part of it was it was unstructured and
they could make it whatever they wanted. So we started loaning them to libraries and
we found libraries didn't want to give them back.
(Chuckles). >> MICHELLE PERERA: So you get the repo truck
to go out there -- no. Is so we thought wait a minute why aren't we selling these to other
libraries so we created a plan to try to sell these to libraries. We loaned them to about
35 libraries. We have since sold them to about 40. And it has just been a great experience.
I mean imagine being the nerdy librarian that I am wherever I am I'm going to pop into the
local library and we were in Fulsome for some reason we went no -- into the library and
there was a play and learn library I was walking around looking like a creepy adult in the
childrens room and the librarian said let me tell you about that I thought I created
that and it was cool. I saw a -- I went to another library and say a play and learn library
and ripped us off and copied our design. But as part of the process for creating these
we also trademarked them. Really came up with a good plan in order to try to sell them to
other libraries. So that -- having a little bit of money that
we earned from these helps us to sustain this project. It allows us to create more for our
library, it allowed us to design a few more but that's not really what's going to sustain
the project money as hard as it is to find isn't the hardest thing to find in the world.
What sustaining it are staff so we really spent a lot of time training our staff in
early childhood development and brain development and that cognitive learning skills because
that's what's going to sustain it they are excited about it they feel confident about
this, they want to create more so by giving them the knowledge and not just saying we're
going to embark on a project and not give you any training or resources which is what
we often do a little too often, that really got them excited and bought into the project
and what's going to keep it going so now I'm going to segue to our staff innovation fund
which is also about training and sustainability because we realized well if we train them
on this area early learning, and they are keeping this project alive very easily what
if we expanded that so we created a ten part training program, grants public speaking,
marketing and branding, dealing with conflict, putting together teams, trend watching and
by putting that training series together, our staff really was able to gain a lot of
confidence and be able to sustain any project they wanted to do because they had the skills
to do it so now they are out there trying to find money, they are out there looking
for experts and looking for resources it's not beholden upon me or one person or two
people in my organization to do that so the staff innovation fund, first part was ten
training sessions. The second part is an innovation fund where
they can apply for money to try to create the projects they want and some of them have
gone onto create projects that deal with the Play and Learn Islands one created a renewable
Play and Learn Islands which has a working solar panel, teaches kids about wind energy,
about water, and it -- it's just a great thing to kind of create something and because staff
is trained and they feel confident, they sustain it and they go off and do it themselves. Our
staff innovation fund is something that's been funded by the California State Library
and they have continued to fund it over the years not just for our library but they have
continued to fund it for other libraries in California and to date about 25 libraries
in California have gone through this program, done the ten-part training and also have been
able to apply for money to do the passion projects that really benefit the community
in their own libraries. Thank you. (Applause).
>> ROBERT KARATSU: Okay. So if there's anyone out in the livestream world or in the room
that has questions, comments for the panel and since I have the microphone I've got one
because you all make these things sound so easy but I suspect along the way you ran into
a few road bumps so would you mind sharing some of those things with us.
>> MICHELLE PERERA: I would be happy to share one. I did make it sound great those Play
and Learn Islands came out as well as they did but one of them which is the one we don't
talk about totally sucked. We got this. We thought we looked at the schematics really
well and then it showed up and instead of being a really versatile Play and Learn Islands
has German wheels can turn things easily move around the library go outside this thing weighed
150 pounds we couldn't move it. There were no wheels and it had this -- it was supposed
to be about movement. Yeah, I know. (Chuckles).
>> MICHELLE PERERA: Ironic and it had this pulley that you crank. Well the first crank
we did, it died and the top crank looked like a guillotine for something for kids having
something that looks like a guillotine can chop off their heads not so great and then
we had a book wheel the idea was to look like a ferris wheel of books going around and we
saw this thing and it was taller than we were and kids were trying to climb into it because
they wanted to go around But the best part about this was the ball run that we had on
here which you connect the pipes together and you drop the ball and it goes through
was a choking hazard. So that was a big failure but what we learned from that well, one, learn
what a choking hazard is early and often but then really it's okay if something doesn't
work out because all of the rest were really successful and as we moved forward we learned
we need to have something that moves easily we need to have something that's versatile
we need to have something that kids are going to enjoy. But still be very safe, as well.
So it was a good learning experience. >> ROBERT KARATSU: Okay. Anyone else have
anything people might find interesting that maybe didn't work out the way you thought
it was going to? >> JULIE WALKER: Well we certainly did have
to do a lot of talking to our libraries out in Georgia when we were working on our Evergreen
project because they kind of got to the point of what are those people going to come up
with next and do we really trust that these four guys sitting in the office in Atlanta
drinking Mellow Yellow and eating Snickers bars are going to come up with an ILS that
we can actually trust and use with all of our data and as I said a couple of the things,
one was that everybody in the world looked in on the very first day but also we had a
lot of brand-new staff the day we went live and we had worked because all of the PINES
libraries had to go live the same day we couldn't pilot this. This was an all or nothing proposition.
So our entire staff had been up for four straight days to bring -- migrate all of the patrons,
all of the items all of the transactions, everything to go live the day after Labor
Day. So we were utterly exhausted that first day before it even started so that was probably
not the best idea in the world but we learned to keep our partners and our friends very
close to us to not spring anything on them, to not make any sudden moves they weren't
prepared for so they really felt confident I guess the only other thing that was a little
bit dicey and still to this day there are vendors that I feel like snarl at us once
we walk by because once you have displaced a major vendor product you are never going
to be their best friend anymore so we have learned that.
>> ROBERT KARATSU: Do we have in the room or did anything come in over Twitter?
>> Michelle, are the training programs, are they online or are this face-to-face or how
do you deliver those segments? >> MICHELLE PERERA: They are face-to-face
what we found by having a large number of our staff go through it together or in these
other libraries a large number of staff go through it up to about 35, 40. Is it works
-- it's working -- it works as a good team building experience activity the interaction
between a adult person a childrens person a teen person the library Director and Assistant
Director everybody on the same level of that training was really invaluable and having
the Director and Assistant Director sit through everything really is something that shows
and sets a good example for staff, as well. And they can bounce ideas off and we found
that as a result of that, you're not going to go through the training and the next day
just be more innovative but by going through the trainings and be able to talk and bounce
ideas off of each other you do become more innovative.
>> Hi, Christy with OCLC I love hearing these stories of a sustainable -- of sustainable
projects. I wonder if you could, any of you, talk about how your evaluation approach, either
hindered or helped your ability to become sustainable once the initial startup funding
was spent. >> MARCIA MARDIS: Well, I would say in our
case, our evaluation approach really helped us kind of keep the faith that we were onto
something. And just what form that something needed to take was something that we had to
kind of work through. Because sometimes I think when there's money on the table and
you're asking people for money and they need to invest in you particularly from the private
sector, they are very, very, very skeptical. And you know first question that they want
to know is so how are you going to make me money? And so there were times where we did
have various existential crisis and in fact I'm still going through one now between the
Open Source route and going a commercial route. I mean that's something that I wrestle with
every day. But a good evaluation and good ongoing dialogue with your evaluator is a
way that you can just really keep the faith in your idea I think from my perspective that's
important. >> MICHELLE PERERA: For us with the Play and
Learn Islands project really looking back at how we created them, what we did, what
we needed to tweet what was response was from the public certainly looking at the whole
debacle with the movement one, that really helped us fine tune the design and fine tune
what we wanted to do and as a result of that fine tuning actually brought the price down
about 35%. So we just seemed to have more stuff because when you first do something,
it's like going to the grocery store when you're hungry you just throw everything in
the cart that's what it was like because we were so excited but then if you make that
shopping list and really figure out what works you're kind of able to make it as efficient
and as cost effective as possible. >> JULIE WALKER: And I think the evaluation
effort we're undertaking on our B4 project will be important because we need to show
our libraries that this project really does yield results or maybe we'll show it doesn't
and it's not something that we'll pursue. But if we're asking them to do something new
and different, I mean they are stretched to the limit. They don't have any extra staff
resources, they don't have any extra money, they are overworked and while they are very
game to always try new things I think at this point we really need to prove to them that
something is going to be effective before we ask every library in the state to participate
in it so I think that's going to be the important thing about the evaluation piece of this project
is what do we get at the end and then what can we tell our libraries that will make them
believers. >> MARCIA MARDIS: If I could tag one thing
on to the end there I'm East Coast time I'm starting to wind down. Something that I do
think was part of the evaluation so there's the Web2MARC IMLS grant and the evaluation
that that -- the role that evaluation played in helping us kind of keep the faith but then
in iCore when we had to do 103 customer discovery encounters in four weeks and we were talking
to everybody, their mothers, brothers, cousins, everything we talked to people from Flickr,
Google, enterprise companies, everybody, to be able to actually listen to what people
are telling you and one of the things with iCore you cannot tell them what your technology
is. You just have to ask them so in this sort of situation like with information where do
you find your information, how do you organize it? What's difficult for you? What's easy
for you? If you had to solve your problem, how would you solve that problem? And just
kind of being quiet and listening and not being tempted to say, oh, we have the tool
for you. And really listening that's important, too and as a professor I don't have access
to large groups of people on a day-to-day basis you would think so but in an online
teaching environment that's not necessarily the case. And it was just really important
so that's a luxury all of you have the fact you can walk out and come face-to-face with
the people who make best use of your services I think is probably the best source of ongoing
evaluation you can have. Because you really miss it when it's not there.
>> ROBERT KARATSU: Okay I don't see any more questions it looks like our time is up but
I would really like to thank the panel, again thank all of you and Maura and Robin, take
us home. (Applause).