eIFL-FOSS ILS workshop on Evergreen, day one

The following summary is taken almost directly from an email I wrote to one of the would-be participants who was, sadly, prevented from making it to Yerevan due to travel complications. I meant to clean this up earlier and post it, but have not yet found the time - so I might as well just post it as is with most names obfuscated and possibly some additional editorial comments. Those who are new to installing and configuring Evergreen might find this useful; and reading through it, I remembered a few challenges I planned to tackle

---

Shortly after I arrived on Monday, I was able to try out the

install of Evergreen 1.2.1.4 that A. and G. from the Fundamental

Science Library (FSL) had completed with only two email exchanges with me.

I was very happy to see that they had successfully completed the install!

There was only one minor problem with the structure of the "organizational

unit" hierarchy that I had to fix. After that, we confirmed that we were

able to import bibliographic records from Z39.50 and attached call numbers and

copies to those records. Finally, we tried searching for the records in

the catalogue and were delighted to see that everything was working as

we had hoped. That allowed me to sleep well on Monday, in preparation for the

first day of the workshop on Tuesday.

After the introductions of the workshop participants on Tuesday, I gave the

introduction to Evergreen presentation and Henri Damien Laurent of BibLibre

demonstrated Koha. Both Henri Damien Laurent and I showed our respective

library systems running with an Armenian interface, thanks to the translation

efforts of Tigran! Then we broke into separate Koha and Evergreen groups to

work together on our respective library systems. Of the attendees of the

workshop, E. was the most

interested in migrating his library (with 40,000 volumes) to Evergreen. A.,

from one of the 29 branches of the American University of Armenia (AUA), also

attended most of the Evergreen session. Even though his institution is mostly

interested in Koha, he wanted to be able to compare the two systems. Albert's

colleague S. attended the Koha training session so they would be able to

compare their experiences later. Our group also had R. from the Netherlands

and A., G., and A. from FSL -- apparently Tigran is considering

running Evergreen as a union catalogue, so his IT people are very interested

in learning more.

Our first exercise was to model the organizational unit hierarchy using the

configuration bootstrap interfaces in the /cgi-bin/config.cgi. We began by

drawing the hierarchy on a whiteboard. The "Yerevan Consortium"

represented the Evergreen system as a whole; we added the FSL, MSU, and AUA

systems as children of the Yerevan Consortium, and then added specific branches

as children of each of these systems. While we were creating this hierarchy, I

showed the participants how the organization unit type defines the labels used

in the catalogue as well as the respective depth in the hierarchy for each type.

We then ensured that the systems and branches in the hierarchy had the right

types, and that the types were defined with valid parent-child relationships. We

found a few types that were children of themselves, which causes a problem in

searching. There was also some confusion about the role of types to

organization units, resulting in the creation of types with labels like "FSL"

rather than "Library System". After a few minutes of explanation and working

through correcting the exercises, I think the participants were better able to

understand the relationship between types and organization units.

After we were satisfied with the structure of the organization unit hierarchy, I

ran the autogen.sh script to update the catalogue and staff client

representations of the hierarchy. Well, first I demonstrated how search in the

catalogue will quickly be broken if you do not run the autogen.sh script

Our next step was to register new users with the Evergreen staff client. This

helped introduce the participants to the staff client, as well as giving them

a quick introduction to some parts of Evergreen that still need to be localized

to allow regional variations on postal code formats, telephone numbers, and

forms of identification. The default Evergreen staff client still enforces

American conventions, but fortunately I have had to create patches for Evergreen

to support my own country's standards so I can assure you that it is relatively

easy to change or remove these format checks. In the future, it would be

wonderful to include a localization pack for each locale interested in using

Evergreen that supports regional variations on date formats, phone number

patterns, etc. The participants were pleased with the feedback mechanism in

the staff client that summarized all of the remaining problems with the current

patron record (missing address, invalid phone number, etc) and made it easy to

switch between screens without losing any of the data they had already entered.

Once we had registered new users for each of our branches, we went to work

importing new bibliographic records and attaching call numbers and copies to

those records. This gave us a good opportunity to see how changing the scope

of a search in Evergreen from "Everywhere" down to a specific branch changes

the search results, and demonstrated how the organization type labels are

displayed in the catalogue. As an aside, I should point out that in Evergreen

1.4 (due by the end of this summer), the labels are internationalized so that

different labels can be displayed depending on the locale in which you are

using the catalogue or staff client. Good news for those of us who work in

bilingual or multilingual libraries!

Now that we had records with copies attached and patrons registered in our

Evergreen instance, we were able to use the catalogue's "My Account" features

to try out features like sharable bookbags, account preferences, and the

account summary. Users also have the ability to specify their

own user names and to log in with those instead (which means that they can

simply remember their unique nickname rather than, say, a 14-digit barcode).

The first feature that the participants discovered, of course, was the

strong password enforcement feature. When a patron is registered, the system

automatically generates a random 4-digit password; however, this is not

considered to be a safe password, so when they log in they are forced to

change it to a longer password containing both numbers and letters.

At this point, we also discovered a data validation bug: in the staff client,

it is possible to enter a user barcode that consists of letters and numbers.

However, in the catalogue, user barcodes containing letters are considered

invalid and the system will not even attempt to log that user in; it simply

rejects the barcode. I plan to ask E. to report this bug to the Evergreen

mailing list; it would be an excellent outcome of the workshop if participants

felt comfortable reporting problems to the mailing list, and reporting this

problem in particular would help improve the quality of Evergreen.

Things were going reasonably well, but we noticed that the system was

running into a problem if you tried to edit a bibliographic record after

you had already created or imported the record. I had rather fortunately

already experienced this problem (it is a result of different behaviour

regarding XML namespaces between different versions of LibXML2) and knew

that it had been fixed in 1.2.2.1. So rather than trying to fix the problem

with the installed version of 1.2.1.4, I decided to try upgrading our

Evergreen system to the recently released 1.2.2.1 to demonstrate to the

participants that the upgrade process was fast, reasonably well documented,

and not nearly as complicated as the install process. This was, by the way,

something Randy had urged me to do, so I blame him for the subsequent problems

we experienced (hah!).

The first problem is that the change from 1.2.1.x to 1.2.2.x requires the

installation of a new Perl module from CPAN (JSON::XS). This is not much of a

problem in itself, as the module is very easy to install and compile; however,

given our internet connection I had to wait a long time for the CPAN

repository metadata to be downloaded. The participants were still able to use

the system while this was happening, but we ended up hitting the coffee break

still waiting for CPAN to finish. (As an aside, Irakli and I were discussing

the possibility of having the eIFL-FOSS coordinators investigate setting up

local mirrors of FOSS resources like CPAN to speed up access to frequently

used resources).

When we returned from the coffee break, the JSON::XS install had finished but

the participants were having problems searching and using the staff client. I

checked the logs (using the "grep ERR /openils/var/log/*" command to start

with) and saw that our database connections were dying for some reason. On a

hunch, I checked the system logs ("dmesg") and discovered that the Linux "out

of memory (OOM) killer" had started killing random processes to try to free up

memory. It was killing the PostgreSQL processes, the Evergreen processes -

anything! I was lucky, because I had been reading about the OOM on Linux

after hearing about a Linux user that had run into a similar

problem, and knew that the way to disable the OOM was to prevent Linux from

overcommitting memory to processes in the first place. Wondering why our

system had started running out of memory in the first place, I ran "free" and

saw that it had been set up with no swap space; I confirmed this by running

fdisk to see that there were no swap partitions. Here, however, I made a

mistake. I ran "echo '2' > /proc/sys/vm/overcommit_memory" to prevent Linux

from overcommitting memory to new processes and to prevent the OOM killer from

killing any more random processes. But this also meant that I was immediately

unable to launch any new programs - so I could not safely shut down PostgreSQL

and Evergreen, and we had to turn the power off to the system.

Fortunately, the system started up cleanly again (hurray for journalled

filesystems) and I was able to complete the upgrade before the rest of our

hands on session for the day was finished. A few things that are missing in the

current upgrade instructions:

1. You have to compile the new version of Evergreen. The easiest way to do

   this is to copy install.conf over from your previous version of Evergreen and

   run "make config" to ensure that all of the settings are still correct, then

   run "make" to build the new version of Evergreen.

2. Very important: Before installing the new version of Evergreen, you must

   prevent the database schema from being completely recreated or it will destroy

   any data that is already in your system. One way of doing this is, during the

   "make config" step, to list all of the Evergreen targets _except for_

   openils_db. I am simply incapable of remembering all of those targets, so my

   dirty workaround is to open Open-ILS/src/Makefile in an editor and modify the

   "install: " make target by removing the "storage-bootstrap" make target. What

   we really need is an "upgrade" target for "make config" that simply installs

   everything except for the database schema.

3. Confirm that the new version of Evergreen has been installed by running

   the srfsh command "request open-ils.storage open-ils.system.version".

For tomorrow (today, by the time you receive this), A. and G. are going to

create a swap file to enable the system to swap memory to disk if need be; the

system has 1 GB of RAM, which is enough for a small Evergreen system but when

one is compiling programs at the same time as running Evergreen swap space

really is necessary. This was a very good lesson learned for all of us!

5. also interested in learning more about basic Linux

administration. His institution currently runs on an entirely Windows

infrastructure, so the requirement to learn Linux is a fairly high hurdle.

I'm hoping that the eIFL-FOSS list will be a good resource for him to start

that journey. He has also asked to go over the step-by-step instructions for

installing Evergreen, so I'm considering starting that in a VMWare session so

that we can run through the steps. Our major goal for tomorrow is to migrate

some data from FSL's legacy system into Evergreen. Wish us luck!

Editorial comment: The combination of Armenian and Russian MARC records refused to load into the Evergreen 1.2.2.1 system, but on the flight home I confirmed that they loaded perfectly and were searchable on my Evergreen development system. As the development version will become this summer's 1.4 "internationalization" release, we are in good shape.

Editorial comment 2:On the second day, while running in circles trying to figure out why the records were refusing to load into the 1.2.2.1 system, I decided to try the #openils-evergreen IRC channel. Yerevan is 9 hours ahead of the Toronto/Atlanta time zone, so at noon Yerevan time I was hardly expecting any of the current core Evergreen developers to be online - yet, to our amazement, Mike Rylander responded. This was a pretty convincing demonstration to the attendees that the core developers really aren't far away or hard to contact at all.