Coffee|Code : Dan Scott - Evergreen

Evergreen on FLOSS Weekly: the aftermath!

dan@coffeecode.net (Dan Scott) — Tue, 31 Aug 2010 22:50:05 -0400

The recorded version of the Evergreen episode of the FLOSS Weekly show was released over the weekend. I'm happy to say that Lynn watched it without looking too pained at any given point, and the Evergreen project has already had several responses to our plea for assistance so far, particularly on the packaging front, which is fantastic! Just having one more skilled helping hand makes all the preparation for and stress about the show worth it.

Several points that amused me about the show as I glanced over Lynn's shoulder:

In Randal's introduction, he said that I "worked for Coffee|Code", full-stop. Aside to Leila Wallenius, the University Librarian of Laurentian University: no, there's nothing I need to tell you, I'm still a full-time employee at the University and I'm not planning on going anywhere! (That said, Coffee|Code Consulting is a registered sole proprietorship that provides small blocks of consulting services for Evergreen software in my spare time).
For the first half of the show, my affiliation was shown as the (misspelled) http://coffecode.net. So of course I immediately ran out and bought that domain.
Co-host Dan Lynch expressed a wish that his own show, Linux Outlaws, had a guest list like FLOSS Weekly. Oddly enough, some time ago when the subject of librarians and their fanatical devotion to open access to information came up on Linux Outlaws, I had submitted a feedback form on their site saying (essentially) "hey, if you want to talk to a Linux-loving free software-developing librarian some time, I'm around..." but I think that comment went into the ether.

If I ever do a video interview like this again, I'm going to try to:

Prop my laptop up on a couple of LCSH books or attach a separate web cam at a proper height so it doesn't appear that I'm looking down on the viewer.
Cut down on the "uhm"s and "ahh"s and stare a bit more robotically at the camera instead of rolling my eyes as I rack my brains to come up with an answer
Stop rambling and trust the interviewers to take the show in the direction that their audience will be interested in instead of trying to jam in points that I think are important or interesting
Ensure that my erstwhile partner in crime has a better Internet connection

Non-stop Evergreen, or "What I'm doing on my summer vacation"

dan@coffeecode.net (Dan Scott) — Wed, 25 Aug 2010 20:40:45 -0400

Last week, I started my summer vacation with a weekend at a friend's cottage. By Tuesday I was deeply engrossed in some Evergreen enhancement work for the International Institute of Social History. I'm building an authorities management user interface that properly exposes Evergreen's powerful authority support in the 2.0 release: browsing authority lists, editing authorities and having the updates ripple through to the bibliographic records with controlled fields, merging and deleting authorities... here's a screenshot of the interface in progress: . The numbers represent the number of bibliographic records linked to each authority record. These are still early days, but I think there are some cataloguers that are going to be pretty excited about this functionality when they get their hands on it.

This week, I'm on location in the Robertson Library at the University of Prince Edward Island doing some Evergreen consulting work for them. The good people at UPEI have put my family and I up in a nice cottage on the island, so I'm toiling away at improving Evergreen during the day while my family explores the island. Melissa Belvadi and Grant Johnson have put together a list of pain points that they would like me to address that happen to mesh nicely with general pain points that have come up over the years on the Evergreen mailing lists. My first priority has been to make working with spine labels a little less aggravating. I'm happy to say that after a day and a half, I've been able to teach the spine label editor how to (*gasp*) move up and down with the arrow keys and (*ooh-ahh*) insert and delete new lines and (*w00t*) have the spine label defaults come from library settings that only have to be set once instead of being individually set by each cataloguer. Oh, and I've added font size, font weight, and font family to those settings so that you can have 20 pt. bold Helvetica spine labels if you want them.

All of this code is being committed to Evergreen trunk as I hit functionality milestones; much of the authority work has made its way into the Evergreen 2.0 alpha release that was cut on Monday (although not yet announced officially). On Monday I also cut the OpenSRF 1.6.0-alpha release and uploaded a virtual image built on Debian Squeeze reflecting the OpenSRF/Evergreen alpha releases to http://evergreen-ils.org/~denials/Evergreen_trunk_2010_08_23.zip (note that it's 500 MB, and does not come with X installed, so it's primarily aimed at users that are already familiar with Evergreen and just want to see the new stuff without having to go through the entire install process).

I did take some time off of Evergreen development this afternoon, as I was honoured to be one of the two guests on the FLOSS Weekly podcast. Mike Rylander and I were there to discuss Evergreen with the hosts, Randal Schwartz and Dan Lynch. Unfortunately for Mike, me, and the audience, Mike's Skype connection kept dropping and I had to do the bulk of the talking. Despite missing the contributions from Mike's massive brain, I'm told that the show went well. So if you're interested in hearing a bit about Evergreen and why I do what I do, keep an eye open for the interview at http://twit.tv/floss132 - it should be edited and online by Friday, August 27th at the latest. I tried not to swear too often so they wouldn't have to do much editing work - heh.

Finally, somewhere in there I celebrated another birthday. Oh yeah! Older? Yes! Wiser? Probably not.

Classification scheme-aware call number sorting in Evergreen

dan@coffeecode.net (Dan Scott) — Sun, 08 Aug 2010 21:37:49 -0400

As a librarian who works at a library that primarily uses the Library of Congress classification scheme, I have been interested for a long time in teaching Evergreen to be aware of call number schemes other than Dewey. The problem, in a nutshell, is that Evergreen simply applies an alphabetical sort against the the uppercased version of the call number when generating call number browser displays - resulting in LC call numbers that sort incorrectly, like:

K 215 .E53 W37 1997
K 22 .U748 v.18

When the subject recently came up on the open-ils-general mailing list, I decided to follow up with some code. So, as of this weekend, Evergreen trunk now has a generalized infrastructure for generating sort keys for call numbers. The broad strokes of the current implementation are:

The classification scheme is set the level of the call number.
Classification schemes are defined in the asset.call_number_classification table with a pointer to a database function to call to generate a normalized sort key for the given call number.
Three classification schemes are available out of the box:
- Generic (the default) - a simple normalization approach that produces reasonable results in the absence of special rules for Cutters, etc
- Dewey (DDC) - a normalization routine taken from the Koha C4::ClassSortRoutine::Dewey Perl module
- Library of Congress (LC) - a normalization routine that simply wraps Bill Dueber's excellent Library::CallNumber::LC Perl module
- and adding more classification schemes is just a matter of adding another row to the asset.call_number_classification table and the appropriate sortkey-generating database function.

Note that this is the first time, to my knowledge, that Koha code has been adopted directly by Evergreen. I included attribution for the copyright holders in both the Generic and Dewey normalization functions. I wrote the Generic implementation in Evergreen from scratch shortly after taking a look at Koha's approach, so in some corners my work would be considered a "derived work". Koha's Dewey normalization function was (somewhat surprisingly) the only open-source implementation that I could find for Dewey, so it made perfect sense to me to adopt that for use in Evergreen. Many thanks to Koha for their use of the GPL v2 or later licence!

There are still some limitations and low-hanging fruit that I hope to address in the near future:

Right now you can only manipulate classification schemes via SQL. The Holdings Maintenance dialogue needs to give cataloguers the ability to set the classification scheme for each call number, because I'm sure they don't want to drop down to the command line. This setting should probably be sticky during a given session, so that if they're processing a cart of government docs, they won't have to change the scheme from the default to CODOC for each item.
Speaking of defaults, each library needs to be able to define a default classification scheme - so your consortium can have a Dewey library and an LC library and a SUDOC library, and their preferences won't trample each other. This can just be a simple org-unit setting.
Following on Mike Rylander's advice, the asset.call_number_classification table should gain a new column that lists the field/subfield combinations used to find the appropriate call number (if any) for each scheme in a given bibliographic record. Then the Holdings Maintenance dialogue can offer the appropriate call number based on the classification scheme.

Authorities in Evergreen: an Amsterdam trip report

dan@coffeecode.net (Dan Scott) — Mon, 19 Jul 2010 14:52:06 -0400

As part of the informal partnership between the International Institute of Social History (IISH) and Project Conifer, I was pleased to be able to spend the last two weeks in Amsterdam, working side-by-side with one of the Institute's developers, Ole Kerpel, on augmenting the support for MARC21 authorities in Evergreen. To prepare for the work session, I had posted a blueprint for the authorities work on the Evergreen Launchpad instance and circulated the list of requirements we had been asked to address to the broader Evergreen development community. We were fortunate to have the attention of Mike Rylander on the proposal, who not only supplied suggestions for how to implement some of the items, but also committed significant code contributions to the effort that greatly assisted our efforts. Here is a summary of the goals we accomplished in the current development branch of Evergreen (targeted for the 2.0 release), followed by a list of the outstanding items and my finger-in-the-air estimate of how much more time it would take to accomplish each of the tasks:

Accomplishments

Controllable control numbers
While not, strictly speaking, a requirement for authority control in and of itself, the ability to ensure that the behaviour of the 001/003/035 fields all conformed to the MARC21 specifications was an important requirement for IISH. They plan to provide external access to their authority and bibliographic records, so making the official identifier fields linkable based on the underlying record ID was an important aspect of the work. We implemented this feature as an optional database-level trigger to ensure that the control numbers and control number identifiers are always perfectly in sync with the internal identifier of the particular system on which the records are stored.
Links
Where having Mike Rylander participate in your review process pays off, part one... Before I even arrived in Amsterdam, Mike implemented a tricky database trigger that tracks the links between a given bibliographic record and the authority records to which it links. The links are tracked at the database level, as well as directly in one or more 0 subfields in each field that is controlled by an authority record. Yes, a given field in a bibliographic record can be controlled by two authority records and it all works. Nice, Mike!
Syncs
Where having Mike Rylander participate in your review process pays off, part two... Mike also implemented the bulk of the logic for automatically updating bibliographic records that are linked to a given authority record when that authority record is modified. Yes, folks, when you add a death date to an authority record, it will automatically appear in the corresponding bib records.
Control an uncontrolled set of bibliographic records
You may have dealt with library systems in the past that use some sort of string matching to implement authority support. As noted above, Evergreen is not like that. However, this means that many of us, when migrating to Evergreen, have bibliographic records lacking the 0 subfields that are required for full authority support. Towards that end, I wrote a script that will walk through a set of bibliographic records, search for matching authority records for each controllable field in each bibliographic record, and add the required 0 subfields to the bibliographic records. It certainly won't be a fast solution, but you should only need to do it once, and it worked on the limited test cases that we had ready at hand.
Teach the MARC editor about authority records
The MARC editor knew all about fixed fields for bibliographic records, and provided a handy grid for editing those fields. However, it didn't even know how to recognize authority records, and presented a fixed field grid that was absolutely meaningless. I spent a chunk of time laboriously transcribing the fixed field rules from MARC documentation into the MARC editor and now the MARC editor presents a reasonable fixed field grid for your editing convenience.
Merge authority records
Something that often happens in a library is that two authority records are created that identify the same thing. Eventually somebody notices the problem and wants to merge the authority records together. Towards this end, I added a database-level stored procedure that supports the merging of authority records, such that the linked bibliographic records will automatically point to the winning authority record.
Authority browse interfaces
Where having Mike Rylander participate in your review process pays off, part the third... Mike also implemented basic browse interfaces that presents a series of authority records in MARCXML format matching your requested authority type (author, title, subject, topic) and the matching substring at the /opac/extras/browse and /opac/extras/startwith URL entry points. While still raw at this point, these can provide the basis for classic authority browse interfaces for those who desperately desire them.

Remaining to-do items

Note that any estimates are based on how long I think it would take me to implement, based on my own familiarity with MARC and Evergreen and all things Perl and JavaScript and PostgreSQL, and provided with the granularity of no less than one day. Actual implementation times may vary, of course; if related work items are worked on consecutively, then it is likely to take less time to achieve than if the items are tackled sporadically.

Add an authority in the flow
When you're working in the MARC Editor and you find that there is no match for an entry that you really think should be controlled, IISH wants to make it easy for a cataloguer to add an authority record for that entry. We thought that there might be two options that we would want to expose - a direct "create an authority record from this field" option that takes no further input, and a "create an authority record from this field and open it in another MARC editor to let me tweak it" option. Estimate: 2 person days
Highlight controlled fields
This is really a two-part problem. First, for uncontrolled fields, we want to teach the Validate button to offer the kind of automatic matching that the script does and add the required 0 subfield. Second, we want to highlight fields that are explicitly controlled by authority records with a subfield differently from fields that simply match an authority record, but which are not controlled by it. Estimate: 1 person day
Simplify authority record selection
This two-part requirement would mask many of the fields that are currently offered as options when you right-click on an uncontrolled subfield to display matching authority records. For example, it is a little weird to offer a "See from" heading to a cataloguer; we're trying to avoid adding new records with those headings, right? Heh. Second, we want to introduce the ability to invoke the authority browse list in this interface so that the cataloguer can see a given set of headings in context and select the heading to apply from there. Estimate: 2 person days
Delete authority record
There is currently no cataloguer-friendly way to delete authority records. We need to expose a list of authority records (probably reusing that browse list again) and make it possible for cataloguers to delete an authority record. When that record is deleted, all bibliographic records that link to it need to have their links removed - and ideally, the cataloguer would be able to tell how many bibliographic records link to that authority before the delete takes place. Estimate: 1 person day
Edit and merge authority records
Although the database-level support now exists for merging authority records, we need to expose a means for cataloguers to select the authority records that they want to edit or merge. This could just be a slightly evolved version of the "Delete" interface. Estimate: 1 person day
Expose authority records via SRU/Z39.50/crawlable interface
One of the goals of the IISH is to be able to share their authority records with other institutions. One of the standard methods is SRU + Z39.50 server support; we should be able to build on the SRU/Z39.50 server support for bibliographic records in Evergreen to provide a basic solution for authority records. Interest has also been expressed in having a crawlable implementation that would give the linked data crowd something to play with. Estimate: 2 person days for an SRU/Z39.50 server, 1 person day for a very basic crawlable linked-data implementation

In summary - hurray for Mike Rylander for helping us out to such an extent, and many thanks, again, to IISH for giving me an opportunity to focus on Evergreen development for an extended period of time, and to Laurentian University for supporting my efforts. I hope that between Ole and myself that it will be possible to finish the rest of these work items prior to the Evergreen 2.0 release. It has been exhilarating to see far Evergreen's authority support has come in less than a month, and given a little more time I suspect that Evergreen's authority support will be the envy of other library systems.

Got funds for enhancing Evergreen? Looking for places to spend it?

dan@coffeecode.net (Dan Scott) — Fri, 02 Jul 2010 15:37:35 -0400

As an Evergreen developer, I believe our project has a few significant gaps that projects like RSCEL might be able to help address for the overall good of the community by bringing in outside resources to the project. Or perhaps there are skills within the community that don't feel like they've been called on yet; when I say that we lack skills, I'm basing that on the lack of patches and offers of assistance that I've seen in these areas. I would be delighted to be proven wrong! Either way, I submit this for the community's consideration.

3rd party security audit: Before Conifer adopted Evergreen, I had hoped that we would be able to fund a security audit of the code by a trusted and competent 3rd party like OmniTI (from a previous life, I believe that OmniTI employs some of the best people in the business, thus the plug - but there are certainly other options out there). As developers, we try our best to avoid vulnerabilities, but as the recently disclosed vulnerability in open-ils.pcrud attests, we're not experts in security. An audit of the public-facing interfaces (the catalogue, feeds, etc) would be a great help to the project. I would expect a prioritized list of areas that need to be addressed, along with recommendations on how to address those problems (whether they be cross-site scripting, session fixation attacks, authentication encryption attacks, etc). Our community's process (or lack thereof) for reporting and addressing security vulnerabilities might be an appropriate subject for an audit as well.
Testing framework: Our project is woefully short on tests, either human-powered or automated, for determining the state of the code at any given point in a release cycle. Thus, we have put out release after release that either won't install cleanly, or won't upgrade successfully from a previous release. The trunk version of the code had a error that meant that Evergreen couldn't be compiled; that problem existed for three weeks before somebody noticed and fixed it. I'm not pointing fingers, here; if I did that, I wouldn't have enough fingers to point back at myself for all of the problems I've introduced that other people have had to fix. Johnathan Nightingale in The Most Important Thing … or How Mozilla Does Security and What You Can Steal provides a great overview of Mozilla's philosophy about and approach to testing. There is all kinds of goodness in this presentation, but one of the most interesting points is that "money can be exchanged for services" -- that is to say, if your existing development team doesn't have the skills or time to implement a testing infrastructure, there are companies that do have the ability to put together a test infrastructure for a given project. Once that infrastructure is in place, it tends to get extended and used by the existing development team because it makes their lives easier; they don't need to manually test a given code path every time in the future, or deal with regressions that aren't noticed until months in the future when the changes they were making are no longer fresh in their minds. It sometimes requires a culture change, though.
Continuous integration: Hand in hand with a testing framework is a continuous integration server that provides testing feedback on every commit to the Evergreen repository for a given set of branches. Even without a testing framework, it is possible to have a continuous integration server run through the process of installing all prerequisites, configuring the code, building and installing the code, and creating the database schema to at least determine whether the basics can be accomplished successfully to confirm that a branch is ready for release. This arguably also goes hand in hand with a team's process for addressing a security vulnerability: if you have a continuous integration server that can tell you if a given fix does not introduce basic build and install errors, then you can get a new release out with much more confidence that you're not going to be encouraging your users to jump to a broken package. Note that Equinox ran a continuous integration server for OpenSRF and Evergreen trunk for a while, but that was killed and replaced by a call for volunteers to build a new continuous integration service (I can't find a more to-the-point call for volunteers, so perhaps it just hasn't been advertised widely enough - or again, perhaps we lack the skills in the community to get a standard CI service like Hudson running.)
Packaging: To decrease the difficulty of installing and configuring Evergreen, we need more investment in packaging Evergreen and all of its unpackaged dependencies. The idea is that a user should be able to run "aptitude install evergreen" or "yum install evergreen" and have the entire system installed and configured, and then run "aptitude upgrade" or "yum upgrade" to have newer versions installed. Right now the process is still rather onerous and requires a great deal of manual effort, although it has improved significantly since the early days of 2007. Again, this requires a particular set of skills that the Evergreen community does not appear to possess in depth: autoconf, automake, APT and RPM packaging - and perhaps some redesign of elements like skins to make local customizations easier to incorporate and keep up to date. This would be a natural complement to a continuous integration service, but much of the effort could also be done on its own.