Coffee|Code : Dan Scott

SFX target parser for Evergreen and some thoughts about searching identifiers

dan@coffeecode.net (Dan Scott) — Mon, 29 Jun 2009 13:00:48 -0400

Laurentian University is part of the Ontario Council of University Libraries (OCUL), and a user of the centrally hosted Ontario Scholars Portal SFX link resolver, so one of the things we needed when we migrated to Evergreen was a target parser for our link resolver. This is the target associated with Search the library catalogue that is the last resort when the resolver fails to turn up any full-text resources for a given OpenURL - so hopefully it won't need to be invoked too often, as we have a very rich set of full-text electronic resources at Laurentian University.

The code

Here is a quick implementation of a target parser that generates search URLs based on ISSN, ISBN, book title, or journal title. Pretty impoverished from an OpenURL perspective, but it maintains the same level of functionality from our previous system. In TargetParser/Evergreen/Conifer.pm I created a target parser called Evergreen::Conifer that implements a subset of the Parsers::TargetParser API for SFX as follows:

package Parsers::TargetParser::Evergreen::Conifer;
use Parsers::TargetParser;
use base qw(Parsers::TargetParser);
use strict;

sub getHolding {
  my ($this,$genRequestObj) = @_;
  
  my $objectType = $genRequestObj->{'objectType'};
  my $ISBN = $genRequestObj->{'ISBN'};
  my $eISBN = $genRequestObj->{'eISBN'};
  my $ISSN = $genRequestObj->{'ISSN'};
  my $eISSN = $genRequestObj->{'eISSN'};

  my $CODEN = $genRequestObj->{'CODEN'};
  my $bookTitle = $genRequestObj->{'bookTitle'};
  my $journalTitle = $genRequestObj->{'journalTitle'};

  # Canonical search results URL for simple searches:
  # http://laurentian.concat.ca/opac/en-CA/skin/lul/xml/rresult.xml?rt=keyword&tp=keyword&t=0895-2779&l=105&d=2&f=&av=

  my $svc = $this->{svc};
  my $egHost = $svc->parse_param('eg_host');
  my $egLocale = $svc->parse_param('eg_locale');
  my $egSkin = $svc->parse_param('eg_skin');
  my $egOrgUnit = $svc->parse_param('eg_org_unit');
  my $egDepth = $svc->parse_param('eg_depth');

  my $path = "http://${egHost}/opac/${egLocale}/skin/${egSkin}/xml/rresult.xml?l=${egOrgUnit}&d=${egDepth}";

  my $searchString = '&rt=keyword&tp=keyword&t=';

  if (defined($ISSN)) {
    if ($ISSN =~ m/x/i) {
      # Current indexer doesn't deal well with ISSNs containing an X, so break it up
      $ISSN =~ s/^(\d{4})-?(\d+)x/$1 -$2 x/i;
      $searchString .= $ISSN;
    } else {
      $searchString .= "\"$ISSN\"";      # format 9999-9999 for MARC
    }
  } 
  elsif (defined($ISBN)) {
    # Evergreen doesn't force ISBNs to be stripped of hyphens, so take whatever
    $searchString .= "\"$ISBN\"";
  }
  elsif (defined($journalTitle)) {
    # Restrict searches to title index, with bibliographic level = s
    $searchString .= "ti:${journalTitle}&bl=s";
  }
  elsif (defined($bookTitle)) {
    # Restrict searches to title index, with bibliographic level = m
    $searchString .= "ti:${bookTitle}&bl=m";
  }

  return ($path . $searchString);

}

1;

And here's the help that I added to the corresponding Conifer.hlp file:

General Information
Target - LOCAL_CATALOGUE_EVERGREEN_CONIFER
Service - getHolding
Parser - Evergreen::Conifer

Information needed in the Target Service:
In the PARSE_PARAM field, replace the following information:
eg_host = $$LOCAL_CATALOGUE_SERVER
eg_locale = Locale (en-US, en-CA, fr-CA, etc)
eg_skin = algoma, default, lul, nohin, uwin
eg_org_unit = 103, 1, etc
eg_depth = 0, 1, 2, 3, etc

Findings and wishlists

While it's quite easy to set up Evergreen as a searchable resource, thanks to its straightforward URL syntax, one of the things that leaps out at me is that Evergreen, by default, has no identifier index for limiting searches by ISBN / ISSN / LCCN / OCLCnum. Ideally, we would disable full-text indexing on this index so that we can more accurately search for ISSNs that include an x. Right now we have to split ISSNs with an "x" into constituent parts and generate searches on those parts, which results in false hits from across the database. This would also be useful for limiting Z39.50 searches.

I would also like to teach Evergreen about ISBN-10/ISBN-13 equivalence, to broaden the search while maintaining precision. And I would like to automatically normalize ISSN and ISBN formats so that I don't have to worry about whether a cataloguer entered hyphens or not - and the same for incoming search terms.

Finally, to support services like xISBN that search for multiple formats and editions of a given work by generating a shotgun blast of ISBNs for all known representations, I would love to teach Evergreen how to accept a list of identifiers as search input.

Don't ask me when these things will happen, though; if it requires work from me, it will probably be 2010 before any of it happens.

Globalization presentation at Evergreen International Conference 2009

dan@coffeecode.net (Dan Scott) — Thu, 04 Jun 2009 22:12:00 -0400

I was fortunate to be invited to give a talk (OpenOffice.org Impress / PDF ) on Evergreen's progress on the globalization front at the first ever Evergreen International Conference. My friend Tigran Zargaryan from the Fundamental Science Library of the National Academy of Sciences of the Republic of Armenia gave a talk at almost the same time about his library's progress in adopting Evergreen. Tigran himself was responsible for the translation of the Evergreen catalogue and staff client into Armenian, and he confided that he also expected to make significant progress towards a Russian translation during the lengthy layovers at airports that are part of his normal travel routine.

So, my goal was to provide an overview of the progress we have made in taking Evergreen from its American English roots and enabling it to support not just translated interfaces, but properly localized content display - and to provide some pointers towards where we need to go next. We have been making progress towards a more formalized translation process, so keep an eye out for a call for translations in the next week or two when the Evergreen 1.6 release candidate is made available for testing. We currently sport Armenian, Canadian English, Canadian French, and Czech translations, and welcome both new translations and revisions to our current translations.

To make it easier for translators to collaborate, we need to take our Pootle translation server from a beta service running on my poor little VPS to a real server. We have some technical challenges to overcome - providing translation support for the Template::Toolkit framework, for example. And we have some basic grunt work to do to replace the hard-coded display of numbers, currencies, dates, and times with localized variations throughout our code.

I was pleasantly surprised by the number of people attending the session; I hadn't expected such an interest in the topic, despite it nominally being an international conference. My only regret was that I rushed off the stage without taking questions in the mistaken belief that I had used up all of my time and was eating into my successor's presentation timeslot; as it turned out, there was a built-in 15 minute buffer that I had overlooked. Ah well. Thanks to everyone who came out, and for everyone else who wasn't able to make it to the session, I hope you'll find the slides a good introduction to the state of globalization in Evergreen. And if you have the skills to contribute, please consider pitching into the globalization enablement effort!

Evergreen International Conference hackfest results: Evergreen serials support

dan@coffeecode.net (Dan Scott) — Wed, 27 May 2009 14:23:40 -0400

Yes, all of a sudden and rather quietly, Evergreen has serials support.

A few weeks ago, I finished hooking up a rudimentary serials holdings display based on David Fiander's MFHD parsing code to our production instance of Evergreen. We loaded our MFHD records from our legacy system into Evergreen and that gave us enough breathing room to keep working on the problem. By rudimentary I mean:

limited to displaying one MFHD record per bibliographic record (a problem for journals for which you have separate sets of holdings in microfiche, print, etc)
serials holdings were displayed for a given bibliographic record no matter what library scope you were searching in (more of a problem in theory than in practice as we currently have one copy of a given bibliographic record per library... that will change over time...)
no way to edit the MFHD records, which is a problem as the issues we have received since migrating to Evergreen three weeks ago are starting to pile up
limited to English labels in the interface

Here's the rudimentary serials holdings display:

The operative phrase is was rudimentary. In the past two weeks, things have come a long way in Evergreen. The primary result of my afternoon of work at the Evergreen International Hackfest, with lots of help from Mike Rylander and Bill Erickson in navigating the impressive new Dojo toolkit-based Evergreen JavaScript widgets and services in the upcoming Evergreen 1.6 release, was to add an Edit button to the holdings display that shows up when the record is viewed in the staff client. When pressed, the Edit button invokes a MARC editor so that you can copy an 86[345] field and fill in the pertinent information; or collapse holdings in the 86[678] fields, etc. It seems like a minor victory, but it was a real result from the hackfest, and that cannot be discounted!

Here's the MARC editor in action:

Since then, I've been on fire... or maybe on a slow burn, as I put a few hours in here and there, and am happy to say that when Evergreen 1.6 is released, serials support will feature:

support for display unlimited MFHD records per bibliographic record
holdings display scoped by library search context - so you'll only see holdings for the part of the library hierarchy that you're searching, rather than the whole consortium
the Edit button for editing the raw MFHD record
internationalization support for interface labels, based on Dojo string substitution

I have already committed these features to the Evergreen trunk, but I hope to add a few more pieces to the mix before the Evergreen 1.6 release is cut. We need to display the 852 field contents to identify the location of each set of holdings, and we need to give cataloguers the ability to edit some of the attributes (such as owning library).

Here are the slides I presented (largely screenshots of the serials display and edit button) for the hackfest results lightning talk that I gave with Jeff Godin of Traverse Area District Library. Jeff did some interesting work in his own right on generating feeds for recently added titles based on copy location during the hackfest.

Conifer lives: Ontario launches a consortial academic library system built on Evergreen

dan@coffeecode.net (Dan Scott) — Mon, 11 May 2009 17:21:24 -0400

I awoke around 4:48 am today. At the time, I thought it was just our baby kicking away excitedly. However, later this afternoon, I realized that it had been almost exactly a week ago, around 4:30 am on Monday, May 4th that I sent a broadcast email message to librarians and staff at 24 different libraries. The Conifer consortial library system, built on the solid foundations of the Evergreen open-source library system, had gone live - and I was exhausted after a long weekend of migrating all of that data. I was proud to see the Laurentian catalogue sporting a completely different look and new functionality - reviews! book covers! sharable book bags! format & edition grouping! - and excited by the promise of more to come.

Conifer represents the first flowering of an effort that began back in July 2007 with a hand-shake agreement between Laurentian University, McMaster University, and the University of Windsor to build a provincial, primarily academic, library system on Evergreen. The system is centrally hosted by the top-notch IT team at the University of Guelph. Things change, and along the way Algoma University and the Northern Ontario School of Medicine joined us as full partners, and McMaster University opted to continue contributing to the common development effort but withdrew from the centrally hosted system.

As noted, we went live on Monday, May 4th and we survived the first day. On Tuesday, May 5th we corrected a problem in our configuration that had caused some instability (thanks to Mike Rylander for providing the patch that set things straight). Since then, we have been slowly refining aspects of the system - setting up circulation rules, migrating records and items that had been missed over the weekend, polishing the Z39.50 server, fine-tuning the permissions scheme - but the core of the system is solid. We have a consortial system that stretches from the southern-most tip of Ontario to the north-west corner of the province (hello, Thunder Bay!), and so far connectivity seems good and the reliability of the system - which, upon launch, has probably become the second largest Evergreen implementation by number of bibliographic records - has been superb.

A few interesting statistics about Conifer... (have I mentioned how much I love that Evergreen is built on PostgreSQL because it becomes so simple to generate basic reports in plain SQL?):

Number of staff and user accounts per library in Conifer

conifer=# SELECT aou.name, count(au.id)
    FROM actor.org_unit aou
    INNER JOIN actor.usr au
    ON aou.id = au.home_ou
    GROUP BY aou.name
    ORDER BY 2 DESC;

                    name                    | count 
--------------------------------------------+-------
 Leddy Library                              | 19468
 J.N. Desmarais Library                     | 11921
 Algoma University, Wishart Library         |  2431
 University of Sudbury                      |  1100
 Hearst, Bibliothèque Maurice-Saulnier      |  1043
 Huntington College Library                 |   834
 Paul Martin Law Library                    |   592
 Northern Ontario School of Medicine (West) |   284
 HRSRH Health Sciences Library              |   261
 Northern Ontario School of Medicine (East) |   224
 Xstrata Process Support Centre Library     |   122
 NOHIN                                      |   121
 Instructional Media Centre                 |     9
 Laboratoire de didactiques, E.S.E.         |     7
 Vale Inco                                  |     4
 Mines Library, Willet Green Miller Centre  |     2
 Art Gallery of Sudbury                     |     1
 Curriculum Resource Centre                 |     1
 Sault Area Hospital                        |     1
 Centre Franco-Ontarien de Folklore         |     1
 Conifer                                    |     1
(21 rows)

Number of copies held per library in Conifer

conifer=# SELECT aou.name, count(ac.barcode)
    FROM actor.org_unit aou
    INNER JOIN asset.copy ac
    ON aou.id = ac.circ_lib
    GROUP BY aou.name
    ORDER BY 2 DESC;

                    name                    |  count
--------------------------------------------+---------
 Leddy Library                              | 1373197
 J.N. Desmarais Library                     |  614380
 Paul Martin Law Library                    |  229391
 Algoma University, Wishart Library         |  115156
 University of Sudbury                      |   42154
 Hearst, Bibliothèque Maurice-Saulnier      |   34276
 Huntington College Library                 |   12517
 Laboratoire de didactiques, E.S.E.         |   10284
 Mining and the Environment Database        |    9940
 HRSRH Health Sciences Library              |    7512
 Music Resource Centre                      |    7511
 Xstrata Process Support Centre Library     |    5477
 Centre Franco-Ontarien de Folklore         |    4365
 Northern Ontario School of Medicine (East) |    3779
 Northern Ontario School of Medicine (West) |    3301
 NOHIN                                      |    2647
 Mines Library, Willet Green Miller Centre  |    2617
 Curriculum Resource Centre                 |    2583
 Sault Area Hospital                        |    2515
 Art Gallery of Sudbury                     |    2237
 Hearst Timmins, Centre de Ressources       |    2202
 Hearst Kapuskasing, Centre de Ressources   |    2007
 Vale Inco                                  |    1106
 Instructional Media Centre                 |    1095
(24 rows)

What about acquisitions, serials, and reserves?

One of the reasons we had a hard migration date of early May was because it matches nicely with the fiscal year-end for those institutions who were running a traditional acquisitions system on their legacy ILS. We normally shut down all purchases for a period of weeks while we roll over the encumbrances into the next fiscal year and set up our budgets. This year, we're migrating all of the old financial data twice: first, and foremost, into the most sophisticated set of spreadsheets you'll ever see attached to a library system (as pulled together by the inestimable Art Rhyno); and second, into the Evergreen acquisitions system that will launch with Evergreen 1.6. The first migration of a given set of data is always the hardest part, so once we have the fund / order / provider data in spreadsheets, the migration into Evergreen proper will be trivial. This will give us the summer to use both systems side-by-side and refine what we need from Evergreen.

We have migrated all of our serials data from the legacy system, I just haven't enabled the display of that data in our live system. A prototype was running on my laptop for a few days until I accidentally blew it away - ah well, anything worthwhile doing is better the second time around anyway. This, too, will be part of the Evergreen 1.6 release, and will feature full MFHD compliance built on the code that David Fiander has been writing on behalf of Equinox. I should note that this first cut at serials is in some ways relatively basic; while the system in Evergreen 1.6 will be fully MFHD compliant, down to the point of letting you to edit an MFHD record to "check in" a new issue by adding a new 863 field, it won't associate barcodes with individual issues. Most of the database schema exists to support that, but there's still a large amount of code to be written on top of the schema and we need Something That Works Right Now I'm confident that that's coming not too far down the road, though.

Finally, what would an academic library be without reserves? Art Rhyno (again!) has been working with Graham Fawcett for the past six months on Syrup - a really impressive melding of the world of electronic reserves and traditional physical library system reserves that uses SIP and Z39.50 to talk to Evergreen. Syrup is just about at a full boil now, so in a few more weeks we should have it deployed so that we can savour its sweetness through the relatively slow summer months before ensuring that the taste is just right for all of our incoming students and faculty in the fall.

All in a day's work: defending book reviews in our catalogue

dan@coffeecode.net (Dan Scott) — Thu, 07 May 2009 16:26:10 -0400

The following was written in response to a faculty member's complaint that a review containing a negative statement about a book that the faculty member had authored was attached to the book record in our catalogue. The facult member asked that we delete the review from the catalogue because [i]t is inappropriate for the Laurentian library to be highlighting attacks on Laurentian's faculty, undermining their work in the eyes of students and other faculty members.

My response:

On the matter of the reviews that appear as a supplemental part of the Conifer catalogue, we contracted with a third-party supplier of review content that draws from a number of sources.

These reviews are automatically matched to books displayed in the catalogue by correlating their ISBN. There is no selection being performed on the part of the Library, nor is there the ability to select or hide specific reviews for a given book in the catalogue.

It is regrettable that the one review available for your book from our added content supplier should include one negative statement; however, I'm sure you can appreciate my position that it would be an extremely dangerous policy for the Library to deliberately suppress content if said content does not support our institution.

As to your concerns about catalogue browsers confusing the providence of the review, I believe most readers would find the byline "CHOICE Copyright © American Library Association, used with permission." a reasonably clear indication of the source of the review. I agree, however, that the reviewer's name should be included, and will request that our added content supplier make this amendment to their service.

I do empathize with your position on this matter, but I hope that you can see that we are trying to draw from a broad range of review sources, and that these sources are generally considered reputable, and that as a general policy I cannot support hiding reviews that are not uniformly supportive of our institution.

I had expected the addition of third-party reviews to the catalogue to be a universally welcomed enhancement, and I naively failed to anticipate this possibility. In retrospect, it's easy to see that enriched content that helps library users select material might not please everyone. However... just as we wouldn't rip the article out of the pertinent copy of Choice if we had it on our shelves, I think we're perfectly justified in maintaining our third-party reviews in the catalogue.

Have other librarians run across similar complaints?

Evergreen iPhone application? Unnecessary!

dan@coffeecode.net (Dan Scott) — Mon, 13 Apr 2009 00:29:58 -0400

This Easter weekend I had the opportunity to play with someone's iPod Touch. Of course, the only thing I tried was the Evergreen 1.4 catalogue interface. Lo and behold, it came up just fine on Safari in all of its heavily dynamic JavaScript and less-than-XHTML-compliant glory - even sporting several Dojo widgets. Nice. So we don't have to worry about writing an iPhone-specific application to access Evergreen; users of such devices can just use the normal dynamic catalogue with full functionality.

Evergreen doesn't fare quite as well with Microsoft's rather decrepit PocketExplorer browser on my HTC Touch smartphone (it's a Windows Mobile monstrosity, sigh), but it does work well with the Opera Mobile 9.5 beta browser. I eagerly anticipate the first good release of Fennec for Windows Mobile (coming soon!), as I'm confident that's going to improve my mobile Web browsing experience even further.

I predict that in another year or two the idea of building mobile-specific Web portals to complement your full-function Web site will be pretty passé. I already get really irritated when Web sites think they're being helpful by automatically redirecting my smartphone to an extremely limited interface; in most cases, the full site runs fine. Give me the option, sure, but don't force me down that path. As hardware costs continue to drop, and 3G networks expand, and more people upgrade to more capable mobile devices, one full-function Web site will be all we need--as long as that site is written in (X)HTML and CSS and JavaScript.

Those sites that decide to push core functionality into Flash or SilverLight, on the other hand, can go straight to hell, thankyouverymuch. I'm looking at you, PTOnTheNet. This is a site to which Lynn has been a paying customer for years. It recently announced that it was revising the Web site, which is all well and good. What's not so good is that they adopted SilverLight: not just for pretty effects here and there, but as a core technology. Problem: Lynn has been using Linux at home since I introduced her to it somewhere around eight years ago, and last year bought one of the early models of the Linux-based Asus EEE netbook. Not only did the site redesign destroy the personal training programs she had set up for her clients over the years (breaking site redesign rule #1: Thou shalt not destroy your clients' data), but it also renders her netbook useless for that site.

Even with the Moonlight plugin installed, it looks like the cretinous site developers are using detection scripts to prevent the plugin from even trying to render the content. With Linux-based netbooks on the rise--and with netbooks being the right form factor and price for personal trainers who want to throw them into their backpacks and not weep too bitterly if their netbook suffers the misfortune of being knocked around or sweated to death--this seems very much like a technology choice that was not based on the needs of the customers. Worst of all, they deliberately chose to exclude Linux, when a (X)HTML, CSS, and JavaScript platform would have supported almost any modern platform: not just Linux netbooks, but other mobile devices like the iPhone and smartphones that are so well-suited to the personal trainer. So, at least one customer is going to be walking away, and if there's a competing Web site out there that caters to a broader clientele, I bet there will be far more customers moving in that direction.

Transparent acquisitions budgets and expenditures for academic libraries

dan@coffeecode.net (Dan Scott) — Thu, 19 Mar 2009 13:30:35 -0400

In my most recent post over at the Academic Matters site, after a general discussion about "new books lists" in academic libraries, I tackle one of the dirty laundry areas for academic libraries: exposing how collection development funds are allocated to departments. Here's a relevant quote:

For 2008-2009, we decided to adopt a much more transparent approach our collection development decisions and began publishing publicly available collection development reports on a weekly basis. These reports mirror our library budgets, which are broken down by a per-department, per-material type, per-item language hierarchy. We make the allocated, encumbered, and paid budget amounts available for all to see - inside and outside the Laurentian University community. ... We recognize that there can be political consequences of exposing this data, particularly if departments feel that they deserve more of a piece of the collection development budget than they currently receive - but those conversations need to be held, and making the data as transparent as possible can only help facilitate those discussions.

Earth-shaking stuff, eh? If you want to read more, go check it out! Maybe you can be the first person to comment on one of the More than Bookends blog posts that Amy Greenberg, Anne Fullerton, and I are slowly publishing

One big library, one little device: Evergreen staff client on Nokia N810

dan@coffeecode.net (Dan Scott) — Mon, 02 Mar 2009 00:23:17 -0500

It's hard to take good photos of these devices

Almost exactly a year ago, Jason Etheridge (the primary developer of the Evergreen staff client) and I managed to get our hands on a developer edition of the Nokia N810 Internet tablet device. It's a nifty little handheld computer that packs 128 MB of memory, a touch screen, and a beautiful 800x480 screen, and I've had my hands on it from almost the beginning. The primary rationale of the Nokia developer program was to encourage developers to put together useful applications for their platform, of course... and as the months ticked by and I did nothing of interest, my guilt slowly grew.

Well, today I feel a little bit better. Here's what happened: when I was attending FSOSS 2008 at Seneca College, I ran into Madhava Enros. Madhava and I had worked together on some help UI designs back when we were both DB2 employees; since then, he had joined the Mozilla Foundation and was working on Fennec, the mobile version of Firefox targeting the N810 device (to begin with, at least). The first alpha of Fennec had been released to coincide with FSOSS 2008, so I gave it a shot a few days later. Madhava's team made some great innovative decisions for Fennec's UI, but what really caught my eye was that they had packaged a port of XULRunner-1.9 to the N810.

See, the Evergreen staff client is built on XUL, the same XML/JavaScript/CSS foundation as Firefox and Thunderbird and Fennec - and to run XUL, you need XULRunner. At the time, though, the Evergreen staff client needed the 1.8 version of XULRunner; it simply wouldn't work with 1.9. So, I stuffed the N810 back into its case and forgot about it for a few more months while I focused on other things like the never-ending effort to improve Evergreen's internationalization support.

Over the last few weeks, though, Jason has been steadily enhancing the staff client in Evergreen trunk - and the comment for one of his recent commits was we're kicking xulrunner 1.8 to the curb with trunk. I had a spare hour or two on my hands today, so I copied a staff client build from Conifer's Evergreen trunk test box to the N810, kicked off the XULRunner command, and waited... expecting failure. Instead, I found that the staff client worked almost exactly as it does on my laptop - the major difference being that some of the default function key mappings on the staff client conflict with the mappings of special buttons on the N810 (like the full screen toggle gets mapped to F6 - Record In-House Use on the staff client). Otherwise, the client did a great job of adjusting to the available screen width, and even Dojo-based interfaces like the Vandelay MARC batch importer/exporter and the pop-up calendar worked. Very cool!

So, if I can find a barcode scanner with a mini-USB attachment, I could have a nice little inventory tool on my hands. Or a mobile circulation station. All because the Evergreen developers made the decision years ago to build on XUL as a cross-platform framework... this should be sweet confirmation that they made a good choice. XUL continues to be ported to more platforms, and anyone using the Evergreen staff client benefits from the optimizations and bug fixes that go into XULRunner. Nice. When we cut a release from Evergreen trunk that supports XULRunner 1.9, I'll do my best to package up a version of the staff client for the N810, and some of my guilt will be assuaged. Yes!

Updated 2009-03-02 10:35 am: Correcting Madhava's name; I shouldn't write past midnight without proof-reading! Sorry Madhava.

Arik and Amber update

dan@coffeecode.net (Dan Scott) — Sun, 01 Mar 2009 08:38:01 -0500

Arik is just over eight weeks old now (and over fifteen pounds!), so I'm long overdue to post some photo updates of the family. The following photo was taken when Arik was about six weeks old. He loves the sling; so does his Mommy, and and so does his Daddy (who has an extra-large version to fit his extra-large frame). Amber prefers us to use the Austin Powers-style baby carrier, though.

Three most important people

Janet sent us this fabulous soft, warm quilt for Arik that will undoubtedly accompany him for years to come. In this picture, he's lying on the floor in our living room soaking up a bit of the sunshine on a warm February day in Sudbury (it happens, honestly!):

A study in contrasts (quilt courtesy Janet)

Spook loves Arik (and the quilt, and sunshine) too

Amber loves Arik. Loves holding him, loves singing to him, loves being sung songs featuring both of them that Daddy makes up on the spot... only sometimes is that a detriment when Arik is sleeping and she wants to wake him up

Big sister doesn't look that big next to Arik

Amber also loves the camera. (For the dentists out there, she is on a weaning program for her pacifier, don't worry)

Mugging

Arik apparently likes the camera too.

Mugging, part two

Updated: 2009-03-01 Thumbnails were all linked to the same image... fixed!

Seven things

dan@coffeecode.net (Dan Scott) — Mon, 09 Feb 2009 00:56:35 -0500

I was tagged by Lukas for the "7 things" meme, and meant to do something about it, but I've been kind of preoccupied with the new baby and the sprinting toddler and work. Anyway, it seems like a heck of a lot more reasonable than the evil Facebook's "25 things" meme, so I'm going to take a few minutes to try to play along.

I was an early riser until I was around 15 or 16 years old and discovered the surrealists. At that point, I began experimenting with sleep deprivation as a means of stimulating my prose and poetry. This is also when I began drinking coffee. After about a month, I was no longer capable of being an early riser--and the fruit of writing experiments was, uh, not too impressive.
Rather than going directly to university after high school, I elected to take what is now termed a gap year. No trips to Europe for me, though; the goal was to refine my bass-playing and music-reading skills and head to a post-secondary music program. I recorded a few prog-rock tracks in a studio with a fantastic couple of guys (hey Pete and Mike!), but ultimately didn't put enough effort into my bass to carry out the plan. Let me assure you that a year of working night shifts at a convenience store in the entertainment district of a small city is not a waste of time; I can't count the number of experiences that I'm thankful for having had during that time.
Although I roast and grind my own coffee, I'm not a coffee snob. In fact, I possess almost no sense of smell and I suspect that my sense of taste is limited in comparison to most people, and I'm quite happy to drink diner coffee. I cannot stand the taste of Starbucks coffee, however.
The first time I was able to run a full kilometre without walking was when I was eighteen. Since then I've run a couple of 5K races and and a sprint duathlon.
I'm pretty sure I was destined to become a systems librarian. When I was 10, I used to hang out at the local college's computer room until the students would log me onto a completely restricted Gandalf mainframe account so I could pretend to be Matthew Broderick in WarGames. My first real job, when I was 14, was as a "computer page" at the Barrie Public Library Children's Annex. It was my responsibility to oversee the use of the bank of Commodore 64s that the library made available to children, luring them in with games but requiring them to complete their allotment of educational software first. Oh the power.
I occasionally wrote reviews for random CDs that came into the campus newspaper office. Nobody else wanted to review this orange CD called Tragic Kingdom by some West-coast band, so I took it on. I gave it a savage review; I wasn't impressed with faux-ska and couldn't stand the lead singer's voice. Six months later No Doubt's "SpiderWebs" was in high rotation on every radio station in North America (look, folks, that song is repetitive enough without being played twice an hour!). I'm sure that my negative review still gnaws at Gwen Stefani today as she weeps bitterly in her platinum mansion.
In grade one, my report card read Dan is too critical of his classmates. In my defence, if they weren't so stupid--come on, sound it out buddy--I wouldn't have been critical. Okay, not much of a defence.
I am not a very demanding friend. I (almost) never call, (almost) never write, and (almost) never visit. Okay, scratch that: I'm a crappy friend. Most of my close friends found out that we were expecting a second child only through Lynn's Facebook account. I called one couple shortly after Arik was born and his quasi-namesake (one of the Eric's in our life who bring honour to the noble name) asked me after a few minutes: "So, uhh... did we know that you were expecting a baby?". No, no you didn't, and that's not your fault. Man I suck.
I'm really good at arithmetic.

Link your original tagger(s), and list these rules on your blog.

Share seven facts about yourself in the post — some random, some weird.
Tag seven people at the end of your post by leaving their names and the links to their blogs.
Let them know they’ve been tagged by leaving a comment on their blogs and/or Twitter.

Wow, that was fun. Lemme see, I'm going to break the rules and just tag two people: Helmut, because he's one of the only other people who worked on the ibm_db2 PHP driver out of passion rather than as a job assignment. And Gabriel because I like his style.

Unicorn to Evergreen migration: rough notes

dan@coffeecode.net (Dan Scott) — Sun, 08 Feb 2009 16:32:48 -0500

Updated 2009-02-25 00:29 EST: Corrected setuptools installation step.

Updated 2009-02-08 23:39 EST: Trimmed width of some of the <pre> code sections for better formatting. Created bzr repository for unicorn2evergreen scripts at http://bzr.coffeecode.net/unicorn2evergreen

I did this once a long time ago for the Robertson Library at the University of Prince Edward Island. For our own migration to Evergreen, I have to load a representative sample of records from our Unicorn system onto one of our test servers. This has been a good refresher of the process... and a reminder to myself to post the other part of the Unicorn to Evergreen migration scripts in a publicly available location. Okay, they're posted to this bzr repository called unicorn2evergreen

Export bibliographic records from Unicorn using Unicorn's catalog key (basic sequential accession number) as the unique identifier (I plopped the catalog key into the 935a field/subfield combo). I use the catalog key because the "flexkey" is not guaranteed to be unique within a single Unicorn instance - and because the catalog key makes it easy for us to match call numbers and copies.
For each item, export call number / barcode / owning library / current location / home location / item type using the catalog key as the identifier.
Set up the organization unit hierarchy on your Evergreen system. You can dump it from an existing Evergreen system into a file named "orgunits.dump" like so:
```
pg_dump -U evergreen --data-only --table actor.org_unit_type \
    --table actor.org_unit > orgunits.sql
```
Then drop all of the existing org_units and org_unit_types and load your custom data in a psql session:
```
BEGIN;
SET CONSTRAINTS ALL DEFERRED;
DELETE FROM actor.org_unit;
DELETE FROM actor.org_unit_type;
\i orgunits.sql
COMMIT
```
Import bibliographic records using the standard marc2bre.pl / direct_ingest.pl / pg_loader.pl process. Point the --idfield / --idsubfield and --tcnfield / --tcnsubfield options for marc2bre.pl at 935a (yes, this sucks for title control numbers, but as noted above they are not guaranteed to be unique in Unicorn and we need uniqueness in Evergreen). We need the bibliographic record entry ID field to be the catalog key to set up subsequent call number/barcode matches.

Enable the subsequent addition of new bibliographic records by setting the sequence object values to avoid conflicting ID / TCN values by issuing the following SQL statements:

SELECT setval('biblio.autogen_tcn_value_seq',
     (select max(id) from biblio.record_entry) + 100);
SELECT setval('biblio.record_entry_id_seq',
     (select max(id) from biblio.record_entry) + 100);

Process holdings records.
1. Call numbers might have MARC8 encoded characters, so process'em and convert to UTF8. Theoretically "yaz-iconv -f MARC-8 -t UTF-8 < holdings.lst > holdings_utf8.lst" should do it, but instead it eats linefeeds and creates an unusable field. Ugh. We use a little Python script instead that requires pymarc, which in turn requires a version of setuptools (0.6c5) newer than Debian Etch's packaged version (0.6c3). So:
```
wget http://pypi.python.org/packages/2.4/s/setuptools/setuptools-0.6c9-py2.4.egg
sudo sh setuptools-0.6c9-py2.4.egg
sudo easy_install pymarc
```
2. Now actually generate the 'holdings_utf8.lst' file.
```
cat holdings.lst | python marc8_to_utf8.py
```
3. Adjust parse_unicorn.py to match up the holdings fields (added flexkey to the start). Then parse the holdings_utf8.lst to generate an SQL file (holdings_eg.sql) that we can load into the import staging table.
```
python parse_unicorn.py
```
  Note that the holdings data for the item with barcode 30007007751786 didn't process cleanly and won't load. Weird - possibly a corrupt character in the item data? Augh, no - there are flexkeys and callnumbers that contain '|' characters (16 occurrences for "|z", 37 for "|b"), which is of course also what we are using as our delimiters. ARGH. I deleted it for now with:
```
grep -v '|z' holdings_utf8.lst > holdings_clean.lst
grep -v '|z' holdings_clean.lst > holdings_clean.lst2
mv holdings_clean.lst2 holdings_clean.lst
```
  Adjust parse_unicorn.py to match the new input name and generate a new holdings_eg.sql.

Create the import staging table:

psql -f Open-ILS/src/extras/import/import_staging_table.sql

Load the items into the import staging table:
```
psql -f holdings_eg_clean.sql
```
We discover that some more of our data sucks - for example, one item ("Research in autism spectrum disorders", HIRC PER-WEB) has a create date of '0' which is not a valid date format because the barcode is "1750-9467|21". For now, grep it out as above and reload.

Investigate possibilities of collapsing unnecessary duplicate item types:

SELECT item_type, COUNT(item_type)
FROM staging_items
GROUP BY item_type
ORDER BY item_type;

 item_type  | item_count 
------------+------------
 ATLAS      |        162
 AUDIO      |        792
 AUD_VISUAL |       1790
 AV         |         69
 AV-EQUIP   |        182
 BOOK       |        996
 BOOKS      |     581592
 BOOK_ART   |          1
 BOOK_RARE  |       4949
 BOOK_SHWK  |          5
 BOOK_WEB   |      49163
 COMPUTER   |         33
...
(40 rows)

How about locations?

SELECT location, COUNT(location)
FROM staging_items
GROUP BY location
ORDER BY location;

  location  | count  
------------+--------
 ALGO-ACH   |     13
 ALGO-ATLAS |    148
 ALGO-AV    |   1837
...
(212 rows)

Now we can collapse categories pretty simply inside the staging table. For example, if we want to collapse all of the BOOK types into a single type of BOOK:

UPDATE staging_items
SET item_type = 'BOOK'
WHERE item_type IN 
('BOOKS', 'BOOK_ART', 'BOOK_RARE', 'BOOK_SHWK', 'BOOK_WEB', 'REF-BOOK');

Update legacy library names to new Evergreen library short names (we're using OCLC codes where possible). Some will be straightforward old names to new names. Others will require a little more logic based on location + legacy library name; we're splitting the DESMARAIS collection into multiple org-units (Music Resource Centre, Hearst locations, hospital locations, etc).
```
-- Laurentian Music Resource Centre
UPDATE staging_items
SET owning_lib = 'LUMUSIC'
WHERE location = 'DESM-MRC';

-- Hearst - Kapuskasing location
UPDATE staging_items
SET owning_lib = 'KAP'
WHERE location LIKE 'HRSTK%';

-- Hearst - Timmins location
UPDATE staging_items
SET owning_lib = 'TIMMINS'
WHERE location LIKE 'HRSTT%';
```
Generate the copies in the system:
```
psql -f generate_copies.sql
```
Make the metarecords:
```
psql -f quick_metarecord_map.sql
```

Ah, recognize that any electronic resources (which don't have associated copies) won't appear. Check for 856 40 and change the bre source to a transcendent one mayhaps?

-- Create a new transcendant resource; 
-- this autogenerates an ID of 4 in a default, untouched system
INSERT INTO config.bib_source (quality, source, transcendant)
VALUES (10, 'Electronic resource', 't');
-- Make the electronic full text resources (856 40) transcendant
-- by setting their bib record source to the new bib_source value of 4
UPDATE biblio.record_entry 
SET source = 4 
WHERE id IN (
    SELECT DISTINCT(record) 
    FROM metabib.full_rec 
    WHERE tag = '856' AND ind1 = '4' AND ind2 = '0'
);

And no transcendence. Hmm. Oh well, worry about that later.

Evergreen Exposed: introduction to Evergreen development (OLA 2009)

dan@coffeecode.net (Dan Scott) — Sun, 01 Feb 2009 15:21:19 -0500

Update 2009-02-19: uploaded diffs from Evergreen 1.4.0.2 (EG_exposed.tar.gz) for adding details to record summary; and Bill Erickson's slides and code examples are also available for download

The slides: Evergreen exposed, part 1 (OpenOffice)

My second presentation at the OLA SuperConference 2009 was Evergreen Exposed: hacking the open library system, which promised to take attendees on a tour of the architecture and source code of the Evergreen library system. I was very fortunate to have Bill Erickson, one of the original Evergreen developers, agree to join me as a co-presenter. Given the hour-and-fifteen-minute time slot that we were allotted, we opted to take an incremental approach to introducing parts of Evergreen to the audience, starting with basic tasks and working up to more complex customisations. We also tried to focus on answering questions that had been posted to the Evergreen mailing lists to ensure that we would satisfy our target audience's interests.

Dan starts with the basics

I started the session with an introduction of how to create a different skin for the catalogue, starting with text, CSS, JavaScript, and images and extending to the translation and customization framework. We talked about how to future-proof your customizations against future upgrades and how consortia can use skins to provide not just different look-and-feel, but different functionality, for each member of the consortium. Not much more than XML entities defined by DTDs, massaged via Apache server side includes (SSI), but it's an important conceptual building block for both the catalogue and the staff client.

I then ran through the exercise of adding a new metadata export format that brought the Federal Geographic Data Committee's Content Standard for Geospatial Data Metadata (FGDC CSGDM) format to Evergreen's existing list of supported formats. On the one hand: big deal, another metadata format. Hold that thought in that one hand; we'll come back to it later.

I also walked through two other common requests on the mailing lists: how do I define a new index or tweak the behaviour of an existing index and how do I hide or show more information on the detailed record display page? I'll follow up with separate posts for each of these pieces to augment what you have before you in the slides; suffice to say that there's a lot of MODS, a little bit of JavaScript, a smidgin of XPath, a dollop of Evergreen's interface definition language (IDL), and a slice of Perl mixed together. Along the way, I peeled back the covers to show a bit of OpenSRF in operation, setting up Bill's part of the show...

Bill leads us into the promised land

Note I'll update this with a link to Bill's slides when he manages to post them!

Bill gave a quick "big picture" view of how OpenSRF operates, including a much clearer overview of Evergreen's object-relational IDL that maps objects to relational tables. He also covered the cstore OpenSRF application that offers access to the underlying database without requiring SQL but still with support for full transactions (commit/rollback) and sub-transactions (savepoints). During Bill's demonstrations of these features, he exercised srfsh in a way that was new to me - he used the introspect command with a partial method name to perform a left-anchored search for matching method names. Cool!

Oh, and he also showed that if OpenSRF would normally return a reference to an object defined in the IDL, you can ask it to flesh the object in-place with its complete set of attributes instead; and of course if any of those attributes are object references, you have the option of fleshing those as well. It's a lovely way to cut down on chattiness in your application.

From there, Bill whipped out DojoSRF, the OpenSRF-aware extensions for dojo, the JavaScript toolkit that Evergreen adopted as its core JavaScript framework in release 1.4. In 90 lines of HTML and JavaScript code, he implemented a basic but workable catalogue - and then, with a few more lines of code, he gave the audience the payoff for that FGDC CSGDM (geographic metadata) format that I had earlier hacked into Evergreen. As part of the transform separates out the geographic coordinates of the subject matter (in the case of our demo data, maps of Northern California), Bill was able, in just a few more lines of code, to easily extract the coordinates from the FGDC CSGDM representation of the bibliographic material and plot the bounding box for the coverage area on a Google Map image. Very cool.

We had about 15 to 20 people attend our session, and I was happy with that attendance given the extremely technical content and relatively niche product. If as a result we end up adding just one more developer to the Evergreen community, that would be a great outcome. And for myself, I was forced to learn much more of Evergreen - just in time for Project Conifer, I hope

Project Conifer update session at OLA SuperConference 2009

dan@coffeecode.net (Dan Scott) — Fri, 30 Jan 2009 11:28:08 -0500

Updated 2009-02-02 to add PDF formatted slides because the free and libre formats just isn't good enough for some people - heh

The slides, up front and center:

Last year I gave a presentation at the OLA SuperConference 2008 on The State of Evergreen. Yesterday, John Fink and I gave an update on the state of Project Conifer, the partnership between Algoma University, Laurentian University, Northern Ontario School of Medicine, and the University of Windsor to mount a consortial instance of Evergreen for our respective academic libraries.

McMaster University (John Fink's employer) is another Project Conifer institutional partner, albeit with a slightly different relationship. They are contributing resources towards development of academic features, but working towards their own Evergreen instance on their own timeline. Their relationship in the project changed the week before our presentation, so John and I had a fun time adjusting our presentation to match the new reality

In comparison to last year, which was largely an introduction to Evergreen and the state of its various features, this session was much more focused on Project Conifer. John gave the background of the project and the importance of having an open source library system at the core of our academic libraries, particularly given the short-term challenges that most of the Project Conifer participants face with their/our current library systems. I focused on the challenges and lessons learned in managing the project, with most of the challenges being the difficulty of getting skilled resources to work on our development requirements, and most of the lessons learned being in working out cost-sharing agreements and priority-setting procedures early on in the project.

The session was well-attended, and there is clearly growing interest in Evergreen as a viable option, as well a a bit of frustration at the pace of development of some of the features that academics in particular are interested in. These are "interesting times" for academic libraries - this week an announcement has been rippling through the Ontario library community that the BiblioCentre consortial library system that has served many Ontario college libraries since 2003 is being shut down. If Evergreen's academic features were already in place, it would have been a slam-dunk to put together a business case for a centrally hosted Evergreen system to serve the same constituency. As those features are still in active development, it's not quite as easy to make that business case.

Happily, Art Rhyno and Graham Fawcett have taken support for academic reserves for managing both print and electronic materials from ground zero to a reasonable interface in just a few months. They expect to start wiring in direct Evergreen support over the next few months so that we will have a functioning reserves system that goes far beyond our current library system's capabilities ("our" being Laurentian University, in this case).

After an exciting drive from Buffalo on a very snowy Wednesday afternoon, Bill Erickson of Equinox Software Incorporated gave Project Conifer participants a demo of the current state of acquisitions on Wednesday night, and it's not too far from meeting our base requirements. Equinox has hired a second developer to contribute to acquisitions development, documentation is being concurrently produced, and one of Project Conifer's contractors is working on adding EDI support. So we're optimistic that a functioning base acquisitions system will be in place in May - although, as one of our collection development librarians has wryly noted, our budgets might not have any room for book purchases in the coming fiscal year in any case.

A highlight of the session was when I asked Susan Downs, CEO of the Innisfil Public Library, to talk about their success story. In October 2008, Innisfil announced to the library world that they had migrated to Evergreen without any vendor assistance - certainly the first known instance in Ontario, and possibly the first self-migrated and self-supported public library on Evergreen in the world. It was great to meet the people behind that project and I was glad to let Susan share some of her energy, enthusiasm, and insights with our audience.

I had some feedback from one attendee who was happy to see a presentation on an in-process project, with warts and all exposed, rather than the usual post-project stories that quickly put the rough patches behind them (or forget them entirely). I'm happy to do as good a job as I can to represent an objective look at the project - for one thing, it's my job as project manager - and I hope that in some small way I've been able to help others prepare for similar projects.

Adding a new metadata format to Evergreen in a dozen lines of code

dan@coffeecode.net (Dan Scott) — Mon, 26 Jan 2009 00:29:00 -0500

Just like my last entry, this is a preview of one part of my upcoming session at the OLA SuperConference, Evergreen Exposed: Hacking the open source library system. We know from the last entry that Evergreen internally converts MARC21 to MODS to support item display; and in fact it also includes support for exposing records as OAI, RDF, SRW, and HTML. Today, we're going to be looking at adding support for an entirely new metadata format to Evergreen.

Back in November, 2008, George Duimovich requested "I would like to hear from anyone on the process for adding an additional supported format" in the specific context of the FGDC metadata format for digital geospatial data. George did a great thing to support his request and included links to the metadata format itself, along with a pointer to an XSLT stylesheet that the inestimable Terry Reese had written and published for converting MARC21 to FGDC XML. His request has been burning at the back of my mind since then, partially because I had quickly responded with the oh-so-helpful:

Assuming that we can get over the licensing hump, it should be a relatively straightforward matter of dropping the transform into Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm and Open-ILS/src/perlmods/OpenILS/WWW/SuperCat/Feed.pm (using something like MODS32 as a template).

Simple and straightforward, right? Well... yes and no. I had just gone through the process of adding MODS 3.2 support because I needed the more granular treatment of URLs to fix an item display problem, so I was pretty comfortable with the code at the time. After a few months, that familiarity goes away and one gets to go through the discovery process again. (Oh, and about a week after the MODS 3.2 support went in and Mike Rylander went the extra mile to update all of the indexes to use MODS 3.2, MODS 3.3 was released to the world. Sigh).

Without further ado, following are the diffs required to roughly support FGDC as a SuperCat format:

dbs@dbs-laptop:~/source/Evergreen-rel_1_4$ svn diff Open-ILS/src/perlmods/
Index: Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm
===================================================================
--- Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm	(revision 11952)
+++ Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm	(working copy)
@@ -143,6 +143,18 @@
 	# and stash a transformer
 	$record_xslt{rss2}{xslt} = $_xslt->parse_stylesheet( $rss_xslt );
 
+	# parse the FGDC xslt ...
+	my $fgdc_xslt = $_parser->parse_file(
+		OpenSRF::Utils::SettingsClient
+			->new
+			->config_value( dirs => 'xsl' ).
+		"/MARC21slim2FGDC.xsl"
+	);
+	# and stash a transformer
+	$record_xslt{fgdc}{xslt} = $_xslt->parse_stylesheet( $fgdc_xslt );
+	$record_xslt{fgdc}{docs} = 'http://www.fgdc.gov/metadata/csdgm/index_html';
+	$record_xslt{fgdc}{schema_location} = 'http://www.fgdc.gov/metadata/fgdc-std-001-1998.xsd';
+
 	register_record_transforms();
 
 	return 1;

If you're still with me after that whack of code, and you're counting, that's about 12 lines of code. Okay, I'm cheating - the diff doesn't include the MARC21 to FGDC stylesheet - for one thing, I'm still waiting to see a version of the stylesheet with a license attached to it. For another, do you really want to see all that XSL? After you patch your copy of OpenILS::Application::SuperCat.pm, copy the MARC21 to FGDC stylesheet into /openils/var/xsl, and restart the Evergreen Perl services, you'll be able to take advantage of the new functionality. That's it!

What's going on in this code? This patch against Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm enables SuperCat (and therefore unAPI) support for the new format. We just add an entry to the hash of XSLT stylesheets that SuperCat knows about, and the rest is visible in URLs like:

http://localhost/opac/extras/supercat/formats/record - list of supported record formats
http://localhost/opac/extras/supercat/retrieve/fgdc/record/1 - display record #1 in FGDC format
http://localhost/opac/extras/unapi?id=tag:localhost,2009:biblio-record_entry/1 - display the record formats that unAPI can return
http://localhost/opac/extras/unapi?id=tag:localhost,2009:biblio-record_entry/1&format=fgdc - return record #1 in FGDC format via unAPI

So who cares about this? Well, George cares, and (I'm guessing wildly here), perhaps it's because his organization has tools that can import FGDC but that also want to maintain the data in their library catalogue because they love MARC. That might be sufficient reason. Another reasonable use case would be to use the FGDC transform to populate spatial data tables built on the geospatial extensions offered by PostGIS and index these for lightning-fast retrieval of maps and map data that cover a given range of coordinates.

I'm sure the same approach could be used for other specialized metadata formats. This is just one example of why I'm sold on Evergreen's capability as a platform for the future of our library.

Fetching item availability from Evergreen using the OpenSRF HTTP gateway

dan@coffeecode.net (Dan Scott) — Tue, 20 Jan 2009 10:57:37 -0500

This is a preview of one part of my upcoming session at the OLA SuperConference, Evergreen Exposed: Hacking the open source library system. In the Conifer implementation of Evergreen, at least one of the partners plans to use a decoupled discovery layer rather than the Evergreen OPAC. So we needed to answer the typical question "How do I retrieve the availability of copies for a given work at my institution?" Note that this mini-tutorial is based entirely on OpenSRF 1.0 / Evergreen 1.4; OpenSRF 0.9 will generate different JSON output, and the URL for the OpenSRF gateway will be different.

Learning from the old masters: how the Evergreen OPAC does it

The Evergreen OPAC itself relies heavily on JavaScript to dynamically flesh out item details and retrieve item status, so it's actually pretty easy to work out how to do this without even delving too deeply into OpenSRF. First, let's use the Firebug Mozilla extension to follow network requests for a given "title details" page in the OPAC search results for the title: The new world guide to beer. Open up Firebug, enable network monitoring for the OPAC site, and watch the requests flood past for the title details page. We can see that there are a number of POST requests to http://dev.gapines.org/osrf-gateway-v1:

POST request #1 parameters
- method = open-ils.search.biblio.record.mods_slim.retrieve
- service = open-ils.search
- locale = en-US
- param = 8526
This is how we retrieve the title / author / ISBN and other bibliographic details of interest for display; as we're talking about a decoupled discovery layer, we won't need to worry about this piece of the puzzle.

POST request #2 parameters

method = open-ils.search.config.copy_status.retrieve.all
service = open-ils.search
locale = en-US

This is how we retrieve the list of all possible copy statuses that have been configured for this Evergreen system; here's the response (truncated for legibility):

{
   "status" : 200,
   "payload" : [
      [
         {
            "__c" : "ccs",
            "__p" : [
               null, null, null, "f", 3, "Lost", "f"
            ]
         },
         {
            "__c" : "ccs",
            "__p" : [
               null, null, null, "t", 0, "Available", "t"
            ]
         },
         {
            "__c" : "ccs",
            "__p" : [
               null, null, null, "t", 1, "Checked out", "t"
            ]
         },
         {
            "__c" : "ccs",
            "__p" : [
               null, null, null, "f", 2, "Bindery", "t"
            ]
         }
      ]
   ]
}

We're getting a response in JavaScript Object Notation (JSON) format - the nice, compact, easy-to-read data interchange format that almost every programming language under the sun can interpret and generate. Yay!

POST request #3 parameters

method = open-ils.search.biblio.copy_counts.summary.retrieve
service = open-ils.search
locale = en-US
param = 8526
param = 1
param = 0

This is how we retrieve the call numbers, copies, and copy status for a given title. We pass in the the TCN input parameter ("8526"), the numeric ID of the organization being searched ("1" = "every branch"), and the depth of the organization ("0" = top of the hierarchy). The response for this request is:

{
   "status" : 200,
   "payload" : [
      [
         [
            "127",
            "663.42 JACKSON, MICHAEL",
            {
               "0" : 1
            }
         ],
         [
            "130",
            "663.42 JACKSON, MICHAEL",
            {
               "0" : 1
            }
         ],
         [
            "125",
            "663.42 JACKSON, MICHAEL",
            {
               "0" : 1
            }
         ],
         [
            "34",
            "R 641.23 JACKSON, MICHAEL",
            {
               "0" : 1
            }
         ]
      ]
   ]
}

Interpreting the HTTP requests and responses

Okay, so we've found a couple of requests that are pertinent to our goal. And you might be able to guess that the fifth element of the __p entry in the copy status response is the numeric identifier for the copy status, while the sixth element is the copy status name (which, as of OpenSRF 1.0 / Evergreen 1.4, if you pass a different locale value can return a translated value).

You might even be able to guess that the response from the copy_counts.summary request returns an array of responses consisting of the organization ID, the call number, and a hash of copy status and the respective counts for each copy status. And you would be guessing correctly. But why guess, when you can get an authoritative interpretation by looking up the class hint (the __c value in the copy_status response of "ccs") in Evergreen's intermediate definition language file /openils/conf/fm_IDL.xml:

<class id="ccs" controller="open-ils.cstore" 
  oils_obj:fieldmapper="config::copy_status" oils_persist:tablename="config.copy_status">
  <fields oils_persist:primary="id" oils_persist:sequence="config.copy_status_id_seq">
    <field name="isnew" oils_obj:array_position="0" oils_persist:virtual="true" />
    <field name="ischanged" oils_obj:array_position="1" oils_persist:virtual="true" />
    <field name="isdeleted" oils_obj:array_position="2" oils_persist:virtual="true" />
    <field name="holdable" oils_obj:array_position="3" 
      oils_persist:virtual="false" reporter:datatype="bool"/>
    <field name="id" oils_obj:array_position="4" 
      oils_persist:virtual="false" reporter:selector="name" reporter:datatype="id"/>
    <field name="name" oils_obj:array_position="5" 
      oils_persist:virtual="false"  reporter:datatype="text" oils_persist:i18n="true"/>
    <field name="opac_visible" oils_obj:array_position="6" 
      oils_persist:virtual="false" reporter:datatype="bool"/>
  </fields>

So now, by taking our first steps into Evergreen's object persistence model, we can determine authoritatively that the order of values in the __p array maps to "isnew", "ischanged", "isdeleted", "holdable", "id", "name", and "opac_visible". As for the response from the copy_counts.summary call, well, these are not Evergreen objects (they don't have a __c class hint) - but you can use the OpenSRF shell "srfsh" introspect command to view the documentation for the applicable method:

bash$ srfsh
srfsh# introspect open-ils.search
... (truncated for legibility) ...
Received Data: {
  "__c":"OpenILS_Application",
  "__p":{
    "api_level":1,
    "stream":0,
    "object_hint":"OpenILS_Application_Search_Biblio",
    "package":"OpenILS::Application::Search::Biblio",
    "remote":0,
    "api_name":"open-ils.search.biblio.copy_counts.summary.retrieve",
    "signature":{
      "params":[
        
      ],
      "desc":"returns an array of these: [
         org_id,
         callnumber_label,
         ,
         ,
        ...
      ]       where statusx is a copy status name.  the statuses are sorted by id.",
      "return":{
        "desc":null,
        "type":null,
        "class":null
      }      
    },
    "server_class":"open-ils.search",
    "notes":"\treturns an array of these:\n\t\t[
       org_id,
       callnumber_label,
       ,
       ,
      ...
    ]    \n\t\twhere statusx is a copy status name.  the statuses are sorted\n\t\tby id.\n",
    "method":"copy_count_summary",
    "argc":0
  }

The introspect output is a bit rough - it's really intended for the doxygen API help interface - but it's good enough for our purposes. If we want to dig into what's going on under the covers, we can follow the package_name value "OpenILS::Application::Search::Biblio" to read the source code for the OpenILS::Application::Search::Biblio Perl module, and look up the method "copy_count_summary" as indicated by the "method" value in the introspect output. That reveals that the input arguments are "($self, $client, $rid, $org, $depth)". Every OpenSRF method automatically receives $self and $client as the first two arguments, so $rid (record ID), $org (organization unit ID), and $depth (organization unit depth) are the variables over which we have control.

Zeroing in on the copies for a particular library or library system

If we want to retrieve the visible copies for just a single organization unit in the entire Evergreen system, we just have to adjust the values of the organization unit ID and organization unit depth parameters accordingly. If we ask for the visible copies for just org_unit ID "125" at depth "2", we narrow down our results to a single hit:

{
   "status" : 200,
   "payload" : [
      [
         [
            "125",
            "663.42 JACKSON, MICHAEL",
            {
               "0" : 1
            }
         ]
      ]
   ]
}

So, with all of that ammunition at your disposal, you can write an Evergreen copy status lookup in any decoupled discovery layer that supports HTTP POST or GET requests. Which should be pretty much any discovery layer, right?

Frequently used tools and methods for Evergreen / OpenSRF hacking

Note, the first: you can easily play with different parameter values for the HTTP POST requests using the json_xs command to pretty print the JSON response:

curl -d service=open-ils.search   -d locale=en-US \
  -d method=open-ils.search.biblio.copy_counts.summary.retrieve \
  -d param=8526 -d param=1 -d param=0 \
  http://dev.gapines.org/osrf-gateway-v1 | json_xs -t json-pretty

Note, the second: the OpenSRF gateway also supports GET requests; simply concatenate the request parameters in a single URL like this