Coffee|Code : Dan Scott

Updated 2009-02-02 to add PDF formatted slides because the free and libre formats just isn't good enough for some people - heh

The slides, up front and center:

• OpenOffice Impress format (ODP)
• Portable Document Format (PDF)

Last year I gave a presentation at the OLA SuperConference 2008 on The State of Evergreen. Yesterday, John Fink and I gave an update on the state of Project Conifer, the partnership between Algoma University, Laurentian University, Northern Ontario School of Medicine, and the University of Windsor to mount a consortial instance of Evergreen for our respective academic libraries.

McMaster University (John Fink's employer) is another Project Conifer institutional partner, albeit with a slightly different relationship. They are contributing resources towards development of academic features, but working towards their own Evergreen instance on their own timeline. Their relationship in the project changed the week before our presentation, so John and I had a fun time adjusting our presentation to match the new reality

In comparison to last year, which was largely an introduction to Evergreen and the state of its various features, this session was much more focused on Project Conifer. John gave the background of the project and the importance of having an open source library system at the core of our academic libraries, particularly given the short-term challenges that most of the Project Conifer participants face with their/our current library systems. I focused on the challenges and lessons learned in managing the project, with most of the challenges being the difficulty of getting skilled resources to work on our development requirements, and most of the lessons learned being in working out cost-sharing agreements and priority-setting procedures early on in the project.

The session was well-attended, and there is clearly growing interest in Evergreen as a viable option, as well a a bit of frustration at the pace of development of some of the features that academics in particular are interested in. These are "interesting times" for academic libraries - this week an announcement has been rippling through the Ontario library community that the BiblioCentre consortial library system that has served many Ontario college libraries since 2003 is being shut down. If Evergreen's academic features were already in place, it would have been a slam-dunk to put together a business case for a centrally hosted Evergreen system to serve the same constituency. As those features are still in active development, it's not quite as easy to make that business case.

Happily, Art Rhyno and Graham Fawcett have taken support for academic reserves for managing both print and electronic materials from ground zero to a reasonable interface in just a few months. They expect to start wiring in direct Evergreen support over the next few months so that we will have a functioning reserves system that goes far beyond our current library system's capabilities ("our" being Laurentian University, in this case).

After an exciting drive from Buffalo on a very snowy Wednesday afternoon, Bill Erickson of Equinox Software Incorporated gave Project Conifer participants a demo of the current state of acquisitions on Wednesday night, and it's not too far from meeting our base requirements. Equinox has hired a second developer to contribute to acquisitions development, documentation is being concurrently produced, and one of Project Conifer's contractors is working on adding EDI support. So we're optimistic that a functioning base acquisitions system will be in place in May - although, as one of our collection development librarians has wryly noted, our budgets might not have any room for book purchases in the coming fiscal year in any case.

A highlight of the session was when I asked Susan Downs, CEO of the Innisfil Public Library, to talk about their success story. In October 2008, Innisfil announced to the library world that they had migrated to Evergreen without any vendor assistance - certainly the first known instance in Ontario, and possibly the first self-migrated and self-supported public library on Evergreen in the world. It was great to meet the people behind that project and I was glad to let Susan share some of her energy, enthusiasm, and insights with our audience.

I had some feedback from one attendee who was happy to see a presentation on an in-process project, with warts and all exposed, rather than the usual post-project stories that quickly put the rough patches behind them (or forget them entirely). I'm happy to do as good a job as I can to represent an objective look at the project - for one thing, it's my job as project manager - and I hope that in some small way I've been able to help others prepare for similar projects.

Just like my last entry, this is a preview of one part of my upcoming session at the OLA SuperConference, Evergreen Exposed: Hacking the open source library system. We know from the last entry that Evergreen internally converts MARC21 to MODS to support item display; and in fact it also includes support for exposing records as OAI, RDF, SRW, and HTML. Today, we're going to be looking at adding support for an entirely new metadata format to Evergreen.

Back in November, 2008, George Duimovich requested "I would like to hear from anyone on the process for adding an additional supported format" in the specific context of the FGDC metadata format for digital geospatial data. George did a great thing to support his request and included links to the metadata format itself, along with a pointer to an XSLT stylesheet that the inestimable Terry Reese had written and published for converting MARC21 to FGDC XML. His request has been burning at the back of my mind since then, partially because I had quickly responded with the oh-so-helpful:

Assuming that we can get over the licensing hump, it should be a relatively straightforward matter of dropping the transform into Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm and Open-ILS/src/perlmods/OpenILS/WWW/SuperCat/Feed.pm (using something like MODS32 as a template).

Simple and straightforward, right? Well... yes and no. I had just gone through the process of adding MODS 3.2 support because I needed the more granular treatment of URLs to fix an item display problem, so I was pretty comfortable with the code at the time. After a few months, that familiarity goes away and one gets to go through the discovery process again. (Oh, and about a week after the MODS 3.2 support went in and Mike Rylander went the extra mile to update all of the indexes to use MODS 3.2, MODS 3.3 was released to the world. Sigh).

Without further ado, following are the diffs required to roughly support FGDC as a SuperCat format:

dbs@dbs-laptop:~/source/Evergreen-rel_1_4$ svn diff Open-ILS/src/perlmods/
Index: Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm
===================================================================
--- Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm	(revision 11952)
+++ Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm	(working copy)
@@ -143,6 +143,18 @@
 	# and stash a transformer
 	$record_xslt{rss2}{xslt} = $_xslt->parse_stylesheet( $rss_xslt );
 
+	# parse the FGDC xslt ...
+	my $fgdc_xslt = $_parser->parse_file(
+		OpenSRF::Utils::SettingsClient
+			->new
+			->config_value( dirs => 'xsl' ).
+		"/MARC21slim2FGDC.xsl"
+	);
+	# and stash a transformer
+	$record_xslt{fgdc}{xslt} = $_xslt->parse_stylesheet( $fgdc_xslt );
+	$record_xslt{fgdc}{docs} = 'http://www.fgdc.gov/metadata/csdgm/index_html';
+	$record_xslt{fgdc}{schema_location} = 'http://www.fgdc.gov/metadata/fgdc-std-001-1998.xsd';
+
 	register_record_transforms();
 
 	return 1;

If you're still with me after that whack of code, and you're counting, that's about 12 lines of code. Okay, I'm cheating - the diff doesn't include the MARC21 to FGDC stylesheet - for one thing, I'm still waiting to see a version of the stylesheet with a license attached to it. For another, do you _really_ want to see all that XSL? After you patch your copy of OpenILS::Application::SuperCat.pm, copy the MARC21 to FGDC stylesheet into /openils/var/xsl, and restart the Evergreen Perl services, you'll be able to take advantage of the new functionality. That's it!

What's going on in this code? This patch against Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm enables SuperCat (and therefore unAPI) support for the new format. We just add an entry to the hash of XSLT stylesheets that SuperCat knows about, and the rest is visible in URLs like:

• http://localhost/opac/extras/supercat/formats/record - list of supported record formats
• http://localhost/opac/extras/supercat/retrieve/fgdc/record/1 - display record #1 in FGDC format
• http://localhost/opac/extras/unapi?id=tag:localhost,2009:biblio-record_entry/1 - display the record formats that unAPI can return
• http://localhost/opac/extras/unapi?id=tag:localhost,2009:biblio-record_entry/1&format=fgdc - return record #1 in FGDC format via unAPI

So who cares about this? Well, George cares, and (I'm guessing wildly here), perhaps it's because his organization has tools that can import FGDC but that also want to maintain the data in their library catalogue because they love MARC. That might be sufficient reason. Another reasonable use case would be to use the FGDC transform to populate spatial data tables built on the geospatial extensions offered by PostGIS and index these for lightning-fast retrieval of maps and map data that cover a given range of coordinates.

I'm sure the same approach could be used for other specialized metadata formats. This is just one example of why I'm sold on Evergreen's capability as a platform for the future of our library.

This is a preview of one part of my upcoming session at the OLA SuperConference, Evergreen Exposed: Hacking the open source library system. In the Conifer implementation of Evergreen, at least one of the partners plans to use a decoupled discovery layer rather than the Evergreen OPAC. So we needed to answer the typical question "How do I retrieve the availability of copies for a given work at my institution?" Note that this mini-tutorial is based entirely on OpenSRF 1.0 / Evergreen 1.4; OpenSRF 0.9 will generate different JSON output, and the URL for the OpenSRF gateway will be different.

Learning from the old masters: how the Evergreen OPAC does it

The Evergreen OPAC itself relies heavily on JavaScript to dynamically flesh out item details and retrieve item status, so it's actually pretty easy to work out how to do this without even delving too deeply into OpenSRF. First, let's use the Firebug Mozilla extension to follow network requests for a given "title details" page in the OPAC search results for the title: The new world guide to beer. Open up Firebug, enable network monitoring for the OPAC site, and watch the requests flood past for the title details page. We can see that there are a number of POST requests to http://dev.gapines.org/osrf-gateway-v1:

• POST request #1 parameters
  • method = open-ils.search.biblio.record.mods_slim.retrieve
  • service = open-ils.search
  • locale = en-US
  • param = 8526
  This is how we retrieve the title / author / ISBN and other bibliographic details of interest for display; as we're talking about a decoupled discovery layer, we won't need to worry about this piece of the puzzle.
• POST request #2 parameters
  • method = open-ils.search.config.copy_status.retrieve.all
  • service = open-ils.search
  • locale = en-US
  This is how we retrieve the list of all possible copy statuses that have been configured for this Evergreen system; here's the response (truncated for legibility):

  {
     "status" : 200,
     "payload" : [
        [
           {
              "__c" : "ccs",
              "__p" : [
                 null, null, null, "f", 3, "Lost", "f"
              ]
           },
           {
              "__c" : "ccs",
              "__p" : [
                 null, null, null, "t", 0, "Available", "t"
              ]
           },
           {
              "__c" : "ccs",
              "__p" : [
                 null, null, null, "t", 1, "Checked out", "t"
              ]
           },
           {
              "__c" : "ccs",
              "__p" : [
                 null, null, null, "f", 2, "Bindery", "t"
              ]
           }
        ]
     ]
  }
  

  We're getting a response in JavaScript Object Notation (JSON) format - the nice, compact, easy-to-read data interchange format that almost every programming language under the sun can interpret and generate. Yay!
• POST request #3 parameters
  • method = open-ils.search.biblio.copy_counts.summary.retrieve
  • service = open-ils.search
  • locale = en-US
  • param = 8526
  • param = 1
  • param = 0
  This is how we retrieve the call numbers, copies, and copy status for a given title. We pass in the the TCN input parameter ("8526"), the numeric ID of the organization being searched ("1" = "every branch"), and the depth of the organization ("0" = top of the hierarchy). The response for this request is:

  {
     "status" : 200,
     "payload" : [
        [
           [
              "127",
              "663.42 JACKSON, MICHAEL",
              {
                 "0" : 1
              }
           ],
           [
              "130",
              "663.42 JACKSON, MICHAEL",
              {
                 "0" : 1
              }
           ],
           [
              "125",
              "663.42 JACKSON, MICHAEL",
              {
                 "0" : 1
              }
           ],
           [
              "34",
              "R 641.23 JACKSON, MICHAEL",
              {
                 "0" : 1
              }
           ]
        ]
     ]
  }
  

Interpreting the HTTP requests and responses

Okay, so we've found a couple of requests that are pertinent to our goal. And you might be able to guess that the fifth element of the __p entry in the copy status response is the numeric identifier for the copy status, while the sixth element is the copy status name (which, as of OpenSRF 1.0 / Evergreen 1.4, if you pass a different locale value can return a translated value).

You might even be able to guess that the response from the copy_counts.summary request returns an array of responses consisting of the organization ID, the call number, and a hash of copy status and the respective counts for each copy status. And you would be guessing correctly. But why guess, when you can get an authoritative interpretation by looking up the class hint (the __c value in the copy_status response of "ccs") in Evergreen's intermediate definition language file /openils/conf/fm_IDL.xml:

<class id="ccs" controller="open-ils.cstore" 
  oils_obj:fieldmapper="config::copy_status" oils_persist:tablename="config.copy_status">
  <fields oils_persist:primary="id" oils_persist:sequence="config.copy_status_id_seq">
    <field name="isnew" oils_obj:array_position="0" oils_persist:virtual="true" />
    <field name="ischanged" oils_obj:array_position="1" oils_persist:virtual="true" />
    <field name="isdeleted" oils_obj:array_position="2" oils_persist:virtual="true" />
    <field name="holdable" oils_obj:array_position="3" 
      oils_persist:virtual="false" reporter:datatype="bool"/>
    <field name="id" oils_obj:array_position="4" 
      oils_persist:virtual="false" reporter:selector="name" reporter:datatype="id"/>
    <field name="name" oils_obj:array_position="5" 
      oils_persist:virtual="false"  reporter:datatype="text" oils_persist:i18n="true"/>
    <field name="opac_visible" oils_obj:array_position="6" 
      oils_persist:virtual="false" reporter:datatype="bool"/>
  </fields>

So now, by taking our first steps into Evergreen's object persistence model, we can determine authoritatively that the order of values in the __p array maps to "isnew", "ischanged", "isdeleted", "holdable", "id", "name", and "opac_visible". As for the response from the copy_counts.summary call, well, these are not Evergreen objects (they don't have a __c class hint) - but you can use the OpenSRF shell "srfsh" introspect command to view the documentation for the applicable method:

bash$ srfsh
srfsh# introspect open-ils.search
... (truncated for legibility) ...
Received Data: {
  "__c":"OpenILS_Application",
  "__p":{
    "api_level":1,
    "stream":0,
    "object_hint":"OpenILS_Application_Search_Biblio",
    "package":"OpenILS::Application::Search::Biblio",
    "remote":0,
    "api_name":"open-ils.search.biblio.copy_counts.summary.retrieve",
    "signature":{
      "params":[
        
      ],
      "desc":"returns an array of these: [
         org_id,
         callnumber_label,
         ,
         ,
        ...
      ]       where statusx is a copy status name.  the statuses are sorted by id.",
      "return":{
        "desc":null,
        "type":null,
        "class":null
      }      
    },
    "server_class":"open-ils.search",
    "notes":"\treturns an array of these:\n\t\t[
       org_id,
       callnumber_label,
       ,
       ,
      ...
    ]    \n\t\twhere statusx is a copy status name.  the statuses are sorted\n\t\tby id.\n",
    "method":"copy_count_summary",
    "argc":0
  }  

The introspect output is a bit rough - it's really intended for the doxygen API help interface - but it's good enough for our purposes. If we want to dig into what's going on under the covers, we can follow the package_name value "OpenILS::Application::Search::Biblio" to read the source code for the OpenILS::Application::Search::Biblio Perl module, and look up the method "copy_count_summary" as indicated by the "method" value in the introspect output. That reveals that the input arguments are "($self, $client, $rid, $org, $depth)". Every OpenSRF method automatically receives $self and $client as the first two arguments, so $rid (record ID), $org (organization unit ID), and $depth (organization unit depth) are the variables over which we have control.

Zeroing in on the copies for a particular library or library system

If we want to retrieve the visible copies for just a single organization unit in the entire Evergreen system, we just have to adjust the values of the organization unit ID and organization unit depth parameters accordingly. If we ask for the visible copies for just org_unit ID "125" at depth "2", we narrow down our results to a single hit:

{
   "status" : 200,
   "payload" : [
      [
         [
            "125",
            "663.42 JACKSON, MICHAEL",
            {
               "0" : 1
            }
         ]
      ]
   ]
}

So, with all of that ammunition at your disposal, you can write an Evergreen copy status lookup in any decoupled discovery layer that supports HTTP POST or GET requests. Which should be pretty much any discovery layer, right?

Frequently used tools and methods for Evergreen / OpenSRF hacking

Note, the first: you can easily play with different parameter values for the HTTP POST requests using the json_xs command to pretty print the JSON response:

curl -d service=open-ils.search   -d locale=en-US \
  -d method=open-ils.search.biblio.copy_counts.summary.retrieve \
  -d param=8526 -d param=1 -d param=0 \
  http://dev.gapines.org/osrf-gateway-v1 | json_xs -t json-pretty

Note, the second: the OpenSRF gateway also supports GET requests; simply concatenate the request parameters in a single URL like this

If you, like me, install Skype in a Windows XP VirtualBox guest instance running on an Ubuntu host on a ThinkPad T60 with an Intel 2300 dual-core 32-bit processor, it might throw Windows exceptions and generate error reports as reported in VirtualBox ticket #1710.

If you then go into your ThinkPad BIOS and enable the virtualization extensions (under the Config->CPU menu) and reboot your Ubuntu host, you might get the same problem again. I'll tell you right now that the problem is that you need to power off your system after changing the BIOS option, rather than deciding to just Save and Exit with an immediate reboot.

You might then try loading the KVM module with the Intel extensions to test whether your system actually does have the Intel virtualization extensions enabled - and you might get a FATAL: Error inserting kvm_intel (/lib/modules/2.6.27-9-generic/kernel/drivers/kvm/kvm-intel.ko): Operation not supported error message. Perhaps you'll end up consulting a page like the Ubuntu KVM wiki entry and think "Man, I'm doing everything right... what's going on?"; you then re-read the VirtualBox ticket #1710 and decide to try updating the BIOS on your laptop just in case it has some problem enabling the virtualization extensions.

Huzzah! After updating the BIOS, you can then load the kvm and kvm-intel modules. (Aside: this has nothing to do with the updated BIOS, and everything to do with the enforced power-off cycle that comes with the update process.) You triumphantly fire up your Windows XP VirtualBox guest instance, ready to get Skype going, and... the image won't even boot, dying with the horrible error: VirtualBox can't operate in VMX root mode. VBox status code: -4011 (VERR_VMX_IN_VMX_ROOT_MODE). What the...? You do a bit more searching of the forums, and all of the solutions (like Ubuntu bug 292588) tell you to disable KVM. By this time, you're thoroughly confused and have convinced yourself that KVM is necessary for the virtualization extension that will satisfy Skype.

Eventually, you rmmod kvm-intel and try VirtualBox again - and much to your surprise, the image loads and Skype is stable. Huzzah! (Aside: this is because VirtualBox uses its own kernel module, vboxdrv.ko, for virtualization support, and the kvm / kvm-intel kernel modules won't co-exist with that... so your little test to see if the virtualization extensions were working ended up preventing your virtualization application from working.)

I'm sure you're wondering why I was going to all of this effort to get Skype running on a virtual machine in the first place, when there's a native package for Ubuntu available that works. The punch line to all of this is that I convinced my mom to start using Skype as a backup for when her phone line goes dead (which it does, all too often), but for some really weird reason I haven't been able to find her as a contact in the Linux Skype client, even though I know her exact user ID. On a hunch, I wanted to try searching for her in a Windows Skype client. And you know what?

I was able to get exactly one hit, the right result, under the Windows Skype client. That is both baffling and frustrating. Skype is both a proprietary application and service and there's nothing I can do to fix the search problem. It seems that Skype's APIs are messed up enough that they can't even support a simple search reliably on a cross-platform basis. Hopefully that doesn't carry over to, say, being unable to maintain an accurate count of your credit balance.

Ah well. I don't think that the open source alternatives like Ekiga or QuteCom are solutions yet for my Mom's needs; we need the service to match the application. Maybe in a parallel universe Skype's services would support any SIP client, and their contact search API would be open. But in this universe, I have both learned how to properly enable virtualization extension support in VirtualBox on Ubuntu and I have finally added my Mom as a contact in a proprietary application / service. That's good enough for today.

Update a few minutes later: Hmm, looks like Tpad might be close to the service that I'm looking for... Perhaps I'll follow up later with a report.