Okay, this is definitely a lame thing to be thinking about at midnight on a Saturday, but I was just playing with the shelf browser in the Evergreen representation of our 780,000 bibliographic records (okay, that is definitely the wrong thing to be doing at midnight on a Saturday). For some reason, I was wandering through the subject collection pertinent to librarians (pray for my soul), noticed a book that probably should have been discarded years ago, and thought "Gee, i don't want to deal with this right now, but wouldn't it be nice if I could just mark this Weed me and forget about it until Monday?"

Then I realized that that wouldn't be a stretch at all. In Evergreen, users have "bookbags" to which they can add items. These bookbags can be shared as RSS feeds and otherwise easily exported into other formats. If we were running Evergreen for real, I could create a "Weed me!" bookbag, add in the suspect along with a bunch of other festering tomes, and send the RSS feed to a student to perform the manual labour. Or perhaps the RSS feed gets aggregated with other weeders' feeds and a weeding list gets generated on a monthly basis for efficient labour practices. You get the idea.

Of course, you would really want to have more information than just the stock shelf browsing interface at hand when making weeding decisions. For example, you would need a tally of recorded uses displayed beside the item, with the ability to drill down for totals by year. If you participate in a consortial "last copy standing" program, you would want a quick check to see if any other institutions still hold a copy of the resource. So, an enhanced interface would be needed to provide an experience that combines the traditional weeding approach of roaming the stacks and generating reports of items matching some minimum age and minimum usage criteria.

Think about it a little further though (I'm sure you're thinking a lot faster than me at this point; you're probably having the luxury of reading this at the beginning of the day, coffee in hand, invigorated after an early morning run in the lingering late spring chill... or not), and there are points in our institutional workflows where we could naturally introduce weeding activities. How do we get to the point of having three editions of a given text on the shelf? If I have the 1995, 2003, and 2007 editions of a text, I can assure you that when I ordered the 2007 edition I had already checked our ILS to see if we had a copy of that edition already, and would have noticed the previous editions. At that point, I should have the ability to say "Oh - get rid of the 1995 edition now and once the 2007 edition is processed and on the shelf, cull the 2003 edition to boot." If I was designing an acquisitions module today, that's certainly something I would consider as a nice-to-have. Ahem.

Weeding 2.0 may not be a sexy subject. Google and Yahoo each turn up exactly four hits, none of them related to libraries, which is remarkable in this overly-hyped everything 2.0 world. But it's something we should consider in the design and tailoring of our library systems; and while it's not going to rank in my top level of priorities for Evergreen, it will work its way in there somewhere, sometime. Hopefully before the stacks in my subject areas buckle under the weight of unused, out-of-date books.

Ramping up

This year, we hosted a small party focusing on the little ones in Amber's life: a few of her friends from day care, and a friend from up the street.

The amazing cat cake

Lynn used the same carrot cake recipe as last year (nice and tasty!), but this year it came in the appearance of Amber's favourite animal.

She blew out the candle herself. Self! SELF!

Blowing out the candle was a huge success.

Cake distribution went smoothly

Very little cake was wasted in the making of this birthday. Most of the cake was consumed rather than applied to faces or clothes.

Daddy and Amber wound down with a book in the window on a rainy day

Thanks to everyone for their cards and calls and emails celebrating Amber's birthday!

Update 2008-05-01: Fixed a typo for sysctl: -a parameter simply shows all settings; -w parameter is needed to write the setting. Duh.

Once you have decided on and acquired your test hardware for Evergreen, you need to think about tuning your PostgreSQL database server. Once you start loading bibliographic records, you might notice that after 100,000 records or so that your search response times aren't too snappy. Don't snarl at Evergreen. By default, PostgreSQL ships with very conservative settings (something like machines with 256 MB of RAM!) so if you don't tune those settings you're getting a false representation of your system's capabilities.

The "right" settings for PostgreSQL depend significantly on your hardware and deployment context, but in almost any circumstance you will want to bump up the settings from the delivered defaults. To give you an idea of what you need to consider, I thought I would share the settings that we're currently using on our Evergreen test server at Laurentian University. You might be able to use these as a starting point and adjust them accordingly once you've run some representative load tests against your configuration. And it's useful documentation for me to fall back on in a few months, when all of this has escaped my grasp

The defaults (as shipped in Debian Etch)

The defaults in Debian Etch are quite conservative. Consider that our test server has 12GB of RAM. The default only allocates 1MB of RAM to work memory (which is critical for sorting performance) and only 8MB of RAM to shared buffers. Following are the defaults set in /etc/postgresql/8.1/main/postgresql.conf:

# - Memory -

#shared_buffers = 1000                  # min 16 or max_connections*2, 8KB each
#temp_buffers = 1000                    # min 100, 8KB each
#max_prepared_transactions = 5          # can be 0 or more
# note: increasing max_prepared_transactions costs ~600 bytes of shared memory
# per transaction slot, plus lock space (see max_locks_per_transaction).
#work_mem = 1024                        # min 64, size in KB
#maintenance_work_mem = 16384           # min 1024, size in KB
#max_stack_depth = 2048                 # min 100, size in KB

# - Free Space Map -

#max_fsm_pages = 20000                  # min max_fsm_relations*16, 6 bytes each
#max_fsm_relations = 1000               # min 100, ~70 bytes each

Our test server settings

Our test server has 12 GB of RAM. Assuming that the PostgreSQL defaults were set for a system with 1 GB of RAM, we should be able to multiply the memory-based settings by at least a factor of 12. We're a little bit more aggressive than that in our settings. Note, however, that this is a single-server install of Evergreen, so we're also running memcached, ejabberd, Apache, and all of the Evergreen services as well as the database - oh, and a test instance of an institutional repository, among other apps - so we're not nearly as aggressive as we would be in a dedicated PostgreSQL server configuration. Please note that I'm making no claims that this is the optimal set of configuration values for PostgreSQL even on our own hardware!

# shared_buffers: much of our performance depends on sorting, so we'll set it 100X the default
# some tuning guides suggest cranking this up to as much 30% of your available RAM
shared_buffers = 100000 # 8K * 100000 = ~ 0.8 GB

# work_mem: how much RAM each concurrent process is allowed to claim before swapping to disk
# your workload will probably have a large number of concurrent processes
work_mem=524288 # 512 MB

# max_fsm_pages: increased because PostgreSQL demanded it
max_fsm_pages = 200000

After you change these settings, you will need to restart PostgreSQL to make the settings take effect.

Kernel tuning

In addition to PostgreSQL complaining about max_fsm_pages not being high enough, your operating system kernel defaults for SysV shared memory might not be high enough to support the amount of RAM PostgreSQL demands as a result of your modifications. In one of our test configurations, we had cranked up work_mem to 8GB; Debian complained about an insufficient SHMMAX setting, so we were able to adjust that by running the following command as root to set the kernel SHMMAX to 8GB (8*1024^2):

sysctl -w kernel.shmmax=8589934592

To make this setting sticky through reboots, you can simply modify /etc/sysctl.conf to include the following line:

# Set SHMMAX to 8GB for PostgreSQL
#kernel.shmmax=8589934592

Other measures

Debian Etch comes with PostgreSQL 8.1. The first version of PostgreSQL 8.1 was released in November 2005. That's a long time in computer years. Version 8.2, which was released less than a year later, "adds many functionality and performance improvements" (according to the release notes). If you're not getting the performance you expect from your hardware with Debian Etch, perhaps a backport of PostgreSQL 8.2 would help out.

Further resources

This is just a shallow dip into PostgreSQL tuning for Evergreen - hopefully enough to alert you to some of the factors you need to consider if you're putting Evergreen into a serious testing environment or production environment. Here are a few places to dig deeper into the art of PostgreSQL tuning:

• PostgreSQL manual, resource consumption section of server configuration: version 8.1 and version 8.2
• An annotated version of the 8.0 parameters with more explicit advice is available at
• Some good advice is buried about halfway down Christopher Browne's page under the heading "Tuning PostgreSQL", along with links to further resources
• The "Performance Whack-A-Mole" presentation at PowerPostgreSQL is a great tutorial for holistic system tuning

Occasionally on the #OpenILS-Evergreen IRC channel, a question comes up what kind of hardware a site should buy if they're getting serious about trying out Evergreen. I had exactly the same chat with Mike Rylander back in December, so I thought it might be useful to share the strategy we developed in case other organizations are interested in piggy-backing on our research. We came up with three different scenarios, depending on the funding available to the organization and how serious the organization is about testing, developing, and deploying Evergreen.

You can also look at the scenarios as stages, as the scenarios enable progressively more realistic testing. An organization can always start with a single server and add more servers over time; if you can swing a significant discount for buying in bulk, however, it might make sense to bite the bullet early.

Some pertinent facts about our requirements: we will eventually be loading around 5 million bibliographic records onto the system. We're an academic organization, so concurrent searching and circulation loads will be low relative to public libraries.

Scenario 1: A single bargain-basement testing server

In this scenario, the organization purchases a single server for the short term, and configures it to run the entire Evergreen + OpenSRF stack:

• database
• Web server
• Jabber messaging
• memcached
• OpenSRF applications

This server needs to have powerful CPUs, large amounts of RAM, and many fast (10K RPM or higher) hard drives in a striped RAID configuration (the latter because database performance typically gets knee-capped by disk access). A "higher education" quote online from a reputable big-name vendor for a rack-mounted 2U database server with 2x4-core CPU, 16GB RAM, 6x73GB RAID 5 drives comes in at approximately $7000.

This scenario is fine for development and testing with a limited number of users, but if you intend to do any sort of stress testing with this server or throw it open to the public, performance will likely grind to a halt. Note: This is close to the system that we're currently running at http://biblio-dev.laurentian.ca - 12 GB of RAM, 2 dual-core CPUs - with 800K bibliographic records and pretty snappy search performance. It's certainly nothing to sneeze at.

Scenario 2: one database server, one network server

In this scenario, you purchase a database server and a network server. We'll use the same specs from scenario 1 for the database server, and a CPU + RAM-oriented server for the network server (disk access isn't a factor for the network apps, so you just buy two small mirrored drives). The stock higher education quote for a rack-mounted 1U network server with 2x4-core CPU, 16GB RAM, 2x73GB RAID 1 drives is approximately $5250.

This scenario will support development and testing, as well as enable you perform relatively representative stress testing runs with a significant number of simultaneous users.

Scenario 3: two database servers, two or three network servers

In this scenario, you purchase two database servers so that you can test database replication, split database loads between search and reporting, and two or three network servers to test different distributions of the caching and network apps across the servers to determine the configuration that best meets your expected demands. The cost of the five servers adds up to less than $30,000 - less than a single traditional proprietary UNIX server - and would be less if you can negotiate a bulk discount.

The third scenario supports development and testing, and will give you practical experience with a configuration that would approximate your production deployment of servers. When you go live, you could move one of the database servers and all but one of the network servers over to the production cluster, and revert back to scenario one for your ongoing test and development environment.

The Conifer approach

We opted to go with the third scenario to build a serious test cluster for our consortium. However, the "scenarios as stages" approach ended up being our strategy as our original choice of Dell servers came with RAID controllers that do not work well under Debian. After returning the servers to Dell, we were forced to press one of our backup servers into service as a scenario-one style server while waiting for our new order from HP to arrive.

Hmm. I think I'm in trouble if the support site itself is incapable of displaying accented characters properly.

Corrupted characters in a problem report about corrupted characters. Oh dear.

My analysis of the problem is that the content in the middle is contained within a frame, and is actually encoded in ISO-8859-1 - but doesn't have an encoding declaration. And the containing HTML page, of course, declares that it is UTF-8. So poor Mozilla gets very confused. And our poor users continue to get corrupted characters in their reminder and overdues notices.

Note: Some information removed from the screencap to protect the innocent - the client care people are actually excellent folk, and I'm sure they're just as frustrated by the problem reporting system as we are.

Project Conifer is the effort by McMaster University, University of Windsor, and Laurentian University to put together a consortial instance of Evergreen. A few weeks back, we agreed that May 2009 would be our go-live date. So the clock is ticking quite loudly in my ears.

Today I got an Evergreen test server up and running, loaded with the records from the consortium of Laurentian partners. I hit a few bumps on the road, but eventually successfully loaded about 800,000 bibliographic records and about 500,000 items. I also turned on the Syndetics enrichment data, so some items offer cover images, tables of contents, reviews, and author information. The response time is pretty snappy (it's running on a 4-core server with 12GB of RAM).

Things that made my task harder than it probably should have been:

• yaz-marcdump generated invalid XML when I converted our MARC records from MARC21 to MARC21XML format. Maybe this problem is fixed in later versions of yaz-marcdump (I was using the stable Debian Etch version, 2.1.56, which is crazy old), or I could have tried marc4j or MarcEdit instead to try for better results, but I didn't, and it cascaded into problems with...
• Dumping all of the holdings as part of the bibliographic records threw things off when some of the records had so many holdings attached (think a weekly periodical that a library circulates and therefore each issue has its own barcode) that they spilled over MARC's record length limit, resulting in multiple MARC records just to hold the holdings - which causes some problems for the basic import process. I eventually punted on trying to parse the MARC21XML for holdings and just dumped the data I needed directly from Unicorn in pipe-delimited format.
• Not tuning PostgreSQL before starting to load data into the database was just plain stupid. The defaults for PostgreSQL are incredibly conservative, and must be modified to handle large transactions and to perform. Here are the tweaks I made for our 12GB machine, starting with the Linux kernel memory settings:

  # -- in /etc/sysctl.conf --
  # Set SHMMAX to 8GB for PostgreSQL
  kernel.shmmax=8589934592
  

  # -- in /etc/postgresql/8.1/main/postgresql.conf --
  # Crank up shared_buffers and work_mem
  shared_buffers = 10000
  work_mem=8388608 # 8 GB, equal to our kernel.shmmax
  max_fsm_pages = 200000
  

• Evergreen depends on accurate fixed fields to determine the format of an item. Unfortunately, many of our electronic resources appear not to have been coded as such... so we have some data clean-up to do.

Ah well: as Jerry Pournelle used to say in his Chaos Manor column, "I do these things so that you don't have to." Hopefully it makes a smoother path for others to get to Evergreen.

On Wednesday, I was fortunate enough to join a distinguished panel of speakers and a crowded music hall at VALE's Next Generation Academic Library System Symposium at The College of New Jersey. I had been invited to present an update on the state of acquisitions support in Evergreen, as well as to provide a brief overview of Project Conifer (the collaboration between Laurentian University, McMaster University, and the University of Windsor to create a consortial implementation of Evergreen).

To summarize what I intended to be the main points of my presentation (which may or may not have come through in real life):

• Project Conifer is an existing effort to create a shared consortial implementation of Evergreen for academic institutions; we would be delighted to have others join forces with us
• If acquisitions isn't as far along as we would have hoped by now, it's because
  • We (the Project Conifer institutions) haven't contributed enough development resource to the effort thus far - although we are planning to correct this problem in the near term by hiring one or more developers to work on the requirements that we, as academic institutions, need for a successful Evergreen experience. If you're interested in a position as an Evergreen developer for Project Conifer, let's talk.
  • Creating an enterprise-grade acquisitions system demands much more effort and attention to detail than creating a simplistic acquisitions system that would be acceptable for a small library. If it took two years to build Evergreen's circulation, cataloging, reporting, and OPAC functionality from scratch, it's not unreasonable that it should take a year or more to build an acquisitions system to the same standards as the rest of Evergreen
• Evergreen acquisitions has made significant progress since December 2007, and at this pace we expect a complete set of basic functionality to be in place by the end of April. By "basic functionality" I mean that the manual acquisitions mode should be supported with a minimalist user interface. MARC order record batch loading, EDI send/receive support, and a more polished user interface will take some more time - probably September-ish 2008. You can see the in-development, regularly updated bare-bones interface at http://acq.open-ils.org/oils/acq/base/index.

I have to say that Equinox is making incredible progress considering that they're still doing the bulk of the work with the same amount of development resource that they had before Georgia PINES went live on Evergreen, and they started their own company, and they started bringing BC PINES on line, and they began receiving an onslaught of requests for visits and presentations and conference calls... imagine what we could do with Evergreen, together, if a few more sites or consortiums were able to devote human or financial resources to enhancing Evergreen.

Here are my slides in OpenOffice and PowerPoint format. If you're going to look at my slides, I highly recommend reading the presenter notes that I wrote; I've recently realized that presenter notes are as much for the benefit of a disconnected audience as they are useful preparation material for the presenter. In the absence of a full paper on the subject matter at hand, presenter notes should help flesh out the brevity forced by slideware.

A huge thanks to Ed Corrado, Anne Hoang, and Kurt Wagner for making the overall experience so enjoyable. I was honoured to be part of such a high-quality panel of speakers.

Oh, and as an aside - the entire symposium was videotaped, and the presentations and question and answer sessions will be made available from the VALE Web site. I will update this post when those become available. I wonder if Ed got this idea from code4lib... in any case, I certainly applaud the initiative.

Update: Umm, more polished acquisitions will likely be available in Sept. 2008, not 2007... thanks to Brad Lajeunesse for pointing out that time travel would be required to make that happen

Well, the talk about CouchDB (an open-source document database similar in concept to Lotus Notes, but with a RESTful API and JSON as an interchange format) wasn't as much of a train wreck as it could have been. I learned a lot putting it together, and had some fun with the content - and even though it was a marked departure from the style of many of the other presentations, I think it was generally positively received (at least, from what I could glean from the backscroll in #code4lib and from comments).

I veer towards the "here's how you do stuff" technical angle because that tends to be what I'm interested in hearing from other people. And even though a 20 minute slot is probably the wrong venue for technical information, CouchDB is so simple in some respects that it's actually enough to get the core message across.

Here are the slides for your amusement and enlightenment. At some point I'm going to write down the OpenOffice.org secret that lets you change the colour of hypertext links - I've learned and forgotten that a number of times already.

• CouchDB: Delicious Sacrilege (OpenOffice Impress)
• CouchDB: Delicious Sacrilege (PDF)

Well, despite getting less than four hours of broken sleep before my 9:00 am presentation, I think I successfully delivered an update on Evergreen:State of the Open ILS to approximately 45 people at the OLA Super Conference today. There were some great questions from the audience that kept me on my toes. Thank heavens David Fiander was there to provide colour commentary and solid advice. Overall, the talk seemed to be well received.

Perhaps the most pleasant surprise of the session was when I discovered that one of the libraries close to my old home town has been working for the last six months on migrating to Evergreen. Marvelous!

If you want the slides from my presentation, I've licensed them under a Creative Commons 2.5 By Attribution license. Presentations available below in two different formats:

• Evergreen:State of the Open ILS (OpenOffice Impress)
• Evergreen:State of the Open ILS (Microsoft PowerPoint)

PHP is getting a native doubly-linked list structure. This is fabulous news; when I wrote the File_MARC PEAR package, I ended up having to implement a linked list class in PEAR to support it. File_MARC does its job today (even though I haven't taken it out of alpha yet), but due to its reliance on userspace data structures it's an order of magnitude slower than packages like marc4j so it's not the best choice for processing hundreds of thousands of MARC records... today. It hurts a little that the VuFind project has to use a non-PHP solution for populating its Solr indices - although I'm delighted that they have started using File_MARC for some on-demand processing.

Now, when I get a chance (insert raucous mocking laughter here), I hope to be able to make File_MARC use splDoublyLinkedList and see how it fares with 500K records. Should be good fun! After that, it just needs to be taught how to convert MARC8 to UTF-8, and we'll have ourselves a fully featured standard MARC package for PHP.

We don't watch a lot of TV; maybe four hours a week. But we noticed in the past year or two that a few of the shows we enjoy have muddy soundtracks due to speech being mixed over top of background music and various sound effects that makes the words almost indistinguishable. My hunch was that it was due to the increasing popularity of surround sound mixing, so much so that the producers had chosen to disrupt the viewing experience of plain jane viewers relying only on their television's built-in speakers. But for various reasons (largely - having better things to do), I never bothered to confirm this hunch.

I took advantage of a Boxing Day sale to pick up a set of surround sound speakers and a Dolby-enabled amplifier; nothing fancy. After many hours of threading wires behind drywall and through the basement (reminding me that I really do have better things to do), we sat down to watch an episode of one of the offending shows (Torchwood). And the difference in clarity (largely due to the separation of the sound into front left, center, and front right channels; the back left and right channels have much less impact) was immediately evident; Lynn mentioned it within five seconds of the beginning of the show.

Our household has entered a new age of audio-visual enjoyment. I know I've been manipulated into buying more electronics, but at the same time - wow, what a difference. It makes me want to watch all of season one of Battlestar Galactica again.

My employer, Laurentian University, issued a press release today announcing that we have selected Evergreen as our future library system. I wrote more about this on the Evergreen blog, but what I didn't say was ... yay!

We still have a long road ahead of us, but knowing that we'll be migrating to a system that I can poke with a sharp stick and make it do my bidding goes a long way towards making me feel warm and fuzzy inside.

I predict that we'll see a few more announcements from universities and colleges in North America joining the Evergreen development effort / adoption process in 2008. Outside of Ontario, I know about the University of Utah' s interest and interest from a New Jersey consortium of academic institutions (see session h. "Open Library Systems and NJ: From Vision to Transformation")... are there other academics who have made public statements of interest in Evergreen that I'm missing out on?

What a night. I upgraded Serendipity, DokuWiki, Drupal, involving four different servers and three different Linux distros, and shifted one application from one server to another (with seamless redirects from the old server to the new) with close to no downtime. I think this is the first time I've completed a single upgrade without having to manually patch at least one file.

Either I'm getting smarter, or the software is getting more robust.

Wow. Eileen R. Kontrovitz, a board member of CODI (Customers of Dynix, Inc.) wrote, as part of her summary of the recent CODI conference:

Many very nice things happened at the conference but the buzz, the thing everyone who was not there wants to hear about is the unannounced, invitation only meeting with Martin Taylor from Vista Equity Partners about open source. Mr. Taylor began the meeting in a very cordial way and with a charm about him that belied the basic message he proceeded to give. That message was, we know more than you do and if you don't like it you can go to some other "happy place". He actually said that on more than one occasion.

I guess Vista feels quite threatened by open source library systems, even though they currently account for only 1% of the US public library market today, if they're willing to have their top brass lay on the fear, uncertainty, and doubt campaign to their top customers behind closed doors. It also seems that Mr. Taylor's tactics have backfired, at least in this case; after praising the common SirsiDynix employees, Eileen goes on to say that Vista's attitude has almost ensured that her library (Ouachita Parish Public Library) will not move to Symphony. This, even though Eileen says:

...I happen to agree with his assessment of open source for a library ILS at this point in time and the many problems with the very nature of the beast without some kind of regulating body...

Perhaps this should be the subject of another blog post, but I believe there are actually a number ways Evergreen is regulated. First is that an open source project is not an "anything goes" project: the committers for the project act as a level of quality control for Evergreen. If code doesn't further the Evergreen mission by contributing towards stability, robustness, flexibility, security, or user-friendliness, then it's simply not going to go into the project proper. Second is that at least one company (Equinox) is staking its success on Evergreen, and others are starting to build up business around Evergreen. They're not going to sit back and rest easy; they know that they have to enhance Evergreen beyond its current core strengths if they want to build inroads into markets like academic libraries. Third is that Evergreen's open source license also acts as a regulator - the code that comprises Evergreen can never be pulled from the market, so the future of Evergreen is always in the hands of the community.