Know your sources: Evergreen / Koha comparisons

Correction update: 2007/06/26Wow. I am incredibly embarassed. Somehow, I made a very stupid mistake in my summary of the State Library of Ohio ILS Options Discussion Meeting Minutes - April 24, 2007. The mistake was that I incorrectly attributed Joshua Ferraro of LibLime with making statements about Evergreen at that meeting when he was not even present. All of the statements about Evergreen should have been attributed to Stephen Hedges. I apologize profusely to Josh for this mistake, and will repeat this correction and apology in the section of the blog entry closer to the text.

I have been in the process of gathering information about the possible future of our library system, with a focus on SirsiDynix's Rome, Evergreen, and Koha, for a number of months now. This results in having to sift through claims from a number of different sources about the capabilities (present and future) for all of these systems. In the context of a recent post on the open-ils.org blog (Lies, Damned Lies, and Library Automation Software), as well as all of the shilling that will undoubtedly be going on on the exhibit floor and in the hospitality rooms of ALA, and finally building on Karen Coombs' post on Bias, Objectivity and Authority, I would like to make a point that ideally shouldn't need to be made (especially for librarians!), but sadly seems to be necessary in the context of discussions about Evergreen and Koha.

The point? "Know your sources." And "Check your facts." When you've been given information about something, you don't blindly accept the information as given - you check the references and determine the authority of the source. This is par for the course for reference librarians educating patrons performing research in their libraries, but oddly enough seems to be a common blind spot when it comes to performing evaluations of the software that powers your libraries.

Problem #1: Evaluating information about a given product from the company or organization that stands to benefit from your adoption of that product.

This is especially hard when dealing with companies offering proprietary products that won't hand you an evaluation copy to try out in your own organization; or that run closed mailing lists; or that don't make their documentation or support infrastructure openly available.

But it can also be hard with companies or organizations offering open source products or support for open source products. The company or organization may point you at an online demo of their product, but that demo may reflect a heavily customized, bleeding-edge version of the product that mere mortals cannot install - or even more insidiously, may include code that is not currently included in the open source repository.

The good news on the open source front is that independent contributors have made VMWare images of some of the most popular library systems available (Evergreen, Koha) for download that reflect a standard install of the product directly from the open source repository.

Note that a common approach to marketing a product is to provide a feature list - basically a checklist of features. A naive decision maker might assume that more is better, which often results in products breaking down the features that they do well into many sub-features. It's a form of checklist inflation - but as long as you've got your eyes open, at least it's more information rather than less. For each feature you're actually interested in, you have to ask a couple of additional questions:

• How many other libraries are currently using this feature? It may be great that the software you're looking at includes LDAP authentication as an option, but if there's only one other library using the feature it's unfortunately likely that they will be using a different LDAP directory product than you (Novell eDirectory vs. MS ActiveDirectory vs. OpenLDAP vs. IBM Directory) and they will be using it in a different way.
• Is this feature part of the base package, or is it an optional extra that's going to cost me more? Not such a problem with the open source options, although it depends in that case whether you're buying commercial support for the product. The model that the company uses may be all-inclusive or a menu of different costed support options.
• Is this a massive feature that hasn't been broken down into sub-features? The danger here is that, for the purposes of looking good in feature comparisons, a product may have added a number of "features" that really just scratch the surface of what comparable products offer.

For example, if a library systems product says it has a serials module and an acquisitions module, you have to dig into what that really means. Does the "serials module" just mean that it will spit out a routing list whenever you check in a new issue of a given serial? Or does it mean that it handles predictions, claiming, holdings, etc., in the way that meets your library needs? By "acquisitions module", does the product mean that it simply records the cost of each item that you have acquired? Does it allow you to make on-order items visible in the catalogue, with the ability to place holds? Does it include EDI capabilities? Does it provide a complete fiscal management system with funds and reporting and electronic record import / export hooks for the ERP system that your university or municipality uses so that costs and invoices don't have to be manually entered multiple times in multiple systems?

Perhaps most importantly, does the system have the flexibility to adapt to your needs, or does the system require you to adapt to its needs? Can you live with the 80% of functionality that most sites need, or does your site live in the long tail of requirements. In the case of serials, for example, do you need the ability to specify any pattern, or can you just deal with irregular patterns those as exceptions?

Problem #2: Evaluating information about a given product from a company or organization that offers a competitive product.

Sales people make it their business to know their competitors so that they can accomplish two goals:

1. Focus attention on (and often embellish) their own product's strengths, and know how to spin responses to their own product's weaknesses that might be identified by a competitor.
2. Identify the weaknesses of their competitor's product, particularly when their own product has comparable strengths in that same area. Note that these weaknesses don't necessarily have to be real, they just have to be believable and hard to disprove.

Among the proprietary options, without having access to a hands-on test system, the system documentation, or the product mailing lists, it is incredibly hard to verify claims about a products' strengths or weaknesses. Even for claims about the future development of a proprietary product that you already have access to, the company cannot be held liable if plans change. Companies can, for example, cancel an entire product even after beta versions of the product have been released into the wild for "development partners." Horizon 8, anyone?

Development partners: test our beta for us!

Oh - on the topic of "development partners" - this is typically a euphemism for "we'll give you a discount on product XXX if you put it into production and report all the bugs you find." Companies love this approach because it gives them visibility in the marketplace ("Look, we already have deployed product XXX to five sites! It's proven and ready for you!") while enabling them to effectively continue development on product XXX and hope to have a polished product ready in time for the bulk of their potential customer base to actually adopt it. In the past, Microsoft very effectively used the "product announce" to prevent customers from purchasing a competitor's product that offered compelling features and stalling the decision long enough to then develop and bring their own product to market.

Disinformation and open source projects

Surprisingly, the disinformation approach works even in open source projects. For example, I have read and heard claims about Evergreen like: "Oh, Evergreen is just for massive consortiums / it needs 40 servers to run / it doesn't scale down to just a single library." You can see how this could be believable if you don't push too hard on the claims, because Evergreen's was developed for a consortial library system and much has been made about the impressive server cluster that GPLS runs Evergreen on -- however, having run Evergreen on a single VMWare machine on my laptop, I can personally attest to its ability to scale down to a single server (or portion thereof). And you can run that same VMWare image on your own laptop or spare desktop machine and disprove that claim yourself; but many of the decision makers do not have the technical skills, time, or interest to get hands-on with products like Evergreen. So they have to trust what they read or hear, hopefully from the most trustworthy of sources.

Another swipe at Evergreen is that it is not a true open source project; that its history as a top-down project initiated by GPLS means that there is no real development community around Evergreen. If you've followed the Evergreen development mailing list, you wouldn't believe a claim like this, and you would proclaim it a blatant lie. To disprove this claim, you just need to browse through the open-ils-dev mailing list and look for the emails with the subject keyword "PATCH" and you'll see that some of us have indeed been contributing patches to the source code. Beyond that, you'll also see that there are many volunteer contributors for install and configuration support, documentation, creation of VMWare images. So how could someone make such a claim about Evergreen and get away with it?

It's all about trusting "authorities", not checking sources, and integrity (or perhaps a lack thereof). Here's an excerpted quote about Evergreen from the State Library of Ohio ILS Options Discussion Meeting Minutes - April 24, 2007

The documentation for the process is very poor, which is typical because it is the last thing developers are thinking about. ... The source code is open but they don't really follow the "playground" rules for the open source production process.

Here's where you need to really know your sources and check your references. Note that the claims about the nature of Evergreen as not being a true open source project are credited to the introductory speaker, Stephen Hedges. Who is Stephen Hedges? He was the director of Nelsonville Public Library (NPL) when he worked with Joshua Ferraro to install Koha as the NPL integrated library system. In addition, he is listed as the contact for Koha documentation submissions. It seems, then, that he has a fairly significant personal stake in the success of Koha, and if the meeting minutes accurately capture his statements about Evergreen, it sounds like he was interested in dissuading attendees from seriously considering Evergreen as an option. Subjectivity alert: as one of the volunteer contributors of code, documentation, install assistance, and a VMWare image of Evergreen from outside GPLS, this quote got me pretty hot under the collar; I've contributed to other open source projects, such as the Linux Documentation Project and PHP, and you always have to prove that you understand the project before being granted commit access.

Correction update: 2007/06/26Wow. In the following paragraph, I somehow made a very stupid mistake by incorrectly attributing Joshua Ferraro of LibLime with making statements about Evergreen at that meeting when he was not even present. All of the statements about Evergreen should have been attributed to Stephen Hedges. I apologize profusely to Josh and LibLime for this mistake.

Who was Stephen introducing as the guest speaker of honour on the subject of open source ILS options in libraries? The speaker was Joshua Ferraro, president of LibLime, the company best known for offering commercial support for Koha. LibLime did announce that they would offer commercial support for Evergreen, and have added sections about Evergreen to their Web site, so it would on the surface seem to be a logical choice to invite a LibLime employee as a one-speaker-fits-all host to cover both Koha and Evergreen. However, LibLime has a rather unusual relationship with Evergreen.

It seems that LibLime has positioned Evergreen among their other offerings as such a high-end product that only a handful of potential customers would qualify for that market:

Evergreen
For consortia who need:
* Scalability to hundreds of libraries, tens of millions of records
[LibLime Products]

It sounds impressive, but way too high-end for the vast majority of libraries. So of course people browsing the LibLime Web site will focus on the Koha options instead. It seems like a deliberate bait-and-switch move to attract libraries interested in Evergreen after the successful launch in Georgia, but to get them to buy support for Koha instead. Consider: LibLime has not contributed a single patch to the Evergreen development (open-ils-dev) mailing list. LibLime has not contributed a single line of documentation to the Evergreen wiki. LibLime does not include Evergreen among their demos. LibLime hasn't made an Evergreen sale. So I think it's a fair question to ask how committed LibLime really is to Evergreen - is LibLime's claim to support Evergreen just a means to get people in the door, in hopes that they'll walk out with a copy of Koha under their arms? I think so. You can come to your own conclusions.

In case you think that Ohio quote was just an unfortunate one-off, and that I'm making a big deal about nothing, here's a more recent quote from the Open Source Session Q&A of the "Everything You Ever Wanted to Know about Open Source" conference held on June 6th, 2007 that caught my attention (and which apparently no-one in attendance at the meeting was capable of providing a rebuttal to):

Q. Contrast Koha & Evergreen?

A. Major difference: Koha was grassroots: started w/rural libraries, distributed organization, bottom-up decision making. Evergreen: PINES library system; top-down decision making. Koha: 800 libs worldwide, 8 years old; Evergreen: 1 year old, 1 consortium.

So there's the comment about the "top-down" nature of Evergreen again, and this time Evergreen is being attacked for being immature and not very widely used. (Note: on that very day, the British Columbia Ministry of Education announced the BC PINES Website - so another consortium is getting on board the Evergreen express.) If there really are 800 libraries using Koha, I'm shocked at how many basic install, config, and runtime problems are being reported on the Koha mailing lists with the current 2.2.9 release... but I'm getting off-topic. The speaker was once again Joshua Ferraro, who:

... talked with us about open source integrated library systems, specifically Koha and Evergreen, and about his company, LibLime... [reference]

If your definition of "talking about" is "praise the product that pays your bills and criticize the product that represents a major threat", then mission accomplished. You can't blame the speaker for being in a perfect position to pitch his product at the expense of a competing product, while being credited with being an objective authority on both products. But I suspect the audience actually wanted a balanced presentation about the two products.

So what's my point? Know your sources. If you invite someone to speak on a broad topic, such as the State Library of Ohio meeting, where [t]he invitation was expanded to include any library interested in the possibility of open source integrated library systems (ILS), you might want to ensure that any personal biases are very much out in the open for your audience (both the in-person audience and the audience reading the meeting minutes at home). If you're the speaker in such a situation, you should reveal any such biases. If you're a company selling a product or services related to a product, perhaps it's inevitable that the profit motive is going to override ethics in such opportunities - but I can dream. If you read the full minutes from the State Library of Ohio meeting, you can see that in terms of an open source ILS option, Evergreen is given only the most cursory coverage and the major focus is on selling Koha.

Getting a fair comparison

For a fair comparison of Koha and Evergreen, please consider either hosting two separate presentations (you wouldn't consider asking SirsiDynix to give a balanced presentation on all of the proprietary ILS options, would you?), or try to find an independent speaker who can provide a more objective analysis of the products at hand. Ask the speaker if they have any financial ties to the products at hand. Heck, has anybody asked Marshall Breeding and Andrew Pace if they've had any financial ties to ILS companies? I assume the answer is no, but our community relies so much on their analysis of the overall library systems landscape with so much financial implication for the companies at question that it would be comforting to have a positive assertion accompany any "state of the ILS landscape" articles in the future.

Ideally, you would find a member of the development community for each of Evergreen and Koha. At the moment, I'm afraid that I can only qualify as a member of the Evergreen community, but I plan to become more familiar with Koha's codebase over the course of the summer - so maybe I can grow into that position. Of course, then you would have to trust me. C'mon, you can trust me! **grin**

Make technology, not war

What would I like to avoid? I would really like to avoid negative energy being invested in a Koha vs. Evergreen or LibLime vs. Equinox battle royale. That doesn't interest me, but I'm sure it greatly interests the companies offering proprietary products. Instead, I hope that this energy can continue to be invested in making both Koha and Evergreen better by those with the technical skills. Let's have a competition on product design and implementation, rather than on marketing spin or dirty tricks. Everyone benefits from strong open source library systems - even if you don't adopt an open source system, it raises the bar for the proprietary systems to differentiate themselves.