Know your sources: Evergreen / Koha comparisons

Posted on Sun 24 June 2007 in Libraries

Correction update: 2007/06/26Wow. I am incredibly embarassed. Somehow, I made a very stupid mistake in my summary of the State Library

of Ohio ILS Options Discussion Meeting Minutes - April 24, 2007. The mistake was that I incorrectly attributed Joshua Ferraro of LibLime with making statements about Evergreen at that meeting when he was not even present. All of the statements about Evergreen should have been attributed to Stephen Hedges. I apologize profusely to Josh for this mistake, and will repeat this correction and apology in the section of the blog entry closer to the text.

I have been in the process of gathering information about the possible future

of our library system, with a focus on SirsiDynix's Rome, Evergreen, and Koha,

for a number of months now. This results in having to sift through claims from

a number of different sources about the capabilities (present and future) for

all of these systems. In the context of a recent post on the open-ils.org blog (Lies, Damned Lies, and Library Automation Software),

as well as all of the shilling that will undoubtedly be going on on the exhibit

floor and in the hospitality rooms of ALA, and finally building on

Karen Coombs' post on Bias, Objectivity and Authority, I would

like to make a point that ideally shouldn't need to be made (especially for

librarians!), but sadly seems to be necessary in the context of discussions about Evergreen and Koha.

The point? "Know your sources." And "Check your facts." When you've been given information about

something, you don't blindly accept the information as given - you check the

references and determine the authority of the source. This is par for the

course for reference librarians educating patrons performing research in their

libraries, but oddly enough seems to be a common blind spot when it comes to

performing evaluations of the software that powers your libraries.

Problem #1: Evaluating information about a given product from the company or

organization that stands to benefit from your adoption of that product.

This is especially hard when dealing with companies offering proprietary

products that won't hand you an evaluation copy to try out in your own

organization; or that run closed mailing lists; or that don't make their

documentation or support infrastructure openly available.

But it can also be hard with companies or organizations offering open source

products or support for open source products. The company or organization may

point you at an online demo of their product, but that demo may reflect a

heavily customized, bleeding-edge version of the product that mere mortals

cannot install - or even more insidiously, may include code that is not

currently included in the open source repository.

The good news on the open source front is that independent contributors have

made VMWare images of some of the most popular library systems available

(Evergreen,

Koha) for

download that reflect a standard install of the product directly from the open

source repository.

Note that a common approach to marketing a product is to provide a feature

list - basically a checklist of features. A naive decision maker might assume

that more is better, which often results in products breaking down the

features that they do well into many sub-features. It's a form of checklist

inflation - but as long as you've got your eyes open, at least it's more

information rather than less. For each feature you're actually interested in,

you have to ask a couple of additional questions:

  • How many other libraries are currently using this feature? It may be

    great that the software you're looking at includes LDAP authentication as an

    option, but if there's only one other library using the feature it's

    unfortunately likely that they will be using a different LDAP directory

    product than you (Novell eDirectory vs. MS ActiveDirectory vs. OpenLDAP vs.

    IBM Directory) and they will be using it in a different way.

  • Is this feature part of the base package, or is it an optional extra

    that's going to cost me more? Not such a problem with the open source

    options, although it depends in that case whether you're buying commercial

    support for the product. The model that the company uses may be all-inclusive

    or a menu of different costed support options.

  • Is this a massive feature that hasn't been broken down into

    sub-features? The danger here is that, for the purposes of looking good in

    feature comparisons, a product may have added a number of "features" that

    really just scratch the surface of what comparable products offer.

For example, if a library systems product says it has a serials module and an

acquisitions module, you have to dig into what that really means. Does the

"serials module" just mean that it will spit out a routing list whenever you

check in a new issue of a given serial? Or does it mean that it handles

predictions, claiming, holdings, etc., in the way that meets your library

needs? By "acquisitions module", does the product mean that it simply records

the cost of each item that you have acquired? Does it allow you to make

on-order items visible in the catalogue, with the ability to place holds? Does

it include EDI capabilities? Does it provide a complete fiscal management

system with funds and reporting and electronic record import / export hooks

for the ERP system that your university or municipality uses so that costs and

invoices don't have to be manually entered multiple times in multiple systems?

Perhaps most importantly, does the system have the flexibility to adapt to your

needs, or does the system require you to adapt to its needs? Can you live with

the 80% of functionality that most sites need, or does your site live in the

long tail of requirements. In the case of serials, for example, do you need

the ability to specify any pattern, or can you just deal with irregular

patterns those as exceptions?

Problem #2: Evaluating information about a given product from a company or

organization that offers a competitive product.

Sales people make it their business to know their competitors so that they can

accomplish two goals:

  1. Focus attention on (and often embellish) their own product's strengths,

    and know how to spin responses to their own product's weaknesses that might

    be identified by a competitor.

  2. Identify the weaknesses of their competitor's product, particularly when

    their own product has comparable strengths in that same area. Note that these

    weaknesses don't necessarily have to be real, they just have to be believable

    and hard to disprove.

Among the proprietary options, without having access to a hands-on test system,

the system documentation, or the product mailing lists, it is incredibly hard to

verify claims about a products' strengths or weaknesses. Even for claims about

the future development of a proprietary product that you already have access

to, the company cannot be held liable if plans change. Companies can, for

example, cancel an entire product even after beta versions of the product have

been released into the wild for "development partners." Horizon 8, anyone?

Development partners: test our beta for us!

Oh - on the topic of "development partners" - this is typically a euphemism for

"we'll give you a discount on product XXX if you put it into production and

report all the bugs you find." Companies love this approach because it gives

them visibility in the marketplace ("Look, we already have deployed product XXX

to five sites! It's proven and ready for you!") while enabling them to

effectively continue development on product XXX and hope to have a polished

product ready in time for the bulk of their potential customer base to actually

adopt it. In the past, Microsoft very effectively used the "product announce"

to prevent customers from purchasing a competitor's product that offered

compelling features and stalling the decision long enough to then develop and

bring their own product to market.

Disinformation and open source projects

Surprisingly, the disinformation approach works even in open source projects.

For example, I have read and heard claims about Evergreen like: "Oh, Evergreen

is just for massive consortiums / it needs 40 servers to run / it doesn't scale

down to just a single library." You can see how this could be believable if you

don't push too hard on the claims, because Evergreen's was developed for a

consortial library system and much has been made about the impressive server

cluster that GPLS runs Evergreen on -- however, having run Evergreen on a single

VMWare machine on my laptop, I can personally attest to its ability to scale

down to a single server (or portion thereof). And you can run that same VMWare

image on your own laptop or spare desktop machine and disprove that claim

yourself; but many of the decision makers do not have the technical skills,

time, or interest to get hands-on with products like Evergreen. So they have

to trust what they read or hear, hopefully from the most trustworthy of

sources.

Another swipe at Evergreen is that it is not a true open source project; that

its history as a top-down project initiated by GPLS means that there is no

real development community around Evergreen. If you've followed the Evergreen

development

mailing list, you wouldn't believe a claim like this, and you would

proclaim it a blatant lie. To disprove this claim, you just

need to browse through the open-ils-dev mailing list and look for the emails

with the subject keyword "PATCH" and you'll see that some of us have indeed

been contributing patches to the source code. Beyond that, you'll also see

that there are many volunteer contributors for install and configuration

support, documentation, creation of VMWare images. So how could someone make

such a claim about Evergreen and get away with it?

It's all about trusting "authorities", not checking sources, and integrity (or perhaps

a lack thereof). Here's an excerpted quote about Evergreen from the State Library

of Ohio ILS Options Discussion Meeting Minutes - April 24, 2007

The documentation for the process is very poor, which is typical because it is

the last thing developers are thinking about. ... The source code is open but

they don't really follow the "playground" rules for the open source production

process.

Here's where you need to really know your sources and check your references.

Note that the claims about the nature of Evergreen as not being a true open

source project are credited to the introductory speaker, Stephen Hedges. Who

is Stephen Hedges? He was the director of Nelsonville Public Library (NPL)

when he worked with Joshua Ferraro to install Koha as the NPL integrated

library system. In addition, he is listed as the contact for Koha documentation

submissions. It seems, then, that he has a fairly significant personal stake in

the success of Koha, and if the meeting minutes accurately capture his

statements about Evergreen, it sounds like he was interested in dissuading

attendees from seriously considering Evergreen as an option. Subjectivity alert: as one of the volunteer contributors of code, documentation, install assistance, and a VMWare image of Evergreen from outside GPLS, this quote got me pretty hot under the collar; I've contributed to other open source projects, such as the Linux Documentation Project and PHP, and you always have to prove that you understand the project before being granted commit access.

Correction update: 2007/06/26Wow. In the following paragraph, I somehow made a very stupid mistake by incorrectly attributing Joshua Ferraro of LibLime with making statements about Evergreen at that meeting when he was not even present. All of the statements about Evergreen should have been attributed to Stephen Hedges. I apologize profusely to Josh and LibLime for this mistake.

Who was Stephen introducing as the guest speaker of honour on the subject of

open source ILS options in libraries? The speaker was Joshua Ferraro,

president of LibLime, the company best known for offering commercial support

for Koha. LibLime did announce that they would offer commercial

support for Evergreen, and have added sections about Evergreen to their Web

site, so it would on the surface seem to be a logical choice to invite a

LibLime employee as a one-speaker-fits-all host to cover both Koha and

Evergreen. However, LibLime has a rather unusual relationship with Evergreen.

It seems that LibLime has positioned Evergreen among their other offerings as

such a high-end product that only a handful of potential customers would qualify

for that market:

Evergreen

For consortia who need:
* Scalability to hundreds of libraries, tens of millions of records

[LibLime Products]

It sounds impressive, but way too high-end for the vast majority of

libraries. So of course people browsing the LibLime Web site will focus on the

Koha options instead. It seems like a deliberate bait-and-switch

move to attract libraries interested in Evergreen after the successful launch in

Georgia, but to get them to buy support for Koha instead. Consider: LibLime

has not contributed a single patch to the Evergreen development (open-ils-dev)

mailing list. LibLime has not contributed a single line of documentation to

the Evergreen wiki. LibLime does not include Evergreen among their demos.

LibLime hasn't made an Evergreen sale. So I think it's a fair question to ask

how committed LibLime really is to Evergreen - is LibLime's claim to support

Evergreen just a means to get people in the door, in hopes that they'll walk

out with a copy of Koha under their arms? I think so. You can come to your own conclusions.

In case you think that Ohio quote was just an unfortunate one-off, and that

I'm making a big deal about nothing, here's a more recent quote from the

Open

Source Session Q&A of the "Everything You Ever Wanted to Know about Open

Source" conference held on June 6th, 2007 that caught my attention (and

which apparently no-one in attendance at the meeting was capable of providing

a rebuttal to):

Q. Contrast Koha & Evergreen?

A. Major difference: Koha was grassroots: started w/rural libraries, distributed organization, bottom-up decision making. Evergreen: PINES library system; top-down decision making. Koha: 800 libs worldwide, 8 years old; Evergreen: 1 year old, 1 consortium.

So there's the comment about the "top-down" nature of Evergreen again, and

this time Evergreen is being attacked for being immature and not very widely

used. (Note: on that very day, the British Columbia Ministry of Education

announced the BC PINES Website - so

another consortium is getting on board the Evergreen express.) If there really

are 800 libraries using Koha, I'm shocked at how many basic install, config,

and runtime problems are being reported on the Koha mailing lists with the

current 2.2.9 release... but I'm getting off-topic. The speaker was

[STRIKEOUT:once again] Joshua Ferraro, who:

... talked with us about open source integrated library systems,

specifically Koha and Evergreen, and about his company,

LibLime... [

href="http://blogs.umass.edu/ealling/2007/06/06/open-source-session-reflections/">reference]

If your definition of "talking about" is "praise the product that pays your

bills and criticize the product that represents a major threat", then mission

accomplished. You can't blame the speaker for being in a perfect position

to pitch his product at the expense of a competing product, while being credited

with being an objective authority on both products. But I suspect the audience actually wanted a balanced presentation

about the two products.

So what's my point? Know your sources. If you invite someone to speak

on a broad topic, such as the State Library of Ohio meeting, where [t]he invitation was expanded to include any

library interested in the possibility of open source integrated library

systems (ILS), you might want to ensure

that any personal biases are very much out in the open for your audience

(both the in-person audience and the audience reading the meeting minutes at

home). If you're the speaker in such a situation, you should reveal any such biases.

If you're a company selling a

product or services related to a product, perhaps it's inevitable that the

profit motive is going to override ethics in such opportunities - but I can

dream. If you read the full minutes from the State Library of Ohio meeting,

you can see that in terms of an open source ILS option, Evergreen is given

only the most cursory coverage and the major focus is on selling Koha.

Getting a fair comparison

For a fair comparison of Koha and Evergreen, please consider either

hosting two separate presentations (you wouldn't consider asking SirsiDynix

to give a balanced presentation on all of the proprietary ILS options, would

you?), or try to find an independent speaker who can provide a more objective

analysis of the products at hand. Ask the speaker if they have any

financial ties to the products at hand. Heck, has anybody asked Marshall

Breeding and Andrew Pace if they've had any financial ties to ILS companies? I

assume the answer is no, but our community relies so much on their analysis of

the overall library systems landscape with so much financial implication for the

companies at question that it would be comforting to have a positive assertion

accompany any "state of the ILS landscape" articles in the future.

Ideally, you would find a member of the development community for each of

Evergreen and Koha. At the moment, I'm afraid that I can only qualify as a

member of the Evergreen community, but I plan to become more familiar with

Koha's codebase over the course of the summer - so maybe I can grow into that

position. Of course, then you would have to trust me. C'mon, you can trust me! **grin**

Make technology, not war

What would I like to avoid? I would really like to avoid negative energy being

invested in a Koha vs. Evergreen or LibLime vs. Equinox battle royale. That

doesn't interest me, but I'm sure it greatly interests the companies offering

proprietary products. Instead, I hope that this energy can continue to be

invested in making both Koha and Evergreen better by those with the technical

skills. Let's have a competition on product design and implementation, rather than

on marketing spin or dirty tricks. Everyone benefits from strong open source

library systems - even if you don't adopt an open source system, it raises the

bar for the proprietary systems to differentiate themselves.

Evergreen