In which I perceive that gossip is not science

Marshall Breeding published the results of his 2009 International Survey of Library Automation a few days ago. Juicy stuff, with averages, medians, and modes for the negative/positive responses on a variety of ILS and vendor-related questions, and some written comments from the respondents. One would expect the library geek blog niche to light up with reaction to the revelations contained in the data.

Except: there were 2,098 total responses to the survey, with 1,633 from the United States [1]. Responses were limited to a single response per library. The ALA estimates there are 122,356 libraries in the United States [2]. So, slightly more than 1% of the libraries in the United States are represented in this survey. Which, if it was a truly random sampling, might be enough to derive some validity from the results. But these survey responders? They're self-selected, therefore more likely to either have an axe to grind or a selection decision to defend. And the perspective of the responder on which these perceptions have been based, and in turn on which Breeding has drawn his observations, may over-represent one aspect of the system. If a circulation clerk responds, they may base their responses primarily on the speed and ease with which they can handle typical circulation tasks, while ignoring the experience of the patron using search and self-serve account interfaces; or a cataloguer may focus on the ease of copy cataloguing but gloss over the system's integration with authentication and financial systems, etc.

In my opinion, the survey's methodology makes its results more akin to gossip than science, and Breeding himself recognizes the lack of validity of his effort. After summarizing all of the findings, making some observations about open source and companies, and providing an overview of his methodology, he issues a caveat at the very end of all of this that effectively admits that the survey is worthless: "one should not read too much into the survey results."

If that's the case, then why even bother issuing this flawed, unscientific survey and publishing its results? A rigorous comparison of library systems that employs both quantitative and qualitative aspects in the assessments would be a useful contribution to the library world. This survey, however, deserves a swift trip to the round file.