Evaluation Criteria for Electronic Resources
Susan Beck
March 2, 1999
Table of Contents
Additional Criteria Under Development
- Public Service Issues
- Technical Services Issues
- Systems Issues
- Pricing
- Consortia Relationships
- Copyright
- Licensing
Researching A Product
- Are there any evaluative reviews of the product available?
- Does the producer make customer lists available for contact with institutions who have had experience using this resource?
- Has contact been made with current users?
- Ask product related questions on library listservs which will cover all aspects of product purchasing such as collection development and licensing issues, public services issues and technical issues.
Content Evaluation
Authority
- Does the publisher, sponsoring agency, vendor or producer have a good business reputation and is considered reliable?
- Is the authority of the product clearly stated in the user documentation?
- What is the authority of the indexers?
Audience
-
- Who is the primary audience for the product? (Undergraduates, Graduate Students, Faculty, Adults, Children, Layperson)
Scope
Subject Coverage
- What is the stated subject coverage for the resource?
- How many subjects are covered by the resource?
- Is this resource the authoritative file in its discipline?
- Are the titles covered appropriate for the subject area?
- Are the titles considered well respected and of high utility in the stated area of coverage?
Time Coverage
- What is the time period covered in this resource?
- How many years of data are in the database?
Geographical Coverage
- Is it international in scope?
- Does it just cover the United States?
- Are there any additional geographic restrictions?
Currency
- How current is the information in this resource?
- What is the lag time between when a periodical is published as opposed to when it appears in the index?
Update
- How frequently is it updated? (daily, weekly, monthly quarterly?)
Language
- Are there multiple languages represented in the publications in the resource?
- If multiple languages publications are present, are citations and abstracts available in English?
Publications
- What types of materials are covered? journal articles? books, documents, pamphlets? conference proceedings?
- How many publications are covered?
Record Format
- Are abstracts available for all records?
- Are abstracts available for selected records?
- What percentage of the database has abstracts?
- Are articles available in full text?
- What percentage of the titles are available in full text?
- If journals are not indexed cover to cover, are exclusions documented?
- Are images (including photographs, graphs and illustrations) available in all articles?
- What percentage of the resource's titles include full image?
- If images are not all available online, are they available to be faxed?
Relation to Similar Works
- How does it compare with similar works on the same topic?
- Is it similar in scope to other works in your collection?
- Is there significant content overlap between this electronic resource and other currently locally available electronic resources?
- What makes this work unique or distinguishes it from other sources?
Product Comparisons
- If the product is available from multiple vendors, how does one version differ from the others?
- What are the differences in searching interfaces?
- What are the differences in display options?
- What are the differences in and output options?
Availability
- Is the resource available on the web?
- Is the resource available to mount locally?
- Does it require maintenance or serial updating?
Depth of Indexing
Accuracy
- How reliable is the information presented?
- How dirty is the database in terms of the presence of misspelled words?
- What is the error rate?
- Is the indexing quality accurate?
- Are the bibliographic citations accurate?
- What quality control procedures does the producer use?
General Indexing Features
- What are the basic components of the record?
- What fields of the record are searchable?
- Can the full text be searched? Is it optional?
- Can indexing be controlled locally?
Author Indexing
- Are all authors associated with an indexed item included?
- Are author names spelled in the index as they are in the work?
- Are authors listed only by the initials of the author and not the full name as it appears on the publication?
Subject Indexing
- Are all facets of the content of each article indexed?
- How many index terms are assigned to each record?
- How thorough is the indexing?
- Do indexes provide unique access points?
- To what extent is the database's vocabulary controlled?
- Are subject terms used that are appropriate to the discipline?
- Is there an online thesaurus available to help the user formulate effective searches?
- Does the thesaurus include scope notes?
- Can you link directly from the thesaurus to a set of articles?
Journal Indexing
- Are journals indexed from cover to cover?
- Are journals indexed selectively?
- Are letters to the editors and editorials indexed?
- Is other editorial content included in the full text online? (instructions to the author, calendars, news)
- If journals are not indexed cover to cover, are exclusions documented?
Special Features
- What unique features distinguish this source from another?
- What is the image quality of graphs, illustrations, and pictures?
- Is a list of sources indexed available with the product?
- Can local holdings be linked to the database?
- Does the producer offer document delivery services?
Searching
Interfaces
- Is searching easy for average users?
- Are there both novice and expert search interfaces available?
- Does the interface have introductory screens?
- Does the interface have online tutorials?
- Does the interface have prompts and menus?
- Does the interface have function specific help?
- Is it easy to move form one function to another
- Is it easy for a user to access full-text from the citation?
- Are error messages easy to understand?
- Is there an easy and obvious way to logoff or exit out of the system?
Structures
- Is Boolean searching available?
- Is keyword searching available?
- Is full text searching available?
- Is index browsing available?
- Is weighted searching available?
- How are weights assigned?
Inverted Indexes
- What search indexes are available?
Vocabulary Linking
- Is there hyper linked searching capabilities?
- Can thesaurus terms be used without rekeying?
Truncation
- Is truncation available?
- Is truncation automatic?
- What are the truncation symbol(s)?
- Is word stemming available?
Positional Searching
- Is proximity searching available?
- Is there a method to search for littorals and stop words as parts of phrases?
- Can selected items be marked for later use?
- Can search results be saved as sets for reuse?
- Can a search strategy be saved and re-used in another database?
Search Limitations
- What search limits are available?
- Can you limit by full text?
- Can you limit by abstracts only
- Can you limit by language?
- Can you limit by dates?
- Can you limit to publication type?
- Can you limit to a single field?
- What additional special limits are available?
- Are there subject or publication format specific limits available? (Such as limit to human, animal, age, industry information, SIC Codes, newspaper headlines)
Search Display Options
- Are the search screens attractive?
- Are search screens easy to read?
- Are the search screens immediately understandable?
- Can users follow on screen instructions to begin searching immediately?
- Are the screens tailored to the needs of the user?
Results Display Options
- What is the default display?
- Can the user select and define different display options?
- Is local control of display options possible?
- Is it easy for a user to understand the different fields in the citation?
- Does the user need special software to display either full text or images? (Adobe Acrobat)
- Does the citation display with the full text?
- How easy is it to move through the display for both citations and full text options?
- How are images displayed with text?
- Can the search results be sorted for display?
- What sort functions are available for display?
Output Options
- Is it easy to download?
- Is it easy to email?
- Is it easy to print?
- Is the search history attached to the results?
- Can the search history be detached from the results?
- Can search results be sorted?
- Are citations included in the same record with the full text of the article?
- Are user defined record formats available?
- What formats are available for downloading or emailing? (ASCII Text, comma separated)
- Are those formats commonly used for importing into productivity software?
- Are there limits to the number of records which can be delivered in one segment? (e.g. First 50 in set).
- If you limit output to an initial segment of the records, is it difficult to select more records for output?
Cost
- What is the pricing structure of the product?
- How are the costs determined?
- Are costs part of a subscription package?
- Is there a pricing differential for electronic sources if the library also subscribes to the printed source?
- Are costs based on number of users served?
- Are costs based on the number of uses of the product?
- Are costs based on the number of ports available to the users?
- Are costs reduced by aggregation? (consortia vs. university-wide vs. campus)
Documentation
- Does the vendor supply adequate documentation?
- Are navigational aids available for the product?
- Is the user documentation accurate?
- Is the user documentation easy to use?
- Is the user documentation comprehensive?
- Is user documentation free?
- Is there a tutorial available for users?
- Is there online documentation available?
- Is there print documentation available?
- Is the user documentation quickly changed to reflect changes in the product?
- If there is no documentation available, how difficult will it be for the library to produce the necessary documentation?
Training
- Does the producer provide training?
- Who is responsible for staff training? (selectors, instruction, reference)
- Who is responsible for user training?
- Who is responsible for the development of user guides or documentation?
- Are user-friendly tutorials available?
Customer Relations and Support
- Can the users suggest changes?
- Is there a publisher newsletter to make users aware of changes?
- Does the vendor support a listserv?
- Is customer support available during the hours the library is open?
- Does the vendor/producer/publisher offer sufficient backup for the product?
- When you call for information or to report a problem, how long does it take 1) to get an initial response? 2) to get the problem resolved?
- Is customer support available by email and what is the response time?
- Is there a toll free number for customer service? Is it always busy?
- Does the vendor's website provide technical support?
- Does the producer support user advisory groups?
- Does the producer sponsor formal update training sessions?
- How are users informed about product changes?
- Does the producer have a process to deal with errors identified by users of the databases?
- Does the producer provide use statistics?
- What kind of use statistics are available from the producer?
Demonstration and Trials
- Will the vendor provide a product demonstration?
- Will the vendor allow trials?
- Do users participating in trials positively evaluate the product? (Notify specific users during a trial, requesting comments or user input.)
- If a trial is available on our website, include an online evaluation form for users.
Publicity and Promotions
- Will the vendor promote use of the resource?
- Will the vendor supply promotional content for email to interested individuals, faculty newsletters, web pages, student newspapers, fliers?
- What specific media sources can be targeted for effective publicity?
Continuous Assessment
- Determine an assessment cycle (quarterly, biannually, yearly)
- Is the product still useful?
- Has the product changed?
- Does the product continue to meet the needs of our users?
- Are there any new competing products available which might better meet our users' needs?
- Are there use statistics available which can be analyzed to determine use?
- Is there continued demand for the product?
- Do local instructional and support materials need modification?