Evaluation Criteria For Electronic Resources
Susan J. Beck
Table of Contents
Additional Criteria Under Development
- Public Service Issues
- Technical Services Issues
- Systems Issues
- Consortia Relationships
Researching a Product
- Are there any evaluative reviews of the product available?
- Does the producer make customer lists available for contact with institutions who have had experience using this resource?
- Has contact been made with current users?
- Ask product related questions on library listservs which will cover all aspects of product purchasing such as collection development and licensing issues, public services issues and technical issues.
- Does the publisher, sponsoring agency, vendor or producer have a good business reputation and is considered reliable?
- Is the authority of the product clearly stated in the user documentation?
- What is the authority of the indexers?
- Who is the primary audience for the product? (Undergraduates, Graduate Students, Faculty, Adults, Children, Layperson)
- What is the stated subject coverage for the resource?
- How many subjects are covered by the resource?
- Is this resource the authoritative file in its discipline?
- Are the titles covered appropriate for the subject area?
- Are the titles considered well respected and of high utility in the stated area of coverage?
- What is the time period covered in this resource?
- How many years of data are in the database?
- Is it international in scope?
- Does it just cover the United States?
- Are there any additional geographic restrictions?
- How current is the information in this resource?
- What is the lag time between when a periodical is published as opposed to when it appears in the index?
- How frequently is it updated? (daily, weekly, monthly quarterly?)
- Are there multiple languages represented in the publications in the resource?
- If multiple languages publications are present, are citations and abstracts available in English?
- What types of materials are covered? journal articles? books, documents, pamphlets? conference proceedings?
- How many publications are covered?
- Are abstracts available for all records?
- Are abstracts available for selected records?
- What percentage of the database has abstracts?
- Are articles available in full text?
- What percentage of the titles are available in full text?
- If journals are not indexed cover to cover, are exclusions documented?
- Are images (including photographs, graphs and illustrations) available in all articles?
- What percentage of the resource's titles include full image?
- If images are not all available online, are they available to be faxed?
Relation to Similar Works
- How does it compare with similar works on the same topic?
- Is it similar in scope to other works in your collection?
- Is there significant content overlap between this electronic resource and other currently locally available electronic resources?
- What makes this work unique or distinguishes it from other sources?
- If the product is available from multiple vendors, how does one version differ from the others?
- What are the differences in searching interfaces?
- What are the differences in display options?
- What are the differences in and output options?
- Is the resource available on the web?
- Is the resource available to mount locally?
- Does it require maintenance or serial updating?
Depth of Indexing
- How reliable is the information presented?
- How dirty is the database in terms of the presence of misspelled words?
- What is the error rate?
- Is the indexing quality accurate?
- Are the bibliographic citations accurate?
- What quality control procedures does the producer use?
General Indexing Features
- What are the basic components of the record?
- What fields of the record are searchable?
- Can the full text be searched? Is it optional?
- Can indexing be controlled locally?
- Are all authors associated with an indexed item included?
- Are author names spelled in the index as they are in the work?
- Are authors listed only by the initials of the author and not the full name as it appears on the publication?
- Are all facets of the content of each article indexed?
- How many index terms are assigned to each record?
- How thorough is the indexing?
- Do indexes provide unique access points?
- To what extent is the database's vocabulary controlled?
- Are subject terms used that are appropriate to the discipline?
- Is there an online thesaurus available to help the user formulate effective searches?
- Does the thesaurus include scope notes?
- Can you link directly from the thesaurus to a set of articles?
- Are journals indexed from cover to cover?
- Are journals indexed selectively?
- Are letters to the editors and editorials indexed?
- Is other editorial content included in the full text online? (instructions to the author, calendars, news)
- If journals are not indexed cover to cover, are exclusions documented?
- What unique features distinguish this source from another?
- What is the image quality of graphs, illustrations, and pictures?
- Is a list of sources indexed available with the product?
- Can local holdings be linked to the database?
- Does the producer offer document delivery services?
- Is searching easy for average users?
- Are there both novice and expert search interfaces available?
- Does the interface have introductory screens?
- Does the interface have online tutorials?
- Does the interface have prompts and menus?
- Does the interface have function specific help?
- Is it easy to move form one function to another
- Is it easy for a user to access full-text from the citation?
- Are error messages easy to understand?
- Is there an easy and obvious way to logoff or exit out of the system?
- Is Boolean searching available?
- Is keyword searching available?
- Is full text searching available?
- Is index browsing available?
- Is weighted searching available?
- How are weights assigned?
- What search indexes are available?
- Is there hyper linked searching capabilities?
- Can thesaurus terms be used without rekeying?
- Is truncation available?
- Is truncation automatic?
- What are the truncation symbol(s)?
- Is word stemming available?
- Is proximity searching available?
- Is there a method to search for littorals and stop words as parts of phrases?
- Can selected items be marked for later use?
- Can search results be saved as sets for reuse?
- Can a search strategy be saved and re-used in another database?
- What search limits are available?
- Can you limit by full text?
- Can you limit by abstracts only
- Can you limit by language?
- Can you limit by dates?
- Can you limit to publication type?
- Can you limit to a single field?
- What additional special limits are available?
- Are there subject or publication format specific limits available? (Such as limit to human, animal, age, industry information, SIC Codes, newspaper headlines)
Search Display Options
- Are the search screens attractive?
- Are search screens easy to read?
- Are the search screens immediately understandable?
- Can users follow on screen instructions to begin searching immediately?
- Are the screens tailored to the needs of the user?
Results Display Options
- What is the default display?
- Can the user select and define different display options?
- Is local control of display options possible?
- Is it easy for a user to understand the different fields in the citation?
- Does the user need special software to display either full text or images? (Adobe Acrobat)
- Does the citation display with the full text?
- How easy is it to move through the display for both citations and full text options?
- How are images displayed with text?
- Can the search results be sorted for display?
- What sort functions are available for display?
- Is it easy to download?
- Is it easy to email?
- Is it easy to print?
- Is the search history attached to the results?
- Can the search history be detached from the results?
- Can search results be sorted?
- Are citations included in the same record with the full text of the article?
- Are user defined record formats available?
- What formats are available for downloading or emailing? (ASCII Text, comma separated)
- Are those formats commonly used for importing into productivity software?
- Are there limits to the number of records which can be delivered in one segment? (e.g. First 50 in set).
- If you limit output to an initial segment of the records, is it difficult to select more records for output?
- What is the pricing structure of the product?
- How are the costs determined?
- Are costs part of a subscription package?
- Is there a pricing differential for electronic sources if the library also subscribes to the printed source?
- Are costs based on number of users served?
- Are costs based on the number of uses of the product?
- Are costs based on the number of ports available to the users?
- Are costs reduced by aggregation? (consortia vs. university-wide vs. campus)
- Does the vendor supply adequate documentation?
- Are navigational aids available for the product?
- Is the user documentation accurate?
- Is the user documentation easy to use?
- Is the user documentation comprehensive?
- Is user documentation free?
- Is there a tutorial available for users?
- Is there online documentation available?
- Is there print documentation available?
- Is the user documentation quickly changed to reflect changes in the product?
- If there is no documentation available, how difficult will it be for the library to produce the necessary documentation?
- Does the producer provide training?
- Who is responsible for staff training? (selectors, instruction, reference)
- Who is responsible for user training?
- Who is responsible for the development of user guides or documentation?
- Are user-friendly tutorials available?
Customer Relations and Support
- Can the users suggest changes?
- Is there a publisher newsletter to make users aware of changes?
- Does the vendor support a listserv?
- Is customer support available during the hours the library is open?
- Does the vendor/producer/publisher offer sufficient backup for the product?
- When you call for information or to report a problem, how long does it take 1) to get an initial response? 2) to get the problem resolved?
- Is customer support available by email and what is the response time?
- Is there a toll free number for customer service? Is it always busy?
- Does the vendor's website provide technical support?
- Does the producer support user advisory groups?
- Does the producer sponsor formal update training sessions?
- How are users informed about product changes?
- Does the producer have a process to deal with errors identified by users of the databases?
- Does the producer provide use statistics?
- What kind of use statistics are available from the producer?
Demonstration and Trials
- Will the vendor provide a product demonstration?
- Will the vendor allow trials?
- Do users participating in trials positively evaluate the product? (Notify specific users during a trial, requesting comments or user input.)
- If a trial is available on our website, include an online evaluation form for users.
Publicity and Promotions
- Will the vendor promote use of the resource?
- Will the vendor supply promotional content for email to interested individuals, faculty newsletters, web pages, student newspapers, fliers?
- What specific media sources can be targeted for effective publicity?
- Determine an assessment cycle (quarterly, biannually, yearly)
- Is the product still useful?
- Has the product changed?
- Does the product continue to meet the needs of our users?
- Are there any new competing products available which might better meet our users' needs?
- Are there use statistics available which can be analyzed to determine use?
- Is there continued demand for the product?
- Do local instructional and support materials need modification?