Library Philosophy and Practice Vol. 7, No. 1 (Fall 2004)

ISSN 1522-0222

Assessing Library Instruction Assessment Activities

Jim Kapoun

Introduction

Common questions that instruction librarians may ask before creating an assessment instrument for their classes are; "What are other libraries doing?" or "What should I ask and what question(s) will really assess outcomes?" As a continuing effort to examine our instructional assessment at Minnesota State University, Mankato, I decided to assess library instruction assessment tools/surveys. This research will examine and reflect on how academic libraries conduct or administer their instructional classroom assessment. We wanted to know what types of questions were asked and how they were delivered to the students. I identified 320 peer libraries from across the nation who have instruction programs and sent a letter inquiring about the assessment procedures used in their instruction program, and asking them to send a paper or e-mail copy of the assessment tool(s). After the information was collected, the documents were analyzed to look for common themes and ideas.

Assessment is not new to library instruction programs, but methods and theories change frequently. At the Minnesota State University, Mankato Library we needed to update our instructional survey but were not sure how to do it or what types of questions to ask. Our old survey assessed the librarians' style and teaching methods and we wanted to change that emphasis. Our campus, like others across the nation, is interested in gathering data that assess student outcomes rather than assessing the style of the instructor. We wondered how other peer libraries with instruction programs were conducting their assessment.

Our university set aside money for faculty members to conduct special research projects on professional research, teaching, or assessment. This program was valuable for evaluating library instruction assessment activities.

Goals and Objectives

Five goals for the project were outlined, with an objective for each goal.

Goal One: Explore how other peer institutions are using assessment tools in the classroom.

Goal Two: Examine documents and create ideas for assessing our instruction program.

Goal Three: Find common themes of assessment from peer institutions.

Goal Four: Prepare a new assessment survey using the findings from this research.

Goal Five: Enhance the overall library assessment program at Minnesota State University, Mankato.

Literature Review

The body of literature on instruction assessment is rather large. This review is limited to material from 2000 and later, and to articles that focus on student outcomes.

Dugan and Hernon (2002) state that universities and regional accrediting bodies focus on learning results and outcomes rather than whether the student was comfortable during the instruction session or could hear the instructor. There are problems with using outputs as a measure of accountability. The authors claim that outputs are intended to measure the application of inputs and do not measure a students' individual learning. Riddle and Hartman (2000) also claim that outputs do not measure changes in skills or attitudes of the individual. Traditionally, libraries were more concerned about the number of students who attended the class, how effective the librarian was as a teacher, what instructional technology was available, and the content of the instruction. These measures, while they may hold some personal or technical value, do not accurately measure student outcomes. The key for assessment is not descriptive inputs and outputs. Rather, it is answering the question, what did the students learn and how do I know they learned it?

As described by Maughan (2001), the question underlying assessment results is what an institution or program has learned about its student learning. The tradition "How am I doing?" might have value to a personal teaching style and might provide insight to the library's physical surroundings but does not address students and their needs. Maki (2002) suggests that assessment be more than a set of questions. She advocates creating cohort groups that chart the academic progress of students throughout their college careers, or at least for significant segments of those careers. Only then can you assess performance and see improvement. The process requires the library to focus its attention outward and partner with other areas of campus. The cohort model starts by determining who will be assessed, e.g., at risk students, underrepresented students, student of traditional college age, international students, those with higher SAT scores, and/or first generation students. Establish a schedule that may include assessments upon matriculation, at the end of a specific semester, upon completion of a course or program, or even a number of years after graduation. Assign a results interpreter who may be a librarian, teaching faculty member, alumnus, or an assessment committee, learning center, or an academic support service. These cohort groups will use pre- and post-testing, integrated assignments that demonstrate information retrieval skills, and observation to examine the student's progress. Maki does not advocate a questionnaire or a one-shot assessment survey.

Meulemans (2002) claims that measuring information competencies is a means of marketing the overall library instruction program. The University of California-Berkeley conducted a broad survey of students, asking various questions about information retrieval. The conclusion was that students think they know more about accessing information and conducting library research than they are able to demonstrate when put to the test. In response to this, the library increased its assessment program with cohort groups and more accurate survey techniques. They developed broader surveys that focused on the student and stopped the traditional questions about "how am I teaching today?" They focused on success at the moment and canvassed students' reactions several days after the formal classroom instruction. This data has allowed the library to respond to student needs.

A common theme in the literature is a shift from "How am I doing?" to "How are you doing?" as an assessment method. Such assessment tells the librarian what areas or services need to be emphasized, regardless of how the student evaluates the librarian. Accrediting agencies are seeking accountability and student performance. Maki believes that academic libraries need to respond to this shift in their assessment programs.

Methodology

The College Net: comparison search engine was used to identify peer institutions. This search identified institutions that are similar to Minnesota State University, Mankato.

Search criteria:

The search resulted in a list of 251 colleges and universities.

The Academic Library Peer Comparison Tool offered online through the National Center for Education Statistics was used to compare these institutions. Variables compared included staff to student FTE ratio, reference statistics, circulation statistics, and materials and operating budget. This searched yielded a list of 142 comparable institutions. This data was cross-referenced with the CollegeNet data and, after duplicates were discarded, the final list of libraries totaled 314. Other regional academic libraries in Minnesota, Wisconsin, South Dakota, and Illinois were added to bring the total to 320.

The preferred contact for each library was an instruction or reference librarian. The mailing consisted of two pieces; a letter explaining the project (Table 1) and a self-addressed postage paid envelope to return the information. Respondents also had the option of sending the information by e-mail. A database was created to record the response.

Results

General information results from survey:

(Table 2)

(Table 3)

(Table 4)

(Table 5)

Specific data from the returned 57 surveys

The largest number of questions on any library's assessment questionnaire was 50. Two surveys asked just one question. The average number of questions was eight and five libraries allowed a name and e-mail option. The rest were anonymous, with 13 requesting the name of the instructor.

Observations

It is still clear that some libraries continue to ask the "How am I doing?" questions? Most libraries prefer paper over electronic assessment; however, most librarians indicated that they would like to move to electronic assessment. Nine libraries did not assess their instruction program, and the reasons varied from lack of staff to lack of support. For example,

Only one librarian indicated a negative attitude towards assessment,

Those libraries that assess their classes vary in time and complexity. The most popular method is a form used at the end of class either filling it out or some use a machine-readable format. Some assessment forms are not immediately returned to the teaching librarian, for example:

Nine libraries gather assessment data from faculty only. None of them indicated why they assess faculty members; however, eight of the nine assess both students and faculty members. One library uses cohort groups that start with the freshman composition class. A sample of students in each class is monitored through their entire undergraduate career. This is conducted through a series of tests given to the student after a special assignment that assesses their information retrieval skills.

Two libraries gave a pre-test and four a post-test . Two libraries grade these post-tests. The graded tests are included as part of the grade in a freshman English composition course. The two non-graded tests were a requirement to pass the freshman English course. This method requires a good partnership between the English department and the library. One library's post-test is a set of questions with several variations. The students are given different questions sets so that only two or three students in each group share exactly the same assignment. Students work in groups of three. The librarian noted that this method requires a lot of work and updating but their results are more valid because more question sets reduces the chance that a few students do the work and most then would copy the answers.

One library has a one credit course offered for credit on a pass/fail bases. This course does not have a formal assessment tool but the librarian says it uses the results from the class exercises to assess the student outcomes. In this one credit course the students are given the option to test out of the course and earn the one credit. They report that on average 90% of the students take a test-out exam and less than 25% pass the exam.

Finally, nine libraries use electronic transfer to record/conduct their assessment. Six have a website for this purpose and three use machine-readable forms. Of those libraries that still used paper, twelve librarians expressed an interest in converting to an electronic format, e.g., ".we still use paper forms. The librarians would like to go via the web but we do not have a person to do it and our campus computer staff is to busy." Maki (2002) advocates creating cohort groups. Only one library indicated they are currently using cohort groups, but provided no data because they just started the process in the fall of 2002. A few libraries used pre- and post-testing but during a single class period, which Maki does not advocate. None of the libraries that do not perform assessment say it is because they lack money. Apparently, the budget does not effect assessment.

Further Study

Based on the information gathered, the instruction team at Minnesota State University library has started a pilot assessment program with three simple, open-ended questions that can be done via e-mail or in the classroom. The questions are:

1. What are the three most important things you learned during the library session?

2. What questions do you still have about library research?

3. What else should we know to help us improve library sessions in the future?

Each librarian submits the responses to the instruction coordinator who compiles them in a semester report. The instruction team then reviews the semester report and may adjust the instruction program goals.

Conclusion

Libraries are still struggling with assessment. Some have an established program that garners good support from the library or university administration. Some libraries do assessment out of obligation, in a way that may or may not really assess the student's progress. And some libraries, because of the lack of support, staff, or time, do not assess. Librarians still struggle with the question of whether students really know how to use library and information retrieval methods.

Works Cited

Dugan, Robert E. ; Hernon, Peter (2002)."Outcomes Assessment: Not Synonymous with Inputs and Outputs," Journal of Academic Librarianship 28, no.6 (Nov.): 376-80.

Riddle, John S. ; Hartman, Karen A. (2000)."But Are They Learning Anything? Designing an Assessment of First Year Library Instruction at the College of New Jersey," College & Undergraduate Libraries 7, no.2: 59-69.

Maughan, Patricia Davitt (2001)."Assessing Information Literacy among Undergraduates: a Discussion of the Literature and the University of California-Berkeley Assessment Experience," College & Research Libraries 62 no.1 (Jan. ): 71-77.

Maki, Peggy L. (2002). "Developing an Assessment Plan to Learn about Student Learning," Journal of Academic Librarianship 28, nos. 1/2 (Jan./Mar. ): 8-13.

Meulemans, Yvonne Nalani (2002)."Assessment City: The Past, Present, and Future State of Information Literacy Assessment," College & Undergraduate Libraries 9 no.2: 61-74.

Table 1: Copy of letter sent to all peer libraries

January 30, 2003

To the Instruction/Reference Librarian: (Or appropriate name)

Address

I am the Instruction Coordinator at Minnesota State University, Mankato Library and our instruction team is currently reviewing its method of classroom assessment. As part of our review we are collecting ideas from other peer university/college libraries. We would appreciate a copy of your assessment approach to your instruction program especially anything involving lower level undergraduate assessment. Any information would help us create policies, develop outcomes, or create assessment forms used in the classroom. You can mail the information in the enclosed envelope or e-mail as an attachment to jim.kapoun@mnsu.edu by March 7, 2003. Thank you for your consideration and for helping us with information and ideas.

Sincerely,

Jim Kapoun

Instruction Coordinator

Minnesota State University, Mankato

PO Box 8419

Mankato, MN 56002

jim.kapoun@mnsu.edu

enclosures

01/03 jmk

Table 2: Returned survey : Type

Table 3: Assessment Totals from the 57 libraries that responded

Table 4: Assessment Groups from the 57 responses

Table 5: Assessment Tools used by the 57 responses

Table 6: Questions that required a Yes or No response:

Table 7: Questions using a Likert scale

Table 9: Other question types

Question (all had one response)

The pace was too fast / too slow

Do you understand the information presented: Yes / I think so / Not really / Still confused

I feel more comfortable using the Library: Yes / I think so / Not really / Still confused

Rate the Librarian: Excellent / Good / Adequate / Substandard / Poor

Level of presentation: Elementary / High School / About Right / Graduate / PhD

The information was Too advanced / Perfect / Adequate / Too elementary