[RSS] [Google]

homepage

contents

contact us

Library Philosophy and Practice 2010

ISSN 1522-0222

Ability, Assistance, and Collaboration in Academic Library Assessment

Gordon J. Hewitt
Assistant Dean of the Faculty for Institutional Research

Rebecca Troendle Hewitt
Reference Librarian

Hamilton College
Clinton, New York USA

Introduction

Assessment continues to play an important role in the management of academic libraries.   As external political entities press institutions of higher education to be more accountable in regards to academic outcomes and use of resources, libraries are under more pressure to show that that they are an integral part of the academic mission, are using their resources efficiently, and continue to contribute to student learning.

While other aspects of the academic enterprise are under the same pressure, libraries face unique problems.  Continually changing technologies makes planning, implementing, measuring and evaluating systems and processes difficult since they quickly become obsolete.  Ever growing external sources of information are also changing the very nature of the library.  The proliferation of information sites on the Internet creates vast reservoirs of accessible information, but makes the mission of information literacy extremely important since users need to decipher what is credible information.  And, while these issues create complexities for academic library managers, they also create a challenging environment for conducting assessment.  It is important for decision-makers to understand what students are learning as well as how they are learning.  With competing external forces, it is essential to show how students are learning through the efforts of the library and related programs.

The need for this study was conceived while one of the authors served on the search committee for the library director position at a private liberal arts college.  The candidates who came to campus for interviews described a wide range of involvement in assessment activities and a wide range of levels of assistance received in conducting assessment from library staff and other campus offices.  It was not clear what expectations these potential leaders had for structures or standards of assessment. 

Literature Review

Assessment in academic libraries is nothing new.  The collection of statistics on university libraries dates back to 1908 (Blixrud , 2001), and throughout the years the reporting of a wide-range on statistics regarding the library, from collection counts to staff members to expenditures, has been standard practice at institutions of higher education.  With the change to electronic processes throughout the library over the past few decades, the availability of data on a wide range of activities – including direct user input -- has been in abundance and has provided library managers with a vast array of data in which to inform decision-making  (Hiller and Self, 2004).

In the past decade, however, the call for assessment in higher education has gone beyond the standard measurement of inputs and resources to a broader need to assess outcomes, or what students actually learn while in college, and libraries must to respond to that need.  The literature on outcomes assessment in libraries has attempted to address this call.  Several have written of the importance of academic libraries to develop a “culture of assessment,” to create library assessment plans, and on the key role data collection and analysis plays in assessment (Hiller and Self, 2004; Matthews, 2007).   Peter Hernon and Robert Dugan outline the process of conducting outcomes assessment in libraries, with great detail on the research process (Hernon and Dugan, 2002).  Beyond traditional managerial assessment, Nancy Fried Foster and Susan Gibbons describe an actual ethnographic study of student use of the library at the University of Rochester.  While this method is more academic in nature, librarians were intricately involved in the research process and ultimately learned a great deal about both data collection and analysis procedures as well as about the students themselves (Fried, Foster and Gibbons, 2007).

 While much of the literature has addressed both the philosophical underpinnings of academic library assessment and the technical and methodological aspects of such research, there have been no empirical studies addressing the resources needed, or used, to conduct assessment.  There is little in the literature on what specific skills a library professional needs to conduct assessment research, and there is nothing on the use of campus resources, such as research or assessment professionals.  Our study was undertaken to begin to fill this void.

Methodology

An email message was sent to the library directors of 117 libraries at liberal arts colleges that are members of either the Higher Education Data Sharing Consortium (HEDS) or the Oberlin Group of Libraries (Oberlin Group).  (HEDS consists of private colleges and universities and assists member institutions in planning, management, institutional research, decision-support, policy analysis, educational evaluation, and assessment, primarily through the facilitation of data and information sharing.  The Oberlin Group is a consortium representing libraries at selective liberal arts colleges in matters of information and resource sharing and other areas of common concern.) Seven of the emails were returned as undeliverable, for a final sample of 110.  The email message, which was followed by a reminder two weeks later, asked library directors to complete an online survey or to forward the link to the survey to someone else at their library who may have assessment responsibilities.  A total of 54 surveys were completed, for a response rate of 49.1%.  Of those who responded, 87.1% identified themselves as library directors, 3.7%  as chief information officers (CIO), 3.7% as combined library and information technology directors, 3.7% as other library staff with assessment responsibilities, and 1.9% as an unidentified “other” (see Table 1).

Table 1.  Job titles of respondents

  %
Library director 87.0
Chief information officer (CIO) 3.7
Library and IT director 3.7
Other library administrator with assessment responsibilities 3.7
Other 1.9

The survey instrument was constructed based on three primary questions.  The question “How important is assessment in your job?” was addressed with items requesting information on the degree to which assessment is an important part of their job and with whom they collaborate on assessment projects.  To answer the question “Do you have the skills to conduct assessment?” a five-point scale was  included for the respondent to identify the relative strength of their own ability and the abilities of their staff in carrying out a series of assessment-related activities.  Finally, items addressing the type and degree of contact the respondent has had with their campus institutional research or assessment office were incorporated to answer the question “To what extent do you use assessment and research resources outside of the library?” The content validity of the instrument was developed through extensive feedback from current academic library directors, professional librarians with assessment expertise, and an institutional research professional with experience in library assessment.

Results

Importance of Assessment

As shown in Table 2, most respondents felt that assessment was one of many important aspects of the job (72.2%), while a small but significant minority felt it was only a small part of their job (24.1%).  When asked with whom they work to carry out assessment activities, almost all (98.1%) identified other library staff, and a significant majority (79.6%) identified the campus institutional research, planning or assessment office.  Other listed sources of collaboration – information technology, individual faculty or academic staff, and campus accreditation or assessment committees – were identified between 35-40% of the time.  Thirty-seven percent also responded that they conduct assessment activities by themselves.  (See Table 3).

Table 2.  The extent to which assessment and evaluation is part of job

  %
Primary focus of job 0.0
One of many important aspects of job 72.2
Small part of job 24.1
Not involved with assessment or evaluation on a regular basis 3.7
Not part of my job 0.0

Table 3.  Collaborators in assessment projects

  %
Other library staff 98.1
Campus institutional  research/planning/assessment office 79.6
Information technology services 38.9
Individual faculty/academic staff 38.9
I do them myself 37.0
Campus accreditation/assessment committee 35.2

Requisite Skills

Table 4 shows how the respondents rated their own ability and their staff’s ability to conduct various assessment-related activities.  In rating their own ability, respondents felt their strongest abilities were the conceptual skills of “developing a research question” (79.6% very or somewhat strong) and “recognizing the need to assess a certain activity” (70.3% very or somewhat strong).  The abilities in which they felt were their weakest were two important components in conducting qualitative research: “analyzing qualitative data” and “developing interview questions” (25.9% very or somewhat weak for both).   

The respondents’ evaluation of their staff show that they are most confident about their abilities to “report findings”  (60.3% very or somewhat strong) and “recognize the need to assess a certain activity” (48.2% very or somewhat strong), and least confident about their abilities to “develop a survey instrument” (20.4% very or somewhat weak ) and “select a research activity (18.6% very or somewhat weak).  Respondents were more inclined to rate their staff as average on most items than they did themselves.

 Table 4.  Evaluation of abilities of self and library staff to conduct assessment activities

Your own ability:

Very strong

%

Somewhat strong

%

Average

%

Somewhat weak

%

Very weak

%

Recognizing the need to assess a certain activity 25.9 44.4 29.6 0.0 0.0
Developing a research question 38.9 40.7 20.4 0.0 0.0
Selecting a research methodology 13.2 41.5 30.2 13.2 1.9
Developing a survey instrument 9.3 31.5 40.7 14.8 3.7
Developing interview questions 7.4 27.8 38.9 18.5 7.4
Analyzing quantitative data 9.3 29.6 46.3 11.1 3.7
Analyzing qualitative data 16.7 20.4 37.0 22.2 3.7
Reporting findings 16.7 38.9 37.0 5.6 1.9
Your staff’s ability:          
Recognizing the need to assess a certain activity 5.6 42.6 48.1 3.7 0.0
Developing a research question 1.9 31.5 51.9 13.0 1.9
Selecting a research methodology 3.7 25.9 51.9 13.0 5.6
Developing a survey instrument 3.7 29.6 46.3 14.8 5.6
Developing interview questions 7.5 34.0 47.1 9.4 1.9
Analyzing quantitative data 7.4 24.1 51.9 16.7 0.0
Analyzing qualitative data 7.4 38.9 40.7 13.0 0.0
Reporting findings 7.5 52.8 34.0 3.8 1.9

 

Use of Campus Research and Assessment Resources

As shown in Table 5, a little over half (51.9%) of the respondents had contact with their institutional research, planning, or assessment office several times per year, and another 35% had contact a few times per year.  Only 11% had infrequent contact, and no respondents had no contact.  Only 2 respondents (1.9%) were unaware of any such office on their campus.

The nature of the contact with the campus IR/planning/assessment office varied.  Sixty-five percent of the respondents contacted the office in order to plan or implement an assessment or evaluation project, and 70% or more of the contact was related to assistance with a specific aspect of a project, regular data requests, or ad hoc data requests (See Table 6).

Table 5. Rate of contact with institutional research/planning/assessment office

  %
Several times per year 51.9
A few times per year 35.2
Infrequently 11.1
Never 0.0
Unaware of any such office on campus 1.9

Table 6. Percent who contact the institutional research/planning/assessment office regarding the following activities:

  %
Planning  and/or implementing assessment and evaluation projects 64.8
Assistance with specific aspects of a project 70.4
Regular or cyclical data requests 72.2
Ad hoc data requests 77.8
Other 1.9

Conclusions

As expected, a large majority identified assessment as one of many important aspects of their job, but almost one-quarter stated that it was only a small part of their job.  And, while it is not surprising that almost all respondents relied on their staff to help them with assessment activities, the extent to which respondents already use campus research and assessment resources was a surprise.  They are using them not only for specific assessment activities but also for other data-related activities. Since the literature doesn’t address the use of campus resources, it was unknown to what degree library professionals were aware of – and using – those resources.  Our results confirm that that there is a high degree of awareness of campus research and assessment resources among academic library directors.

The rating of assessment and research skills shows that the respondents (mostly library directors) rate themselves higher overall in regards to strength of skills than their staff, and they are most confident about their own conceptual skills.  They are less confident in their methodological skills, especially when it comes to qualitative research. 

Limitations and Further Research

This study was limited to smaller, private, mostly liberal arts colleges.  Most of the colleges have no graduate programs, and those that do have relatively small programs.  While there are commonalities between library services at these colleges and larger research universities, it would be hard to generalize these findings to those institutions since library organizations at smaller colleges are different as are the responsibilities of the library administrators and staff.  For example, libraries at larger institutions often have larger staffs with professionals and administrators who specialize in assessment, where at small colleges the library staffs are smaller and have to cover a wider range of duties.

As this was an initial attempt to provide empirical evidence on the skills and resources needed by academic librarians to conduct assessment, there are plenty of opportunities for further research.  A larger and more diverse sample of institutions would provide more robust findings and make them more generalizable to different types of institutions, such as public and research universities.  More in-depth analysis of the nature of assessment work in academic libraries, through interviews or focus groups, would provide more context and richer data.  It would also be useful to understand the perspective of other campus constituents who help with, or have a stake in, library assessment, such as faculty members, institutional researchers, and other academic administrators. 

References

Julia C. Blixrud, “The Association of Research Libraries Statistics and Measurement Program: From Descriptive Data to Performance Measures,” Proceedings, 67th IFLA Council and General Conference, August 16-25, 2001.

N. Fried Foster and S. Gibbons (Eds.), Studying Students (Chicago: Association of College and Research Libraries, 2007).

P. Hernon and R. Dugan, An Action Plan for Outcomes Assessment in Your Library (Chicago: American Library Association, 2002).

Steve Hiller and James Self, “From Measurement to Management: Using Data Wisely for Planning and Decision-Making,” Library Trends 53 (Summer 2004): 129-155.

Joseph R. Matthews, Library Assessment in Higher Education (Westport, CT: Libraries Unlimited, 2007).

homepage

contents

contact us