|
Exploring Examination Equity Issues for Certified Rehabilitation Counselor Candidates who are Deaf or Hard of Hearing By Christine Reid, Ph.D., CRC, Commissioner and Project Coordinator and Patricia Nunez, MA, CRC, CDMS, CCM, Commissioner and Examination & Research Committee Chair Commission on Rehabilitation Counselor Certification (CRCC)
www.crccertification.com The Certified Rehabilitation Counselor (CRC) certification examination is increasingly the national standard in the field and is being recognized by many states in their licensure laws. Attaining CRC certification influences the career paths of many rehabilitation counselors, including their ability to obtain pay raises and promotions. For these reasons, the CRC has become a “high stakes” certification examination. As the CRC is increasingly recognized as a high stakes examination, questions have surfaced about test equity issues for individuals who are deaf or hard of hearing. In recent years, CRCC has intensified its study of accommodations offered for the CRC examination, including attention to potential language barriers within the examination itself. As will be discussed in greater depth in this report, individuals who have not had full access to spoken English generally have not experienced the language’s many meanings and nuances (for additional information see also Mounty & Martin, 2005.) For these individuals, including those for whom American Sign Language (ASL) may be their primary language, the construction of written tests becomes a paramount concern. To explore issues regarding the CRC as a high stakes examination and test equity, the Commission on Rehabilitation Counselor Certification (CRCC) convened an Advisory Panel of individuals with expertise and national recognition for their contributions in matters relating to individuals who are deaf or hard of hearing. The meeting resulted in several recommendations being put forth to be explored further by the Commission’s Examination & Research Committee. Those steps (which will be outlined in greater detail in this report) include:
As a result of convening the Advisory Panel meeting, the CRCC Examination & Research Committee will also conduct future item-writing workshops and review of test items with increased sensitivity to item construction as it relates to deaf or hard of hearing individuals. Item writing guidelines will be reviewed to enhance item accessibility for all candidates, including deaf and hard of hearing individuals and candidates for whom English is not their first or primary language, while maintaining the psychometric integrity of the examination. The CRCC wishes to thank the participants in the Advisory Panel who gave of their time and expertise to explore the issues surrounding high stakes testing and test equity for deaf or hard of hearing individuals with compassion, fairness, and understanding. Discussion: The CRC Certification as a “High Stakes” Examination The Commission on Rehabilitation Counselor Certification (CRCC) convened an Advisory Panel to discuss test equity issues for individuals who are deaf or hard of hearing with regard to the Commission’s Certified Rehabilitation Counselor (CRC) examination. The CRC examination has been given since the Commission was established in 1974. Increasingly, the CRC examination has become a “high stakes” test for rehabilitation counselors in the field because many states recognize the CRC exam in their licensure laws, and many employers require CRC certification for their employees. Additionally, many state vocational rehabilitation agencies recognize CRC certification as an important standard for defining who is a “qualified” rehabilitation counselor under federal Comprehensive System of Personnel Development (CSPD) mandates. Members of the Advisory Panel—including those who work with and/or supervise rehabilitation counselors who are deaf or hard of hearing—raised concerns about the test equity of the CRC examination for people who have hearing impairments. They pointed to failure rates among deaf or hard of hearing rehabilitation counselors, which are known to them anecdotally, largely from incidents involving their own staff members or students of their programs. (To respect the privacy of applicants, the CRCC does not ask applicants who sit for the test if they are deaf or hard of hearing, or possess any other disabilities. The only data to which CRCC is privy in this regard are from those who request specific accommodations.) The concern with the CRC examination for individuals who are deaf or hard of hearing is one of language. People who were born and raised with a significant hearing loss have not had the same opportunity to learn English auditorially, as do hearing individuals. These individuals have not had the ability to hear and learn the nuances and multiple meanings of spoken English in contexts such as conversation, although they may be highly literate in reading and writing (see also Mounty & Martin, 2005.) For many individuals who are deaf, American Sign Language (ASL) is their primary language while English is used as a second language. Given these concerns, the Advisory Panel aired several questions for discussion, such as: How does a lack of access to auditory language present an impediment to understanding the constructs measured by questions on high-stakes examinations, such as the CRC? Does a language barrier exist for rehabilitation counselors who are deaf or hard of hearing because they do not understand the questions or what the questions are asking of them? As a result, does their performance on the CRC certification test not accurately reflect their knowledge, skills, and competency? As these questions were explored, information, opinions, and suggestions were gathered with the ultimate goal of helping the CRCC to take appropriate, reasonable steps to make the CRC examination fair and accessible. The panel discussed test equity from the perspective of the entire examination process: from test development—including the writing and review of test items—to the administration of the examination. The panel discussion culminated in the writing of this white paper, which will serve multiple purposes: 1. The findings will be presented to the CRCC Examination & Research Committee for consideration. The Examination & Research Committee would then present any recommendations for possible further study and/or action to the full Commission for consideration. 2. This white paper will be available for distribution to policymakers and other interested parties to inform them about current issues surrounding test equity for individuals who are deaf or hard of hearing. 3. The Advisory Panel participants will receive copies of the report to help further the discussion and to work collaboratively toward greater understanding of all the issues related to test equity and the CRC examination. The Timing of the CRCC Advisory Panel Discussion The CRCC has intensified its study of test equity and accommodations for individuals with disabilities for several reasons. First, the Commission is migrating to a computer-based test beginning in the fall of 2008, which will be a major change in how the examination is administered. (Extended time for taking the exam has been offered as an accommodation upon request for individuals who qualify, including those who are deaf or hard of hearing, and will continue to be offered in the computer-based format.) The computer-based test will remain a linear examination, versus transitioning to an adaptive format. The CRCC recognizes that migration to a computer-based test format is the opportune time to explore examination issues related to test equity, including reviewing accommodations that are offered to qualified individuals upon request on a case-by-case basis. As the new examination is written and reviewed, it is particularly advantageous to review the existing item bank and to explore the content and structure of the test, especially with regard to language. Exploration of accommodations and test equity is an extension of ongoing efforts of the CRCC Examination & Research Committee, which over the past few years has sought to improve accessibility for all people with disabilities. One of the final issues identified by the committee has been whether accommodations offered to people who are deaf or hard of hearing are adequate, or if additional accommodations need to be made available. As a result of the Advisory Panel meeting, the CRCC affirmed that it will proactively reach out to test-takers to inform them that they may request appropriate accommodations without penalty. One factor that should be considered when discussing performance of any examinees is that the CRC certification examination is scored conjunctively. This means that in order to pass the examination, applicants must achieve a passing score on two separate parts: one related to counseling and the other to rehabilitation. Conjunctive scoring is necessary to test for minimum competency in both counseling and rehabilitation, which is required in the broad practice of rehabilitation counseling. Individuals practicing in the field may be able to compensate for a weakness in one area (e.g. in counseling) with a strength in another (e.g. rehabilitation). With conjunctive scoring, however, individuals must demonstrate a minimum competency in both areas in order to pass the examination and become certified. This could potentially lead to a higher failure rate among those who are stronger in one area than in another. A baseline of competency in rehabilitation and counseling is necessary for all individuals in the field, including those who specialize in a particular area, such as Rehabilitation Counselors for the Deaf (RCDs). In addition, RCDs and those who practice in other specialty areas must possess other skill sets, for which the CRC does not test. Nonetheless, these special skill sets are in addition to the foundation of rehabilitation counseling, which is reflected in the CRC examination. Skill sets necessary for RCDs include sign language fluency, cultural competence in working with clients who are deaf or hard of hearing, and other competencies outlined in the current version of the Model State Plan (Watson, 1990). It should be noted that at the time of the Advisory Panel meeting, an update of the Model State Plan was in preparation, which participants described as including further attention to specific competencies necessary for RCDs. As the CRC migrates to computer-based testing, one suggestion made by the Advisory Panel, and proposed for consideration by the CRCC Examination & Research Committee, is to allow test-takers to mark certain questions for additional review, and return to those questions after answering others. The panel members explained that especially for individuals who are deaf or hard of hearing, being allowed to go back and review certain test items would be important, whether that becomes a standard feature of the examination or is allowed as a requested accommodation. The CRC Examination as a “High Stakes” Examination Another critical component in the timing of the test-equity discussion is the fact that the CRC examination is increasingly the national standard in rehabilitation counseling. In more than a dozen states, the CRC exam is recognized in general counselor licensure laws. Further, the federal Rehabilitative Services Administration requires that state vocational rehabilitation agencies establish qualifications for personnel to assure that they are adequately prepared and trained. If a state does not have its own certification, licensing, or registration requirements for rehabilitation counselors, then the state vocational rehabilitation agency must base its personnel standards on CRC eligibility requirements. Advisory panel members described scenarios in their states, such as Because of the salary and career consequences connected with the passing of the CRC, it is known as a “high stakes” certification examination. Such high-stakes testing has become more common “as a condition of promotion, continuation, or graduation for elementary and secondary school students, and for licensure and certification of a wide range of professionals” (Mounty & Martin, 2005). The concern expressed by several members of the Advisory Panel is whether the CRC certification examination, as a high-stakes credentialing test, poses an unintentional barrier because of language issues for counselors who are deaf or hard of hearing. Participants stressed that rehabilitation and vocational counselors who are deaf or hard of hearing should be held to the same minimum competency standards in rehabilitation and counseling, which the CRC examination is designed to demonstrate. The key issue, however, is whether the language of the test—and specifically how questions are worded and structured—is an impediment for individuals who are deaf or hard of hearing, including those whose primary language is ASL. With regards to the CRC examination, the advisory panel discussed whether it presents a “double cognitive” challenge for deaf or hard of hearing individuals: that they are being tested not only on their knowledge of the field, but also on their knowledge of English, which is a particular challenge to those who do not have auditory access to this language. As one advisory panel member noted: “My main concern is how questions are written and how the language decisions are made…not just the vocabulary, but the syntax that is used and the use of redundancy in the content.” Because of the language issue for individuals who are deaf or hard of hearing, several members of the Advisory Panel questioned whether states should use the CRC as a determinant for personnel standards. Although panel participants acknowledged that the CRC does test for minimum competency in the field of rehabilitation counseling, they observed that the examination is not intended to demonstrate the required knowledge and skill sets for specialties within the field, such as RCDs. RCDs, who may also be deaf or hard of hearing, must possess other specialized skills such as manual communication, knowledge of Deaf Culture, and so forth, as detailed in the Model State Plan (Watson, 1990). It should also be noted that some rehabilitation counselors who are deaf or hard of hearing have a general case load that is consistent with the competencies represented by the CRC credential. As an independent certification organization—held to high standards by its accreditation by the NCCA—CRCC has no role in determining or influencing states’ use of the CRC examination for licensure and/or personnel standards. However, the CRCC does acknowledge the language issue for deaf or hard of hearing counselors and urges careful consideration of employment requirements for these individuals with regard to the CRC certification, particularly during an interim period as the Commission assesses equity in its testing. Advisory panel members also recommended as a resource the Model State Plan for the Rehabilitation of Individuals who are Deaf and Hard of Hearing, developed by the University of Arkansas Rehabilitation Research and Training Center on Deafness and Hearing Impairment and the CSAVR Standing Committee on Services for Individuals Who are Deaf, Hard of Hearing, Late-Deafened, and Deaf-Blind. The Model State Plan, which is currently under revision, “offers standards in the development of rehabilitation services for individuals who are deaf and hard of hearing.” The objective of the plan is to “assist VR agencies in planning and developing strategies which would more effectively address the needs of those seeking VR services; identify exemplary practices that may be recommended for use by state VR agencies and other professionals; and offer a format for staff development and rehabilitation training programs which increase the knowledge and awareness of standards for VR services to clients” who are deaf or hard of hearing. (Watson, 1990.) Possible Action Steps Under CRCC Consideration As a result of the discussions of the Advisory Panel, recommendations will be made to the CRCC’s Examination & Research Committee. They include:
The Examination & Research Committee may also review its accommodations to see if additional offerings should be made and, specific to the population of deaf or hard of hearing professions, whether ASL interpretation of questions should be pursued. Members of the Advisory Panel discussed problems with offering an ASL version of the exam, including: 1) Minimally, the interpretation must be done by a qualified interpreter who fully understands the subject areas and can accurately translate the subtleties of each question. Optimally, a team approach to translation would be most effective, including test developers, bilingual interpreters, persons with linguistic/translation backgrounds, and deaf or hard of hearing professionals who are familiar with testing and are bilingual themselves. 2) Accuracy of the interpretation must be assessed through “reverse translating” the ASL version back into English. Subject matter experts must then compare the original English version to the “reverse translated” version. If they differ, the ASL translation should be revised, then “reverse translated” and compared again to the original, until comparability is demonstrated. 3) The only way to allow such checks on accuracy of interpretation is to videorecord each item on the exam. Given that the exam changes at least twice per year, this would be a significant recurring expense. 4) Not every individual who is deaf or hard of hearing is a fluent user of ASL. Offering an ASL version of the exam would not address the language accessibility concerns of people with hearing impairments who do not use ASL. To properly research the effectiveness of any ASL version of the CRCC exam, it would be important to also have an independent measure of the ASL fluency of examinees taking that version of the test. Otherwise, insufficient command of ASL could be a significant barrier to reliable and valid assessment of the CRC content measured in an ASL version of the exam. Accommodations already provided by the CRCC for individuals with disabilities who qualify include: audio rendition of the examination, Braille format exam, enlarged print examination, sign language interpreter to interpret test instructions and facilitate communication with test center personnel, copy of the monitor’s instructions, scribe to record answers, ability to record answers in the book, extended testing time, extended break time, separate room (particularly for extended testing time), and adjustable height tables.
Members of the Advisory Panel asked whether it would be possible for CRCC to develop specialized subsets of questions pertaining to specific areas of practice, such as RCDs. Technical difficulties with this approach, largely related to the potentially small numbers of examinees taking each subset of questions, were addressed by CRCC’s psychometrician. The CRC is designed to test competency in rehabilitation and counseling and is based on rigorous field research. Developing subsets of questions or additional types of specialty certifications would not be supported by a large enough population of potential certificants to be statistically valid. Further, such endeavors would be cost prohibitive for the Commission to pursue.
Conclusion The CRCC Advisory Panel brought together leaders in the rehabilitation counseling field and experts on test equity and language issues for individuals who are deaf or hard of hearing, establishing an important dialogue to explore the concerns and consequences of the CRC certification as a high stakes examination. The meeting resulted in many concrete steps that the CRCC will explore, such as the convening of a panel of professionals who are deaf and hard of hearing to review items in its test bank, providing training to item writers in order to reduce linguistic barriers within the exam, and participating in the development of exam preparation materials in multiple formats. In addition, the meeting paved the way for future discussions and collaboration to ensure that the CRC certification, as a recognized national standard, maintains the highest integrity for examination content, fairness, and accessibility. CRCC Advisory Panel Meeting Participants Glenn B. Anderson, Ph.D., CRC, Professor and Director of Training at the University of Arkansas Rehabilitation Research and Training Center for Persons who are Deaf or Hard of Hearing Timothy Beatty, Program Manager for Deaf & Hard of Hearing Services, California Department of Rehabilitation Katherine J. Bruni, Outreach Coordinator, Georgia/Florida State Outreach and Technical Assistance Center, PEPNet-South, and English instructor to students who are deaf or hard of hearing, Georgia Perimeter College, Center for Disability Services Deirdre N. Burchette, M.S., CRC, Vocational Rehabilitation Specialist for DC Rehabilitation Services Administration, in Patty C. Conway, CRC, retired Branch Manager, Deaf & Hard of Hearing Services, Kentucky Office of Vocational Rehabilitation Judith L. Mounty, Ed.D., MSW, Rebecca Sills, MA, CSC, Director, Georgia VR Deaf Services Program, Department of Labor/Vocational Rehab Services, Barry C. Stephens, Ph.D., CRC, Associate Professor – School of Professional Counseling, Lindsey Wilson College, and representative of ADARA Robert Weinstock, Director of the Gallaudet University National Academic Bowl for Deaf and Hard of Hearing High School Students, and contributing author to Assessing Deaf Adults: Critical Issues in Testing and Evaluation Also participating: Christine Reid, Ph.D., CRC, Commissioner and Project Coordinator, CRCC; Rehabilitation Counseling Professor at Patricia Nunez, MA, CRC, CDMS, CCM, Commissioner, Vice Chair of the Commission, and Chair of Examination & Research Committee, CRCC; current President of the American Rehabilitation Counseling Association, and manager for case management at CNA Insurance Company F. Jay Breyer, Ph.D., Executive Director, Psychometrics, Prometric Kathy Henrichs, facilitator References: Mounty, J., & Martin D. (Eds.). (2005). Assessing Deaf Adults: Critical Issues in Testing and Evaluation. Watson, D. (Ed.). (1990). |
