Table of Contents
Volume 40(1), 2019
3 Return on Investment for State Vocational Rehabilitation Programs: Introduction to the Special Issue
Robert M. Schmidt, Kevin Hollenbeck, and Kirsten L. Rowe
5 What is ROI?
Kevin Hollenbeck
Abstract: An investment is a transaction in which an investor exchanges resources that the investor owns or controls for an asset that is expected to have value in the future. One type of investment is a vocational rehabilitation (VR) agency providing an individual with services, with the expectation that the individual will experience positive labor market or other lifestyle outcomes. The return on investment, ROI, for such an investment is calculated from the net payoff (benefits minus costs) that occurs as a result of the investment. This article lays out the basic principles used in calculating ROIs, and discusses how the ROIs for VR differ depending on the perspective: the VR agency, the individuals served, or society.
11 Social Return on Investment: An Important Consideration for State Vocational Programs
Betsy Hopkins
Abstract: A more traditional method of estimating the ratio of net benefits and costs or a return on investment (ROI) is one way for agencies to indicate the financial benefits that these programs realize. In contrast, social return on investment (SROI) is a relatively new approach that can be used to help determine the value of social benefits gained by individuals with disabilities who obtain competitive integrated employment (CIE). This paper will discuss and give examples of the benefits of specific SROI methods in assessing employment outcomes for individuals with disabilities, which may be particularly important in estimating the overall value of the public vocational rehabilitation (VR) program.
17 The Return on Investment of Vocational Rehabilitation Services: Some Ethical Considerations
Robert Froehlich, Olivia Bentley, Diona Emmanuel, and Maureen McGuire-Kuletz
Abstract: Ethical codes and principles guide research and practice. From the feedback provided by its Advisory Council, the Vocational Rehabilitation Return on Investment (VR-ROI) Project recommends eight ethical considerations to guide the analysis, interpretation, and application of ROI data. These considerations are grounded in various principles and sections of the Commission on Rehabilitation Counselor Certification’s (CRCC) Code of Professional Ethics for Rehabilitation Counselors. The CRCC Code and suggested ethical decision-making tools provide a framework for Vocational Rehabilitation administrators and decision-makers.
23 Conceptual Issues in Developing Return on Investment Estimates of Vocational Rehabilitation Programs
(Link to ungated pre-publication version)
Christopher M. Clapp, John V. Pepper, Robert M. Schmidt, and Steven N. Stern
Abstract: We provide an overview of the basic conceptual issues involved in estimating the return on investment (ROI) of state vocational rehabilitation (VR) programs. Our aim is to highlight some of the key issues in ROI evaluations, especially those associated with estimating the benefits and costs of VR. Finally, we discuss different ways of implementing ROI calculations and suggest that rate-of return type analysis is appealing for VR evaluations where there is no widely accepted discount rate.
35 Data Issues in Developing Valid ROI Estimates
(Link to ungated pre-publication version)
Steven N. Stern, Christopher M. Clapp, John V. Pepper, and Robert M. Schmidt
Abstract: This paper discusses issues associated with using readily available administrative data in estimating ROI for vocational rehabilitation services. It starts with a discussion of longitudinal outcomes data. The discussion is divided up into labor market outcomes data, other types of outcomes data, necessary sample sizes (power analysis), and ways to deal with people systematically excluded from the outcomes data. Next, the paper focuses on services data. The topics covered include the need for control groups, using service cohort data, different sources of service, and merging service data with outcomes data. Finally, the paper moves to the need for other controlling explanatory variables including discussions of inclusion of demographic explanatory variables and data from local labor markets. Two online appendices to this paper provide additional details through (a) an example of a power analysis to illustrate sample size issues and (b) a discussion of Institutional Review Board issues associated with conducting empirical investigations using administrative data.
47 An Overview of the VR ROI Project and its Approach to Estimating ROI
Kirsten L. Rowe, Joseph Ashley, John V. Pepper, Robert M. Schmidt, and Steven N. Stern
Abstract: The public vocational rehabilitation program (VR) has faced increasing demands to demonstrate its effectiveness in recent decades. Consistent with these rising expectations, the 2014 Workforce Innovation and Opportunity Act (WIOA) introduced new performance accountability requirements, one of which extends agency tracking of employment from the time of case closure to a full year after closure. The VR-ROI Project supplements these requirements by developing and testing a model to address questions of long-term VR impacts and cost-effectiveness. This paper provides an overview of the VR-ROI Project before describing seven key features of the model. In the process, the paper illustrates why rigorous ROI estimates require a partnership between researchers/evaluators well versed in appropriate statistical procedures and VR agency personnel. Through that partnership, the proposed approach offers methodologically sound and feasible strategies for state VR programs to demonstrate their employment impacts and effectiveness for people with disabilities.
57 Applications of the VR-ROI project: ROI estimates for Virginia and Maryland
(Link to ungated pre-publication version)
Robert M. Schmidt, Christopher M. Clapp, John V. Pepper, and Steven N. Stern
Abstract: This paper briefly describes and then implements the VR-ROI (Vocational Rehabilitation Return on Investment) Project’s model for applicants in state fiscal year 2007 to Virginia’s Department for Aging and Rehabilitative Services (VA DARS) and to Maryland’s Division of Rehabilitation Services (MD DORS). We present results that account for differences across disability, agency, VR service type and source, applicant characteristics, and county as well as national economic conditions. This approach provides a rich set of estimates that display considerable heterogeneity within each agency across three disability types (mental illness, physical impairment, and cognitive impairment) and seven service categories (diagnosis & evaluation, training, education, restoration, maintenance, job placement, job supports) within each disability type. Five online appendices providing additional detail on the model and results can be found at https://scholarship.richmond.edu/economics-faculty-publications/56/
73 Extending the VR-ROI Approach to Measure the Return on Virginia’s Investment in the Public Workforce System
Jeffrey Brown and Joseph Ashley
Abstract:This article discusses recent efforts to evaluate the return on investment (ROI) of Virginia’s public workforce development system and provides examples of similar efforts in other states. It considers various challenges in attempting to meet the common expectation of a measurable return on public investments in the workforce system for both participants and taxpayers, and argues for extending to other workforce programs the approach being used in Virginia to develop ROI estimates for the state vocational rehabilitation program.
Introduction to the Special Issue
Robert M. Schmidt
University of Richmond
Kevin Hollenbeck
Upjohn Institute
Kirsten L. Rowe
Virginia Department for Aging and Rehabilitative Services
Return on investment (ROI) and cost-benefit analyses of the public vocational rehabilitation(VR) program in the United States have a long history, from the first published studies by Bellante (1972), Conley (1969), and Worrall (1978) to a number of more recent publications (see, for example, Bua-Iam & Bias, 2011; Bua-Iam, Hampton, Sink, & Snuffer, 2013; Cimera, 2010; Dean, Pepper, Schmidt, & Stern, 2015, 2017; Schmidt et al., 2019 (this issue); Uvin, Karaaslani,& White, 2004; Wilhelm, 2013).While they have taken different approaches, existing VR-ROI studies have attempted to accurately estimate the full cost of VR as well as its economic benefits for program participants and, in some cases, for employers and taxpayers. VR-ROI estimation methods have become increasingly more rigorous in recent decades, and have increasingly focused on state-specific program impacts, as access to state-level administrative data on VR program participants, the VR services they receive, and their employment histories has increased.
The work currently being carried out by the Vocational Rehabilitation Return on Investment (VR-ROI) Project – a Disability and Rehabilitation Research Project funded by the National Institute on Disability, Independent Living and Rehabilitation Research – represents perhaps the most rigorous approach to estimating ROI for state VR programs. The purpose of this special issue is threefold:
- To highlight conceptual, methodological and ethical considerations for VR program administrators and others interested in developing and using VR-ROI estimates;
- To provide an overview of the VR-ROI Project and key features of its approach to estimating ROI for state VR programs; and
- To describe the results of recent VR-ROI Project analyses of the VR programs in two states.
The first article, by Hollenbeck, offers an introduction to basic economic concepts and calculations used in estimating ROI and applies them to the VR context. It is followed by Hopkins’ article on social ROI and its potential importance for the public VR program. Froehlich, Bentley, Emmanuel, and McGuire-Kuletz then discuss ethical considerations for policymakers and practitioners in using the results of ROI analyses.
These articles are followed by Clapp, Pepper, Schmidt, and Stern’s discussion of conceptual issues in developing ROI estimates for state VR programs and Stern, Clapp, Pepper, and Schmidt’s discussion of data issues in developing valid VR-ROI estimates. Rowe, Ashley, Pepper, Schmidt, and Stern then provide an overview of the VR-ROI project and its approach to estimating ROI, and Schmidt, Clapp, Pepper, and Stern describe the results of the project’s recent analyses of program impacts and ROI estimates for the state VR programs in Virginia and Maryland. The final article in this special issue by Brown and Ashley considers ROI estimation in the broader context of evaluating the performance of state workforce development systems.
As discussed by McGuire-Kuletz and Tomlinson, government agencies of all types are being asked by elected officials, the public, and the media: Journal of Rehabilitation Administration, 40(1), pp. 3–4, 2019 What kind of positive impact is being delivered by your agency’s service, and how much are those services costing taxpayers? Return on in- vestment (ROI) studies are one of several powerful tools for vocational rehabilitation (VR) to demonstrate relevance (2015, p. xii).
Our hope is that the articles in this special issue will serve as a resource for VR program administrators and others interested in making effective use of this powerful tool.
References
- Bellante, D. M. (1972). A multivariate analysis of a vocational rehabilitation program. Journal of Human Resources, 7(2), 226–241. doi:10.2307/ 145185
- Bua-Iam, P., Hampton, J. E., Sink, T., & Snuffer, D. W. (2013). Critical issues unique to the selection of a realistic return on investment method for public vocational rehabilitation programs. Journal of Rehabilitation Administration, 37(1), 25–33.
- Bua-Iam, P., & Bias, T. K. (2011). Estimating worklife return on investment of WVDRS youth and older consumers with significant disabilities. Journal of Rehabilitation Administration, 35(1), 17–26.
- Cimera, R. E. (2010). The national cost-efficiency of supported employees with intellectual disabilities: the worker’s perspective. Journal of Vocational Rehabilitation, 33(2), 123–131.
- Conley, R. W. (1969). A benefit-cost analysis of the vocational rehabilitation program. Journal of Human Resources, 4(2), 226–252. doi:10.2307 /144722
- Dean, D., Pepper, J. V., Schmidt, R., & Stern, S. (2015). The effects of vocational rehabilitation for people with cognitive impairments. International Economic Review, 56(2), 399–426.
- Dean, D., Pepper, J. V., Schmidt, R., & Stern, S. (2017). The effects of vocational rehabilitation services for people with mental illness. Journal
of Human Resources, 52(3), 826–858. - McGuire-Kuletz, M., & Tomlinson, P. (2015). Return on investment and economic impact: Determining and communicating the value of vocational rehabilitation (Institute on Rehabilitation Issues Monograph No. 38). Washington, DC: The George Washington University Center for Rehabilitation Counseling Research and Education.
- Uvin, J., Karaaslani, D., & White, G. (2004). Evaluation of Massachusetts’ public vocational rehabilitation program: Final report. Boston, MA: Massachusetts Rehabilitation Commission.
- Wilhelm, S. (2013). The economic impact of Utah’s vocational rehabilitation program. Journal of Disability Policy Studies, 24(3), 148–157.
- Worrall, J. D. (1978). A benefit-cost analysis of the vocational rehabilitation program. Journal of Human Resources, 13(2), 285–298. doi: 10.2307/145363
Acknowledgements
We would like to thank the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR Grant 90DP0070) for generous funding to support the development of this special issue, the authors of the articles in this issue for their contributions, and the anonymous reviewers who provided important feedback to the authors.