National Academies Press: OpenBook

Performance Measurement Tool Box and Reporting System for Research Programs and Projects (2008)

Chapter: chapter 3 - research performance measures.

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

13 CHAPTER 3 – RESEARCH PERFORMANCE MEASURES The research performance measures gathered from the nationwide surveys were considered in conjunction with performance measure information available in the literature. A comprehensive list of research-related performance measures was compiled from the gathered information, and the perceived value of each metric was considered. The comprehensive list is shown in Appendix F. After analysis, the research team recommended 18 performance measures to the NCHRP panel for inclusion in the system to be developed. At the request of the panel, the research team agreed to expand the number of standard performance measures to be provided to 30. In addition, definitions for an additional 10 performance measures will be included to inform users of other metric possibilities. The panel believed that the larger group of performance measures was needed for the system to adequately address the broad range of needs and desires existing among the states. Finally, it was also decided that users should have the ability to manually incorporate any of these other performance measures, or their own agency’s unique performance measures, into performance measure reports available from the system. This utility has been provided in the RPM System. The selected standard performance measures and their definitions are shown in Table 3.

Table 3. Standard Research Performance Measures Included in the RPM System PM Number Short Performance Measure Name Definition Comment Outcome Measurements 1 Dollars Saved Estimated present value dollar savings in the cost of contract work, cost of agency-purchased materials, and cost of employee labor made possible by research products A core justification for research budgets. Very important to agency administrators and all funding appropriators. 2 Lives Saved Projected number of lives to be saved based on the number of fatalities associated with the problem prior to the product implementation and the estimated or determined effectiveness of the research products A core justification for research budgets. Very important to both agency personnel and all elected officials. 3 Crashes Avoided Estimated reduction in number of crashes based on the number of crashes associated with the problem prior to the research product's implementation and the estimated or determined effectiveness of the product A core justification for research budgets. Very important to both agency personnel and all elected officials. Output Measurements 4 Technical Products Number of types of research products improving design processes, specifications, or technical standards or practices Each product will either be a technical product, a management product, or a knowledge product. This is a general measure of the impact of the research program on the agency. 5 Management Products Number of types of research products improving the agency's management procedures, policies, and non-technical training Each product will either be a technical product, a management product, or a knowledge product. This is a general measure of the impact of the research program on the agency. 6 Knowledge Products Number of types of research products improving basic knowledge or understanding in the subject area without creating a specific technical or management product These are the products of basic research projects. This measure may be used to establish or maintain the desired level of basic research being funded by the agency. 7 Environmental Products Number of types of research products improving or protecting the natural environment Very important, and can be of primary importance to some state and federal appropriators and others.

Table 3. Standard Research Performance Measures Included in the RPM System (cont.) 8 Congestion Mitigating Products Number of types of research products reducing or eliminating traffic congestion and other transportation system delays Very important to the general public and all elected officials. 9 Traveler Comfort Products Number of types of research products improving the physical or psychological comfort of the traveler or enhancing the aesthetic quality of the system or improving system security (safety products not included unless traveler comfort or well-being is improved in non-crash situations) Believed to be one of the most important factors to the traveling public. 10 Quality of Life Products Number of types of research products improving quality of life, which is defined as the total of those product types meeting the criteria for Environmental Products, Congestion Mitigating Products, or Traveler Comfort Products Important to the traveling public, the most important transportation agency customer. 11 Safety Products Number of types of research products improving design methodologies, traffic management, roadside safety devices, and any other innovation or enhancement for the transportation system which improves safety for anyone on or near the transportation system Safety is always a top priority. This is an indirect measure of the number of lives saved and reduced crashes made possible by the research program. 12 Cost-Saving Products Number of types of research products reducing the cost of contract work, cost of agency-purchased materials, and cost of employee labor This is an indirect measure of the amount of cost savings being obtained for the agency by the research project or program. 13 Research Reports Number of published research reports and other technical publications emanating from completed research projects during the evaluation year This measure combines two measures currently used by agencies: “Number of Papers Written as a Result of Program” and “Number of Research Reports Completed per Year.” 14 Graduate Students Total number of graduate students financially supported or otherwise involved in transportation research The value of the training given to future transportation professionals has been generally understated in the past. Resource Allocation Measurements 15 Dollar-Saving Projects Number of research projects pursuing lowered cost to provide the transportation system This measure monitors funding balance in the research program and the extent to which agency cost savings are being pursued.

Table 3. Standard Research Performance Measures Included in the RPM System (cont.) 16 Safety Projects Number of research projects pursuing safety enhancements This measure monitors funding balance in the research program and the extent to which improved transportation safety is being pursued. 17 Quality of Life Projects Number of research projects pursuing improved quality of life This measure will be obtained by adding the number of projects including environmental products, traveler comfort products, and traffic congestion mitigating products. 18 Total Contractors Number of unique entities with research projects that were active for any length of time during the evaluation period If proposals are competitively awarded, this is an indirect measure of competitiveness. 19 Minority Contractors Percentage of total research program contract budget that is awarded to minority universities, as defined by the US Department of Education and applicable federal regulations A federal requirement, reported at least annually. 20 In-House Percentage Percentage of the total funding for research projects being performed by agency personnel This can be an indicator of growing or declining in-house technical strength. Efficiency Measurements 21 Benefit-Cost Ratio Total present value dollar savings associated with the project(s) compared to either the total present value cost of the project(s) plus implementation effort(s) or to the total present value cost of the fiscal year research program plus related implementation efforts. The system report generator selects the cost basis and enters cost data. A key efficiency measurement for state and federal budget appropriators. 22 % Administrative Costs Dollar value of program overhead expenses divided by the total program cost An internal efficiency measurement. 23 % Requests Funded Number of projects funded divided by number of projects requested A lowering trend indicates probable need for additional research funding. 24 % Projects Implemented Number of projects with at least one product implemented (completely or partially implemented) divided by total number of projects completed during the evaluation period An indicator of quality in the project selection process and research project execution.

Table 3. Standard Research Performance Measures Included in the RPM System (cont.) 25 % Projects On Time Number of projects completed on/before the scheduled completion date divided by total number of projects to have been completed during the evaluation period This target should probably be around 80 percent due to the nature of research. A lower percentage can indicate generally poor contractor efforts in creating proposal work schedules. 26 % Projects within Budget Number of projects completed within budget divided by total number of projects completed during the evaluation period This target should probably be around 80 percent due to the nature of research. A lower percentage can indicate generally poor contractor efforts in creating proposal budget estimates. 27 % Project with Reports Number of projects completed during the evaluation period (FY one year prior) for which all research reports have been submitted within one year of project completion divided by the total number of projects completed during the evaluation period This is a challenging area for most research programs. Monitoring performance and having a target can be used as a tool for the research manager to encourage or require improved contractor performance. Stakeholder Measurements 28 Customer Satisfaction Number of customers reporting satisfied or very satisfied on survey divided by total number of customers surveyed Variations of surveys were reported on survey responses from several states. It is believed that all stated needs can be addressed by the definition of this PM. 29 Agency Participation Number of agency personnel involved in the program overseeing projects, serving on committees, assisting in project selection, etc. Most research programs need the participation of large numbers of agency personnel from outside of the research office. There are a number of benefits to the agency derived from this participation. This number should be provided to agency administrators. 30 Project Needs Statements Number of project needs statements submitted by internal customers This is a key indicator to research program managers for several reasons, particularly in that it shows the degree to which agency personnel understand that research provides solutions to everyday problems.

18 One of the considerations of the research team was to attempt to select performance measures which would not only meet the needs of state transportation agency administrators and research program managers, but that would also provide for the needs of secondary customers such as contract researchers, stakeholders such as state and federal legislators, and other state-level research professionals. It was not cost effective or practical to include every possible performance measure in the system being developed, but the research team endeavored to provide a set of performance measures which was as comprehensive as possible. The standard performance measures are divided into five different types in Table 3. The five types of measures are outcome, output, resource allocation, efficiency, and stakeholder. Outcome measures assess the extent to which a research project or a product of a research project achieves a desired result such as cost savings or reducing crashes. Output measures count the number of deliverable units related to a specific attribute, examples being the number of research projects which improve safety and the number of products from projects which positively impact the environment. The third type of measure deals with resource allocation. Resource allocation performance measures primarily capture the deployment of agency dollars, such as the percent of research funding awarded to minority contactors or the number of research projects being funded in attempts to improve transportation safety. In contrast, efficiency measures, the fourth category, are rates or ratios which compare what is accomplished to the effort expended. Examples of this type of measure are the percent of research products being implemented by the agency and the percent of research projects being completed within budget. The final type of performance metric is the stakeholder measure. Stakeholder measures gage the involvement of customers in the research process as well as their level of satisfaction. These performance measures include the percent of satisfied customers, number of participating agency personnel, and the number of project needs statements submitted. By subdividing the performance measures into these five categories, the user is given the opportunity to better balance the selected set of performance measures to be used.

TRB’s National Cooperative Highway Research Program (NCHRP) Web-Only Document 127: Performance Measurement Tool Box and Reporting System for Research Programs and Projects explores the integration of standard performance measures and tools to assist users in implementing performance measures into the Research Performance Measurement (RPM) System.

Welcome to OpenBook!

You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

Do you want to take a quick tour of the OpenBook's features?

Show this book's table of contents , where you can jump to any chapter by name.

...or use these buttons to go back to the previous chapter or skip to the next one.

Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

To search the entire text of this book, type in your search term here and press Enter .

Share a link to this book page on your preferred social network or via email.

View our suggested citation for this chapter.

Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

Get Email Updates

Do you enjoy reading reports from the Academies online for free ? Sign up for email notifications and we'll let you know about new publications in your areas of interest when they're released.

To read this content please select one of the options below:

Please note you do not have access to teaching notes, a framework for business analytics in performance management.

International Journal of Productivity and Performance Management

ISSN : 1741-0401

Article publication date: 1 January 2013

Increased business competition requires even more rapid and sophisticated information and data analysis. These requirements challenge performance management to effectively support the decision making process. Business analytics is an emerging field that can potentially extend the domain of performance management to provide an improved understanding of business dynamics and lead to a better decision making. The purpose of this positional paper is to introduce performance management analytics as a potential extension of performance management research and practice. The paper clarifies the possible application areas of business analytics and their advantages within the context of performance management.

Design/methodology/approach

The paper employs a literature based analysis and from this a conceptual argument is established. Finally, a business analytical model is presented to be used to undertake future research.

The paper clarifies the possible application areas of business analytics and their advantages within the context of organizational performance management.

Originality/value

The main implication is that the paper provides evidence of the use of business analytics for understanding organizational performance. Several insights are provided for management accounting research and education.

  • Business analytics
  • Performance management systems
  • Performance measurement
  • Management control systems
  • Performance management
  • Decision making

Schläfke, M. , Silvi, R. and Möller, K. (2013), "A framework for business analytics in performance management", International Journal of Productivity and Performance Management , Vol. 62 No. 1, pp. 110-122. https://doi.org/10.1108/17410401311285327

Emerald Group Publishing Limited

Copyright © 2013, Emerald Group Publishing Limited

Related articles

We’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

IMAGES

  1. Performance Management Analytics

    research performance management analytics html

  2. The Importance of Performance Management Analytics: A Case Study in

    research performance management analytics html

  3. What is Performance Analytics?

    research performance management analytics html

  4. How ServiceNow Uses Performance Analytics

    research performance management analytics html

  5. Employee Performance and Appraisal Management System

    research performance management analytics html

  6. Digital Performance Management---A Step Ahead, From Insights To Impact

    research performance management analytics html

VIDEO

  1. What is performance management system software

  2. Replacement Theory (Decision )-Performance (Financial ) Management/Operations Research /PM /QA /FM

  3. Working with updated products for PHP. Part 1 (2023.1)

  4. Behavioural Aspects of Performance Management

  5. Application Performance Monitoring (APM) by Site24x7

  6. Get Your Dream Job! Learn Data Science with Innomatics

COMMENTS

  1. Effectiveness of Performance Management System for Employee

    This article seeks to explore the effectiveness of a comprehensive performance management system in terms of employee performance. Besides, the mediating effect of work engagement was also examined. The performance management system effectiveness (PMSE) was determined by the extent of its accuracy and fairness, as recommended by previous ...

  2. Chapter 3

    A comprehensive list of research-related performance measures was compiled from the gathered information, and the perceived value of each metric was considered. The comprehensive list is shown in Appendix F. After analysis, the research team recommended 18 performance measures to the NCHRP panel for inclusion in the system to be developed.

  3. A framework for business analytics in performance management

    Business analytics is an emerging field that can potentially extend the domain of performance management to provide an improved understanding of business dynamics and lead to a better decision making. The purpose of this positional paper is to introduce performance management analytics as a potential extension of performance management research ...