摘要
为了对科技项目评审专家开展评价,基于元评价理论和实证数据建立了元评价综合模型。该模型包括四个步骤和三个指标,即偏差系数、秩相关系数和二值法系数。此外,三个变量被明确为该模型的调整系数,包括变异系数、参与度和立项率。基于实证数据分析了指标之间的独立性,设计了综合各个指标和调整系数的计算方法和步骤。实证数据来自某政府科技管理部门的实际评价数据。算例结果表明该模型有利于提升元评价的可靠性。最后,针对科技项目管理部门如何提升专家评审质量提出了建议。
Founding science and technology projects is an important approach to support innovations.Most science and technology projects are selected through the evaluation of review experts.How to assess the review experts is necessary and important to select good projects.The evaluation of review experts are also called as meta-evaluation.After reviewing the theories and methods on meta-evaluation,it was found there are some limitations when referring to the existing literatures in estimating review experts in science and technology projects,i.e.,some evaluation criteria used in meta-evaluation can estimate experts from different perspectives and may be used together;some criteria may be duplicated and should not be used together;there are some influential factors on evaluation criteria.Therefore,a comprehensive meta-evaluation model was proposed for the meta-evaluation of science and technology projects by review experts.The comprehensive model regarded that three kinds of information may be used to estimate one review expert in meta-estimation,i.e.,1)the deviation degree between the expert′s scores on each review criterion with the scores given by all experts;2)the deviation degree between the rank of all projects given by the expert and the rank given by all experts;3)the deviation degree between the pass results suggested by the expert and the actual results.Accordingly,the comprehensive model adopted three evaluations criteria based on literature review to calculate the deviation degrees,i.e.,1)deviation coefficients on each review criterion which can be used for estimate the first deviation degree;2)Spearman rank correlation coefficient on all projects which can be used for estimating the second deviation degree,and 3)binary value reliability coefficients on each project which can be used for estimating the third deviation degree.These criteria include all types of data used in the researches on meta-evaluation,cover two research perspectives of meta-evaluation,and include the main meta-evaluation criteria.In addition,three influential factors were identified based on literature review and discussions,i.e.,1)the variation coefficients of each review criterion among all experts which can be measured with the standard deviation of the experts′score given on each estimation criterion;2)the attendance degree of one expert in project review work which can be measured with the ratio of the project quantities that an expert participated in and the maximum quantity of projects that all experts participated in;3)the actual pass ratio of projects that can be measured with the ratio of the final supported project quantities to the total applying project quantities.In order to analysis the relationships among these criteria and other influential factors,an empirical study was adopted.The empirical data was provided by one government department including the review data on 1634 projects by over 1000 experts.99 experts with the most attendance and all of their review data were picked up for this study.Some suggestions can be given based on the statistical analysis of the empirical data.1)The correlation relationships among three estimation criteria are not significant(p=0.01).Though the relationships between Spearman rank correlation coefficients and binary value reliability coefficients are significant on p=0.02,their relationship coefficient is low at 0.24.The data suggests that three selected estimation criteria can make estimation through different perspectives and these criteria can be used together to estimate review experts.2)There is a significant correlation relationship between the deviation coefficients of one expert and the variation coefficients of all experts on review criteria.The correlation relationship coefficient is 0.901(p=0.00).The close relationships suggest that the variation coefficient has a significant impact on the deviation coefficients.It is not suitable to estimate one expert only according to his/her performance on the deviation coefficient on each review criterion.Therefore,a formula was founded to adjust the deviation coefficients based on the variation coefficients.3)The experts′attendance degrees have significant impact on the estimation performance(p=0.01).Therefore,an adjustment model was founded to revise Spearman rank correlation coefficient with experts′attendance degrees.4)There are significant correlation relationships between binary value reliability coefficients and the pass ratios of projects.The higher(lower)the pass rates of science and technology projects is,the more(less)the experts recommend the projects.Accordingly,an adjustment model was built to revise the binary value reliability coefficients with pass ratios of projects.At last,in order to integrate these three criteria in the comprehensive meta-estimation model,their weights were decided based on an empirical survey among 8 specialists from industries and schools.The survey data suggested different weight for each criterion,i.e.,the deviation coefficients on each review criterion(0.2),Spearman rank correlation coefficient on all projects(0.4),and binary value reliability coefficients on each project(0.4).Based on three estimation criteria and three influential factors,a comprehensive meta-estimation model was built.In order to illustrate the reliability of the model,an actual case was adopted for testing.In this case,there are seven review experts participating in reviewing 15 projects.The results of the comprehensive model were compared with the traditional meta-estimation methods.The comparison results show that the results from the comprehensive model can give out more practical results.Finally,some suggestions are provided on how to improve the expert review quality for science and technology project management departments.1)It is not enough to adopt few estimation criteria during meta-estimation of review experts.A comprehensive meta-estimation model is necessary to integrate different criteria and influential factors.2)It is necessary to dynamically manage review experts,and the attendance degree of review experts is also important for review quality improvement.Therefore,it is necessary to improve the stability of review experts.3)It is necessary to strengthen the communication between project management departments and review experts on the pass rate of projects and other relevant information.4)It is helpful to combine quantitative and qualitative estimation criteria so as to reduce the deviations of review data given by review experts.In this study,a single review expert was estimated based on their deviation with the total experts.This may be not good in some cases,such as when reviewing the projects with originality.Some standard projects may be suggested for testing review experts.In addition,this study did meta-estimation only based on the quantitative data,an integrated meta-estimation model based on quantitative and qualitative data may be proposed in the future study.
作者
汪建
王裴裴
丁俊
Wang Jian;Wang Peipei;Ding Jun(School of Management,Shanghai University,Shanghai 200444,China;Shanghai Economic and hiGinnalion Technology Commission,Shanghai 200125,China)
出处
《科研管理》
CSSCI
CSCD
北大核心
2020年第2期183-192,共10页
Science Research Management
基金
国家自然科学基金面上项目(71572104,2016.01-2019.12)
教育部哲学社会科学重大课题攻关项目(13JZD025,2013.10-2018.12)。
关键词
科技管理
项目评审
元评价
science and technology management
project review
meta-evaluation
作者简介
汪建(1974-),男(汉),湖北浠水人,副教授,研究方向:运营管理、创新管理;通讯作者:王裴裴(1993-).女(汉),江苏宿迁人,硕士研究生,研究方向:供应链运营及评价;丁俊(1981-).女(汉),浙江慈溪人,经济师,研究方向:科技项目管理。
