Studies on Evaluation

back

Das österreichische Forschungsförderungssytem: Systemevaluiert

Year:

2009

Title:

Das österreichische Forschungsförderungssytem: Systemevaluiert

Client:

Joanneum Research

Evaluator:

Andreas Schibany

Contents:

Mit der vorliegenden Nummer des TIPpolicybrief wird eine neue Reihe eröffnet, in der wir in unregelmäßigen Abständen aktuelle nationale und internationale Themen der Technologie-, Innovations- und Forschungspolitik diskutieren und kommentieren. Wir richten uns dabei an ein breites Publikum, wobei politische Entscheidungsträger im Mittelpunkt stehen. Die im vertretenen Meinungen sind jene der jeweiligen AutorInnen.

Language:

German

Pdf:

TIPpolicybrief

Der European Innovation Scoreboard: vom Nutzen und Nachteil indikatorengeleiteter Länderrankings

Year:

2007

Title:

Der European Innovation Scoreboard: vom Nutzen und Nachteil indikatorengeleiteter Länderrankings

Client:

BMWF, BMVIT und BMWA

Evaluator:

Joanneum Research: Andreas Schibany, Gerhard Streicher, Helmut Gassler

Contents:

For quite some time now, European countries have felt the need to carry out comparisons between their respective performances through the application of benchmarking and scoreboard tools, the findings of which are then typically processed into country rankings, which in turn guarantee a large degree of publicity from the media and the general public. Within the scope of such activities, the European Innovation Scoreboard (EIS) has, over the past years, established itself as probably the most attention-grabbing benchmarking tool in the discussion of European technology policy.

Language:

German

Pdf:

European Innovation Scoreboard

IST Evaluation and Monitoring

Year:

2007

Title:

IST Evaluation and Monitoring

Client:

Evaluator:

Wolfgang Polt (Joanneum Research), Nicholas Vonortas (G.Washington University–Msl/Aueb) (Coordinators) Philippe Baudouin (Idate), Irwin Feller (Penn State University), Luke Georghiou (Prest), Lawrence Green (Prest), Tarmo Lemola (Advansis Oy), Georg Licht

Contents:

The European research policy is oriented towards the ‘Lisbon objectives’ of transforming the European Union into the most competitive and dynamic knowledge-based economy in the world, capable of sustainable economic growth with more and better jobs and greater social cohesion. Investments in research have key role in this transformation and the Framework Programme (FP) in Research, Technological Development and Demonstration (RTD) is the key research policy instrument of the European Union. The Panel was tasked to explore future perspectives of an evaluation strategy for IST RTD and put forward a proposal for a coherent, systematic evaluation framework in which the basic evaluative questions, core methodologies, and content specifications are identified together with the background supporting material in relation both to the just concluded FP6 and to the new FP7 schedules and deadlines.

Language:

English

Pdf:

IST_Eval_Monitoring

What Can Be Achieved By Special R&D Funds When There is No Special Leaning Towards R&D Intensive Industries?

Year:

2006

Title:

What Can Be Achieved By Special R&D Funds When There is No Special Leaning Towards R&D Intensive Industries?

Client:

n/a

Evaluator:

WIFO: Rahel Falk, Hannes Leo

Contents:

This paper explores the effects of Austria’s recent Special Funds initiative on the R&D expenditures of its private corporate sector. It is the first one to approach the due evaluation from a macro perspective.

Language:

English

Pdf:

special_r&d_funds

Measuring the effects of public support schemes on firms' innovation activities. Survey Evidence

Year:

2006

Title:

Measuring the effects of public support schemes on firms' innovation activities. Survey Evidence

Client:

n/a

Evaluator:

WIFO: Rahel Falk

Contents:

This paper discusses conceptual frameworks for measuring the effects of innovation policy and begins with applying conventional descriptive methods to explore how firms rate and rank the merits of public intervention.

Language:

English

Pdf:

Effects_public_schemes

Methods for the impact analysis of RTD policy measures: Challenges and developments

Year:

2006

Title:

Methods for the impact analysis of RTD policy measures: Challenges and developments

Client:

Austrian Council for Research and Technology Development

Evaluator:

ARC systems research: Wolfram Rhomberg, Claudia Steindl, Matthias Weber

Contents:

The study proposes a portfolio of different methods for the impact analysis of RTD policy measures, and discusses their potential as well as the challenges that arise with impact analysis and different methods applied. Against this background and the experiences made in Austria with RTD evaluation and impact analysis so far, the authors develop a general framework for impact analysis; what methods might be applied in different policy context, for different kinds of policy measures and types of R&D investigated. Additionally, they suggest a set of methods and give examples for indicators that can be applied when measuring the impact of the Austrian "Offensivprogramme" (a portfolio of policy measures that got recommended for implementation by the Austrian RTD Council). Finally, recommendations on central aspects for future impact analysis of the Austrian "Offensivprogramme" are given.

Language:

German

Pdf:

0108_Methoden_Wirkungsanalyse_endg

How to evaluate Special R&D Funds’ Programs

Year:

2004

Title:

How to evaluate Special R&D Funds’ Programs

Client:

Austrian Council for Research and Technology Development

Evaluator:

co-operation of Platform members

Contents:

The Austrian Council for Research and Technology Development is advising the Austrian Federal Government in R&D issues. For the eva-luation of the use of these funds the council, made up of a group of experts and compiled a set of rules, principles and basic guidelines for the development of a broad evaluation system.

Language:

German

Pdf:

Modul-II

Road MAP - Good practices for the management of Multi Actors and Multi Measures Programmes (MAPs) in RTDI policy

Year:

2004

Title:

Road MAP - Good practices for the management of Multi Actors and Multi Measures Programmes (MAPs) in RTDI policy

Client:

European Commission

Evaluator:

FFG: Birgit Baumann, Dorothea Sturn, Sabine Mayer; Fraunhofer ISI: Susanne Bührer; MFPL: Harald Hochreiter; WWTF: Michael Stampfer; Heather Greer; Paul Simmonds

Contents:

MAP stands for Multi-Actors and Multi-Measures Programmes (MAPs), which are RTDI funding programmes addressing not an individual firm or research institution but whole (sub-) systems of innovation (e.g. science-industry cooperation). Nevertheless a huge variety of features and management practices was observed, depending both on the National Innovation System (NIS) in the respective country (especially the institutional setting) and on the problems addressed.

Language:

English

Pdf:

roadMAP

Training Workshops on Evaluation - Documentation

Year:

2004

Title:

Training Workshops on Evaluation - Documentation

Client:

Plattform Research and Technology Policy Evaluation

Evaluator:

Klaus Zinöcker, Sonja Sheikh, Michael Stampfer, Leonhard Jörg, Wolfgang Polt

Contents:

In the first half-year general meeting of the Platform Research and Technology Policy Evaluation in 2004 it was decided to highlight the topic ‘training’. In this context the co-ordination team arranged the one-day workshop “Evaluation of Research and Technology Policy” that was given at six associated organisations of the Platform. Regarding to the different requirements and the content-related alignments of every associate it was figured out rather early that a uniform workshop would not lead the training to the intented success. The contents of the eight given workshops were modified over time. These changements are documented in detail.

Language:

German

Pdf:

Ausbildung-2004

The RECORD Manual – Benchmarking Innovative Research Organisations in European Accession Countries

Year:

2004

Title:

The RECORD Manual – Benchmarking Innovative Research Organisations in European Accession Countries

Client:

European Commission – funded under FP5 (STRATA)

Evaluator:

Balázs Borsi, Katalin Dévai, Gábor Papanek, Howard Rush, Adolf Filacek, Klaus Schuch (ZSI), Amir Fazlagic (ZSI), Peter Stanovnik et al.

Contents:

The Manual – together with the Experimental Map – is the final publication of the RECORD network. RECORD is an acronym for Recognising Central and Eastern European Centres of RTD. The main objective of RECORD was to assist in learning the practice of benchmarking RTDI organisations. Initially the network selected some Accession States research organisations that were considered innovative and for which there was an expectation of successful integration in the ERA.

Language:

English

Pdf:

record_manual_finalEU

Behavioural Additionality Effects of R&D Subsidies. Empirical Evidence from Austria

Year:

2004

Title:

Behavioural Additionality Effects of R&D Subsidies. Empirical Evidence from Austria

Client:

n/a

Evaluator:

WIFO: Rahel Falk

Contents:

This paper has addressed long-term behavioural changes emerging from FFFparticipation, so-called "behavioural additionality". Descriptive evidence from the survey data revealed that FFF-funding is indeed generating various dimensions of behavioural additionality: Around 80-85 percent of the sample firms experience some degree of project additionality. • Acceleration additionalities arise for two in three firms. • The share of companies appreciating scale additionalities ranges between 60-74 percent.

Language:

English

Pdf:

behaviouraladditionality

EU Evaluation Practice and Toolkit – Applicability for Japan

Year:

2004

Title:

EU Evaluation Practice and Toolkit – Applicability for Japan

Client:

Mitsubishi Research Institute

Evaluator:

PREST: Luke Georghiou, Joanneum Research: Wolfgang Polt

Contents:

The paper aims to describe the practice and development of evaluation of research and development and related activities in the programmes of the European Commission and the methods available more generally before concluding by assessing the applicability of these approaches to the Japanese situation.

Language:

English

Pdf:

Evaluation_Practice_Japan

The implementation of an evaluation system in RTD programmes

Year:

2003

Title:

The implementation of an evaluation system in RTD programmes

Client:

Technologie Impulse Gesellschaft (TIG)

Evaluator:

Joanneum Research: Klaus Zinöcker

Contents:

Evaluation systems have the task to structure the different phases of programmes and projects. In addition their objectives are to summarise, optimise and systematise these steps. The minimum components of an evaluation system are timing (ex ante, interim, ex post), objectives (program, project, policy) and contents (concept evaluations, design evaluations, “black box evaluation” and impact analysis) that should be coordinated and operationalised. The Austrian spin off project AplusB was taken to illustrate the implementation of an evaluation system.

Language:

German

Pdf:

Evaluierungssystem

A Toolkit for Evaluating Public R&D Investment - Models, Methods, and Findings from ATP's First Decade

Year:

2003

Title:

A Toolkit for Evaluating Public R&D Investment - Models, Methods, and Findings from ATP's First Decade

Client:

Evaluator:

Rosalie Ruegg, Irwin Feller

Contents:

This report draws from a body of 45 studies commissioned by ATP between 1990 and 2000 and analyzes the methods and techniques used and examines the findings of those studies. These studies have increased understanding not only of ATP but also of the dynamics of innovation systems and the relationships between public and private sector funding of R&D. The findings examined are organized around five major themes: firm/industry effects, collaboration effects, spillover effects, interfaces and comparisons with other programs, and measures of overall program performance.

Language:

English

Link:

Toolkit

E-PUB - Socio-Economic Evaluation of Public RTD policies

Year:

2002

Title:

E-PUB - Socio-Economic Evaluation of Public RTD policies

Client:

European Commission, Project within the STRATA (Strategic Analysis of Specific Policy Issues) Programme of the 5th Framework Programme: Research, Technological Development and Demonstration (RTD)

Evaluator:

S. Arvanitis (ETH Zürich) , M. Boden , L. Georghiou, J: Rigby (PREST), S. Bührer, S. Kuhlmann (FhG-ISI), H. Capron, M. Cincera (Université Libre de Bruxelles), R. Cowan (MERIT), J. Eaton (Boston University and NBER), G. Fahrenkrog, A. Tübke (IPTS), M. Kei

Contents:

Working together in a ‘Thematic network", partners from 12 countries and the Institute for Prospective Technology Studies (IPTS) of the EU, will explore methods and approaches appropriate for measuring and assessing socio-economic impacts of RTD programmes. On the basis of this exploratory study, recommendations on the evaluation procedures of the EU are formulated.

Language:

English

Pdf:

epub

Assessing the Socio-Economic Impact of the Framework Programme (ASIF)

Year:

2002

Title:

Assessing the Socio-Economic Impact of the Framework Programme (ASIF)

Client:

European Commission

Evaluator:

L. Georghiou, J. Rigby, H. Cameron, S. Kuhlmann, T. Henize, W. Polt, A. Schibany, O. Fritz, K. Guy et al.

Contents:

This study was undertaken in order to examine the socio-economic assessment of the impacts of the Community Framework Programmes and to improve understanding of how such impacts might be designated, defined and measured. The rationale for the work lies in the mounting appreciation of the importance of research and technological development (RTD) for economic growth and the emergence of the knowledge-based economy.

Language:

English

Pdf:

ASIF_report

Evaluating Actions and Measures Promoting Female Entrepreneurship – A Guide

Year:

2002

Title:

Evaluating Actions and Measures Promoting Female Entrepreneurship – A Guide

Client:

European Commission, DG Enterprise

Evaluator:

KMU Forschung Austria: Sonja Sheikh, Nadia Steiber

Contents:

The Evaluation guide includes information on how to prepare and manage an evaluation, which evaluation criteria and indicators to apply and which techniques for data collection to use when assessing measures promoting female entrepreneurship. The guide addresses programme managers, rather than evaluation specialists. It is not intended to serve as an exhaustive instructional manual for conducting evaluation, but provides a framework for thinking about evaluation as a relevant and useful programme tool and for commissioning and supervising evaluations of programmes targeted at the promotion of female entrepreneurship.

Language:

English

Pdf:

Evaluation_Guide

Quantitative Methods for the Evaluation of Technology Policy Programmes

Year:

2002

Title:

Quantitative Methods for the Evaluation of Technology Policy Programmes

Client:

Federal Ministry of Transport, Innovation and Technology (bmvit)

Evaluator:

W. Polt, D. Sturn, W. Pointner, B. Woitech, A. Geyer, G. Hutschenreiter, C. Rammer

Contents:

This report contains two research projects of the Austrian research centre Seibersdorf and Joanneum Research concerning the topic evaluation in research and technology policy. The results were discussed in a workshop.

Language:

German

Pdf:

Qeva

EvinA - evaluations of innovative actions in technology, structure and labour market policy

Year:

2000

Title:

EvinA - evaluations of innovative actions in technology, structure and labour market policy

Client:

Federal Ministry of Science and Transport

Evaluator:

O. Fritz, K. Grasenick, M. Gruber, C. Hartmann, M. Novakovic, W. Polt, D. Sturn, B. Woitech, K. Zinöcker (Joanneum)

Contents:

The aim of the Evina project is to analyse and to advance evaluation in structure, technology and labour market policy. Methods and criteria for evaluation, possibilities for quantification and agenda objectives are the main topics.

Language:

German

Pdf:

EvinA

Wie kann oder wie soll Österreichs akademische Forschung evaluiert werden?

Year:

1999

Title:

Wie kann oder wie soll Österreichs akademische Forschung evaluiert werden?

Client:

Federal Ministry for Science and Transport

Evaluator:

Institute for Advanced Studies (IHS): B. Felderer; D. Campell

Contents:

Language:

German

Pdf:

Academic_Research

Evaluation of RTI-Programmes: between Best-Pratice Development and Austrian Structures

Year:

1997

Title:

Evaluation of RTI-Programmes: between Best-Pratice Development and Austrian Structures

Client:

Federal Ministry of Science and Transport

Evaluator:

Joanneum Research: O. Fritz, D. Sturn; WIFO: G. Hutschenreiter

Contents:

Language:

German

Pdf:

EvalR&Dstudies

Evaluation of the ITF-Programme Management “Energy Technology”

Year:

1997

Title:

Evaluation of the ITF-Programme Management “Energy Technology”

Client:

Bundesministerium für Wissenschaft undVerkehr, Bundesministerium für wirtschaftliche Angelegenheiten, Innovations- und Technologiefonds (ITF)

Evaluator:

Kanzlei Dr. Bruck, Ingenieurskonsulenten: Manfred Bruck, Sabine Gasser

Contents:

Language:

German

Pdf:

ITF_Energietechnik