The screening of projects in the government agencies will also tend to create a selection bias in the estimated impact. More precisely, if there is a positive correlation between a firm hitting particularly promising projects that tend to generate above average performance growth in subsequent years, and the chance of the firm receiving R&D support, the ‘difference-in-differences’ estimator will overestimate the impact of the R&D-support on the performance of the supported firms. Previous studies of the effectiveness of R&D subsidies in stimulating private R&D spending has been criticized by Kauko (1996) along these lines.
The econometric literature has suggested that such biases can be reduced or eliminated by augmenting the ‘difference-in-differences’ estimator, incorporating conditioning variables reflecting the pre-program performance27. That is, differences in longitudinal changes in performance between supported and non-supported firms should control for pre-program, temporary shocks that influence the probability of being supported, e.g. pre-program changes in R&D or firm growth. Similarly, one would also like to control for anticipated future temporary shocks that influence the probability of being supported by conditioning on forward looking variables, in particular physical and R&D investment and perhaps also hiring or firing.
In the review of the study of SEMATECH in section 2, we raised the issue that the members and non-members in SEMATECH were to a large extent quite different firms in terms of size and closeness to the technological frontier. As emphasized in Heckman et al. (1998), such differences make the evaluation results critically dependent on assumptions about functional forms, both in terms of the performance equation and the selection equation, and Heckman et al. find that this tend to generate substantial biases in the case they examine.
Exploring various matching-procedures as well as regression methods, Heckman et al. conclude that evaluation results are only reliable when they are based on ‘treated’ units (cf. supported firms) which are similar to some of the ‘non-treated’ units (cf. non-supported firms). For the supported firms that can not be adequately ‘matched’, the comparison to non-supported firms can give quite misleading inference of the impact.
Spillovers and the counter factual: ‘Catch-22’? Using the non-supported firms to evaluate what would have happened to the supported firms if they had not been supported assumes that there is no spillover effects of the R&D support scheme to the non-supported firms, which is clearly a strong assumption. The question is whether the performance of the non-supported firms can be considered independent of the support given to the supported firms28. One could argue both wrays in terms of the bias this problem introduces in the estimated impact of the
R&D program; the impact will be underestimated if the non-supported firms tend to benefit e.g. from pure knowledge spillovers from the R&D in the supported firms, while the impact will be overestimated if the non-supported firms are hurt as they loose relative competitiveness to the supported firms.