Data mining Science policy
Data mining: Cross-boundary collaboration and innovation
September 6, 2017
, , , , , , ,

In our data mining project for the European Commission, two of the six case studies treated levers for promoting innovation, and we’ll start to tease those apart here. In brief, collaboration across disciplinary and sectoral boundaries is believed to promote innovation, while innovation in turn is believed to support broader economic and social prosperity. Even though these premises are often used as the foundations on which to build policies for science and the economy, there is surprisingly little evidence about whether they’re true—no matter how intuitively appealing they may be, these premises are still quite speculative. These are important gaps in our knowledge base, and we’ll look here to start filling in the evidence around levers of innovation.

Measuring multidisciplinary collaboration

In brief, our study design used indicators of cross-boundary collaboration within authorship groups on peer-reviewed, scholarly research papers, and used citations of those papers in granted patents as an indicator of their uptake in innovation. Of course, these indicators do not capture the full breadth of cross-disciplinary or cross-sectoral collaboration, of innovation, or of social and economic prosperity; we’ll return to this point at the end.

Do interdisciplinary #research & public-private partnership promote #innovation? Scant evidence Click To Tweet

The indicator we used for multidisciplinary research (MDR) collaboration integrates three elements: the number of disciplines represented in the authorship group, the balance of representation amongst the disciplines involved, and their thematic proximity to or distance from one another. (This approach is based on the Rao–Stirling index.) Disciplines are identified by the departmental affiliations of the authors listed for a paper, noting that some algorithmic and manual data preparation was required in order to standardize department names into thematic bins, and that some departments (or centres, or institutes) were too heterogeneous to identify conclusively with a single discipline.

MDR scores on this indicator range from 0 to 1. Here is how some score ranges can be interpreted a bit more intuitively:

Data mining technical framework, multidisciplinarity, multidisciplinarity scale

The question we examined was whether papers with a higher MDR score had better chances of being cited by a patent than papers with a lower MDR score. What we found was that multidisciplinarity increases the likelihood of uptake in innovation: the association was statistically significant in about one third of subfields, most of them within the health sciences. The strength of the effect varied from one subfield to the next, but we’ll analyze orthopaedics here, which had the strongest effect.

Data mining technical framework, multidisciplinarity, patent citation

As shown in the inset chart above, about 20% of articles in orthopaedics had an MDR score of 0, while about 60% were in the 0.2–0.4 range, and the remaining 20% were above 0.4. Odds ratios of being cited in a patent increased by about 50% with each increase of 0.1 in the MDR score. That means that a notably multidisciplinary paper (score 0.4) had about five-fold higher chances of being taken up in innovation than a monodisciplinary paper (score 0) had. However, that five-fold increase translates into an improvement to a 1/40 chance, up from a 1/200 chance—still not a very high chance, and recall that orthopaedics was the subfield with the strongest effect. Furthermore, the model is effective for predicting outcomes in the aggregate, but not whether an individual paper will or will not be cited by a patent.

A notably multidisciplinary paper has about 5-fold higher chances of being taken up in innovation Click To Tweet

Measuring multi-sectoral collaboration

As undertaken for collaboration across disciplinary boundaries, research collaboration across sectoral boundaries is tracked here by examining authorship lists on scholarly publications; in this case, the institutional affiliations of authors are algorithmically assigned to either the public or the private sector. This data preparation enables us to code a sectoral indicator for authorship groups: private-only, public-only, public–private partnership.

About 3% of author addresses are linked to the private sector, and about 4%–4.5% of all articles arise from public–private partnership (PPP)—a finding that holds at the world level and for the EU-28, for the 2000–2015 period. The applied sciences have the highest share of PPP, with some fields up around 10%, while most natural science and health science fields are in the 5%–7% range, economics is at 3%, social sciences just over 1%, and the fields of the arts and humanities are all below 1%.

data mining technical framework, cross-sectoral collaboration, public–private partnerships

Once again, we examined here the relationship between private-sector involvement and uptake in innovation (relative to the baseline of patent citations to public-only publications). The association between PPP and innovation was positive, multiplying the odds of patent uptake by about 3.5 times. Looking across thematic areas, the association was positive in 98% of subfields, and statistically significant for 85% of subfields. The effects were strongest in the arts and humanities, where public–private partnership and uptake in innovation are both incredibly rare. Otherwise, the effect size was largest for economic & social sciences (about a 5-fold increase in uptake for PPP vs. public-only), followed by health sciences and natural sciences (about a 3-fold increase, in each case) and finally applied sciences (just over a 2-fold increase).

Policy implications of the findings

To put the findings on public–private partnership into context with those presented above for multidisciplinary research teams, engaging across sectoral boundaries has about the same magnitude of effect as engaging substantively across disciplinary boundaries (where “substantive” engagement here is interpreted as a balance of contributing disciplines, represented by an MDR score of about 0.4).

Also worth noting is that private-sector-only research teams had even higher odds of patent uptake than public–private partnerships did. On the basis of this finding, one might speculate that it is in fact the involvement of the private sector that does the heavy lifting in terms of increasing uptake in innovation, rather than the collaboration of the public and private sectors; this hypothesis is highlighted for future development and testing. If indeed that is the case, then public policymakers may yet be wise to promote public–private partnership: private-sector research might be more innovation-ready, but if encouragement from the public sector is an important motivator to get the private sector involved in research in the first place, then public–private partnership would indeed promote innovation, albeit through mechanisms less direct than previously suspected.

Private-sector #research leads more often to #innovation—whether academia/gov involved or not? Click To Tweet

The chances of a paper being cited by a patent are very small; furthermore, the model put together here is effective at predicting outcomes in the aggregate but not for individual papers. These qualifications of the findings have two meaningful messages that are highly relevant for policy consideration. First, research being taken up into innovation remains a very rare event. Even if multidisciplinarity and public–private partnership multiply the odds of uptake several fold, a many-fold increase still translates into quite small absolute chances of uptake in innovation—the large majority of research is never taken up in innovation, even research produced in collaboration across disciplinary and sectoral boundaries. Second, we might expect that the major determinants of innovation would provide a model that is effective at predicting outcomes for individual cases; if indeed that expectation is warranted, then the fact that the present models are useful only in the aggregate suggests that we have not yet apprehended the primary drivers of innovation, which remain at large.

These findings raise some delicate questions: What share of our research do we expect to be taken up in innovation (directly)? And how much do we expect our policy interventions to increase that innovation uptake? What is our anticipated return on investment in promoting collaboration across disciplinary and sectoral boundaries? The findings here suggest (to me) that the effect sizes we’re looking at are quite small, and that we haven’t yet hit on the main drivers of innovation.

By how much do we expect #research #policy to increase #innovation? Is that level reasonable? Click To Tweet

General lessons on using research to inform policy

Recall that these findings come from pilot studies and need corroboration. However, with so little study done on these topics, it seems that the findings presented here hold the weight of evidence, even though our examinations are still limited in various ways. If indeed the effect sizes documented here challenge our use of these policy mechanisms to promote innovation, further study could turn out a few different ways:

  • First, perhaps these study findings cannot be reproduced, in which case we’re back to a situation where we’re left searching for solid evidence on which to base our policy positions.
  • Second, perhaps these study findings can be reproduced, but we acknowledge that the study design does not capture the full breadth of “cross-disciplinary collaboration,” “cross-sectoral collaboration” and “innovation.” That’s fine as well, though the onus is then on those maintaining this position to specify what it is they mean exactly by these terms, and how they could be operationalized in order to gather the appropriate evidence.
  • Third, perhaps the study findings are reproducible and robust, and we concede that these indicators are indeed capturing the full sense of “cross-disciplinary and cross-sectoral collaboration” and of “innovation”—this would suggest that we were wrong in taking for granted that these types of collaboration promote innovation. In this case, one could reasonably hope that we would change our policies.

Some of the findings presented here may seem intuitive, and this response may prompt one to question the value of even doing the study in the first place. “Why bother wasting time proving something so obvious?” In that case, I would urge that we consider what would have happened if we had tested an intuitively obvious hypothesis only to find that it’s simply not true. Remember that we used to think that the sun revolves around the earth, and that the size of an object doesn’t change depending on how fast it’s moving. The history of science is littered with intuitively self-evident hypotheses that simply turned out to be false. One measure of whether a hypothesis is worth testing—in policy contexts, as more broadly— is not whether it seems obvious or not; the importance of testing depends on how heavily we rely on it being true, and how much (or little) evidence we have of its truth.

Even if 'obviously' true, test hypotheses we rely on for policy & have little evidence about Click To Tweet

Our next case study covers innovation and its influence on growth and prosperity. You can read it here.


Science-Metrix’s final report for this data mining project is available from the Publications Office of the European Union.

Data Mining. Knowledge and technology flows in priority domains within the private sector and between the public and private sectors. (2017). Prepared by Science-Metrix for the European Commission. ISBN 978-92-79-68029-8; DOI 10.2777/089


All views expressed are those of the individual author and are not necessarily those of Science-Metrix or 1science.


About the author

Brooke Struck

Brooke Struck is the Senior Policy Officer at Science-Metrix in Montreal, where he puts his background in philosophy of science to good use in helping policy types and technical types to understand each other a little better every day. He also takes gleeful pleasure in unearthing our shared but buried assumptions, and generally gadfly-ing everyone in his proximity. He is interested in policy for science as well as science for policy (i.e., evidence-based decision-making), and is progressively integrating himself into the development of new bibliometric indicators at Science-Metrix to address emerging policy priorities. Before working at Science-Metrix, Brooke worked for the Canadian Federal Government. He holds a PhD in philosophy from the University of Guelph and a BA with honours in philosophy from McGill University.

Related items

/ You may check this items as well

Impact assessment stories: decisions, decisions

It appears that the research & innovation policy c...

Read more

Budget 2018: The Evidence Budget

In our post last week on the 2018–19 Canadian fe...

Read more

Budget 2018: the fundamental question of research funding

Science has been quite prominent on the Canadian p...

Read more

There are 0 comments