Sitra’s impact evaluation 2: Driver of change and playmaker in the economy

Authors: Vesa Salminen, Kimmo Halme, Julia Wiikeri (4FRONT) & Mika Nieminen, Kirsi Hyytinen, Sampsa Ruutu (VTT)

Our evaluation has studied the impact of The Finnish Innovation Fund Sitra from the standpoint of the strategic objective area Moving to a regenerative and collaborative economy. The purpose of the evaluation is to provide an independent view to support strategy, management and development activities at Sitra, and inform the stakeholders about the means Sitra uses to further sustainability and welfare.

The evaluation questions addressed are grouped into three areas.

1. Strategic choices: Have the chosen goals been aligned with the overall strategic objectives and purpose of Sitra? Has the project portfolio been aligned with the goals and objectives?
2. Contribution: How has Sitra contributed to achieving the goals? What are the paths, mechanisms, partnerships and other means that have created the impact? What factors have impeded or hindered impact? What has been Sitra’s role in facilitating systemic change?
3. Progress towards the goals: What progress has been made towards the goals and what or who, besides Sitra, has contributed? What has been achieved nationally and internationally? What actions are needed in the future to further sustainable welfare?

Our aim was to focus on creating an overall picture of the impact of Sitra’s activities, specifically from the perspective of the objective area Moving to a regenerative and collaborative economy (Talous uudistavaksi ja yhteisölliseksi). Consequently, the evaluation does not present a detailed picture of each activity.

The evaluation was conducted between February and June 2017 by 4FRONT and VTT, in parallel with the evaluation of Sitra’s activities related to the objective area Addressing well-being in a holistic way.

Here’s a detailed summary of that report that was originally written in Finnish.

Evaluation approach

Framework

Sitra is a challenging object for evaluation, as its goals imply large, systemic changes that involve multiple actors besides Sitra. The impact of Sitra is thus predominantly indirect and dependent on the actions of others. The impacts that are sought are mostly behavioural changes and regulatory or legislative. Thus Sitra’s activities involve facilitating change by forming opinions and creating partnerships and networks to effect change.

Because of the nature of Sitra’s goals and activities, evaluation by traditional impact metrics would be challenging and possibly misleading. The evaluation focuses on creating a rich picture of the systemic impacts of Sitra’s activities through the identification of impact pathways and systemic changes that are the results of Sitra’s activities. The theoretical framework guiding the evaluation is the Multi-Level Perspective, which entails an analysis of regimes composed of different actors who work within the rules of the regime’s institutions. The regime generally is self-sustaining and driven by exogenous environmental pressure on the one hand, and local niche experiments on the other. Given enough external pressure, the regime cannot renew itself and a new regime is formed, often from the niche solutions.

This framework is used to reflect Sitra’s role in facilitating renewal of the regime and cultivating new solutions. The regime, as defined by Sitra, is composed of:

1. politics and government; politicians, policymakers and civil servants (nationally, regionally and locally);
2. the private sector, enterprises and entrepreneurs;
3. science, research and technology development (universities, colleges and research institutes);
4. the third sector (NGOs)
5. private citizens
6. the media and culture (journalists, media houses, cultural influencers).

Data and methods

The main data was gathered from:

• Sitra programme and project reports and other materials;
• stakeholders’ documents (strategy papers, reports, studies and web pages), which were used to assess the progress towards the goal and Sitra’s role in it;
• Sitra’s internal evaluation and monitoring data;
• interviews with Sitra’s programme and project managers and stakeholders (altogether 14 Sitra personnel and 45 representatives of the stakeholders from 41 different organisations);
• Three workshops

The main methodological approach was outcome harvesting, where the starting point was to form a picture of the changes that constitute progress towards the selected goals. After that Sitra’s role and contribution to the change was assessed through document analysis, interviews and workshops. To complement the analysis, a soft-system dynamics approach was used to model Sitra’s contribution to the development of the Finnish economy.

Conclusions and recommendations

Strategic choices

Have the chosen goals been aligned with the overall strategic objectives and purpose of Sitra?

Key elements in Sitra’s ability to have an impact are the recognition of the opportunity to influence and create change and choosing the timing of its action. The analysis highlighted the importance of choosing and defining the specific goals and designing the programmes and partners to implement them. Besides setting goals and programming, another key area is exit planning, which further highlights the importance of the choice of projects and partners to ensure sustained impact.

Consequently, foresight, strategy and programming are essential for Sitra to make a greater impact. Sitra needs to recognise relevant challenges and its own niche in time to design programmes and form networks. According to the findings, Sitra has succeeded in this, and, overall, the data suggests that Sitra does the right things, at the right time and with the right tools. Furthermore, even if the timing has not been perfect in all of its former programmes, Sitra has been able to assess the situation, lean and adapt.

While Sitra has generally been successful in making strategic choices, Sitra should pay further attention to engaging and informing other actors when making the choices.

“

Sitra’s added value is greatest when addressing systemic challenges.

Sitra has been involved with ambitious and complex initiatives, including a basic-income trial, the circular economy and social impact bonds. At the same time, not all of Sitra’s activities exhibit the same level of progressive thinking; for example, the Sustainable economic policy training does not to challenge current thinking as much as it could. Sitra’s added value is greatest when addressing systemic challenges that span policy areas, which are difficult to address by others. Sitra has a unique place in Finnish society in being viewed as an “impartial” non-politicised actor with governmental gravitas and strong substance expertise, which enables Sitra to act as a change agent in otherwise difficult areas.

Has the project portfolio been aligned with the goals and objectives?

Sitra’s programmes have clearly contributed towards their set objectives, and the interlinking nature of the projects is recognisable. At the level of substance, the goal Moving to a regenerative and collaborative economy is essentially a programme with several projects that are implemented and followed independently. These projects are loosely related, and as they are defined in practical terms, the connection to the overall purpose and different levels of goals, and between each other, is perceived to be loose, according to the data. The data also suggests that even those people implementing the projects have difficulty seeing their contribution to the purpose and greater goals of Sitra, which makes it even harder for the stakeholders. While it is necessary to allow the programmes some autonomy in their implementation, clarifying the connections between the goals and the projects and their interconnections would be more likely to contribute to further advancing the programmes.

Going further, making the different sets of goals and their relations to the programmes and individual projects clear and transparent would benefit Sitra and stakeholders. A detail that contributes to the confused perception is that Sitra uses its own non-standard language to describe its activities, instead of programmes or projects, Sitra talks about “focus areas”, and instead of outputs, outcomes and impacts, Sitra uses terms “adoption targets or objectives”. Also, in Sitra’s own reports, adoption targets, goals and impacts of the projects are used side by side to discuss overlapping matters. Simplifying and standardising language alone would be likely to clarify the purpose and goals, and their relationship to the programmes.

Contribution

How has Sitra contributed to achieving the goals? What are the paths, mechanisms, partnerships and other means that have created an impact?

Sitra uses different mechanisms to reach a variety of stakeholders that are necessary to effect systemic changes. The six key mechanism or impact pathways recognised in this evaluation are as follows.

1. “Sense making” about new concepts and ideas.
2. Starting conversations about societal challenges and new solutions.
3. Network building around recognised challenges or solutions.
4. Pilot projects and trials to test potential new solutions.
5. Establishment of new behaviours or practices.
6. Building a political and administrative base for change by influencing policymakers.

The most easily traceable impact pathway is building a political and administrative base, as exhibited by Sitra’s influence in the basic-income trial, where Sitra had a direct influence on building the legislative base and paving the way for the trial. In the future, addressing the general public on the one hand and the private sector either directly or together with agencies on the other have been proposed as ways to enhance impact.

What has been Sitra’s role in facilitating systemic change?

Sitra’s main added value is addressing systemic change. Sitra has the ability to comprehensively address and engage different actors through various means and channels in a way not possible for other traditional operators. A good example is Sitra’s activities in the circular economy, where it has influenced governmental strategy documents, facilitated the creation of business ecosystems and pilots, and influenced public discourse through the media.

The broader impact of Sitra’s activities is materialized over relatively long periods of time, and through the activities of other actors, which makes verifying the exact impact challenging. Despite this, Sitra’s role and impact mechanism are clearly recognisable. Sitra has contributed to shaping public discussion, influencing policymakers, facilitating the formation of networks, setting common goals and supporting pilot projects. Some of Sitra’s activities have achieved international attention, chief among them the basic-income trial and Sitra’s advocacy of the circular economy.

Progress towards the goals

What progress has been made towards the goals and what or who, besides Sitra, has contributed? What has been achieved nationally and internationally? What actions are needed in the future to further sustainable welfare?

There has been clear progress towards the goals assessed in the evaluation. The circular economy, experiments and impact investing are examples of areas where concrete changes and progress on different levels and within different dimensions have been identified. They are also areas in which Finland is seen as one of the forerunner countries. Although this progress is the result of activities by several different actors, the evaluation evidence suggests that Sitra’s role in initiating the changes has been significant. The main challenge for the future is to ensure that the systemic progress continues  even without the active role of Sitra.

Recommendations

This last section condenses the recommendations derived from the analysis for the specific goal Moving to a regenerative and collaborative economy and general recommendations for the strategic and operative management of Sitra.

Goal-specific recommendations

Recommendation 1: Sitra should concentrate on systemic cross-sectoral programmes. Sitra has the ability to address systemic issues that span policy areas better than others in the Finnish system as a result of its independent status and unique position as an impartial organisation. This does not imply major changes to Sitra’s existing model, but does underline that Sitra should focus primarily on issues that are difficult for others in society.

Recommendation 2: Sitra should further broaden its impact pathways. Sitra has many ways of addressing issues. Its strengths are influencing policy and societal discourse. In the 2010s, Sitra’s activities have broadened towards the grass-roots level, especially in activities relating to the circular economy. The data suggests that this has had a positive effect on Sitra’s impact and it is recommended that Sitra develops the capacity to address private citizens when implementing its projects. Sitra could also take a larger role in supporting the capacity of stakeholders to implement pilot projects and facilitate networking.

Recommendation 3: Sitra should pay more attention to transparency and outreach in choosing and preparing programmes. Sitra should pay more attention to dialogue with stakeholders and general transparency at the preparation stages of its programmes. Sitra should also outline the niches it addresses, and is about to address, more clearly and transparently to its stakeholders, to enable the stakeholders to prepare and commit to the activities. One specific method could be the publicising of Sitra’s own strategy to stakeholders.

General recommendations for strategic and operative management

Recommendation 4: Sitra should link the programme objectives more directly to the goal Moving to a regenerative and collaborative economy. In practice, this means setting clear goals for each programme and project derived from the goal Moving to a regenerative and collaborative economy. All activities should relate or be positioned to Sitra’s purpose and vision. The work had already started during the evaluation and it should continue. In the future, the overall vision and goals should be made clear and be understood within Sitra itself on all levels, as well as by stakeholders.

Recommendation 5: Sitra should pay more attention to planning exits from the programmes. All Sitra programmes include an exit plan, defining how Sitra will scale back its activities once certain objectives and conditions are met. This is an example of good project planning. However, the data indicates Sitra’s exit is often abrupt and sudden, and it is not always planned with or communicated to stakeholders. Having a clear exit plan is very much recommended in the future. What also should be done differently is that exit conditions should be communicated in advance for other involved parties and stakeholders, and the activities should be followed-up to ensure the effected change is sustainable. Additional action could include low-level activity in the projects to keep the ideas and activities going and linking the previous programmes explicitly to new ones to ensure continuity.

Recommendation 6: Sitra should develop links between programmes across areas and structures for internal learning. Since Sitra works with large systemic challenges, it is imperative that people working on different (sub) goals within the overall vision work in collaboration. There are good examples of learning between programme areas, but the data suggests that a structure for systematic knowledge exchange and learning is needed. Linking the programmes more strongly to the overall goals and vision can help in this, but, in practice, forums and resources for knowledge exchange and employee development are also needed.

Recommendation 7: Sitra should develop monitoring and evaluation by developing a clear common goal hierarchy and using standard language. Sitra’s own monitoring and evaluation should be developed by instituting a common framework for measurement, monitoring and evaluation across goal areas and programmes. Attention should be paid to setting goals in relation to Sitra’s overall goals and adopting a common form of monitoring and reporting the projects. Additional attention should be paid to the follow-up phase after the exit. Furthermore, it is recommended that Sitra adopt common standard terms for programme and project management to make it easier to communicate within Sitra and with stakeholders especially.

This article is a summary of Sitra’s impact evaluation report written by 4Front Oy and VTT Oy: Vaikuttavuusarviointi 2: Talouden muutosajuri ja pelinrakentaja