Evaluation

From Open Government Pioneer Project
Jump to: navigation, search

Introduction[edit | edit source]

How do we know that the Open Government Pioneers UK initiative is making a difference?

  • The evaluation of the core activities funded by the Big Lottery Fund UK will be delivered in partnership with an external consultant with expertise in this area selected by the Project board. Budget has been allocated for the consultants fee.
  • The evaluation of the equality activities will involve an equality impact assessment and further consultancy budget for specific equality expertise has been allocated for this.

Aims and objectives[edit | edit source]

We want the learn from the mistakes we make and build on the successes we create. Is there evidence that a diverse range of citizens and civil society are any closer to engaging the decisions that affect them as a result of the Open Government Pioneers Project UK?

Objective[edit | edit source]

Track progress towards the following four big outcomes.

  1. Empowered engaged citizens, leveraging their knowledge of SDGs.
  2. Political representatives have greater awareness of SDGs and types of actions required.
  3. Government officials have a better plan for involving citizens.
  4. Open government seen as instrumental, and a common good towards progressing SDGs.

Overall approach[edit | edit source]

  • To use an open and agile approach to monitoring, learning and evaluation
  • To assess how successful the project is at opening up government progress against the Sustainable Development Goals to citizen participation
  • To inform future activities that support Open Government approaches to delivering the Sustainable Development Goals

Beneficiary involvement[edit | edit source]

Beneficiary Involvement Sharing learning
Influencers - politicians, government officials, institutional partners As critical friends, advisors on positioning of the initiative and its outputs Learning shared via briefings and presentations
Participants - civil society activists, engaged citizens Directly involved in shaping the project's design, analysis and presentation from the start (using the Wiki and other digital platforms) Learning shared via collaborative Wiki and through Forums, including Forum polling
General Public e-petitions, social media, comments on new sites Learning shared via mainstream media reflective articles

Key questions[edit | edit source]

Question Suggestion Comments Agreed
Are the objectives SMART (i.e. achievable and measurable)?
Is the overall approach acceptable and ethical?
Have we got the right audiences to involve in this? Anyone missing?
Other?

How should the project be evaluated?[edit | edit source]

Proposed evaluation framework

STREAM INDICATORS EVIDENCE COLLECTION METHOD RESPONSIBILITY WHEN REPORTED
NI Outputs 4 Webinars Theme of webinar; date delivered; numbers registered; participation feedback Analytics, narrative & participant survey NIEL Annually
4 Equality Events Theme; date delivered; numbers attended Analytics & narrative report NIEL Annually
1 National Conference Sessions delivered, themes, attendee numbers & feedback, outputs Analytics, narrative & participant survey All Post Conference
Blogs & practice to policy paper Themes, number produced, blog reads Analytics All Bi-annually
STREAM INDICATORS EVIDENCE COLLECTION METHOD RESPONSIBILITY WHEN REPORTED
Awareness (Capacity) Articulation of issues Events, blogs, forum posts, wiki posts, website updates Analytics & narrative report All Bi-annually
Publicity & Promotion Tweets, articles, media coverage Analytics & narrative report All Bi-annually
Sharing (Learning) New approaches & tools used Itemisation of new tools & methods used Narrative report All Bi-annually
Methods & approaches shared with project partners and other stakeholders Details of new tools & methods shared Narrative report All Bi-annually
Participation (Engagement) New participants/members Forum registrations, twitter followers Analytics & narrative report All Bi-annually
New collaborations Number of new partners & collaborators Analytics & narrative report All Bi-annually
Digital platform interaction Number of website/platform visits & interactions Analytics All Bi-annually
Event & webinar attendees Number & demographic Analytics All Bi-annually
Influence (Outcomes/Impact) Progress with SDGs Implementation plan Narrative report All Bi-annually
Citizen involvement Participation opportunities created Narrative report All Bi-annually
Policy influence Influence on government policy, for example, the PfG, SDG implementation plan Analytics & narrative report All Bi-annually
Transparency & accountability of process Government utilisation of OG methods in the planning & implementation process Narrative report All Bi-annually

Agile learning approach[edit | edit source]

  • Initial data collected in the first quarter of the project will form a baseline (against the Data subjects in the table below)
  • An assessment will be made of any changes when reported (quarterly, monthly or annually as appropriate). Advice from Reference Group partners will be sought as appropriate.
  • A recommendation will be made to the Project Board by the Delivery Group to change project elements based in this assessment.

Data underpinning evaluation[edit | edit source]

Stream Data subject Data Relevant targets Collection method Responsibility When reported
Capacity Opengov forum Registrations

Posts and replies

Increase forum registrations from 1000 to 5000 and aim for fivefold increase in each nation Website usage stats Each nation lead partner Quarterly
Capacity Opengov pioneers Wiki Visits

Active contributions

Citations

Tbc after setting baseline Website analytics

Google analytics

SCVO Monthly
Capacity Opengov collaboration Contributions to crowdsourcing platforms

New initiatives between citizens sparked as a result of project activities

Submissions from 20% of members during lifecycle of project Crowdsourcing platform analytics

Forum polling

Involve UK Quarterly
Engagement Opengov website Blog reads

Google ranking

tbc Website analytics

Google analytics

Involve UK Quarterly
Engagement Positioning of Open Government agenda in delivery of SDGs Stakeholder analysis Qualitative Influencer interviews / meetings Each nation lead partner Quarterly
Engagement Impact on SDG progress Policy documents (government) Qualitative & # of citations Rapid review and appraisal Consultant Annually
Learning Knowledge of what works in using Open Government to progress SDGs Blogs and articles from members Submissions from 20% of members during lifecycle of project Media analysis Each nation lead partner Annually
Learning Engagement between Pioneer civil society on what works Pioneer forum engagement

Posts and replies

Forum polls

Tbc after setting baseline Website usage stats SCVO & Involve UK Quarterly
Learning Engagement between UK nations on what works Stakeholder analysis Qualitative Influencer interviews / meetings NIEL Annually

Independent evaluator[edit | edit source]

We need to ensure that the project success is properly and independently assessed by an independent evaluator. This is for the following reasons:

  • The use of an independent evaluation consultant was agreed by project core partners and specified in the bid to the Big Lottery Fund UK. There will be an expectation of this by all partners who have reviewed the application and the plans which was published on this Wiki at that time
  • An independent evaluation will allow a more objective evaluation of the success of the project than project core partners interviewing or getting feedback themselves from project beneficiaries
  • A single independent evaluation rather than separate evaluations in each UK home nation will allow us to stay within budget and capture differences and common experiences across the nations.
  • We will use the process towards recruiting the independent evaluator to generate a conversation between existing and prospective delivery partners on our evaluation approach. In particular, we will seek advice on how to get a clearer line of sight between the project output delivery and progress towards outcomes

We will openly advertise and recruit an evaluation partner (consultant) to deliver the independent evaluation, and another evaluation partner to deliver the equalities evaluation. The project's Delivery Group's recommendation to the project board will be to use a single evaluator for both the main and equalities evaluation of the project. This is to recognise the importance of embedding equalities in the overall evaluation approach. It would also fit into the outcome Empowered engaged citizens, leveraging their knowledge of SDGs, emphasising the focus on engaging people whose voices are least heard.

In order to inform the process used to recruit the evaluation consultant, we will draw on the pilot approach we used to recruit the equalities trainer for the project team. This involved an open approach, where potential delivery partners self-identified their interest on this Wiki on the Partners page. They were subsequently added to the Opengovernment Pioneers Project Partners forum and were encourages to comment on the overall process and brief openly to other partners before making their pitch.

The evaluation will involve interviews with project partners, project participants and government, and will be supported by the data collected as part of the project's monitoring activities and a discussion with the project board and input from project's forums around evaluation priorities. The evaluator will be expected to make recommendations about how to continue and develop the engagement of citizens and civil society in using open government to secure progress towards SDGs.

Self-evaluation[edit | edit source]

It is important that this project does not rely completely on the independent evaluation to understand the difference it is making.

Therefore:

  1. The UK Conference will include a workshop component to facilitate stakeholder analysis, where project participants collectively deliberate around what works and how the project is making a difference
  2. The Activity sprint on international learning in Summer 2018 will also provide international level learning about what works and external perspective on how the project is making a difference based on International webinars with Pioneer civil society cohort
  3. Crucially, the independent evaluation will be accompanied by six monthly self-evaluations by each core partner for each home nation which will be captured and reported through this Wiki.

Timescales for evaluation[edit | edit source]

Milestone Timescale Status
Evaluation plan on Wiki

Delivery Group Self-evaluation

May 17 In progress
Meeting with Project Board

Paper outlining equalities impact proofing project activities

Jun 17 Scheduled
Recruitment process agreed Aug 17
Recruitment of evaluators

Delivery Group Self-evaluation

Dec / Jan 18
Evaluator session with Project Board Jan 18
Evaluation delivery

Delivery Group Self-evaluation

May 18
Evaluation report and recommendations Jul 18
UK wide sprint to review project and legacy Sep 18

How will we ensure a strong equalities focus?[edit | edit source]

This Open Government Pioneers Project is primarily about raising awareness and supporting the implementation of the SDGs through Open Government principles and methods, with an emphasis on reaching groups who have difficulty in having their voices heard.

We will use the following approaches to ensure an equalities focus is embedded at all stage and all activities of the project:

  • Prioritising budget and engagement with partners and groups supporting those without a strong voice in opengovernment and SDG discussions
  • Securing strong buy-in to this project from a variety of equalities-focused civil society organisations
  • Ensuring all project activities seek wider engagement than the 'usual suspects' - e.g. using livestreaming, non-jargon language, accessibility checks for digital platforms.
  • Ensuring all project activities are as open and accessible as possible at the design stage to people who face additional barriers to engaging the decisions that affect them.
  • We will learn from the existing evidence base available to us, starting with the research and reports referenced at Analysis and theory of change#Who participates.3F

Key questions[edit | edit source]

Question Suggestion Comments Agreed
Do the proposed evaluation indicators link closely enough with the objectives?
Is the timescale proposed realistic?
Is the proposal to ensure a strong equalities focus adequate and realistic?

Project Impact[edit | edit source]

Updated May 2017

Overall impact assessment[edit | edit source]

  • The project has facilitated a renewal of interest in the Sustainable Development Goals which had been flagging since the launch of the SDGs in late 2015.
  • The project has also increased interest and visibility in the Open Government Partnership and action plans amongst the policy communities in Scotland and Wales.
  • The project has secured a greater appreciation by government and civil society participants in the open government partnership of the link between OGP and SDGs in Northern Ireland and England.
  • The project has not yet broadened awareness and interest in OGP and SDGs in the broader public and media. However, a number of structures and planned engagements are now in place to progress this during the next six months of the project.

Impact against outcomes[edit | edit source]

Outcomes Beneficiary Impact assessment Equalities impact assessment Areas for improvement
Political representatives have greater awareness of SDGs and types of actions required. Influencers - politicians, government officials, institutional partners In Scotland, project partner Scotland Malawi Partnership has used the leverage of our new globalgoals.scot platform to get 100 MSPs to sign up to commit to SDGs. The project was discussed as promising area of innovation in the recent UK Stakeholders for Sustainable Development conference in London, March 2017.

In Wales, a clear connection has now been secured between the aims of the open government approach to sustainable development goals and the Wales Wellbeing of Future Generation Goals with political buy-in from the Office of the Future Generations Commissioner into the aims of this project. Very willing central government in Wales to engage with civil society on this.

The SDGs were included in the NIEL response to the NI programme for Government. While the draft PFG makes reference to the SDGs, the collapse of the NI Executive poses a direct challenge to progressing the SDG agenda. The network has met with relevant government officials, but it is clear that government are not progressing as quickly as expected and seem to be awaiting Ministerial direction. The Network continues to send information (such as blogs reports and project activities) to relevant government officials. The Network has also directed efforts at local government. A presentation was given to officers from all local authorities to raise awareness and understanding of the project and the relevance of the SDGs at local level.

In Scotland, politicians have begun to see SDGs as a domestic policy issue. Many of the MSPs who have signed up to the SDGs have identified domestic poverty and inequality as priority themes.

In NI, the SDGs are included in the NI PFG. There is overlap between the Goals and PFG outcomes framework, particularly in the area of poverty, health and climate adaptation.

Wales: engagement with local government will be important as they will be delivering wellbeing plans from this year, reporting against progress on the wellbeing goals.

NI: political literacy around the SDGs is still low. Continue to raise understanding through all activities.

Government officials have a better plan for involving citizens. Participants - civil society activists, engaged citizens Around 1400 professionals have now joined the OGN network across the UK - around 300 since this project started. Most new sign-ups have taken place in Scotland and England. A series of active forum discussions have taken place around data, freedom of information and democracy. The challenge now is to turn this into collective action.

Scottish Government officials have now committed to integrating SDGs into their National Performance Framework and making links with this project and our Globalgoals.scot platform to support participation.

Welsh Government officials are collaborating with our Network to crowd source the next Wales Open Government Action Plan Commitments 2018-20.

In Northern Ireland, government have begun an initial exercise to map the SDGs with the PFG. However, government is aware of our activities and on restoration of the Institutions we will become an essential partner for facilitating citizen input.

In Scotland, a joint roundtable between Government and civil society has identified widening engagement to those (not round the table) in open government as a priority focus for Scottish Government around social security, public service reform and participatory budgeting.

NI OGN is working with officials from the Dept of finance to plan an open policy making pilot to enhance citizen engagement in the policy development process. The pilot will explore, develop and trial creative and effective open policy-making and public engagement methods and share the learning across government. The objective is to embed a culture of proactive and meaningful engagement with the public across government departments to ensure that public input contributes in a meaningful way to policy formulation.

In Scotland, we need a clearer strategy for follow through for interest in open government and SDGs by citizens and civil society nurtured by this project.

NI: Given the deficiencies in citizen engagement as acknowledged by the NI Executive, this pilot will contribute to more meaningful engagement.

Empowered engaged citizens, leveraging their knowledge of SDGs. General Public Around 200 people have joined the Meetup groups set up for people and communities to engage with OGP and SDGs in Edinburgh, Glasgow, Dundee and Cardiff. A number of people have stepped forward to host their own meetups.

In Wales, buy -in from the Co-Production Network Wales, ensures engagement with third sector organisations, citizens to engage in co-production with Government.

In Scotland, our two meetups held directly with citizens (rather than our OGN events) demonstrated that people are keen for a more open government approach and see the relevance of SDGs to framing their asks. This contrasts with our OGN events, where organisations see the value of open government to improving their operating environment but are not yet familiar with the offer/relevance of SDGs to their work.

In Northern Ireland, the OG Network has engaged diverse audiences to raise awareness and understanding of the SDGs. For example, a seminar was held at the Ulster University Pushing Boundaries Seminar Series with 30 academics exploring open gov and the SDGs. An SDG Democracy Games themed event was held in conjunction with NI Grassroots. This fun event engaged approx. 60 children from Special Schools in Northern Ireland. 51 citizens registered for our first webinar exploring the relationship between OG and the SDGs. A session was delivered for comm & vol organisations and students from QUB exploring the benefits of open policy making and the rules of engagement between citizens and government.

A database of stakeholders with specific interest in the SDGs has been established providing a multi-sectoral Network for circulating news and information.

In Wales, project has identified that local well-being plans have not yet been communicated across to wider groups of citizens. Project partner ERS have planned crowdsource project to find out from non-voters what barriers are.

In Scotland, our event with CEMVO has demonstrated that BME equalities organisations have a strong interest in SDGs and are keen to use this as a hook for engagement with the wider BME community.

In Scotland, at two of our meetups with citizens and our first equalities roundtable with CEMVO, people were keen to take the project forward in conversation with other citizens but several said they needed ‘resources’. In particular, the Project team were asked if we could come to future events to present.

In NI, the children from special schools embraced the SDG themed democracy Games event. The young people had some serious fun and it was an enjoyable and empowering learning experience.

We will need a better plan in place to support people who are interested in hosting meetups, as simply providing more staff time will exhaust team's resources.

NI: design equally effective event formats for working with other equality groups

Open government seen as instrumental towards progressing SDGs. Media, academia, funding bodies, policy professionals, charities, private sector, international networks The Guardian Society has referenced a research report that claims that 30% of charities do not see the relevance of SDGs to UK. Meanwhile private sector is demonstrating increasing interest in engaging SDGs. This sets an external baseline for our engagement with Media, and charities over the course of this project.

In Scotland, third sector policy networks have expressed increasing interest in meeting to discuss open government, but less so on SDGs at the moment. Engagement with British Council has reached a collaborative application agreed and submitted under their ERASMUS plus strand for Youth voice in open democracy in May 2017. Academics from Edinburgh, Dundee and Stirling Universities have agreed to link with the themes of this project and work with us for towards an application to the the Scottish Universities Insight Institute - 'Data and Democracy in the Future City' May 2017.

In Wales, there was virtually zero awareness of the open government agenda in civil society. There is now interest and awareness in civil society and civil society practitioners are asking about the process - language will be important.

In Northern Ireland, the first webinar explored the instrumental value of open government in progressing the SDGs. Webinar participants included academics, policy professionals and NGO's from across the UK and further afield. The webinar recording and an OECD report on this topic were shared across communication channels to maximise the audience engaged.

In Scotland, YoungScot have endorsed our Wiki approach to planning equalities activities stressing the importance of clear mechanisms to engage young people at the co-design stage of our activities with them.

In NI, one of the main purposes of the Grassroots Democracy Games organised for young people with Special needs was to make the connection between democracy and policy making and the SDGs.

SDGs have not yet received mainstream buy-in from civil society organisations.

Project hasn't achieved mainstream media coverage.

NI: continue to adapt event formats for other groups and audiences

Engagement statistics[edit | edit source]

Indicator Commentary (updated May 2017) Nov 2016 Jan 2017 Feb Mar Apr May Jun Jul
Pioneers project partners (organisations registered on forum as Pioneers Project partners) Most organisations have one individual registered. Delivery partner organisations have 2-3 people registered. See https://forum.opengovernment.org.uk/groups/opengovpioneers 55 57
Open Government Forum registrations We have seen substantial increases in forum memberships in Scotland. All: 1149

Scot: 121

Wales: 29

NI: 299

All: 1207

Scot: 133

Wales: 29

NI: 303

All: 1310

UK: 402

Scot: 160

Wales: 34

NI: 326

Wales: 36

Wales: 48
Open Government Forum posts and replies Wales: 12 Wales: 19
Wiki visits Unable to track visits due to Miraheze platform restrictions - hoping this will change soon.
Wiki contributions Registration not required to contribute anonymously. Therefore we anticipate those that choose to register plan to contribute regularly or on multiple occasions. Wales: 1 32 contributors signed up

888 page edits

Wiki citations Difficult to track this properly due to limitations of platform. May need to review this indicator. Currently using Google links 150
Digital platform pageviews (globalgoals,scot) Platform not formally launched yet 741 1251 1663
Digital platform contributions *Contributions largely posted by SCVO for now 120*
Digital platform citations Platform not formally launched yet
Open Government Network blog reads 645
Open Government Network ranking (google)
Open Government Network blog/article contributions Wales: 1 Wales: 1

NI: 3

Scot: 1

UK: 2

Open Government Network ref group posts and replies
Meetup.com registrations (see graph below) UK (Scot): 99 UK (Scot and Wales): 163 UK (Scot and Wales): 200
Twitter followers Scot: 175

Wales:

104

Wales: 120 Wales: 143

Scot: 231

Wales: 189
Meetup.com graph for growth of meetup registrations for Opengovpioneers shows rapid growth to 200 people
Meetup.com graph for growth of meetup registrations for Opengovpioneers