Evaluation

From Open Government Pioneer Project
Jump to: navigation, search

Introduction[edit | edit source]

How do we know that the Open Government Pioneers UK initiative is making a difference?

  • The evaluation of the core activities funded by the Big Lottery Fund UK will be delivered in partnership with an external consultant with expertise in this area selected by the Project board. Budget has been allocated for the consultants fee.
  • The evaluation of the equality activities will involve an equality impact assessment and further consultancy budget for specific equality expertise has been allocated for this.

Aims and objectives[edit | edit source]

We want the learn from the mistakes we make and build on the successes we create. Is there evidence that a diverse range of citizens and civil society are any closer to engaging the decisions that affect them as a result of the Open Government Pioneers Project UK?

Objective[edit | edit source]

Track progress towards the following four big outcomes.

  1. Empowered engaged citizens, leveraging their knowledge of SDGs.
  2. Political representatives have greater awareness of SDGs and types of actions required.
  3. Government officials have a better plan for involving citizens.
  4. Open government seen as instrumental, and a common good towards progressing SDGs.

Overall approach[edit | edit source]

  • To use an open and agile approach to monitoring, learning and evaluation
  • To assess how successful the project is at opening up government progress against the Sustainable Development Goals to citizen participation
  • To inform future activities that support Open Government approaches to delivering the Sustainable Development Goals

Beneficiary involvement[edit | edit source]

Beneficiary Involvement Sharing learning
Influencers - politicians, government officials, institutional partners As critical friends, advisors on positioning of the initiative and its outputs Learning shared via briefings and presentations
Participants - civil society activists, engaged citizens Directly involved in shaping the project's design, analysis and presentation from the start (using the Wiki and other digital platforms) Learning shared via collaborative Wiki and through Forums, including Forum polling
General Public e-petitions, social media, comments on new sites Learning shared via mainstream media reflective articles

How should the project be evaluated?[edit | edit source]

Agile learning approach[edit | edit source]

  • Initial data collected in the first quarter of the project will form a baseline (against the Data subjects in the table below)
  • An assessment will be made of any changes when reported (quarterly, monthly or annually as appropriate). Advice from Reference Group partners will be sought as appropriate.
  • A recommendation will be made to the Project Board by the Delivery Group to change project elements based in this assessment.

Data underpinning evaluation[edit | edit source]

Stream Data subject Data Relevant targets Collection method Responsibility When reported
Capacity Opengov forum Registrations

Posts and replies

Increase forum registrations from 1000 to 5000 and aim for fivefold increase in each nation Website usage stats Each nation lead partner Quarterly
Capacity Opengov pioneers Wiki Visits

Active contributions

Citations

Tbc after setting baseline Website analytics

Google analytics

SCVO Monthly
Capacity Opengov collaboration Contributions to crowdsourcing platforms

New initiatives between citizens sparked as a result of project activities

Submissions from 20% of members during lifecycle of project Crowdsourcing platform analytics

Forum polling

Involve UK Quarterly
Engagement Opengov website Blog reads

Google ranking

tbc Website analytics

Google analytics

Involve UK Quarterly
Engagement Positioning of Open Government agenda in delivery of SDGs Stakeholder analysis Qualitative Influencer interviews / meetings Each nation lead partner Quarterly
Engagement Impact on SDG progress Policy documents (government) Qualitative & # of citations Rapid review and appraisal Consultant Annually
Learning Knowledge of what works in using Open Government to progress SDGs Blogs and articles from members Submissions from 20% of members during lifecycle of project Media analysis Each nation lead partner Annually
Learning Engagement between Pioneer civil society on what works Pioneer forum engagement

Posts and replies

Forum polls

Tbc after setting baseline Website usage stats SCVO & Involve UK Quarterly
Learning Engagement between UK nations on what works Stakeholder analysis Qualitative Influencer interviews / meetings NIEL Annually

Independent evaluator[edit | edit source]

We need to ensure that the project success is properly and independently assessed by an independent evaluator. This is for the following reasons:

  • The use of an independent evaluation consultant was agreed by project core partners and specified in the bid to the Big Lottery Fund UK. There will be an expectation of this by all partners who have reviewed the application and the plans which was published on this Wiki at that time
  • An independent evaluation will allow a more objective evaluation of the success of the project than project core partners interviewing or getting feedback themselves from project beneficiaries
  • A single independent evaluation rather than separate evaluations in each UK home nation will allow us to stay within budget and capture differences and common experiences across the nations.

We will openly advertise and recruit an evaluation partner (consultant) to deliver the independent evaluation, and another evaluation partner to deliver the equalities evaluation. The project's Delivery Group's recommendation to the project board will be to use a single evaluator for both the main and equalities evaluation of the project. This is to recognise the importance of embedding equalities in the overall evaluation approach. It would also fit into the outcome Empowered engaged citizens, leveraging their knowledge of SDGs, emphasising the focus on engaging people whose voices are least heard.

The evaluation will involve interviews with project partners, project participants and government, and will be supported by the data collected as part of the project's monitoring activities and a discussion with the project board and input from project's forums around evaluation priorities.

Timescales for evaluation[edit | edit source]

Evaluation plan on Wiki
Meeting with Project Board
Recruitment process agreed
Recruitment of evaluators
Evaluator session with Project Board
Evaluation delivery
Evaluation report
UK wide sprint to review project and legacy

Project Impact[edit | edit source]

Updated May 2017

Overall impact assessment[edit | edit source]

  • The project has facilitated a renewal of interest in the Sustainable Development Goals which had been flagging since the launch of the SDGs in late 2015.
  • The project has also increased interest and visibility in the Open Government Partnership and action plans amongst the policy communities in Scotland and Wales.
  • The project has secured a greater appreciation by government and civil society participants in the open government partnership of the link between OGP and SDGs in Northern Ireland and England.
  • The project has not yet broadened awareness and interest in OGP and SDGs in the broader public and media. However, a number of structures and planned engagements are now in place to progress this during the next six months of the project.

Impact against outcomes[edit | edit source]

Outcomes Beneficiary Impact assessment Equalities impact assessment Areas for improvement
Political representatives have greater awareness of SDGs and types of actions required. Influencers - politicians, government officials, institutional partners Project partner Scotland Malawi Partnership has used the leverage of our new globalgoals.scot platform 100 MSPs have signed up to commit to SDGs. The project was discussed as promising area of innovation in the recent UK Stakeholders for Sustainable Development conference in London, March 2017.

A clear connection has now been secured between the aims of the open government approach to sustainable development goals and the Wales Wellbeing of Future Generation Goals with political buy-in from the Wellbeing Commission into the aims of this project.

Government officials have a better plan for involving citizens. Participants - civil society activists, engaged citizens Around 1400 professionals have now joined the OGN network across the UK - around 300 since this project started. Most new sign-ups have taken place in Scotland and England. A series of active forum discussions have taken place around data, freedom of information and democracy. The challenge now is to turn this into collective action.

Scottish Government officials have now committed to integrating SDGs into their National Performance Framework and making links with this project and our Globalgoals.scot platform to support participation.

Empowered engaged citizens, leveraging their knowledge of SDGs. General Public Around 200 people have joined the Meetup groups set up for people and communities to engage with OGP and SDGs in Edinburgh, Glasgow, Dundee and Cardiff.
Open government seen as instrumental, and a common good towards progressing SDGs. Media, academia, funding bodies, policy professionals, charities, private sector, international networks Engagement with British Council has reached a collaborative application agreed and submitted under their ERASMUS plus strand for Youth voice in open democracy in May 2017.

Academics from Edinburgh, Dundee and Stirling Universities have agreed to link with the themes of this project and work with us for towards an application to the the Scottish Universities Insight Institute - 'Data and Democracy in the Future City' May 2017.

Project hasn't achieved mainstream media coverage.