Evaluation

From Open Government Pioneer Project
Jump to: navigation, search

Introduction[edit | edit source]

How do we know that the Open Government Pioneers UK initiative is making a difference?

  • The evaluation of the core activities funded by the Big Lottery Fund UK will be delivered in partnership with an external consultant with expertise in this area selected by the Project board. Budget has been allocated for the consultants fee.
  • The evaluation of the equality activities will involve an equality impact assessment and further consultancy budget for specific equality expertise has been allocated for this.

Aims and objectives[edit | edit source]

We want the learn from the mistakes we make and build on the successes we create. Is there evidence that a diverse range of citizens and civil society are any closer to engaging the decisions that affect them as a result of the Open Government Pioneers Project UK?

Objective[edit | edit source]

Track progress towards the following four big outcomes.

  1. Empowered engaged citizens, leveraging their knowledge of SDGs.
  2. Political representatives have greater awareness of SDGs and types of actions required.
  3. Government officials have a better plan for involving citizens.
  4. Open government seen as instrumental, and a common good towards progressing SDGs.

Overall approach[edit | edit source]

  • To use an open and agile approach to monitoring, learning and evaluation
  • To assess how successful the project is at opening up government progress against the Sustainable Development Goals to citizen participation
  • To inform future activities that support Open Government approaches to delivering the Sustainable Development Goals

Beneficiary involvement[edit | edit source]

Beneficiary Involvement Sharing learning
Influencers - politicians, government officials, institutional partners As critical friends, advisors on positioning of the initiative and its outputs Learning shared via briefings and presentations
Participants - civil society activists, engaged citizens Directly involved in shaping the project's design, analysis and presentation from the start (using the Wiki and other digital platforms) Learning shared via collaborative Wiki and through Forums, including Forum polling
General Public e-petitions, social media, comments on new sites Learning shared via mainstream media reflective articles

Key questions[edit | edit source]

Question Suggestion Comments Agreed
Are the objectives SMART (i.e. achievable and measurable)?
Is the overall approach acceptable and ethical?
Have we got the right audiences to involve in this? Anyone missing?
Other?

How should the project be evaluated?[edit | edit source]

Proposed evaluation framework

STREAM INDICATORS EVIDENCE COLLECTION METHOD RESPONSIBILITY WHEN REPORTED
NI Outputs 4 Webinars Theme of webinar; date delivered; numbers registered; participation feedback Analytics, narrative & participant survey NIEL Annually
4 Equality Events Theme; date delivered; numbers attended Analytics & narrative report NIEL Annually
1 National Conference Sessions delivered, themes, attendee numbers & feedback, outputs Analytics, narrative & participant survey All Post Conference
Blogs & practice to policy paper Themes, number produced, blog reads Analytics All Bi-annually
STREAM INDICATORS EVIDENCE COLLECTION METHOD RESPONSIBILITY WHEN REPORTED
Awareness (Capacity) Articulation of issues Events, blogs, forum posts, wiki posts, website updates Analytics & narrative report All Bi-annually
Publicity & Promotion Tweets, articles, media coverage Analytics & narrative report All Bi-annually
Sharing (Learning) New approaches & tools used Itemisation of new tools & methods used Narrative report All Bi-annually
Methods & approaches shared with project partners and other stakeholders Details of new tools & methods shared Narrative report All Bi-annually
Participation (Engagement) New participants/members Forum registrations, twitter followers Analytics & narrative report All Bi-annually
New collaborations Number of new partners & collaborators Analytics & narrative report All Bi-annually
Digital platform interaction Number of website/platform visits & interactions Analytics All Bi-annually
Event & webinar attendees Number & demographic Analytics All Bi-annually
Influence (Outcomes/Impact) Progress with SDGs Implementation plan Narrative report All Bi-annually
Citizen involvement Participation opportunities created Narrative report All Bi-annually
Policy influence Influence on government policy, for example, the PfG, SDG implementation plan Analytics & narrative report All Bi-annually
Transparency & accountability of process Government utilisation of OG methods in the planning & implementation process Narrative report All Bi-annually

Agile learning approach[edit | edit source]

  • Initial data collected in the first quarter of the project will form a baseline (against the Data subjects in the table below)
  • An assessment will be made of any changes when reported (quarterly, monthly or annually as appropriate). Advice from Reference Group partners will be sought as appropriate.
  • A recommendation will be made to the Project Board by the Delivery Group to change project elements based in this assessment.

Data underpinning evaluation[edit | edit source]

Stream Data subject Data Relevant targets Collection method Responsibility When reported
Capacity Opengov forum Registrations

Posts and replies

Increase forum registrations from 1000 to 5000 and aim for fivefold increase in each nation Website usage stats Each nation lead partner Quarterly
Capacity Opengov pioneers Wiki Visits

Active contributions

Citations

Tbc after setting baseline Website analytics

Google analytics

SCVO Monthly
Capacity Opengov collaboration Contributions to crowdsourcing platforms

New initiatives between citizens sparked as a result of project activities

Submissions from 20% of members during lifecycle of project Crowdsourcing platform analytics

Forum polling

Involve UK Quarterly
Engagement Opengov website Blog reads

Google ranking

tbc Website analytics

Google analytics

Involve UK Quarterly
Engagement Positioning of Open Government agenda in delivery of SDGs Stakeholder analysis Qualitative Influencer interviews / meetings Each nation lead partner Quarterly
Engagement Impact on SDG progress Policy documents (government) Qualitative & # of citations Rapid review and appraisal Consultant Annually
Learning Knowledge of what works in using Open Government to progress SDGs Blogs and articles from members Submissions from 20% of members during lifecycle of project Media analysis Each nation lead partner Annually
Learning Engagement between Pioneer civil society on what works Pioneer forum engagement

Posts and replies

Forum polls

Tbc after setting baseline Website usage stats SCVO & Involve UK Quarterly
Learning Engagement between UK nations on what works Stakeholder analysis Qualitative Influencer interviews / meetings NIEL Annually

Independent evaluator[edit | edit source]

We need to ensure that the project success is properly and independently assessed by an independent evaluator. This is for the following reasons:

  • The use of an independent evaluation consultant was agreed by project core partners and specified in the bid to the Big Lottery Fund UK. There will be an expectation of this by all partners who have reviewed the application and the plans which was published on this Wiki at that time
  • An independent evaluation will allow a more objective evaluation of the success of the project than project core partners interviewing or getting feedback themselves from project beneficiaries
  • A single independent evaluation rather than separate evaluations in each UK home nation will allow us to stay within budget and capture differences and common experiences across the nations.
  • We will use the process towards recruiting the independent evaluator to generate a conversation between existing and prospective delivery partners on our evaluation approach. In particular, we will seek advice on how to get a clearer line of sight between the project output delivery and progress towards outcomes

We will openly advertise and recruit an evaluation partner (consultant) to deliver the independent evaluation, and another evaluation partner to deliver the equalities evaluation. The project's Delivery Group's recommendation to the project board will be to use a single evaluator for both the main and equalities evaluation of the project. This is to recognise the importance of embedding equalities in the overall evaluation approach. It would also fit into the outcome Empowered engaged citizens, leveraging their knowledge of SDGs, emphasising the focus on engaging people whose voices are least heard.

In order to inform the process used to recruit the evaluation consultant, we will draw on the pilot approach we used to recruit the equalities trainer for the project team. This involved an open approach, where potential delivery partners self-identified their interest on this Wiki on the Partners page. They were subsequently added to the Opengovernment Pioneers Project Partners forum and were encourages to comment on the overall process and brief openly to other partners before making their pitch.

Access the evaluator tender brief here.

The evaluation will involve interviews with project partners, project participants and government, and will be supported by the data collected as part of the project's monitoring activities and a discussion with the project board and input from project's forums around evaluation priorities. The evaluator will be expected to make recommendations about how to continue and develop the engagement of citizens and civil society in using open government to secure progress towards SDGs.

UPDATE: Mutualgain has been appointed as the independent evaluator. See Partners page for more about them.

Self-evaluation[edit | edit source]

It is important that this project does not rely completely on the independent evaluation to understand the difference it is making.

Therefore:

  1. The UK Conference will include a workshop component to facilitate stakeholder analysis, where project participants collectively deliberate around what works in delivering open government approaches to SDGs.
  2. The Activity sprint on international learning in Summer 2018 will also provide international level learning about what works and external perspective on how the project is making a difference based on International webinars with Pioneer civil society cohort
  3. Crucially, the independent evaluation will be accompanied by six monthly self-evaluations by each core partner for each home nation which will be captured and reported through this Wiki.

Timescales for evaluation[edit | edit source]

Milestone Timescale Status
Evaluation plan on Wiki

Delivery Group Self-evaluation

May 17 Completed
Meeting with Project Board

Paper outlining equalities impact proofing project activities

Jun 17 Completed
Recruitment process agreed Aug 17 Completed
Recruitment of evaluators

Delivery Group Self-evaluation

Dec / Jan 18 Completed
Evaluator session with Project Board Jan 18 Scheduled
Evaluation delivery

Delivery Group Self-evaluation

May 18
Evaluation report and recommendations Jul 18
UK wide sprint to review project and legacy Sep 18

How will we ensure a strong equalities focus?[edit | edit source]

This Open Government Pioneers Project is primarily about raising awareness and supporting the implementation of the SDGs through Open Government principles and methods, with an emphasis on reaching groups who have difficulty in having their voices heard.

We will use the following approaches to ensure an equalities focus is embedded at all stage and all activities of the project:

  • Prioritising budget and engagement with partners and groups supporting those without a strong voice in opengovernment and SDG discussions
  • Securing strong buy-in to this project from a variety of equalities-focused civil society organisations
  • Ensuring all project activities seek wider engagement than the 'usual suspects' - e.g. using livestreaming, non-jargon language, accessibility checks for digital platforms.
  • Ensuring all project activities are as open and accessible as possible at the design stage to people who face additional barriers to engaging the decisions that affect them.
  • We will learn from the existing evidence base available to us, starting with the research and reports referenced at Analysis and theory of change#Who participates.3F

Key questions[edit | edit source]

Question Suggestion Comments Agreed
Do the proposed evaluation indicators link closely enough with the objectives?
Is the timescale proposed realistic?
Is the proposal to ensure a strong equalities focus adequate and realistic?

Project Impact[edit | edit source]

Updated Jan 2018

Overall impact assessment[edit | edit source]

  • The project has facilitated a significant renewal of interest in the Sustainable Development Goals which had been flagging since the launch of the SDGs in late 2015.
  • The project has also increased interest and visibility in the Open Government Partnership and action plans amongst the policy communities in Scotland and Wales.
  • The project has secured a greater appreciation by government and civil society participants across the UK and has engaged a broader constituency of interests in the open government partnership of the link between OGP and SDGs in Northern Ireland and England.
  • The project is beginning to broaden awareness and interest in OGP and SDGs in the broader public and media. There has been a steady increase in engagement with the structures and planned engagements that are now in place to progress this over the previous six months of the project.
  • The project has for the first time brought engagement around open government and SDGs to the attention of young people, and people engaged with minority and disability groups.

Impact against outcomes[edit | edit source]

Outcomes Beneficiary Impact assessment Areas for improvement
Political representatives have greater awareness of SDGs and types of actions required. Influencers - politicians, government officials, institutional partners UK level:
  • UKSSD are leading a UK-wide action plan for SDG and have engaged directly with the opengovpioneers project at the highest level in order to open up the process for this action plan development.

Scotland:

  • The Sustainable Development Goals Network has become an active group of around 59 diverse and influential members across sectors and society.
  • With facilitation support they have developed a plan for increasing the momentum towards SDGs in Scotland which is now being put into action.
  • Visibility and engagement with the globalgoals.scot platform is growing pending a formal launch early this year.
  • Politicians are warm to the SDGs, and now see this as a domestic policy issue, but it has not yet entered everyday parliamentary debate. Domestic poverty and inequality as priority themes are key drivers for interest in SDGs.

Northern Ireland:

  • Difficult situation with Assembly suspension. Talks process ongoing and hoping for decision on future governing arrangements. The NI Open Government Network has contributed to efforts to pilot a citizens Assembly in NI. A pilot should be run in 2018 and the topic received widespread mainstream and political attention in recent weeks.

Wales: In Wales the focus of the project is on the Well-being Goals. There is a particular focus on the engagement & involvement elements of the Well-being of Future Generations Act (the 5 ways of working) and a number of network meetings between civil society & Government officials have now taken place ensuring better involvement in the most integrated way, and learning is shared.

England:

  • Civil society has been scoping city-level political representatives with a view to informing them of the need to implement SDGs.
Wales: engagement with local government will be important as they will be delivering well-being plans from this year, reporting against progress on the well-being goals.

NI: political literacy around the SDGs is still low. Continue to raise understanding through the Pioneer conference.

Government officials have a better plan for involving citizens. Participants - civil society activists, engaged citizens UK Opengov networks:
  • Government has been responsive and engaged with our forums in Scotland, Wales and England.
  • UK and SG Government are planning to collaborate around bringing OGP approach to UK cities. Potential now for an application by a UK city to the next cohort of subnational pioneers.

Scotland:

  • Civil society has been invited to contribute to a very closely aligned national performance framework with SDGs being developed by Scotgov,
  • We are seeing the incorporation of SDGs into a number of civil society organisational plans.
  • A ministerial roundtable was also held with civil society, where the Scottish Government re-committed to a new round of co-produced open government action plans for the next year, and to facilitate engagement with this collaboratively with the opengovpioneers project. A number of structures have already been set up to facilitate this.

Northern Ireland:

  • Close working with NI Government, but difficult to progress without Ministerial direction.
  • Engagement with the UK OGP action plan will be key influence.

Wales: A plan is currently being initiated with Welsh Government’s Digital team who are responsible for the development of the Welsh commitments for the 4th NAP. Involvement of policy leads across Welsh Government such as Head of Local Government, the Equalities dept and the Cabinet Office (who head up the SDGs reporting of Welsh Government) will ensure wider involvement.

England:

  • Government officials have provided informal feedback on whether new metro-mayors could be invited to submit application as subnational OGP pioneer.
In Scotland, we need a clearer strategy for follow through for interest in open government and SDGs by citizens and civil society nurtured by this project.

NI: Encourage the Open Government Implementation Group to incorporate an SDGs commitment in the new OG Action Plan. This will be difficult.

Wales: It will be important to engage civil society from the outset of developing themes and commitments rather than just consulting them. There is a risk of this at the moment, and our project will be key to making sure this happens.

Empowered engaged citizens, leveraging their knowledge of SDGs. General Public Scotland and Wales:
  • 316 people have joined the Meetup groups set up for people and communities to engage with OGP and SDGs in Edinburgh, Glasgow, Dundee and Cardiff.
  • The Meetups themselves are currently not as engaged as we would like them, and we are developing partnerships with equalities organisations in order to address this.

Equalities engagement:

  • In Scotland, our events with young people, disabled people and ethnic minorities, has seeded a number of new partnerships and applications to funders.
  • There was a clear interest in greater capacity building from equalities groups to develop engagement around SDGs and opengov. As a project we are therefore investing more resources now into building the capacity of our equalities partners to deliver project outputs.

Northern Ireland:

  • Our publicity SDGs is surfacing people/orgs who are not the usual suspects.
  • Youth engagement programme has resulted in considerable awareness raising and capacity building around the SDGs and OG amongst youth groups and special needs youth groups in NI. The Democracy Games methodology is a colourful and entertaining way of engaging youth groups and inspiring children around the topic of open gov and sustainable development. Three events engaged approx. 150 young people and 20 teachers/coordinators.

Wales: Through the 3 equalities projects that received funding from the Wales Open Government Network engagement is now being undertaken in earnest with grassroots groups & individuals with protected characteristics. This will link to the Well-Being Goals and SDGs. A project bid is being developed currently to develop the Wales GlobalGoals website.

England:

  • We have been identifying key local groups to engage in open government to bring in the SDG aspect to the action plan process
Scotland: We will need a better plan in place to support people who are interested in hosting meetups, as simply providing more staff time will exhaust team's resources.

NI: Encouraging teachers to adopt our framework/methodology to run their own democracy games sessions in schools, so the approach can be sustained after the lifetime of the project. Adjust the methodology for 16-24 year olds.

Wales: There will be a real need to share the learning from citizens, third sector and government on achievements of well-being goals / SDGs.

Open government seen as instrumental towards progressing SDGs. Media, academia, funding bodies, policy professionals, charities, private sector, international networks UK wide
  • Around 1600 professionals have now joined the OGN network across the UK - around 500 since this project started. Most new sign-ups have taken place in Scotland and England.
  • A series of active forum discussions have translated into letters and campaigns by members of the network around data, freedom of information and democracy.
  • We note increasing international interest in the link between SDGs and Open Government being pioneered across the UK. The OGP symposiums at the Haig and in Washington noted strong interest in our opengovpioneers approach to linking OGP with SDGs.

Scotland

  • First joint engagement with mainstream media journalists which reflects increasing interest in meeting to discuss open government, but less so on SDGs at the moment.
  • For civil society organisations and third sector intermediaries, there is a real recognition now that SDGs captures sector values well and can act as a useful framing tool for collaboration and policy development.

Northern Ireland:

Wales:

  • WCVA are linking with the Office for the Future Generations Commissioner, who oversee the well-being plans of local authorities, as part of the Well-Being of Future Generations Act. There is now a clear links between open government approach and the approach being taken by the Well-being Act. There is an opportunity therefore to really embed open government in the delivery of the Act.
  • Engagement with open government project is primarily through workshops and events, although interest and use of the digital tools and the forum is increasing.

England:

  • Initiation of 4th National OGP UK action plan, is using SDG themes for the first time to frame workshops.
Project hasn't achieved mainstream media coverage, but interest amongst journalists is now clear and can be built on.

Democracy engagement approaches pioneered in NI are now being spread to the other countries via this project.

Wales: Still early days in building open government momentum in third sector, but this is the first year where the link has been made (primarily by this project). Internet capacity still limits digital engagement for Wales citizens and third sector, and is an area for further development.

Engagement statistics[edit | edit source]

Stats are cumulative totals, unless otherwise specified

Indicator Commentary (updated Jan 2018) Nov 2016 Mar

2017

May

2017

Dec 2017
Pioneers project partners (organisations registered on forum as Pioneers Project partners) Most organisations have one individual registered. Delivery partner organisations have 2-3 people registered. See https://forum.opengovernment.org.uk/groups/opengovpioneers 0 55 57 87
Open Government Forum registrations We have seen a doubling in forum memberships for Scotland and Wales, and a notable increase for both UK and NI forums. All: 1149

Scot: 121

Wales: 29

NI: 299

All: 1310

UK: 402

Scot: 160

Wales: 34

NI: 326

Wales: 36

All: 1595

UK: 428

Scot: 260

Wales: 63

NI: 378

Open Government Forum posts and replies Activity around posting and replies are in line with numbers of registrations. Wales: 12 UK: 411

Scot: 164

Wales: 28

NI: 185

Wiki visits Unable to track visits due to Miraheze platform restrictions - unlikely this will change soon.
Wiki contributions Registration not required to contribute anonymously. Therefore we anticipate those that choose to register plan to contribute regularly or on multiple occasions. Wales: 1 32 contributors signed up

888 page edits

50 contributors signed up

1372 page edits

Wiki citations Difficult to track this properly due to limitations of platform. May need to review this indicator. Currently using Google links 150 247
Digital platform unique users (globalgoals.scot) N/A N/A 167 1296
Digital platform page views (globalgoals.scot) Platform has been soft launched and made available to those taking an interest in SDGs. Platform has not been launched yet formally through mainstream media. N/A 147 from new users, 741 including returning 761 from new users, 1663 including returning 4766 from new users, 12,618 including returning
Digital platform contributions *Contributions largely posted by SCVO for now 120*
Digital platform citations Platform not formally launched yet
Open Government Network blog reads analytics> behaviour>content drilldown> 2016/12 + 2017 (unique page views) stats are cumulative from end Nov 16 648 1002 2775
Open Government Network ranking (google)
Open Government Network blog/article contributions statistics are cumulative starting end of Nov 16 (latest stats are until end of december) Wales: 1

NI:1

Scot: 2

UK: 5

Mixed: 4

Wales: 1

NI: 4

Scot: 3

UK: 9

Mixed: 4

Wales: 1

NI: 12

Scot:22

UK: 14

Mixed: 6

Open Government Network ref group posts and replies
Meetup.com registrations Trebling of meetup.com registrations which comprises mainly people not already members of the opengov forums. i.e. engaged citizens. UK (Scot and Wales): 163 UK (Scot and Wales): 200 UK (Scot and Wales): 331
Twitter followers Substantial increases for Wales and Scotland as these are the new networks. Scot: 175

Wales:

104

Wales: 143

Scot: 231

UK: 1997

Wales: 346

Scot: 538

NI: 1320