Aims and objectives
How do we know that the Open Government Pioneers UK initiative is making a difference?
- The evaluation of the core activities funded by the Big Lottery Fund UK will be delivered in partnership with an external consultant with expertise in this area selected by the Project board. Budget has been allocated for the consultants fee.
- The evaluation of the equality activities will involve an equality impact assessment and further consultancy budget for specific equality expertise has been allocated for this.
We want the learn from the mistakes we make and build on the successes we create. Is there evidence that a diverse range of citizens and civil society are any closer to engaging the decisions that affect them as a result of the Open Government Pioneers Project UK?
Track progress towards the following four big outcomes.
- Empowered engaged citizens, leveraging their knowledge of SDGs.
- Political representatives have greater awareness of SDGs and types of actions required.
- Government officials have a better plan for involving citizens.
- Open government seen as instrumental, and a common good towards progressing SDGs.
- To use an open and agile approach to monitoring, learning and evaluation
- To assess how successful the project is at opening up government progress against the Sustainable Development Goals to citizen participation
- To inform future activities that support Open Government approaches to delivering the Sustainable Development Goals
|Influencers - politicians, government officials, institutional partners||As critical friends, advisors on positioning of the initiative and its outputs||Learning shared via briefings and presentations|
|Participants - civil society activists, engaged citizens||Directly involved in shaping the project's design, analysis and presentation from the start (using the Wiki and other digital platforms)||Learning shared via collaborative Wiki and through Forums, including Forum polling|
|General Public||e-petitions, social media, comments on new sites||Learning shared via mainstream media reflective articles|
|Are the objectives SMART (i.e. achievable and measurable)?|
|Is the overall approach acceptable and ethical?|
|Have we got the right audiences to involve in this? Anyone missing?|
How should the project be evaluated?
Proposed evaluation framework
|STREAM||INDICATORS||EVIDENCE||COLLECTION METHOD||RESPONSIBILITY||WHEN REPORTED|
|NI Outputs||4 Webinars||Theme of webinar; date delivered; numbers registered; participation feedback||Analytics, narrative & participant survey||NIEL||Annually|
|4 Equality Events||Theme; date delivered; numbers attended||Analytics & narrative report||NIEL||Annually|
|1 National Conference||Sessions delivered, themes, attendee numbers & feedback, outputs||Analytics, narrative & participant survey||All||Post Conference|
|Blogs & practice to policy paper||Themes, number produced, blog reads||Analytics||All||Bi-annually|
|STREAM||INDICATORS||EVIDENCE||COLLECTION METHOD||RESPONSIBILITY||WHEN REPORTED|
|Awareness (Capacity)||Articulation of issues||Events, blogs, forum posts, wiki posts, website updates||Analytics & narrative report||All||Bi-annually|
|Publicity & Promotion||Tweets, articles, media coverage||Analytics & narrative report||All||Bi-annually|
|Sharing (Learning)||New approaches & tools used||Itemisation of new tools & methods used||Narrative report||All||Bi-annually|
|Methods & approaches shared with project partners and other stakeholders||Details of new tools & methods shared||Narrative report||All||Bi-annually|
|Participation (Engagement)||New participants/members||Forum registrations, twitter followers||Analytics & narrative report||All||Bi-annually|
|New collaborations||Number of new partners & collaborators||Analytics & narrative report||All||Bi-annually|
|Digital platform interaction||Number of website/platform visits & interactions||Analytics||All||Bi-annually|
|Event & webinar attendees||Number & demographic||Analytics||All||Bi-annually|
|Influence (Outcomes/Impact)||Progress with SDGs||Implementation plan||Narrative report||All||Bi-annually|
|Citizen involvement||Participation opportunities created||Narrative report||All||Bi-annually|
|Policy influence||Influence on government policy, for example, the PfG, SDG implementation plan||Analytics & narrative report||All||Bi-annually|
|Transparency & accountability of process||Government utilisation of OG methods in the planning & implementation process||Narrative report||All||Bi-annually|
Agile learning approach
- Initial data collected in the first quarter of the project will form a baseline (against the Data subjects in the table below)
- An assessment will be made of any changes when reported (quarterly, monthly or annually as appropriate). Advice from Reference Group partners will be sought as appropriate.
- A recommendation will be made to the Project Board by the Delivery Group to change project elements based in this assessment.
Data underpinning evaluation
|Stream||Data subject||Data||Relevant targets||Collection method||Responsibility||When reported|
Posts and replies
|Increase forum registrations from 1000 to 5000 and aim for fivefold increase in each nation||Website usage stats||Each nation lead partner||Quarterly|
|Capacity||Opengov pioneers Wiki||Visits
|Tbc after setting baseline||Website analytics
|Capacity||Opengov collaboration||Contributions to crowdsourcing platforms
New initiatives between citizens sparked as a result of project activities
|Submissions from 20% of members during lifecycle of project||Crowdsourcing platform analytics
|Engagement||Opengov website||Blog reads
|Engagement||Positioning of Open Government agenda in delivery of SDGs||Stakeholder analysis||Qualitative||Influencer interviews / meetings||Each nation lead partner||Quarterly|
|Engagement||Impact on SDG progress||Policy documents (government)||Qualitative & # of citations||Rapid review and appraisal||Consultant||Annually|
|Learning||Knowledge of what works in using Open Government to progress SDGs||Blogs and articles from members||Submissions from 20% of members during lifecycle of project||Media analysis||Each nation lead partner||Annually|
|Learning||Engagement between Pioneer civil society on what works||Pioneer forum engagement
Posts and replies
|Tbc after setting baseline||Website usage stats||SCVO & Involve UK||Quarterly|
|Learning||Engagement between UK nations on what works||Stakeholder analysis||Qualitative||Influencer interviews / meetings||NIEL||Annually|
We need to ensure that the project success is properly and independently assessed by an independent evaluator. This is for the following reasons:
- The use of an independent evaluation consultant was agreed by project core partners and specified in the bid to the Big Lottery Fund UK. There will be an expectation of this by all partners who have reviewed the application and the plans which was published on this Wiki at that time
- An independent evaluation will allow a more objective evaluation of the success of the project than project core partners interviewing or getting feedback themselves from project beneficiaries
- A single independent evaluation rather than separate evaluations in each UK home nation will allow us to stay within budget and capture differences and common experiences across the nations.
- We will use the process towards recruiting the independent evaluator to generate a conversation between existing and prospective delivery partners on our evaluation approach. In particular, we will seek advice on how to get a clearer line of sight between the project output delivery and progress towards outcomes
We will openly advertise and recruit an evaluation partner (consultant) to deliver the independent evaluation, and another evaluation partner to deliver the equalities evaluation. The project's Delivery Group's recommendation to the project board will be to use a single evaluator for both the main and equalities evaluation of the project. This is to recognise the importance of embedding equalities in the overall evaluation approach. It would also fit into the outcome Empowered engaged citizens, leveraging their knowledge of SDGs, emphasising the focus on engaging people whose voices are least heard.
In order to inform the process used to recruit the evaluation consultant, we will draw on the pilot approach we used to recruit the equalities trainer for the project team. This involved an open approach, where potential delivery partners self-identified their interest on this Wiki on the Partners page. They were subsequently added to the Opengovernment Pioneers Project Partners forum and were encourages to comment on the overall process and brief openly to other partners before making their pitch.
Access the evaluator tender brief here.
The evaluation will involve interviews with project partners, project participants and government, and will be supported by the data collected as part of the project's monitoring activities and a discussion with the project board and input from project's forums around evaluation priorities. The evaluator will be expected to make recommendations about how to continue and develop the engagement of citizens and civil society in using open government to secure progress towards SDGs.
UPDATE: Mutualgain has been appointed as the independent evaluator. See Partners page for more about them.
You can access all the outputs from the evaluation activities at this webpage - www.mutualgain.org/ogpeval
You can shortly access the full draft of Mutualgain's independent evaluation report (Aug 2018) here.
It is important that this project does not rely completely on the independent evaluation to understand the difference it is making.
- The UK Conference will include a workshop component to facilitate stakeholder analysis, where project participants collectively deliberate around what works in delivering open government approaches to SDGs.
- The Activity sprint on international learning in Summer 2018 will also provide international level learning about what works and external perspective on how the project is making a difference based on International webinars with Pioneer civil society cohort
- Crucially, the independent evaluation will be accompanied by six monthly self-evaluations by each core partner for each home nation which will be captured and reported through this Wiki.
Timescales for evaluation
|Evaluation plan on Wiki
Delivery Group Self-evaluation
|Meeting with Project Board
Paper outlining equalities impact proofing project activities
|Recruitment process agreed||Aug 17||Completed|
|Recruitment of evaluators
Delivery Group Self-evaluation
|Dec / Jan 18||Completed|
|Evaluator session with Project Board||Jan 18||Completed|
Delivery Group Self-evaluation
|Evaluation report and recommendations||Jul 18|
|UK wide sprint to review project and legacy||Sep 18|
How will we ensure a strong equalities focus?
This Open Government Pioneers Project is primarily about raising awareness and supporting the implementation of the SDGs through Open Government principles and methods, with an emphasis on reaching groups who have difficulty in having their voices heard.
We will use the following approaches to ensure an equalities focus is embedded at all stage and all activities of the project:
- Prioritising budget and engagement with partners and groups supporting those without a strong voice in opengovernment and SDG discussions
- Securing strong buy-in to this project from a variety of equalities-focused civil society organisations
- Ensuring all project activities seek wider engagement than the 'usual suspects' - e.g. using livestreaming, non-jargon language, accessibility checks for digital platforms.
- Ensuring all project activities are as open and accessible as possible at the design stage to people who face additional barriers to engaging the decisions that affect them.
- We will learn from the existing evidence base available to us, starting with the research and reports referenced at Analysis and theory of change#Who participates.3F
|Do the proposed evaluation indicators link closely enough with the objectives?|
|Is the timescale proposed realistic?|
|Is the proposal to ensure a strong equalities focus adequate and realistic?|
Updated Jan 2018
Overall impact assessment
- The project has facilitated a significant renewal of interest in the Sustainable Development Goals which had been flagging since the launch of the SDGs in late 2015.
- The project has also increased interest and visibility in the Open Government Partnership and action plans amongst the policy communities in Scotland and Wales.
- The project has secured a greater appreciation by government and civil society participants across the UK and has engaged a broader constituency of interests in the open government partnership of the link between OGP and SDGs in Northern Ireland and England.
- The project is beginning to broaden awareness and interest in OGP and SDGs in the broader public and media. There has been a steady increase in engagement with the structures and planned engagements that are now in place to progress this over the previous six months of the project.
- The project has for the first time brought engagement around open government and SDGs to the attention of young people, and people engaged with minority and disability groups.
Impact against outcomes
NB. Text in italics is due for an update.
|Outcomes||Beneficiary||Impact assessment||Areas for improvement|
|Political representatives have greater awareness of SDGs and types of actions required.||Influencers - politicians, government officials, institutional partners||UK level:
|NI: political literacy around the SDGs is still low. Continue to raise understanding through the Pioneer conference and include OG Implementation Group in deliberative citizen engagement training.|
|Government officials have a better plan for involving citizens.||Participants - civil society activists, engaged citizens||UK Opengov networks:
|In Scotland, we need to explore how to improve enthusiasm in engaging government in partnership on opening up government.
NI: Continue to push for the OG commitments to be progressed by the NICS Board. This will be difficult.
Wales: It will be important to engage civil society from the outset of developing themes and commitments rather than just consulting them. There is a risk of this at the moment, and our project will be key to making sure this happens.
|Empowered engaged citizens, leveraging their knowledge of SDGs.||General Public||Scotland:
|Scotland: We will need a better plan in place to support people who are interested in hosting meetups, as simply providing more staff time will exhaust team's resources.
NI: Encouraging teachers to adopt our framework/methodology to run their own democracy games sessions in schools, so the approach can be sustained after the lifetime of the project. Adjust the methodology for 16-24 year olds.
Wales: There is a need to share the learning from these projects with Welsh Government and other relevant public bodies.
|Open government seen as instrumental towards progressing SDGs.||Media, academia, funding bodies, policy professionals, charities, private sector, international networks||UK wide
While the project has had success in building more interest in open government approaches to SDGs, it is clear that breaking into the mainstream media remains a challenge.
Stats are cumulative totals, unless otherwise specified
|Indicator||Commentary (updated Jan 2018)||Nov 2016||Mar
|Dec 2017||July 2018|
|Pioneers project partners (organisations registered on forum as Pioneers Project partners)||Most organisations have one individual registered. Delivery partner organisations have 2-3 people registered. See https://forum.opengovernment.org.uk/groups/opengovpioneers||0||55||57||87||103|
|Open Government Forum registrations||We have seen a doubling in forum memberships for Scotland and Wales, and a notable increase for both UK and NI forums.||All: 1149
NEW FORUM: All: 191
|Open Government Forum conversations||Activity around posting and replies are in line with numbers of registrations.||Wales: 12||UK: 411
|Wiki visits||Unable to track visits due to Miraheze platform restrictions - unlikely this will change soon.|
|Wiki contributions||Registration not required to contribute anonymously. Therefore we anticipate those that choose to register plan to contribute regularly or on multiple occasions.||Wales: 1||32 contributors signed up
888 page edits
|50 contributors signed up
1372 page edits
|75 contributors signed up
1,849 page edits
|Wiki citations (currently active)||Difficult to track this properly due to limitations of platform. May need to review this indicator. Currently using Google links||150||247||230|
|Digital platform unique users (globalgoals.scot)||N/A||N/A||167||1296|
|Digital platform page views (globalgoals.scot)||Platform has been soft launched and made available to those taking an interest in SDGs. Platform has not been launched yet formally through mainstream media.||N/A||147 from new users, 741 including returning||761 from new users, 1663 including returning||4766 from new users, 12,618 including returning|
|Digital platform contributions||*Contributions largely posted by SCVO for now||120*|
|Digital platform citations||Platform not formally launched yet|
|Open Government Network blog reads||analytics> behaviour>content drilldown> /2016/12/ + /2017/ + /2018/ (unique page views) stats are cumulative from end Nov 16||648||1002||2775||5854|
|Open Government Network blog/article contributions||statistics are cumulative starting end of Nov 16 (latest stats are until 03/08/2018)||Wales: 1
|Meetup.com registrations||Trebling of meetup.com registrations which comprises mainly people not already members of the opengov forums. i.e. engaged citizens.||UK (Scot and Wales): 163||UK (Scot and Wales): 200||UK (Scot and Wales): 331||UK (Scot and Wales): 397|
|Twitter followers||Substantial increases for Wales and Scotland as these are the new networks.||Scot: 175
This section summarises and assess legacy options to build on the project outputs.
Legacy planning suggestions
|Equalities co-production initiative||Supporting a partnership of Equalities organisations across the UK to host an open conversation around engaging people furthest from decision making into plans around SDGs and Open Government.||The Opengovpioneers project opened up engagement with equalities groups, but found a much more intensive capacity building exercise is needed to engage those furthest from the decisions that effect them in using opengov as a way to progress SDGs that affect them.||A consortium of equalities orgs led by a key equality network in each UK home nation, back by an online wiki to document and plan progress.
There is an opportunity for the opengovpioneers project to host an initial engagement event, that can be used to develop a proposal to BIG UK.
|Focus on Equalities - Food Poverty||Food poverty is a growing issue leading to (and resulting from) a range of inequalities. Aim is to support partnership of groups which will progress activity around this and related SDGs (particularly 1,2,3). Develop partnership approach with other cities to progress.||Pioneer project has been good at reaching equalities groups, but there is an argument for taking a more focused approach, and targeting specific issues or SDGs where we feel we can have impact. One such area is food poverty.||Linked to proposal around equalities co-production (above). There is a local food poverty equalities event planned for Autumn, which would bring a range of groups together. Belfast Food Network is the key co-ordinating body - opportunity to link with Nourish Scotland.||Early 2019|
|Open Glasgow initiative||Supporting Glasgow City Council and Glasgow civil society to consider whether to apply for the OGP local programme next year.||Glasgow is in a good place politically, technologically and socially to consider an Open Government Glasgow action plan.||Initial conversations led by GCC and GCVS with SCVO support, documented openly at this wiki.||Late 2019|
|Open Belfast Pioneer City||Belfast to be a Pioneer City championing open government at local level in NI||Belfast has been transformed over the past 2 decades, and the City Council has just developed a new Community Plan - the Belfast Agenda.||Potential for civil society to support BCC in development of OGP Action Plan; progress range of civic engagement methods to strengthen new plan; opportunities in connecting cities such as Glasgow/Belfast.||2019|
|Open Government civil society network Scotland||An independent civil society-led network in Scotland, building on the infrastructure set up by the opengovpioneers project, to maintain the network, forums and engagement with Scotgov Multi-stakeholder opengov forum.||The BIG lottery funded Opengovpioneers project was designed to build the infrastructure, visibility and engagement with Open Government and using this to advance the SDGs. However, the BIG lottery was an accelerator project, and not designed for maintenance of the network. Funding ends 2018, therefore a new plan and delivery model and funding partner(s) is required to secure a sustainable network.||As part of the legacy work, the Opengovpioneers project team will host discussions with project partners on opportunities and interest in leading the next phase of the opengov movement.||Late 2018|
|Gendered approach to securing the SDGs across the UK||Partnership of Open Government pioneer civil society organisations across the UK Home nations using a gendered approach to open government as a means to securing SDGs||We suspect that Gender equality has become one of the single most important factors holding back progress on the SDGs, but Opengovpioneers ddn't take a gendered lens to engagement in the project. Addressing this should help speed progress to achieving the SDGs.||A project led by existing core partners of the opengovpioneers project to advance a gender equalities based approach to the SDGs using open government approaches. Linking into existing civil society networks and groups that focus on gender.||Late 2019|
|Open Government 'beyond the usual suspects'||Partnership of Open Government civil society organisations across the UK using new methods to engage a wider group of people on Open Government and the next Open Government Action Plans for each of the nations.||The key principles of Open Government of transparency, accountability and participation could be game changing for a much wider representation of the public, and yet to date we have had limited engagement with 'harder to reach' groups. This project would have a particular emphasis on engaging more diverse groups of people in the project and development of the next plans, and would follow on from the issues identified in the Welsh network's equality projects.||A project led by existing partners, but working directly with groups that specifically represent groups of people that haven't been engaged to date.||2019|