- 1 Introduction
- 2 Aims and objectives
- 3 How should the project be evaluated?
- 4 Project Impact
Introduction[edit | edit source]
How do we know that the Open Government Pioneers UK initiative is making a difference?
- The evaluation of the core activities funded by the Big Lottery Fund UK will be delivered in partnership with an external consultant with expertise in this area selected by the Project board. Budget has been allocated for the consultants fee.
- The evaluation of the equality activities will involve an equality impact assessment and further consultancy budget for specific equality expertise has been allocated for this.
Aims and objectives[edit | edit source]
We want the learn from the mistakes we make and build on the successes we create. Is there evidence that a diverse range of citizens and civil society are any closer to engaging the decisions that affect them as a result of the Open Government Pioneers Project UK?
Objective[edit | edit source]
Track progress towards the following four big outcomes.
- Empowered engaged citizens, leveraging their knowledge of SDGs.
- Political representatives have greater awareness of SDGs and types of actions required.
- Government officials have a better plan for involving citizens.
- Open government seen as instrumental, and a common good towards progressing SDGs.
Overall approach[edit | edit source]
- To use an open and agile approach to monitoring, learning and evaluation
- To assess how successful the project is at opening up government progress against the Sustainable Development Goals to citizen participation
- To inform future activities that support Open Government approaches to delivering the Sustainable Development Goals
Beneficiary involvement[edit | edit source]
|Influencers - politicians, government officials, institutional partners||As critical friends, advisors on positioning of the initiative and its outputs||Learning shared via briefings and presentations|
|Participants - civil society activists, engaged citizens||Directly involved in shaping the project's design, analysis and presentation from the start (using the Wiki and other digital platforms)||Learning shared via collaborative Wiki and through Forums, including Forum polling|
|General Public||e-petitions, social media, comments on new sites||Learning shared via mainstream media reflective articles|
Key questions[edit | edit source]
|Are the objectives SMART (i.e. achievable and measurable)?|
|Is the overall approach acceptable and ethical?|
|Have we got the right audiences to involve in this? Anyone missing?|
How should the project be evaluated?[edit | edit source]
Proposed evaluation framework
|STREAM||INDICATORS||EVIDENCE||COLLECTION METHOD||RESPONSIBILITY||WHEN REPORTED|
|NI Outputs||4 Webinars||Theme of webinar; date delivered; numbers registered; participation feedback||Analytics, narrative & participant survey||NIEL||Annually|
|4 Equality Events||Theme; date delivered; numbers attended||Analytics & narrative report||NIEL||Annually|
|1 National Conference||Sessions delivered, themes, attendee numbers & feedback, outputs||Analytics, narrative & participant survey||All||Post Conference|
|Blogs & practice to policy paper||Themes, number produced, blog reads||Analytics||All||Bi-annually|
|STREAM||INDICATORS||EVIDENCE||COLLECTION METHOD||RESPONSIBILITY||WHEN REPORTED|
|Awareness (Capacity)||Articulation of issues||Events, blogs, forum posts, wiki posts, website updates||Analytics & narrative report||All||Bi-annually|
|Publicity & Promotion||Tweets, articles, media coverage||Analytics & narrative report||All||Bi-annually|
|Sharing (Learning)||New approaches & tools used||Itemisation of new tools & methods used||Narrative report||All||Bi-annually|
|Methods & approaches shared with project partners and other stakeholders||Details of new tools & methods shared||Narrative report||All||Bi-annually|
|Participation (Engagement)||New participants/members||Forum registrations, twitter followers||Analytics & narrative report||All||Bi-annually|
|New collaborations||Number of new partners & collaborators||Analytics & narrative report||All||Bi-annually|
|Digital platform interaction||Number of website/platform visits & interactions||Analytics||All||Bi-annually|
|Event & webinar attendees||Number & demographic||Analytics||All||Bi-annually|
|Influence (Outcomes/Impact)||Progress with SDGs||Implementation plan||Narrative report||All||Bi-annually|
|Citizen involvement||Participation opportunities created||Narrative report||All||Bi-annually|
|Policy influence||Influence on government policy, for example, the PfG, SDG implementation plan||Analytics & narrative report||All||Bi-annually|
|Transparency & accountability of process||Government utilisation of OG methods in the planning & implementation process||Narrative report||All||Bi-annually|
Agile learning approach[edit | edit source]
- Initial data collected in the first quarter of the project will form a baseline (against the Data subjects in the table below)
- An assessment will be made of any changes when reported (quarterly, monthly or annually as appropriate). Advice from Reference Group partners will be sought as appropriate.
- A recommendation will be made to the Project Board by the Delivery Group to change project elements based in this assessment.
Data underpinning evaluation[edit | edit source]
|Stream||Data subject||Data||Relevant targets||Collection method||Responsibility||When reported|
Posts and replies
|Increase forum registrations from 1000 to 5000 and aim for fivefold increase in each nation||Website usage stats||Each nation lead partner||Quarterly|
|Capacity||Opengov pioneers Wiki||Visits
|Tbc after setting baseline||Website analytics
|Capacity||Opengov collaboration||Contributions to crowdsourcing platforms
New initiatives between citizens sparked as a result of project activities
|Submissions from 20% of members during lifecycle of project||Crowdsourcing platform analytics
|Engagement||Opengov website||Blog reads
|Engagement||Positioning of Open Government agenda in delivery of SDGs||Stakeholder analysis||Qualitative||Influencer interviews / meetings||Each nation lead partner||Quarterly|
|Engagement||Impact on SDG progress||Policy documents (government)||Qualitative & # of citations||Rapid review and appraisal||Consultant||Annually|
|Learning||Knowledge of what works in using Open Government to progress SDGs||Blogs and articles from members||Submissions from 20% of members during lifecycle of project||Media analysis||Each nation lead partner||Annually|
|Learning||Engagement between Pioneer civil society on what works||Pioneer forum engagement
Posts and replies
|Tbc after setting baseline||Website usage stats||SCVO & Involve UK||Quarterly|
|Learning||Engagement between UK nations on what works||Stakeholder analysis||Qualitative||Influencer interviews / meetings||NIEL||Annually|
Independent evaluator[edit | edit source]
We need to ensure that the project success is properly and independently assessed by an independent evaluator. This is for the following reasons:
- The use of an independent evaluation consultant was agreed by project core partners and specified in the bid to the Big Lottery Fund UK. There will be an expectation of this by all partners who have reviewed the application and the plans which was published on this Wiki at that time
- An independent evaluation will allow a more objective evaluation of the success of the project than project core partners interviewing or getting feedback themselves from project beneficiaries
- A single independent evaluation rather than separate evaluations in each UK home nation will allow us to stay within budget and capture differences and common experiences across the nations.
- We will use the process towards recruiting the independent evaluator to generate a conversation between existing and prospective delivery partners on our evaluation approach. In particular, we will seek advice on how to get a clearer line of sight between the project output delivery and progress towards outcomes
We will openly advertise and recruit an evaluation partner (consultant) to deliver the independent evaluation, and another evaluation partner to deliver the equalities evaluation. The project's Delivery Group's recommendation to the project board will be to use a single evaluator for both the main and equalities evaluation of the project. This is to recognise the importance of embedding equalities in the overall evaluation approach. It would also fit into the outcome Empowered engaged citizens, leveraging their knowledge of SDGs, emphasising the focus on engaging people whose voices are least heard.
In order to inform the process used to recruit the evaluation consultant, we will draw on the pilot approach we used to recruit the equalities trainer for the project team. This involved an open approach, where potential delivery partners self-identified their interest on this Wiki on the Partners page. They were subsequently added to the Opengovernment Pioneers Project Partners forum and were encourages to comment on the overall process and brief openly to other partners before making their pitch.
Access the evaluator tender brief here.
The evaluation will involve interviews with project partners, project participants and government, and will be supported by the data collected as part of the project's monitoring activities and a discussion with the project board and input from project's forums around evaluation priorities. The evaluator will be expected to make recommendations about how to continue and develop the engagement of citizens and civil society in using open government to secure progress towards SDGs.
UPDATE: Mutualgain has been appointed as the independent evaluator. See Partners page for more about them.
Self-evaluation[edit | edit source]
It is important that this project does not rely completely on the independent evaluation to understand the difference it is making.
- The UK Conference will include a workshop component to facilitate stakeholder analysis, where project participants collectively deliberate around what works in delivering open government approaches to SDGs.
- The Activity sprint on international learning in Summer 2018 will also provide international level learning about what works and external perspective on how the project is making a difference based on International webinars with Pioneer civil society cohort
- Crucially, the independent evaluation will be accompanied by six monthly self-evaluations by each core partner for each home nation which will be captured and reported through this Wiki.
Timescales for evaluation[edit | edit source]
|Evaluation plan on Wiki
Delivery Group Self-evaluation
|Meeting with Project Board
Paper outlining equalities impact proofing project activities
|Recruitment process agreed||Aug 17||Completed|
|Recruitment of evaluators
Delivery Group Self-evaluation
|Dec / Jan 18||Completed|
|Evaluator session with Project Board||Jan 18||Scheduled|
Delivery Group Self-evaluation
|Evaluation report and recommendations||Jul 18|
|UK wide sprint to review project and legacy||Sep 18|
How will we ensure a strong equalities focus?[edit | edit source]
This Open Government Pioneers Project is primarily about raising awareness and supporting the implementation of the SDGs through Open Government principles and methods, with an emphasis on reaching groups who have difficulty in having their voices heard.
We will use the following approaches to ensure an equalities focus is embedded at all stage and all activities of the project:
- Prioritising budget and engagement with partners and groups supporting those without a strong voice in opengovernment and SDG discussions
- Securing strong buy-in to this project from a variety of equalities-focused civil society organisations
- Ensuring all project activities seek wider engagement than the 'usual suspects' - e.g. using livestreaming, non-jargon language, accessibility checks for digital platforms.
- Ensuring all project activities are as open and accessible as possible at the design stage to people who face additional barriers to engaging the decisions that affect them.
- We will learn from the existing evidence base available to us, starting with the research and reports referenced at Analysis and theory of change#Who participates.3F
Key questions[edit | edit source]
|Do the proposed evaluation indicators link closely enough with the objectives?|
|Is the timescale proposed realistic?|
|Is the proposal to ensure a strong equalities focus adequate and realistic?|
Project Impact[edit | edit source]
Updated Jan 2018
Overall impact assessment[edit | edit source]
- The project has facilitated a significant renewal of interest in the Sustainable Development Goals which had been flagging since the launch of the SDGs in late 2015.
- The project has also increased interest and visibility in the Open Government Partnership and action plans amongst the policy communities in Scotland and Wales.
- The project has secured a greater appreciation by government and civil society participants across the UK and has engaged a broader constituency of interests in the open government partnership of the link between OGP and SDGs in Northern Ireland and England.
- The project is beginning to broaden awareness and interest in OGP and SDGs in the broader public and media. There has been a steady increase in engagement with the structures and planned engagements that are now in place to progress this over the previous six months of the project.
- The project has for the first time brought engagement around open government and SDGs to the attention of young people, and people engaged with minority and disability groups.
Impact against outcomes[edit | edit source]
|Outcomes||Beneficiary||Impact assessment||Areas for improvement|
|Political representatives have greater awareness of SDGs and types of actions required.||Influencers - politicians, government officials, institutional partners||UK level:
Wales: In Wales the focus of the project is on the Well-being Goals. There is a particular focus on the engagement & involvement elements of the Well-being of Future Generations Act (the 5 ways of working) and a number of network meetings between civil society & Government officials have now taken place ensuring better involvement in the most integrated way, and learning is shared.
|Wales: engagement with local government will be important as they will be delivering well-being plans from this year, reporting against progress on the well-being goals.
NI: political literacy around the SDGs is still low. Continue to raise understanding through the Pioneer conference.
|Government officials have a better plan for involving citizens.||Participants - civil society activists, engaged citizens||UK Opengov networks:
Wales: A plan is currently being initiated with Welsh Government’s Digital team who are responsible for the development of the Welsh commitments for the 4th NAP. Involvement of policy leads across Welsh Government such as Head of Local Government, the Equalities dept and the Cabinet Office (who head up the SDGs reporting of Welsh Government) will ensure wider involvement.
|In Scotland, we need a clearer strategy for follow through for interest in open government and SDGs by citizens and civil society nurtured by this project.
NI: Encourage the Open Government Implementation Group to incorporate an SDGs commitment in the new OG Action Plan. This will be difficult.
Wales: It will be important to engage civil society from the outset of developing themes and commitments rather than just consulting them. There is a risk of this at the moment, and our project will be key to making sure this happens.
|Empowered engaged citizens, leveraging their knowledge of SDGs.||General Public||Scotland and Wales:
Wales: Through the 3 equalities projects that received funding from the Wales Open Government Network engagement is now being undertaken in earnest with grassroots groups & individuals with protected characteristics. This will link to the Well-Being Goals and SDGs. A project bid is being developed currently to develop the Wales GlobalGoals website.
|Scotland: We will need a better plan in place to support people who are interested in hosting meetups, as simply providing more staff time will exhaust team's resources.
NI: Encouraging teachers to adopt our framework/methodology to run their own democracy games sessions in schools, so the approach can be sustained after the lifetime of the project. Adjust the methodology for 16-24 year olds.
Wales: There will be a real need to share the learning from citizens, third sector and government on achievements of well-being goals / SDGs.
|Open government seen as instrumental towards progressing SDGs.||Media, academia, funding bodies, policy professionals, charities, private sector, international networks||UK wide
|Project hasn't achieved mainstream media coverage, but interest amongst journalists is now clear and can be built on.
Democracy engagement approaches pioneered in NI are now being spread to the other countries via this project.
Wales: Still early days in building open government momentum in third sector, but this is the first year where the link has been made (primarily by this project). Internet capacity still limits digital engagement for Wales citizens and third sector, and is an area for further development.
Engagement statistics[edit | edit source]
Stats are cumulative totals, unless otherwise specified
|Indicator||Commentary (updated Jan 2018)||Nov 2016||Mar
|Pioneers project partners (organisations registered on forum as Pioneers Project partners)||Most organisations have one individual registered. Delivery partner organisations have 2-3 people registered. See https://forum.opengovernment.org.uk/groups/opengovpioneers||0||55||57||87|
|Open Government Forum registrations||We have seen a doubling in forum memberships for Scotland and Wales, and a notable increase for both UK and NI forums.||All: 1149
|Open Government Forum posts and replies||Activity around posting and replies are in line with numbers of registrations.||Wales: 12||UK: 411
|Wiki visits||Unable to track visits due to Miraheze platform restrictions - unlikely this will change soon.|
|Wiki contributions||Registration not required to contribute anonymously. Therefore we anticipate those that choose to register plan to contribute regularly or on multiple occasions.||Wales: 1||32 contributors signed up
888 page edits
|50 contributors signed up
1372 page edits
|Wiki citations||Difficult to track this properly due to limitations of platform. May need to review this indicator. Currently using Google links||150||247|
|Digital platform unique users (globalgoals.scot)||N/A||N/A||167||1296|
|Digital platform page views (globalgoals.scot)||Platform has been soft launched and made available to those taking an interest in SDGs. Platform has not been launched yet formally through mainstream media.||N/A||147 from new users, 741 including returning||761 from new users, 1663 including returning||4766 from new users, 12,618 including returning|
|Digital platform contributions||*Contributions largely posted by SCVO for now||120*|
|Digital platform citations||Platform not formally launched yet|
|Open Government Network blog reads||analytics> behaviour>content drilldown> 2016/12 + 2017 (unique page views) stats are cumulative from end Nov 16||648||1002||2775|
|Open Government Network ranking (google)|
|Open Government Network blog/article contributions||statistics are cumulative starting end of Nov 16 (latest stats are until end of december)||Wales: 1
|Open Government Network ref group posts and replies|
|Meetup.com registrations||Trebling of meetup.com registrations which comprises mainly people not already members of the opengov forums. i.e. engaged citizens.||UK (Scot and Wales): 163||UK (Scot and Wales): 200||UK (Scot and Wales): 331|
|Twitter followers||Substantial increases for Wales and Scotland as these are the new networks.||Scot: 175