Module 6: Monitoring and Measuring Policy Success

To be effective, policy making must be a learning process which involves finding out from experience what works and what does not, and then ensuring others can learn from it too. This means that data must be collected and analyzed to inform the implementation processes. Policies should be periodically reviewed by the First Nation to assess their impact.

Assessing the criteria for successful policy and ways to measure its effectiveness needs to be addressed at the beginning of the policy development process. The development of success measures helps policy makers to understand whether the policy is working or not working – or whether it is meeting its intended objectives.

The following topics are covered in Module 6:

• Monitoring the effectiveness of policy
• Measuring the success of policies
  

Monitoring the effectiveness of policy

Figure 1: Policy Cycle Stages

Within the policy cycle (discussed in Module 2 and illustrated in Figure 1), one of the stages is monitoring and assessing the impacts of policies (Implement and Evaluate at stage 9). The intent of policy development is to create a policy to address the intended issue or achieve the desired objective. In order to understand if the policy is effective, a framework for measuring and tracking results needs to be developed. Policy monitoring and evaluation goes beyond assessing a policy for its impact. It also involves ensuring a policy decision is acted upon.

A policy review should be undertaken at least every two years and can occur more frequently if issues arise which impact the policy. The review is conducted by the Policy Lead (the Administrator in the First Nation whose role is to develop and manage the specific policy) in collaboration with other individuals who have expertise and knowledge in the area. To achieve an effective review, the Policy Lead can bring together a small team of two to three people with expertise and knowledge of the policy.

The following should be considered when reviewing a policy initiative:

• Has the policy realized its objectives?
• Is the policy still relevant? Does the policy meet community expectations?
• Should the process be applied for future work?
• Are the methods for managing the policy issues working to minimize risks and provide benefits? Are the key performance measures appropriate?
• Has the purpose or the intent of the policy changed? If yes, why?
• What are the impacts, options and costs associated with developing and implementing policies?
• Do new areas need to be addressed? Are new risks or trends emerging?

Measuring the success of policies

Assessing the success of policies in achieving their objectives requires the creation of performance measures and indicators. When a First Nation wants to determine the effectiveness of a policy, it is important to identify what you are measuring to understand progress (the measures) and whether that measure is changing in a positive direction, increasing, or decreasing (the indicators). For example, in the case of a training policy, one measure might be the participation of community members or citizens in training courses on an annual basis and the indicator is the change in participation on a yearly basis. There are several activities that can assist a First Nation in measuring the successful implementation of a policy. These are as follows:

• Establish key measures for determining success of a policy related to its intended objectives (e.g. increased economic activity in the community) – some examples are: increased participation in education, decreased waiting times for member payments, improved budget management and increased on-time home rental payments
• Ensure the measures are simple and practical
• Identify the specific indicators that can support each measure (e.g. revenues generated)
• Ensure that each indicator is realistic and can actually be measured effectively (e.g. that the First Nation has control over the particular indicator)
• Each policy should have its own measures and indicators
• Ensure those leading and managing the policy are involved in the development of the measures and indicators

As a reminder, policy has a better chance for succeeding and being more effective when:

• A clear and transparent process is followed
• It is purposeful rather than reactive
• It is NOT created in a vacuum (if policy is created in a vacuum, it will not attract support)
• It aligns with the overall strategy for the First Nation
• Attention is paid to examining implications of policy options (economic, financial, social, gender equality, environmental, etc.)
• It anticipates and mitigates risks related to the policy
• It is monitored and reviewed once implemented

It is important to identify what you are measuring to understand progress (the measures) and whether that measure is changing in a positive direction, increasing, or decreasing (the indicators)