Effectiveness

Introduction

The effectiveness assessment factor considers the contribution that the proposed solution makes to achieving the potential identified in the strategic fit assessment and to the purpose of the Land Transport Management Act 2003.

Effectiveness assesses how well the proposed activity will deliver the desired results, while considering integration, whether the solution has been correctly scoped, is affordable Demonstrates that adequate funding to deliver the activity, or combination of activities, is available from identified sources, e.g. a funding plan has been identified. , timely and manages risk.

Fit for purpose assessment

The assessment of effectiveness should be fit for purpose, taking into account the scale and complexity of the project or programme and its stage of development.

Examples:

1. For a small, straightforward rural road bridge replacement, the focus of assessment should be on option selection, scope and timing of the project. The assessment of outcomes, integration, affordability and confidence should be a simple and brief exercise. If these latter criteria are satisfied as well as they can be, they should rate as 'High'. The simple nature of the project does not limit the rating of effectiveness.
2. A large project An improvement project with a cost of more than $5 million. that originates from a multi-modal, multi-organisation programme business case would require a focus across all criteria to assess effectiveness. A significant proportion of the evidence around outcomes, integration, affordability and timing should come from the programme business case, with the assessment considering the fit with the business case. The scale, complexity and riskiness of the project, and how it fits against the programme business case, necessitates a detailed assessment of effectiveness, which should be peer reviewed.
3. For a programme, e.g. existing public transport services or road maintenance, where there is limited change requested from previous expenditure and key trends are in line with the Transport Agency's expectations, the assessment of outcomes, integration, affordability, scope and confidence should be a simple and brief exercise. If these criteria are satisfied as well as they can be, they should rate as 'High'.
4. Where significant change A significant change in strategic context The Strategic Context represents the alignment of the proposed investment with the business problem owner’s priorities, regional and national priorities, other programmes and strategies and other organisations’ priorities (if relevant).
   
   It includes:
   the assumptions or view of the future, including transport and population growth, economic and industry change statistics, etc. 
   
   
   * underlying/umbrella strategic documents, such as the Government Policy Statement on Land Transport or regional strategies, to position the outcomes sought within the strategic assessment against wider national and regional outcomes.  A problem with perceived significance may be identified in the strategic assessment but, when placed within a wider strategic context, it is acknowledged that it is not of high significance and does not present value for money Selecting the right things to do, implementing them in the right way, at the right time and for the right price. for investment at this time.
   
   * the proposing organisation’s objectives.
   is:
   
   
   * a change in planning assumptions in relation to demographic (i.e. population or age profile changes) projections as a result of the next official census, or
   
   * a change in planning assumptions relating to the nature and shape of dominant industries in the region, or
   
   * a need to alter a region's RLTS or RPS to address impacts of projected sea level rises, coastal erosion or significantly increased flood risk in areas where key networks are at risk, or
   
   * a legislative requirement to renew a region's RLTS or RPS.
   Significant change pressures within the context of regional growth are:
   
   
   * a change in planning assumptions relating to the nature and shape of dominant industries in the region, or
   
   * impacts from sea level rise, coastal erosion or significant flood risk from climate change that threatens or is projected to threaten local and or key state highway networks. is proposed for a programme and there is considerable cost and risk exposure for the Transport Agency, the programme business case, e.g. the activity management plan, should provide robust evidence around outcomes, integration, affordability, scope and confidence. A detailed assessment of effectiveness is required that considers the scale, complexity and risk associated with the change and how it fits with the business case. The assessment should be peer reviewed.

Need to use judgement in assessment

Assessment of effectiveness as a whole is based on quantitative and qualitative evidence, although for each of the six criteria it may comprise solely one or the other. It requires the use of judgement rather than being a simplistic "box ticking" exercise. For large, complex programmes and projects the assessment should be moderated amongst experienced peer reviewers, to ensure different perspectives are debated to arrive at a consensus.

Evidence

Sufficient evidence must be provided to support the effectiveness rating. This includes as appropriate:

• a strategy or study, project feasibility report (PFR A report, which provides a simple, rapid method of analysis to facilitate the identification of economically viable projects. ), or scheme assessment report (SAR A detailed report that includes a precise problem definition, a description of options available to address the problem, a project evaluation, an assessment of environmental impacts, and a recommendation. ). (for activities still moving through the old life cycle prior to the Business Case Approach)
• a strategic business case
• a programme business case, progressing to detailed business case depending on stage of development
• an Activity Management Plan (AMP An activity management plan prepared in accordance with clause 2 of schedule 10 of the Local Government Act 2002, or a similar plan.
  Activity management planning considers the assets in the context of the services they are supporting, and clarifies the purpose for holding the asset. The goal of good asset management is to support the delivery of a level of service (whatever the service may be) in the most cost effective manner, taking long term sustainability into account.
  Activity Management Plans Plans describe the tactics to give effect to a strategy. They are specific in content, action oriented and outputs focussed, resulting in a tangible set of activities to be delivered within a clear timeframe. should be based on the National Asset Management Steering (NAMS) Group's International infrastructure management manual. )
• a Regional Public Transport Plan (RPTP A plan which specifies how the regional council intends to give effect to the public transport service components of the regional land transport plan that applies to the region. As of June 2013, the contents and management of the plan is defined by Part 5 of the Land Transport Management Act 2003. )
• a Road Safety Action Plan (RSAP Plans Plans describe the tactics to give effect to a strategy. They are specific in content, action oriented and outputs focussed, resulting in a tangible set of activities to be delivered within a clear timeframe. developed at the local level to address road safety issues in the area, as a result of the road safety action planning process. ) or equivalent planning tool
• a recognised regional or local lifelines transport network study

Default rating

The default rating for Work category 001: Regional land transport planning management is high and no further assessment is required.

For all other activities, the default effectiveness is low until evidence is provided to support a higher rating.

Interim rating where evidence is under development

On occasion, early in the development of some complex activities, e.g. before completion of the programme business case, there may be insufficient evidence to make a confident assessment of effectiveness. An interim rating of M* at the time of inclusion in the National Land Transport Programme Interrelated and complementary combination of activities that, when delivered in a coordinated manner, produce synergies – can span more than one work category and more than one activity class, e.g. a programme could include a road improvement and public transport improvement activities. (NLTP A National Land Transport Programme Interrelated and complementary combination of activities that, when delivered in a coordinated manner, produce synergies – can span more than one work category and more than one activity class, e.g. a programme could include a road improvement and public transport improvement activities. adopted by the NZTA under section 19 of the LTMA, as from time to time amended or varied ) indicates there is insufficient information and that further development is required. The M* rating enables an activity to be included in the NLTP, but does not enable funding approval until the information is developed and a firm rating of effectiveness is provided.

What Low and Medium effectiveness mean

A rating of Low effectiveness indicates more work is required to justify the activity. For programmes, e.g. a road maintenance programme The total of the organisation's approved maintenance, operations and renewal activities. , this may result in additional work during NLTP A National Land Transport Programme Interrelated and complementary combination of activities that, when delivered in a coordinated manner, produce synergies – can span more than one work category and more than one activity class, e.g. a programme could include a road improvement and public transport improvement activities. adopted by the NZTA under section 19 of the LTMA, as from time to time amended or varied negotiations to achieve a fundable rating. Alternatively, it may result in a condition of funding, which requires evidence to be provided before funds are released in full or part.

Medium effectiveness means that an activity has not achieved the full potential identified in the strategic fit assessment. This may be due to a deficiency in the development process, e.g. sensitivity analysis should have considered more scenarios to provide a fuller view of the risks, or an acknowledged shortfall in the chosen intervention, e.g. a safety intervention that addresses only part of the crash risk. The deficiency or shortfall is not considered by the assessor as significant to the extent that the activity should not be progressed, more that its effectiveness is not as good as it could be.

Assessing effectiveness

The assessment considers all criteria. The overall effectiveness assessment is reported as the lowest rating for any criterion, i.e. an overall M rating will be given when all criteria and parts have either an M or H rating. An overall L rating will be given if any criterion is rated L.

A low rating means that there is insufficient evidence to show that a solution can successfully address the problem, issue or opportunity. If the effectiveness is assessed as low, the proposed solution should remain in the programme business case stage until the effectiveness criteria assessed as low have been addressed through further development. 

The effectiveness assessment criteria look at how well the proposed activity or programme:

• is outcomes focused
• is integrated
• is correctly scoped
• is affordable Demonstrates that adequate funding to deliver the activity, or combination of activities, is available from identified sources, e.g. a funding plan has been identified.
• is timely
• provides confidence

Criteria for effectiveness

All six criteria set out below are to be assessed for any programme or activity proposed for NLTP A National Land Transport Programme Interrelated and complementary combination of activities that, when delivered in a coordinated manner, produce synergies – can span more than one work category and more than one activity class, e.g. a programme could include a road improvement and public transport improvement activities. adopted by the NZTA under section 19 of the LTMA, as from time to time amended or varied inclusion or funding approval. The following explanations are most appropriate when applied to road improvement activities. Variations of these are appropriate when assessing activities other than road improvements and further explanation is provided in the following section. The explanations are a guide to assessment, highlighting aspects that need to be considered. If any of these aspects is not applicable to the activity then it should not form part of the assessment.

Assessment of activities other than road improvements.

Variations to the above explanations are:

 Walking & cycling

 Road safety promotion

 Public transport existing services

 Public transport improvements

 Maintenance programmes

Assessment of incremental effectiveness

Assessment of incremental effectiveness is required for option selection, optimisation and scope change proposals.

Information to provide in TIO The NZTA's web-based funding allocation system.

The Transport Agency expects that a summary of the effectiveness assessment will be shown in both the project/programme owner's and the Transport Agency's (planning & investment) assessment of effectiveness in Transport Investment Online (TIO The NZTA's web-based funding allocation system. ).

The following example demonstrates how the effectiveness assessment summary should be shown in TIO The NZTA's web-based funding allocation system. . The content will be case specific and reflect the actual evidence for the activity.

Succinct supporting information may be uploaded in TIO The NZTA's web-based funding allocation system. to provide more detail for the assessment if required, provided this is referenced in the summary. Preferably, only the relevant page(s) of a document should be uploaded. If a full document is uploaded, the specific part(s) of the document should referenced in the summary.