Peer review of improvement projects

Introduction

For the latest information, please see the Assessment Framework for the 2015-18 NLTP.

This section sets out the factors that must be taken into account, at a minimum, when undertaking a peer review of improvement projects.  The peer review must include at least a review of the:

• conformity
• credibility
• choice of do-minimum
• identification and selection of alternatives A strategic option that may encompass a mix of modes and/or high level routes and/or land use options. Alternatives would be considered during strategy development, with the preferred alternative being selected and taken through into package and project development. and options
• strategic fit rating
• effectiveness rating
• cost estimate(s)
• economic efficiency evaluation and rating
• risk assessment, analysis and mitigation
• sensitivity analysis.

Selection and independence of peer reviewer

The NZ Transport Agency requires an independent, external peer review for any large (over $5million construction cost) or complex project A project with a construction/implementation cost estimate greater than $20 million and/or considered to be high risk. . Approved Organisations and the NZTA (state highways) are encouraged to have small projects (between $250,000 and $5million construction cost) externally peer reviewed if the cost and/or benefit risks associated with these are considered high or the applicant lacks experience in the development and implementation of such projects. In any event, all small project An improvement project with a construction/ implementation cost of $5 million or less and more than $300,000. evaluations should be internally peer reviewed.

Where an external peer review is required or warranted, the peer reviewer shall be selected and appointed by the applicant, and must:

• be independent of the organisation and the project, unless otherwise  formally agreed with the NZTA (planning and investment);
• take an objective, professional stance at all times in undertaking the peer review, keeping in mind that the key aim of the review is to reduce the risk to the delivery of desired outcomes, including economic efficiency, from the NZ Transport Agency's investment decisions; and
• be competent with respect to the specific nature of the project and must not exceed its level of competence in undertaking the review.

For very large, complex packages and projects, a peer review panel, covering a range of competencies, may be most appropriate.

The NZ Transport Agency reserves the right to undertake its own peer review of any project or to require the Approved Organisation or the NZTA (state highway) to appoint a specific peer reviewer or to establish a peer review panel with appropriate competencies.

Conformity

The reviewer must first determine whether the project is eligible for funding in that it fits the description of one of the activity classes in the current Government Policy Statement on Land Transport Funding.

The reviewer must ensure that the project evaluation conforms to the requirements of this Knowledge Base, including that it has been assessed by the applicant in conformance with the NZ Transport Agency’s Assessment Framework.

Credibility

To check credibility, the reviewer must:

• Ensure the transport issue, priority or opportunity has been identified, is reasonable and is adequately described.
• Critically assess the results of each stage of the project’s economic efficiency evaluation, avoiding unnecessary detail where possible. The test as to the level of detail to consider is whether the conclusion reached in the report is a reasonable and a credible result from the information and data used in the analysis.
• Assess the costs estimated for the project and consider how realistic these are, taking into account current market rates.
• Identify the key benefits and determine whether they are realistic (e.g. are the travel time savings realistic or are excessive delays being forecast under congested conditions in the do-minimum?). Some quick ‘back-of-the-envelope’ calculations are necessary to check the level of forecast benefits.
• Identify the factors or assumptions, particularly forecasted estimates that have a major influence on the evaluation. Describe each of these factors/assumptions and include a commentary on the sensitivity of the evaluation to each factor or assumption.
• Highlight any significant areas of risk for costs and benefits.

Choice of do minimum

The reviewer must assess the do-minimum as stated in the project report and must determine whether it is realistic, and does not represent another option to be considered in the analysis.

Identification and selection of alternatives A strategic option that may encompass a mix of modes and/or high level routes and/or land use options. Alternatives would be considered during strategy development, with the preferred alternative being selected and taken through into package and project development. and options

The reviewer must examine the evaluation and judge whether all feasible alternatives A strategic option that may encompass a mix of modes and/or high level routes and/or land use options. Alternatives would be considered during strategy development, with the preferred alternative being selected and taken through into package and project development. and options have been identified and considered adequately. These should include alternative transport modes, where applicable, and low cost options.

The reviewer needs to be satisfied that the process to select the preferred alternative and option(s) has been robust and includes incremental assessment where appropriate.

Strategic fit rating

The reviewer needs to be satisfied that the strategic fit rating for the activity is correct.

Effectiveness rating

The reviewer needs to be satisfied that the effectiveness rating for the activity is correct.

Cost estimate

The reviewer shall check compliance with parallel cost estimate process requirements, where applicable.

Economic efficiency evaluation and rating

The reviewer must determine whether the economic efficiency evaluation has conformed to all the relevant requirements of the NZ Transport Agency’s Economic Evaluation Manual (Jan 2016). The reviewer must determine whether there are any outstanding issues not addressed in the project report.

If there is a departure from the requirements, or any defect or omission, the reviewer must comment on its significance.

Where the reviewer considers that there have been discrepancies and departures from procedure, or has concerns on cost and/or benefit estimation, the reviewer will determine the project benefit cost ratio (BCR The NZTA uses the BCR as a measure of economic efficiency from a national perspective as defined in the NZTA's Economic Evaluation Manual. The ratio compares the benefits accruing to land transport users and the wider community from implementing a project or providing a service, with that project or service's whole of life costs. ) and compare this with the applicant’s calculations.

The reviewer must determine whether the options identified in the analysis are mutually exclusive options of the same project. If the options identified:

• are mutually exclusive, then the reviewer must determine that an incremental assessment of the options has been carried out correctly, as set out in the Incremental Assessment section of this Knowledge Base; or
• are not mutually exclusive, then by definition they must be either:
  • independent projects, in which case the reviewer must determine that the analysis has been undertaken in terms of independent projects and has been undertaken correctly, and should comment whether they should be resubmitted as separate projects; or
  • inter-dependent projects, e.g. components of a package, in which case the reviewer should consider whether the analysis that has been undertaken is valid.

In special cases, other economic impacts may be considered (e.g. intangibles). These are to be shown as sensitivity analyses, in addition to the EEM The NZTA's Economic evaluation manual. procedure economic analysis.

Where supplementary (third party) funding is involved, a government BCR The NZTA uses the BCR as a measure of economic efficiency from a national perspective as defined in the NZTA's Economic Evaluation Manual. The ratio compares the benefits accruing to land transport users and the wider community from implementing a project or providing a service, with that project or service's whole of life costs. must be determined in addition to the national BCR.

Risk assessment, analysis and mitigation

The reviewer must ensure that:

• risks have been assessed adequately in the applicant’s evaluation
• realistic mitigation measures have been considered
• a full risk analysis has been undertaken for large/complex and high-risk projects.

Sensitivity analysis

The reviewer must consider whether the sensitivity of critical aspects of the project evaluation has been covered off adequately, paying particular attention to:

• key assumptions that underlie the project and its delivery of desired outcomes, in particular future growth and demand assumptions
• information and data values that are ‘out of the ordinary’ or unusual
• the sensitivity of the project’s outcomes to the input parameters.

Raising concerns

The reviewer must raise in writing with the applicant organisation for funding assistance (and its representative) any:

• discrepancies or departures from the requirements of the Knowledge Base or the Economic evaluation manual
• material concerns about the project as a result of the review.

The reviewer must request that the applicant organisation:

• responds to the reviewer within a reasonable period
• provides reasons for any discrepancies or departures and answers to resolve the reviewer’s concerns.

The review must note any outstanding concerns in the review report.