Design for the Evaluation of the National Flood Insurance Program This report was prepared for the Federal Emergency Management Agency by the American Institutes for Research with the Pacific Institute for Research and Evaluation and Deloitte and Touché. Contract Number 282-98-0029 Task Order Number 24 January 2002 1000 THOMAS JEFFERSON STREET, NW | WASHINGTON, DC | 202 944-5300 | WWW.AIR.ORG Table of Contents I. Introduction Background of Evaluation.................................................................................1 Primary Evaluation Questions...........................................................................5 II. Logic Models A Logic Model Analysis of the National Flood Insurance Program..............1 Overall Logic Model of the National Flood Insurance Program....................3 Submodel 1-Floodplain Management and the NFIP.....................................6 Submodel 2- NFIP Floodplain Management Assistance, Monitoring, and Enforcement ..................................................8 Submodel 3- NFIP Risk Assessment and Underwriting................................9 Submodel 4- Mandatory Purchase of Flood Insurance..................................11 Submodel 5- Disaster Assistance and the NFIP.............................................13 Submodel 6- National Flood Insurance Claims Adjustment..........................15 Submodel 7- National Flood Insurance Marketing and Communications.....17 Evaluation Designs III. Area of Inquiry 1: Occupancy and Use of Floodplains Matrices..........................................................................................................1 Narratives.......................................................................................................5 IV. Area of Inquiry 2: Costs and Consequences of Flooding Matrices..........................................................................................................1 Narratives.......................................................................................................10 V. Area of Inquiry 3: Insurance Rating and Indemnity Functions Matrices..........................................................................................................1 Narratives.......................................................................................................18 VI. Area of Inquiry 4: Floodplain Management and Enforcement Matrices..........................................................................................................1 Narratives.......................................................................................................6 VII. Area of Inquiry 5: Hazard Identification and Risk Assessment Matrices..........................................................................................................1 Narratives.......................................................................................................9 VIII. Area of Inquiry 6: Marketing and Communications Matrices..........................................................................................................1 Narratives.......................................................................................................10 IX. Priority Ranking Priority Ranking.............................................................................................1 Evaluation Questions......................................................................................7 X. Steering Committee Role and Responsibilities of the Steering Committee....................................1 Biographical Sketches....................................................................................2 Summary of August 29th, 2001 Meeting........................................................10 Summary of November 16th, 2001 Meeting...................................................12 Acronyms BFE Base Flood Elevation BPAT Building Performance Assessment Team CAV Community Assistance Visit CBRS Coastal Barrier Resource System CRS Community Rating System FEMA Federal Emergency Management Agency FHBM Flood Hazard Boundary Map FIA Federal Insurance Administration FIMA Flood Insurance and Mitigation Administration FIRM Flood Insurance Rate Map FY Fiscal year GAO General Accounting Office HUD Department of Housing and Urban Development LOMA Letter of Map Amendment LOMC Letter of Map Change LOMR Letter of Map Revision NFIP National Flood Insurance Program PL Public Law SFHA Special Flood Hazard Area USGS United States Geological Survey WYO Write Your Own INTRODUCTION The National Flood Insurance Act of 1968 created the National Flood Insurance Program (NFIP), which the Federal Emergency Management Agency (FEMA) administers through the Federal Insurance and Mitigation Administration (FIMA). The NFIP has nearly 20,000 participating communities with more than four million policyholder paying over $1.7 billion in premiums each year. The NFIP seeks to: 1) make flood insurance widely available at actuarially sound rates or with legally mandated subsidies; 2) identify areas that are prone to flooding; 3) reduce the frequency and adverse consequences of flooding and the number of associated insurance claims; 4) minimize the amount of disaster assistance required for recovery from floods; and 5) support the natural, beneficial uses of floodplains. Since its creation, the NFIP has undergone many changes. For example, although the Department of Housing and Urban Development initially administered the program, FEMA, an independent agency, now does so. Similarly, the laws and regulations governing the program’s implementation have changed in important ways as a result of experience with flooding, disaster assistance, and hazard identification and mitigation as well as because of a desire to improve the program. Despite the many changes that have occurred and the program’s significance, the NFIP has not yet been evaluated comprehensively. Given the program’s size, scope, and national importance, it is imperative to know how well it is operating and to identify areas where performance may be deficient or where performance exceeds reasonable expectations. As a consequence, FEMA convened a committee in 1999 to establish a framework for conducting the first comprehensive evaluation of the NFIP. The committee consisted of current FEMA staff, retired government executives, and experts from universities and the private sector. The committee’s task was to produce a set of research questions to guide an evaluation that would assess the NFIP’s effectiveness and efficiency while also providing alternatives to improve the NFIP’s current operations. In response to this mandate, the committee prepared a series of questions designed to cover all aspects of the NFIP and parts of other state and federal programs that affect the NFIP. The questions were placed in six areas of inquiry: 1) Occupancy and Use of Floodplains; 2) Costs and Consequences of Flooding; 3) Insurance Rating and Indemnity Functions; 4) Floodplain Management and Enforcement; 5) Hazard Identification and Risk Assessment; and 6) Marketing and Communications. The complete list of questions can be found in Section IX. To address these questions, FEMA divided the evaluation into two phases. The purpose of Phase I was to design the evaluation. The evaluation will be conducted during Phase II. In September 2000, FEMA selected the American Institutes for Research (AIR) and its subcontractors, the Pacific Institute for Research and Evaluation (PIRE) and Deloitte and Touché, to design the evaluation. In September 2001, FEMA awarded AIR and its subcontractors a multiyear task-order contract to evaluate the NFIP in Phase II. This report summarizes the work products and activities of Phase I PHASE I Phase I required the development of a design for a comprehensive evaluation of the NFIP. The primary goal during this phase was to assess the feasibility and evaluability of the questions in the six areas of inquiry. In addition, the goal was to reduce the number of questions to a more manageable number that would address the issues of highest importance and of most benefit to the NFIP while accommodating the resources available for the evaluation. The evaluation team began Phase I by meeting with staff of FEMA, the General Accounting Office, the Congressional Research Service, and with local floodplain managers, mapping and insurance companies, and other NFIP stakeholders. In addition, the evaluation team twice convened a national steering committee (see Section X). Logic Models The evaluation team developed “logic models,” found in Section II, for each of the NFIP’s major components as well as an overall model that summarizes the linkages among all parts of the NFIP. Each logic model links one or more parts of the NFIP in an organized fashion. The overall logic model summarizes congressional purposes, the NFIP’s mission, simplified goals, objectives, and activities with intermediate and ultimate outcomes. Equally important, the overall model reflects consensus between FEMA and the evaluation team as to what successful implementation of the NFIP should accomplish in terms of ultimate outcomes: Decreased risk of flood losses; Reduced costs and adverse consequences of flooding; ? Reduced demands and expectations for federal disaster assistance after floods; and ??Natural and beneficial values of floodplains are restored and preserved. In turn, the ultimate outcomes also provide a guide for developing indicators of success and accomplishment, the basis for any assessment of performance. Design Matrices and Narratives Once the logic models were completed, the evaluation team developed matrices that outline the proposed evaluation design for each question in each area of inquiry. The matrices display what information is needed to answer each evaluation question, possible sources for the information, the tasks involved in gathering and analyzing the information, and the outcome measures that can be used to answer the question. The matrices, in Sections III through VIII, summarize the design and should be used in conjunction with the design narratives, which follow the relevant matrices for each area of inquiry. The design narratives provide a summary discussion of the evaluation design as well as illustrative research in the area and other pertinent information. The narratives include a brief discussion of the perceived strengths and weaknesses of the design, the relative priority of the question, and an assessment of the feasibility of addressing the topic or issue. The narratives will be used to develop scopes of work for substudies to be performed during Phase II. FIMA staff and the members of the steering committee were given an opportunity to review all the narratives. The narratives reflect all comments received, but do not necessarily reflect the preferences or perspectives of all the reviewers. Priority Ranking Throughout the design phase, FEMA staff and members of the steering committee advised the evaluation team to narrow the focus of the evaluation. The evaluation will be of highest value if it provides definitive answers to high priority issues rather than inconclusive answers to many, less important issues. Consequently, the evaluation team and FEMA staff spent considerable time examining the relative priority of questions. In addition, discussions concerning the priority of various issues served as a major focus of the meetings of the steering committee during Phase I. The result of this collaborative process is a series of “Primary Evaluation Questions,” divided into three groups. The evaluation will begin with questions in Group 1 and then move to those in Groups 2 and 3. The Primary Evaluation Questions can be found immediately after this introduction (see Section IX for the complete priority ranking). Steering Committee In accordance with FEMA’s desire to ensure widespread participation and substantial external review of the design and evaluation, the evaluation team established a national steering committee. It consists of government officials, FEMA staff, and private sector and academic experts. They represent an enormous amount of knowledge concerning the NFIP. Most of the steering committee members have been deeply involved with the NFIP and water resource issues for much of their careers. During the two steering committee meetings and with additional written comments, the steering committee provided valuable feedback and recommendations that assisted the evaluation team in revising the proposed designs. The steering committee also validated the logic models and priority ranking of the evaluation questions. The steering committee will provide guidance and oversight during Phase II. Related Reports In addition to the materials described above, AIR also prepared two related reports during Phase I: A Chronology of Major Events Affecting the National Flood Insurance Program and The National Flood Insurance Program: An Annotated Bibliography. PHASE II The evaluation team believes that with the assistance of the steering committee, FEMA staff, and other NFIP stakeholders, this evaluation design will produce the first comprehensive and independent assessment of the NFIP. This evaluation will contribute to a more effective and efficient program by assessing current operations and accomplishments and by recommending alternative approaches to the achievement of the NFIP’s ultimate objectives when the results of the evaluation suggest that changes in existing procedures may be desirable. ACKNOWLEDGEMENTS The evaluation team would like to thank Claudia Murphy, FEMA’s Task Order Officer, members of the steering committee, FEMA staff, and others who provided assistance during Phase I. The primary evaluation team members are: Rich Tobin, Project Director, Michael Burke, Gray King, and Ben Young from American Institutes for Research, Ted Miller, Principal Investigator, from Pacific Institute for Research and Evaluation and Dick Messick from Deloitte and Touché. Primary Evaluation Questions Group 1 1(b) Has the NFIP changed or affected (positively or negatively) public knowledge, occupancy, and use of the nation’s floodplains and other erosion-prone or flood-prone areas? More specifically, has the NFIP effectively promoted uses (or nonuses) of floodplain and other flood-prone areas that minimize the risks of and damages caused by floods? If so, how? What are the consequences of these changes for flood mitigation and management? 2(a) Has the NFIP affected federal, state, and local governments’ overall costs of response and recovery from flooding? If so, how? Have these costs changed as the number of policyholders has increased? How do pre-FIRM properties and repetitive-loss properties affect these costs? 2(b) Are the NFIP and other government programs reducing, increasing, redistributing, or controlling total flood losses, both governmental and individual? What are the financial impacts of these losses, and how accurate are current methods for estimating these losses? Are the NFIP’s mitigation efforts cost-effective? 2(c) How has the NFIP affected the need for or cost of federal disaster relief? 3(a) Are the types of insurance coverage offered and associated premium rates aligned with the program’s goal of effective floodplain management? If not, how can the coverage and the rates be changed to encourage improved management of floodplains? 3(b) What impact have the NFIP’s minimal building standards for new construction in Special Flood Hazard Areas had on risk exposure and property loss? Which standards are the most and least effective in reducing exposure and property losses due to floods? Is the cost of implementing the major standards commensurate with their benefits? 4(c) To what extent are states and communities effectively implementing the NFIP’s requirements for floodplain management? Does the NFIP have appropriate sanctions available when communities do not perform responsibilities effectively? Does it apply those sanctions appropriately? Does the NFIP monitor compliance adequately and concentrate its investigative resources to maximize the detection of communities with serious problems? 4(e) Are the NFIP’s standards for construction and building design sufficiently stringent so that losses are minimized at a reasonable cost to communities and property owners when flood damage occurs? Has the NFIP responded appropriately when changes are needed in standards for building design and construction? Are the standards and incentives sufficient to protect against flood risks that may be increasing in the future? 5(h) What are the implications of making the 1%-probability flood a threshold for mandatory insurance purchase and flood management ordinances? Specifically, how has this choice affected construction in floodplains, property values, NFIP loss experience, map amendment costs, and federal disaster expenses in areas that face flood hazards below the 1%-probability threshold? What probability levels capture 90%, 95%, and 99% of disaster costs? How would the mapping backlog, annual map amendment costs, insurance sales, insurance rates, federal disaster costs, and total disaster costs be affected by shifting to one of those probabilities as a threshold (selecting the probability based on the steepness of the fall-off in the damage curve)? Cross-cutting 1 What measures does FEMA use to assess and evaluate its performance with respect to its efforts to affect: The occupancy and use of floodplains? The costs and consequences of flooding? Insurance rating and indemnity functions? Floodplain management and enforcement? Hazard identification and risk assessment? Communications and marketing? Are there alternative measures of performance that would better inform policymakers about the costs, benefits, and accomplishments of the NFIP? Cross-cutting 2 Are there limits to FEMA’s funding or legal authority that prevent or interfere with its ability to accomplish its goals and objectives with respect to: The occupancy and use of floodplains? The costs and consequences of flooding? Insurance rating and indemnity functions? Floodplain management and enforcement? Hazard identification and risk assessment? Communications and marketing? Group 2 1(a) Is there a unified national program for floodplain management as envisaged in the National Flood Insurance Act of 1968? If yes, what are its key characteristics and consequences, especially with respect to the NFIP’s effectiveness? If not, why not? Is there evidence of effective coordination, consistency, and compliance among federal agencies in the implementation of Executive Order 11988, Floodplain Management, in support of the NFIP? In what ways has the NFIP promoted the institutionalization of the floodplain management and flood-mitigation strategies in other federal agencies? How do activities of other agencies contribute to the NFIP’s effectiveness? What action can increase the synergy and effectiveness of the NFIP’s relations with other agencies? 1(d) What, if any, are the unintended or unexpected social, and environmental effects (both positive and negative) of the NFIP? If applicable, in what ways do these effects influence achievement of the NFIP’s goals? What cost-effective measures are available to mitigate adverse effects? 2(j) What proportion of flood-prone, low-income people have flood insurance? This analysis should examine regional and rural-urban variation. How well does flood insurance serve low-income people? Are there changes that can be made to the NFIP to better serve this population, or are there other approaches better suited to addressing the needs of low- income people? What are the social and economic consequences of the NFIP’s mitigation activities on low-income populations, especially those in flood-prone areas? 4(b) Are the roles of the states and FEMA properly identified, funded, and integrated within the system? Is the role of state government adequately identified and sufficiently substantive to be effective in contributing to the NFIP’s goals? Has FEMA promoted the institutionalization of floodplain management and flood mitigation strategies in the states? Has FEMA federalized the nation’s flood problem so that states avoid taking responsibility for the problem, or has FEMA encouraged states to develop floodplain management programs? Are there alternative institutional arrangements whereby states (or groups of states) could have their roles in floodplain management (and flood insurance) enhanced while the federal government’s insurance role is diminished? 5(c) What might be the costs and advantages of flood insurance rate maps that reflect anticipated development? Are such maps feasible? Would they contribute to the reduction of flood- related damages and increases in the number of policyholders? What communities would it be cost-effective to target for anticipated development mapping? 6(b) What messages has the NFIP sought to deliver to program constituents? Has the information raised awareness of the NFIP or increased the number of flood insurance policies sold? How effectively and efficiently have the messages of the NFIP been communicated to target audiences? What is the relationship between knowledge of benefits and attitudes toward the NFIP? What strategies are the most and least cost- effective in raising awareness? Are there better strategies for communicating the NFIP’s messages? Group 3 3(c) Are there more effective and efficient ways to risk rate NFIP policies and continue to support improved management of floodplains? How detailed should elevation rating of structures be in order to accomplish insurance rating and floodplain management? 3(f) Is there an optimal level of market penetration for flood insurance to achieve program goals for insurance and floodplain management? What is that level and how can it be determined? 3(i) How effective and synergistic is the current operational model, which is premised on risk assessment and rating on a structure-by-structure basis? Are there compelling alternatives and/or enhancements to the existing framework? If so, what are they? 4(a) How well does FEMA coordinate its flood-related programs, including its mitigation activities, the NFIP, and disaster relief or assistance? How does the NFIP coordinate its efforts with other programs, agencies, and organizations concerned with land-use management, building science, and the mitigation of flood losses? If so, with what consequences? 4(d) What incentives or disincentives exist to encourage (or discourage) states and communities to exceed the NFIP’s minimum floodplain management requirements? Are existing incentives sufficient to promote the NFIP’s objectives in a timely and cost-effective manner? If not, why? Are there additional, politically feasible and cost-effective incentives for state and local governments, builders, realtors, lenders, property owners, or others that can promote the NFIP’s effort to manage floodplains efficiently and effectively? What alternatives are there to eliminate or mitigate disincentives? 5(d) How can new and developing technologies improve the identification of flood hazards? How can these technologies be used to convey risk information more effectively? Does the NFIP have an organizational culture that promotes and facilitates the acquisition and use of new technologies? How does the NFIP assess the potential utility of new technologies? 5(e) How are detailed floodplain maps best financed? What practical and cost effective alternatives are there to fund the identification of flood hazards that would more equitably distribute costs among policyholders, states and communities, taxpayers, and private sector users? What are the advantages to communities that participate in the mapping process, thereby taking an activist approach to managing their floodplains? Does the process encourage an activist approach to floodplain management? 6(a) How successfully has the NFIP been in communicating its insurance component to insurance companies, agents, adjusters, the lending industry, and the public? Has the NFIP been successful in communicating the program’s goals and requirements for floodplain management to these audiences, state and local governments, the building industry, and other concerned groups? How can the NFIP assess changes in its relative success with its communications over time? 6(h) How can the NFIP’s terminology (e.g., “100-year flood”) be improved to communicate with the public more effectively? What is the best way to convince federal, state, and local governments to change their flood terminology?