About the Competition

The following text is an excerpt from the Advanced Research Projects Agency – Energy (ARPA-E) funding Opportunity No. DE-FOA-0001357, “Generating Realistic Information for the Development of Distribution and Transmission Algorithms (GRID DATA),” which will provide the problems and data sets for this Challenge, and the Request For Information for the Competition.

Background

Since the dawn of the age of electrification, electric power system designers and operators have been required to manage (due to the absence of large-scale cost effective electricity storage) the real-time matching of instantaneous electricity generation and demand. Achieving a continuous match between supply and demand requires utilities, grid operators, and other stakeholders to use a variety of sophisticated optimization algorithms operating across a wide range of timescales. These include tools for determining optimal transmission line and power plant siting and construction, maintenance scheduling, and long-term, day ahead, hour ahead, and real-time electricity dispatch rates.

A number of emerging trends, including the integration of high penetrations of renewable electricity generation, changing electricity demand patterns, and the improving cost effectiveness of distributed energy resources (including storage), will substantially alter the operation and control of electric grids over the next several decades. For example, more active optimization and control of electric distribution systems are likely to be required, including the near real-time estimation, optimization, and control of distribution network power flows. The expected growth in system complexity will require the development of substantially improved software optimization and control tools to assist grid operators, and deliver the societal benefits of improved grid performance.

Many new grid optimization methods have been proposed in the research community in recent years. Many theoretical claims have been made regarding the possible benefits that these new algorithms might offer utilities and grid system operators. However, the existing accepted practice for testing emerging optimization algorithms does little to help accelerate adoption. In particular, the vast majority of new reports in the literature only test new algorithms on relatively small-scale models that often must be heavily modified to satisfy the modeling requirements for each individual algorithm. Computational experiments are also typically conducted on a wide range of computational systems (ranging from commodity laptops to large-scale clusters with many thousands of nodes). Variations in modeling assumptions further complicate the comparability of algorithm testing results in the literature (for example, what types of contingency constraints are included and/or how normal vs. emergency ratings are considered). A new paradigm for the testing and evaluation of emerging grid optimization algorithms is needed to accelerate the adoption of these transformational techniques by industry. This competition seeks to lay the foundation for that change.

The first major barrier to fair, transparent, industrially relevant algorithm performance comparisons has historically been the lack of high-fidelity, open-access, large-scale power system models. As described ARPA-E’s recent GRID DATA FOA[1], many of the existing open-access power system benchmark models lack key details, are too small, and do not sufficiently reflect real-world systems. ARPA-E’s recently announced GRID DATA program is aiming to fill this important gap. Specifically, ARPA-E is funding 5 teams to develop new open-access power system models that have been designed specifically to enable the comprehensive evaluation and testing of new OPF algorithms.

The second major barrier to fair, transparent, industrially relevant algorithm testing has been the lack of consistent modeling assumptions and uniform problem definitions. Even small changes in how specific constraints are modeled or which constraints are considered can have significant implications for algorithm performance and solution quality. Variations in the literature make direct, fair comparisons between different approaches to solving problems extremely difficult. ARPA-E intends to fill this gap through the establishment of a common community platform for the fair and consistent evaluation of new algorithms. This existence of this platform will accelerate the use and widespread adoption of new power system optimization and control approaches. The specific design of the OPF competitions will also require the community to investigate the applicability of new algorithms across a wide range of system operating points.

The existence of the competition platform and establishment of consistent, community-defined modeling assumptions will become increasingly important over the next several decades. The rapid growth in variable and uncertain renewable generation and numerous types of distributed energy resources is expected to lead to a critical need for vastly improved approaches to grid optimization and control. Indeed, new approaches to grid optimization and control are expected to facilitate the cost effective integration of larger penetrations of these new resources.

    Opportunities in Grid Power Flow Optimization

    The OPF problem is the central optimization challenge underlying the entire suite of grid planning and operations tools. Simply stated, the OPF problem is that of finding the optimal dispatch settings for power generation, flexible customer demand, energy storage, and grid control equipment that maximize one or more grid objectives.[2],[3],[4] In order to be deployable, the recommended settings must satisfy all physical constraints of electric power infrastructure and applicable operating standards (including, for example, minimum/maximum voltages at each bus, minimum/maximum power generation from all generators, thermal transmission constraints, and constraints related to the security of the system when contingencies occur). For a more complete history and formal problem formulation, we refer the reader to a history authored by the Federal Energy Regulatory Commission (FERC).[5]

    The core OPF solution methods predominantly used in industry today were designed in an era when computers were far less capable and more costly than they are currently and formal general purpose optimization solvers were in their infancy. Therefore grid operators and OPF vendors were required to make a range of simplifying assumptions, most commonly a set of linearizing assumptions which ignore voltage and reactive power optimization referred to as “DC-OPF.”[6] Many proprietary variations on these algorithms have been developed over the past several decades by vendors. Despite improvements in DC-OPF formulations and solvers, there are no tools currently in widespread use in industry that use the full AC power flow equations (without linearizing assumptions) and simultaneously co-optimize both real and reactive power generation (known as “AC-OPF”). The OPF tools in use today often result in conservative solutions that additionally must be iteratively checked for physical feasibility of solutions before implementation. When non-physical solutions are found, the OPF algorithm must be run again with a modified set of constraints to generate a new solution.

    Dramatic improvements in computational power and advancements in optimization solvers in recent years have prompted research on new approaches to grid operation and new approaches to solving OPF and other grid optimization problems.[7] Since the turn of the millennium, the performance of the most powerful supercomputers has increased by almost four orders of magnitude (while the cost per computational step has dropped by approximately the same factor).[8],[9] Improvements in optimization and search methods have evolved similarly, especially those related to Mixed Integer Programming (MIP) and heuristic-based optimization methods. The relative speed of commercial general-purpose solvers such as CPLEX[10] and Gurobi[11] has also increased by over three orders of magnitude on fixed hardware.[12] “Cloud computing as a service,” which can be used to leverage many of these gains, has also started to gain more widespread interest within the power system engineering community.[13]

    In tandem, many new approaches to solving OPF problems have been proposed in the literature in recent years; it appears increasingly likely that scalable and more accurate approaches to solving the full AC-OPF may be within sight. For example, fast and accurate convex relaxations have been formulated where the global minimum can be found efficiently using semi-definite and second order cone programming (under certain system assumptions and conditions).[14],[15],[16],[17] Often it can be shown that these relaxations give global solutions to the original, non-convex problem.[18],[19] Distributed and parallelizable OPF algorithms have also been proposed, for example, using the Alternating Direction Method of Multipliers (ADMM), suggesting that AC-OPF can leverage more advanced computational hardware.[20],[21],[22] These same algorithms could enable the real-time coordination and/or optimization of large numbers of distributed energy resources. Finally, many unique methodologies using techniques such as genetic algorithms, neural networks, fuzzy algorithms and holomorphic embedding have also emerged, claiming, in many cases, to revolutionize solution methods for OPF.[23],[24]

      Competition Motivation

      Despite numerous recent research projects and papers on improved grid optimization strategies, most new advances have struggled to mature past the early research stage. Most recent grid operation optimization advances remain non-validated on realistic, large-scale test models and their operational limits also remain largely unexplored. Few mechanisms currently exist to allow for the direct comparison of these different solution methods. Therefore, it is difficult to know the precise relative strengths and weaknesses of different algorithms.

      Establishing methods for the transparent and fair benchmarking and comparison of different grid optimization algorithms could vastly accelerate the validation and adoption of new algorithms. Indeed, ARPA-E’s GRID DATA program is aiming to contribute to this need by funding the development of largescale, realistic, validated, open-access power system models specifically designed for testing new grid optimization algorithms. We hope that once available to the power systems research community, these models will enable greater reproducibility of research results and will enable more comprehensive evaluations. This will be critical to breaking down the silos that currently exist within the early-stage applied research community today and will help connect innovators and entrepreneurs with the industrial power systems engineering community. This is a particularly acute need for researchers from other technical disciplines whose expertise may have value in application to power systems optimization. Given the dynamics, complexity, and uncertainty of emerging power systems, this broader research community could provide transformative opportunities for achieving timely and effective solutions.

      Beyond providing new models, direct comparisons of new algorithms can be further facilitated by the creation of formal prize competitions. Many other industries that are heavily reliant on mathematical algorithms and/or optimization have successfully employed prize competitions to accelerate algorithm development and validation. In the right context, prize competitions have a number of advantages over traditional research grants such as ARPA-E’s focused programs and OPEN solicitations. When employed properly, innovation prizes can result in better problems and solutions, more efficient use of funding, and engagement across broad communities of stakeholders.

      Competitions develop more impactful problems and solutions

      Prize competitions are nimble and allow participants to rapidly develop solutions to address near-term issues or create new learning curves. Inviting a vast number of people – most of whom would be hard to identify independently – to solve a difficult problem increases the probability of developing a novel solution. Research at Harvard Business School has provided strong evidence that prize competitions can lead to faster, more efficient, and more-creative problem solving.[25] Prize competitions also allow organizations and individuals to bring forward difficult problems. Traditional grants can be more restrictive, especially when problem statements and proposed business solutions have to be made confidential.

      Competitions raise funds and invest capital more efficiently

      Because they only pay for results, innovation prizes can set more radical goals compared with traditional grants. By shifting risk from sponsors to competitors, prizes attract surplus investment, time, and talent from motivated participants. For example, teams competing for the 10 million dollar Ansari X PRIZE collectively spent over $100 million to develop reusable manned spacecraft.[26] Successful prize competitions that produce vetted solutions create momentum towards more ambitious programs and greater financial involvement from the private sector. Since the Ansari X PRIZE concluded in 2004, $1.5 billion has been invested in the nascent space taxi industry.[27]

      Competitions engage a broader community of stakeholders

      Prize competitions increase the number and the diversity of entities that are addressing difficult challenges. Established industries have been transformed by competitions around well posed problems that engage multi-disciplinary teams and those without subject matter expertise. By bridging across disciplines and involving the private sector through problem definition, financial sponsorship, judging, and commercialization, prize competitions create communities in ways that grants cannot achieve. In addition to forming new communities, prize competitions can highlight a range of best practices or opportunities in a given field. Innovation prizes can also expose latent market demand, change the behavior of participants, and influence public perception.

      When are prize competitions the right model?

      Innovation prizes are not the right choice in every situation. For example, highly specialized technical challenges that require sustained investment over long periods may be better suited for traditional research grants. On the other hand, problems that fit the following criteria may be a good fit for prize competitions:

      • Objective is clear, measurable, and achievable within reasonable time frame
      • Relatively large population of potential problem solvers is available
      • Fast and efficient ways exist to test a large number of potential solutions
      • Industry and researchers have yet to converge on the optimal solution
      • Competition participants are willing to bear some of the costs and risks
      • Different problem solvers can focus on specific parts of a modular challenge

      How can prize competitions create value in power systems?

      There are a range of power system challenges that may lend themselves well to prize competitions. Increasing variable renewable generation, flexible customer demand, falling energy storage costs, and innovation in grid control equipment are poised to expose deficiencies in the existing OPF algorithms. A sizable research community has advanced promising AC-OPF algorithms. However, without comprehensive, fair, and transparent comparisons, the utility industry and vendors are unlikely to fully invest in these potentially transformative approaches.

      The next generation of grid optimization algorithms can benefit greatly from the advantages of innovation prizes, and fit the criteria listed above. Most importantly, the power industry needs a medium to vet solutions to these challenging technical problems. ARPA-E is developing a dedicated platform that we believe could be used to manage a wide range of power system algorithm research competitions. It is our hope that once the processes are established and the prize competition model has been validated within this domain, private sector entities or other government agencies will commission and sponsor additional prize competitions. We believe this could usher a new era of innovation in electric power systems optimization research.

      Competition Vision

      For the reasons listed above, ARPA-E believes that a competition structure may be the ideal mechanism for testing new algorithms and spurring innovation in the OPF space. The purpose of this initial phase zero of the competition is to solicit opinions regarding the details—including the baseline problem specifications, competition rules, eligibility for participation, scoring metrics, criteria for winning, prize structure and on-line competition platform details. Phase zero represents a “straw-man” model for each of these competition components. Phase zero participants should comment via the forum, or to the ARPA-E RFI, specifically on their technical and programmatic opinions regarding this model, detailing exactly why or why not these components are optimally structured for delivering innovation in OPF algorithmic advances. Respondents may comment on all or some of the components of this model.

      As detailed above, a necessary first step for any meaningful AC-OPF competition must be the development of many small, medium and large-scale power systems models under a wide variety of operating conditions (reflecting both the current grid and that in the future). It is expected that these models (transmission and distribution) will be developed under the ARPA-E Generating Realistic Information for the Development of Distribution and Transmission Algorithms (GRID DATA) program.[28]

      It is expected that competition participants will develop new modeling approaches and solution algorithms using GRID DATA datasets. It is envisioned that the competition will be multi-phase (with escalating problem and model complexity), with winners announced for each phase. In each phase, a subset of power systems models and scenarios relevant to the chosen problem will be released to participants to facilitate the early development and testing of solution methods. Competitors will also have the opportunity to submit their code for formal evaluation (and scoring) by the competition platform. This should give participants a chance to familiarize themselves with the GRID DATA models and to enable any training of OPF algorithms that may be necessary). Evaluation and scoring of solutions will be automated and transparent, with parameters announced before the competition begins (and possibly evolved from the “straw-man” version contained in this document).

      Participants will interact with the competition via a hosted computational platform with a web front-end portal. A public “leader board” (or several leaderboards) will be maintained during the competition on the web portal. Participation will be unrestricted and may be composed of individuals or teams and may merge during the duration of the competition.

      Please note that the development of this competition is still in its early stages. All of the parameters listed in this document (including those in the paragraphs above) are not completely determined; we request input and feedback on any parameters about which respondents have an opinion.

      Goal

      The overall goal is running a competition that is fair, transparent and results in the accelerated development of disruptive new grid optimization algorithms that can robustly address existing and emerging power system challenges. Participants should proffer an opinion on any structure which results in this ultimate goal. The goal of the OPF Challenge is to identify grid optimization strategies and algorithms that are beyond the state-of-the-art currently available. The OPF Challenge is seeking pioneering solutions to the problem and participants are encouraged to comprehensively redesign their approaches.

      Impact

      Improved OPF algorithms could yield significant benefits. Recent studies have suggested that enhanced OPF algorithms could offer as much as 5-10% reductions in total U.S. electricity cost due to the alleviation of grid congestion (corresponding to $6-$19B saved depending on energy prices).[29],4 In addition to monetary savings, improved optimization algorithms are likely to help ensure reliable system operations as power flows become more dynamic in the future.[30] To fully realize the potential benefits of renewable generation as well as recently developed electric transmission power-flow controllers, distribution automation technologies, distributed generation, energy storage, and demand-side control will require more complex (and fundamentally non-linear) grid operation optimization and dispatch algorithms. Further, as the number of controllable resources connected to electric power systems (at both transmission and distribution voltages) grows substantially, distributed or decentralized versions of OPF algorithms could become increasingly important. The cost effective and reliable operation of future renewable-intensive electric power systems is likely to rely more on algorithm outputs and decision support tools and less on operator intuition.

      Prizes

      The level of effort needed to advance the field is significant. This is a hard problem that has been long studied. Prizes will be awarded to the winner or top performers of each phase of the competition; these may include a cash award or follow-on research funding for an ARPA-E managed project.

      Challenge Statement

      Solve each of the specified set of Optimal Power Flow (OPF) problems posted on the website, producing an accurate and reliable solution with the lowest objective value and the fastest elapsed time less than the maximum time allowed.

      • [1] Advanced Research Projects Agency—Energy (ARPA-E) Financial Assistance Funding Opportunity Announcement No. DE-FOA-0001357, "Generating Realistic Information for the Development of Distribution and Transmission Algorithms (GRID DATA)", https://www.fedconnect.net/FedConnect/PublicPages/PublicSearch/Public_Opportunities.aspx?doc=DE-FOA-0001357&agency=DOE, 2015.
      • [2] J. Carpentier, "Contribution to the economic dispatch problem," Bulletin de la Société Française des Électriciens, ser. 8, vol. 3, pp. 431‐447, 1962
      • [3] H.W. Dommel and W.F. Tinney, "Optimal power flow solutions," IEEE Transactions on Power Apparatus and Systems, vol. 87, no. 10, pp 1866-1876, October 1
      • [4] There are a variety of specific applications for OPF. The specific objective function and most important constraints can vary widely. In many applications, where demand is considered fixed, the objective is considered to be minimization of total generation cost. In the context of electric distribution systems, this problem is often focused on minimization of system losses. 
      • [5] M. B. Cain, R. P. O’Neill, and A. Castillo, "History of optimal power flow and formulations," Federal Energy Regulatory Commission, Washington, DC, August 2013, http://www.ferc.gov/industries/electric/indus-act/market-planning/opf-pa...
      • [6] A. J. Wood, B. F. Wollenberg, and G. Sheblé, Power generation, operation, and control, 3rd ed. Hoboken, NJ: John Wiley & Sons, 2013
      • [7] P. Panciatici et al. "Advanced optimization methods for power systems." Proceedings of the 18th Power System Computation Conference, Wroclaw, Poland, August 2014, pp. 1-18, doi: 10.1109/PSCC.2014.7038504
      • [8] http://www.top500.org/
      • [9] https://intelligence.org/2014/05/12/exponential-and-non-exponential/
      • [10] http://www-01.ibm.com/software/commerce/optimization/cplex-optimizer/
      • [11] http://www.gurobi.com/
      • [12] T. Koch et al., "MIPLIB 2010," Mathematical Programming Computation, vol. 3, no. 2, pp. 103-163, June 2011, doi: 10.1007/s12532-011-0025-9  
      • [13] J. Goldis et al., "Use of Cloud Computing in Power Market Simulations" Presentation at FERC Staff Technical Conference on Increasing Real-Time and Day-Ahead Market Efficiency through Improved Software, Washington, DC, June 2014
      • [14] S. Low, "Convex relaxation of optimal power flow, Part I: Formulations and equivalence," IEEE Transactions on Control of Network Systems, vol. 1, no. 1, pp. 15-27, March 2014, doi: 10.1109/TCNS.2014.2309732
      • [15] S. Low, "Convex relaxation of optimal power flow, Part II: Exactness," IEEE Transactions on Control of Network Systems, vol. 1, no. 2, pp. 177-189, May 2014, doi: 10.1109/TCNS.2014.2323634
      • [16] R. Madani, S. Sojoudi, and J. Lavaei, "Convex relaxation for optimal power flow problem: Mesh networks," IEEE Transactions on Power Systems, vol. 30, no. 1, pp. 199-211, May 2014, doi: 10.1109/TPWRS.2014.2322051
      • [17] D. Molzahn et al., "Implementation of a large-scale optimal power flow solver based on semidefinite programming," IEEE Transactions on Power Systems, vol. 28, no. 4, pp. 3987-3998, April 2013, doi: 10.1109/TPWRS.2013.2258044
      • [18] J. Lavaei and S. Low, "Zero duality gap in optimal power flow problem," IEEE Transactions on Power Systems, vol. 27, no. 1, pp. 92-107, August 2011, doi: 10.1109/TPWRS.2011.216097 
      • [19] L. Gan et al., "Exact convex relaxation of optimal power flow in radial networks," IEEE Transactions on Automatic Control, vol. 60, no. 1, pp. 72-87, June 2014, doi: 10.1109/TAC.2014.2332712 
      • [20] A. Sun, D.T. Phan, and S. Ghosh, "Fully decentralized AC optimal power flow algorithms," Presentation at IEEE Power and Energy Society General Meeting, Vancouver, BC, Canada, July 2013, doi: 10.1109/PESMG.2013.6672864
      • [21] S. Magnússon, P. Weeraddana, and C. Fischione, "A distributed approach for the optimal power flow problem based on ADMM and sequential convex approximations," arXiv preprint arXiv:1401.4621, January 2014
      • [22] B. H. Kim and R. Baldick, "A comparison of distributed optimal power flow algorithms." IEEE Transactions on Power Systems, vol. 15, no. 2, pp. 599-604, May 2000, doi: 10.1109/59.867147
      • [23] X. F. Wang, Y. Song, and M. Irving, Modern power systems analysis, New York, NY: Springer Science & Business Media, 2008
      • [24] A. Trias, "The holomorphic embedding load flow method," Presentation at IEEE Power and Energy Society General Meeting, San Diego, CA, July 2012, doi: 10.1109/PESGM.2012.6344759
      • [25] Pisano, Gary. "You Need an Innovation Strategy" Harvard Business Review https://hbr.org/2015/06/you-need-an-innovation-strategy
      • [26] Bays, Jonathan. "Using Prizes to Spur Innovation" McKinsey & Company http://www.mckinsey.com/insights/innovation/using_prizes_to_spur_innovation
      • [27] ibid.
      • [28] https://www.fedconnect.net/FedConnect/PublicPages/PublicSearch/Public_Opportunities.aspx?doc=DE-FOA-0001357&agency=DOE
      • [29] M. Ilic, "Modeling of hardware and systems related transmission limits: the use of AC OPF for relaxing transmission limits to enhance reliability and efficiency," Presentation at FERC Staff Technical Conference on Increasing Real-Time and Day-Ahead Market Efficiency through Improved Software, Washington, DC, June 2013, http://www.ferc.gov/CalendarFiles/20140411131533-T2-B%20-%20Ilic.pdf
      • [30] GE Energy, "Western wind and solar integration study," National Renewable Energy Laboratory, Technical Report No. NREL/SR-550-47434, May 2010, http://www.nrel.gov/docs/fy10osti/47434.pdf