It is generally recognized that a high-bandwidth and highly available networked communication system should overlay the transmission system topology in order to enable the control and protection envisaged today to make the grid more efficient and more reliable. The specifications for such a communication system have been difficult to develop, however, because it needs to support a great variety of applications, many of which have not yet been developed. Organizations such as the North American SynchroPhasor Initiative (NASPI) are trying to build on this vision of a communication system that can utilize phasor measurement data to initiate fast controllers, including flexible alternating current transmission system (FACTS) devices.
A major hurdle in developing such fast, wide area controls has been the lack of design tools available to do so. In particular, the development of controls that depend on communications to carry the input and output signals and complex software to process these signals requires tools to simulate and analyze such controls. To accurately portray the behavior of such controls, design tools must integrate the dynamic behavior of the power system with the response of the communication and computation system.
We describe here a simulator—GridSim—that can simulate in real time the electromechanical dynamic behavior of the power grid, the IT infrastructure that overlays the grid, and the control systems taking advantage of that IT infrastructure. This simulator was devised for designing and testing new wide area control and protection schemes. GridSim is able to represent a large portion of a grid and runs in real time so that various components running at different sampling rates can be tested together.
The use of time-synchronized, high-data-rate sensor technology is widely viewed as a critical enabler for increasing the reliability of the power grid while allowing the integration of many more stochastically variable renewable energy sources such as solar radiation and wind. For example, the deployment of phasor measurement units (PMUs) is becoming more commonplace. PMUs are capable of sampling frequency, voltage, and current thousands of times per second and outputting accurate, time-stamped measurements 30–120 or more times per second. It is difficult, however, for utilities to take full advantage of these devices due to a lack of tools for designing and evaluating the control systems that exploit them. Furthermore, the behavior of such control systems will also depend on the performance of the wide area communications systems that connect the sensors, control logic, and actuators—wide area communications systems whose design and specifications are themselves still evolving.
Simulation is historically one the principal tools used in the design of power system controls. No existing simulation framework, however, can model at the scale of the power grid the combined behavior of the power system, the communications system that overlays it, and the control system that relies on the latter to monitor and control the former. GridSim is intended to address these issues by providing a very flexible simulation framework that incorporates power system simulation, data delivery, flexible sensor deployments, and the ability to incorporate actual power system components, protocols, and algorithms. Using actual power system artifacts is important for two reasons. First, it allows the artifacts to be tested in the simulation environment, which is one way to increase confidence in a design. Second, it allows existing artifacts such as the Grid Protection Alliance’s openPDC product and the GridStat communication framework to be used as building blocks for GridSim, speeding its implementation. From this decision comes another requirement: that GridSim operate in real time so as to properly interface with these artifacts.
The Overall Design of GridSim
GridSim is a real-time, end-to-end power grid simulation package designed using a default sample rate of 30 samples per second (per sensor). The goal of this project is to simulate power grid operation, control, and communications at gridwide scale (e.g., the Western Interconnection) in order to give utilities the ability to explore new equipment and control system deployments. Possibilities include simulating large-scale PMU installations and power applications able to utilize the vast quantities of data generated in such a situation. With the objective of providing tools to simulate real-world equipment usage and the ability to be used in conjunction with readily available utility industry equipment, GridSim uses the IEEE C37.118 data format standard for all streaming measurement data.
The GridSim platform consists of a number of components falling into four groups: power system simulation, substation simulation, communication and data delivery, and control center applications (see Figure 1). We first describe the overall relationship between these groups and then look at each of them in detail.
The power system simulation calculates the electromechanical dynamics in real time. Sensor data from the simulated power system are fed in C37.118 format to the substation simulation processes at a rate of 30 samples per second. In the substations, data are optionally processed by substation applications and published, along with the outputs of the substation-level applications, to the data delivery component through simulated substation gateways. Delivery to control center applications and other substations occurs via the data delivery system. Note the design choice here: the wide area data delivery system is not involved in connecting simulated sensors within the simulated substations where they are located. Although the substation-level processing of the data is simulated, the data communication within the substation is assumed to be negligible for the current goals of wide area control design.
The data delivery component of GridSim is GridStat, a publish-subscribe, wide area data delivery framework designed from the ground up to meet the emerging needs of electric power grids. Once data are published, the flexibility provided by the GridStat data delivery middleware allows subscribing applications to be easily integrated into the system without massive reconfiguration.
In the current GridSim implementation, published data are used by the two control center applications included in this project: the hierarchical state estimator and the oscillation and damping monitor.
Power System Simulation
Power system simulation in GridSim is provided by a modified version of TSAT, an industry-proven transient stability simulator produced by Powertech Labs, Inc. Unmodified TSAT accepts power system topologies, initial values, and dynamic simulation variables (such as faults at specific times) as inputs. On execution, the simulator loads the input values, then as quickly as possible computes the state of the system over time; on completion it writes the results to a file.
An off-line transient stability simulation such as TSAT does not perfectly meet the needs of GridSim. To obtain real-time performance, the simulator was modified so that simulation time progresses no faster than wall-clock time. This is accomplished by pausing after computing each set of measurements (30 sets per second) until the correct wall-clock time arrives for that set to be published. To extract the measurement sets at the time they are produced by the simulation, certain TSAT functions are used. They directly implement simulated PMUs attached to particular points in the power system topology where they measure frequency, voltage, and current 30 times a second. These sensor data from the simulated PMUs are sent to the measurement generator for postprocessing (see Figure 2).
The measurement generator also bridges the gap between the bus-branch power system model supported by TSAT and the more detailed bus-breaker model that represents the substations. To do this, GridSim’s static data generator creates tables that map the FromBus/ToBus/EquipmentID measurement identification information used in TSAT to the unique CircuitBreaker/BusID/PMUID numbers used throughout the rest of GridSim. Data from the static data generator also allow the measurement generator to synthesize additional measurements, such as breaker currents, from the TSAT outputs. Noise and other real-world attributes can be added within the measurement generator, if desired. Once these operations have been performed, the PMU measurements are sent to a C37.118 encoder and then to the substation simulation processes.
The substation simulation processes host substation-level power applications and substation gateways. Power applications perform computations—both the applications described below have substation-level processing—and submit results to the substation gateway. Measurement generator output for each substation is also published to the data delivery component by the substation gateway.
Communication System and Data Delivery
Data delivery latency and loss rate are important factors in the performance of wide area control and protection applications, but the data delivery infrastructure that will ultimately support those applications is still evolving. GridSim’s data delivery component, GridStat, is a publish-subscribe middleware framework that has influenced the NASPInet effort led by NERC and the U.S. Department of Energy (DOE). Its design centers on the fact that sensor measurements are digitally represented as a periodic stream of data points. Working from this data model, GridStat was designed to allow for efficient, wide area, encrypted multicast delivery of data. GridStat as a component of GridSim is a realistic model for emerging power system data delivery services and at the same time provides great flexibility for configuring and evaluating potential wide area control and protection applications.
GridStat is designed to meet the requirements of emerging control and protection applications that require data delivery latencies on the order of 10–20 ms over hundreds of miles with extremely high availability. The GridStat architecture consists of two communication planes: the data plane and the management plane (see Figure 3). The data plane is a collection of forwarding engines (FEs) designed to quickly route received messages on to the next FE or termination point. The FEs are entirely dedicated to delivering messages from publishers to subscribers. Routing configuration information is delivered to the FEs from the management plane. The forwarding latency through an FE implemented in software is on the order of 100 µs, and with network processor hardware it is less than 10 µs. We believe that the performance of a custom hardware implementation of an FE could match or exceed that of a general-purpose Internet router. Thus, in a typical wide area configuration, GridStat would not add more than 1 ms over the speed of the underlying network while providing quality-of-service (QoS) guarantees tailored to rate-based control and protection applications.
The management plane is a set of controllers, called QoS brokers, that manage the FEs of the data plane. The QoS brokers are organized in a hierarchy to reflect the natural hierarchy in power grids. When a subscriber wishes to receive data from a publisher, it communicates with a QoS broker that designs a route for the data and delivers the routing information to the relevant FEs, creating the subscription. Since path computations are done out of band from data delivery, even heavy loads of new subscription creation do not adversely affect the performance of the data plane. Beyond this, QoS brokers have a privileged view of routing performance and the router graph that allows them to create optimal delivery paths. QoS brokers also implement policies for resource usage, cybersecurity, aggregation, and adaptation.
Because the entire purpose of GridStat is the efficient delivery of data, it includes features providing configurable QoS per subscription while attempting to minimize data delivery costs. A subscriber can request quality-oriented parameters such as data delivery rate, temporal redundancy of data packets, and spatial redundancy of data streams (delivery over multiple independent delivery paths, each of which meets the end-to-end delay requirements). The QoS brokers ensure that each subscriber gets the resources it needs while preserving the needs of existing subscriptions. To conserve network resources, the management plane identifies any shared data paths between a publisher and two or more subscribers. If there is any overlap in these paths, the management plane ensures that data are only sent once for that leg of the journey before being duplicated at the split.
GridStat supports multicast delivery of a given sensor update stream whereby different subscribers can subscribe to different rates yet no update message is ever sent over a network link more than once and it is not forwarded on a link at all if not needed. FEs implement this via a mechanism called rate filtering: only forwarding an update on an outgoing link at the highest rate that any subscriber downstream via that link requires. Some kinds of data place additional restrictions on the rate filtering. GridStat’s rate-filtering algorithms are coordinated across multiple PMU streams in order to ensure that subscribers receive sets of updates from different PMUs taken at the same instant. For example, consider PMUs that send updates at a rate of 120 Hz. While such a high rate would be useful for a few application programs, many applications would not need such frequent updates. For an application subscribing to two different PMU streams at a rate of 20 Hz, five-sixths of the updates will be dropped before reaching it. But GridStat ensures that the same one-sixth of the updates are delivered from the two PMUs, so they can be used as a global snapshot. This synchronized rate filtering is set up when subscriptions are being added and is based on time stamps in the updates, so it does not require any inter-FE coordination when updates are being delivered. So scalability is not harmed by this strong delivery property.
When used as the data delivery layer component of GridSim, GridStat allows for virtual substations to be created or reconfigured and additional subscribers and power applications to be added with minimal changes. This contrasts starkly with the current situation in the power grid, where even minimal changes to the number of sources or consumers of data can require the data delivery system to be completely re-architected. Conversely, GridSim also allows for potential deployments of GridStat to be tested with real-world volumes of data and with different network and power system topologies.
Control Center Applications
Continuing the theme of using existing artifacts as components of the GridSim environment, we now describe two control center applications that have been incorporated into GridSim thus far.
One of the main objectives of GridSim is to allow experimentation with and testing of wide area control and protection applications using PMU and other high-rate, time-stamped data streams. Thus far, two prototype applications have been included in GridSim: a linear, hierarchical state estimator and an oscillation monitoring system.
Both applications were built using components of the Grid Protection Alliance’s openPDC product. Thus, one benefit of incorporating these applications in GridSim is that other openPDC-based applications can easily be brought into the GridSim environment. The openPDC application set is an open-source software system that collects PMU measurements from multiple sources, aligns them according to their time stamps, and processes them with user-defined functions. The openPDC applications also provide numerous advanced functions, such as cybersecurity and device management, that are necessary for industry use. Thus far, however, GridSim uses only the C37.118 protocol parser and the time-alignment functionality.
The openPDC applications contain three kinds of adapters: input adapters, action adapters, and output adapters. GridSim’s applications, however, use only two of these. Input adapters read data and parse them. Although the openPDC applications provide many built-in input adapters that can read data from files, databases, or the network, none of them supports the publish-subscribe communication pattern used in GridSim. New input adapters were therefore developed supporting the GridStat publish-subscribe system. Action adapters receive time-aligned measurements and process them. In GridSim, all of the power system calculations, including substation-level and control center–level state estimation as well as oscillation detection, are implemented using custom action adapters. These new functions embedded in the openPDC applications are not only useful in the simulation environment but can also be run in the real industry environment.
Since the openPDC applications were primarily designed and implemented for field usage, which has different technical requirements from GridSim, work was performed to adapt them for the simulation environment. For example, the openPDC applications provide a user interface for configuring devices, phasors, and measurements. Since GridSim is intended to simulate a variety of systems that may change frequently, manual configuration is too cumbersome and error-prone. A program was therefore created to read the power flow file for TSAT and configure the whole system automatically, saving a lot of effort and simplifying the integration of the openPDC and simulation software.
The Oscillation Monitoring System
The oscillation monitoring system (OMS) application has been developed at Washington State University for real-time monitoring of problematic electromechanical oscillations using wide area PMU measurements. OMS combines advanced signal-processing algorithms with heuristic expert system rules to automatically extract the damping ratio, frequency, and mode shape of poorly damped electromechanical oscillations in a power system from power system measurements. A prototype OMS has been implemented as part of the phasor data concentrator at Tennessee Valley Authority (TVA) since 2007. It is also currently being implemented at Entergy in conjunction with a smart grid investment grant project.
In our GridSim project, the OMS is being used as a real-time application example, both serving to illuminate what GridSim must provide in order to incorporate actual applications and demonstrating how executing an application with simulated real-time test data can help validate the application. The OMS engines are integrated into an action adapter module of the openPDC applications. Thus, the OMS receives real-time simulated PMU data streams from TSAT, via the measurement generator and the data delivery system, which are buffered onto the internal signal-processing engines of the OMS. Results from the OMS can be exported to a custom SQL database that can be visualized and set to trigger alerts or alarms whenever damping levels of oscillatory modes fall below prespecified thresholds. The operator can then take manual action to bring the damping back to acceptable levels.
Unlike the real power system, where the actual modal characteristics of the system are unknown values, the modal properties of the test system in TSAT can be accurately determined from model-based small-signal stability analysis. Comparing the outputs of the OMS engines with the respective model-based modal values is useful for testing and tuning the OMS engines for target power systems. Since GridSim includes communication models, such studies also reveal the effects of communication delays, the loss of PMU channels, and network congestion on the resulting OMS modal estimates. We plan to use GridSim to test automatic control action by the OMS, although such closed-loop feedback will require further modification of TSAT.
The OMS includes two engines, as shown in the flow chart in Figure 4. The event analysis engine, shown on the right side of the flow chart, carries out an expert system–based Prony-type ringdown analysis of system responses following disturbances in the system. The objective for this engine is fast detection of sudden changes in the damping of oscillatory modes from large disturbances in a power system, so that mitigating control actions can be initiated before the damping problems degenerate into widespread blackouts. Typical analysis uses 5–10 s of PMU data at a time, and the calculations are repeated over moving time windows and over different PMU signal groups to ensure the consistency of results. The event monitor engine can typically detect oscillatory problems by using 10–15 s of PMU data, starting from the instant the oscillations begin to appear in a power system.
The complementary damping monitor engine, shown on the left side of the flow chart, estimates the damping, frequency, and mode shape of poorly damped oscillatory modes from ambient PMU measurements. Unlike the event monitor engine, which only works when the system is subject to disturbances, the damping monitor engine is applicable all the time. By using natural power system responses to routine random fluctuations from load variations and generation changes, the damping monitor engine continuously tracks damping levels and mode shapes of poorly damped oscillatory modes. The damping monitor engine uses an extension of a frequency-domain algorithm called frequency domain decomposition (FDD). This engine is aimed at preventive detection of poorly damped oscillations. The damping monitor engine uses about four minutes’ worth of PMU data in every computational run. As with the event monitor engine, the analysis is then repeated over moving time windows and over different signal groups to verify the consistency of modal analysis results.
Figure 5 shows the results from the two engines for a recent event near a major generating plant. The system encountered a routine event at about 830 seconds. The event analysis engine of the OMS carried out moving time-window analysis of the PMU measurements using real-time Prony analysis and concluded at 838 seconds (the vertical dotted line in Figure 5) that the oscillation was from a local 1.2-Hz mode (i.e., one involving mainly one PMU or a few nearby PMUs) with a damping ratio of +1.5%. Subsequently, the damping monitor engine analyzed the real-time ambient PMU data and estimated the dominant oscillatory mode to be the same local mode at 1.2 Hz, with a damping ratio of +1.8%. Thus the results of ringdown analysis and ambient noise analysis match well for this example. The two engines serve as complementary techniques for identifying the dominant poorly damped oscillatory modes of a power system whenever such modes exist.
A two-level linear state estimator has been developed at Washington State University that is an excellent candidate application for testing in the GridSim environment. It is based on PMU data and requires algorithmic processing at the substation level, fast communication of the substation results to the control center, and synchronization of the data at the control center before it finally calculates a state estimate (SE) for the whole system. The power system simulation produces PMU measurements 30 times per second, and the final SE is also calculated at the same rate. Thus errors in the simulation, communication, synchronization, and SE calculation can all be checked during the testing of this application on GridSim.
The processing of this two-level SE is shown in Figure 6 for both the substation level and the control center level. At each substation, the local PMU data are processed using linear estimation algorithms for both current and voltage phasor measurements. This processing has the advantage of estimating and eliminating errors from noise, bad analog data, and bad circuit breaker status data on a small set of measurements. The topology, current, and voltage estimates from each substation are then sent through the communication network to the control center. At the control center, the data are synchronized for the same time stamp, and the whole system states are linearly estimated.
Figure 7 provides some results for this test as carried out on GridSim for an 11-substation power system. For a small system like this, the simulation and communication speeds were not a problem, so the test’s purpose was mainly to check the computation processes and data delivery. When the SE was running perfectly, the figure shows that the bus voltages (a) calculated by the TSAT simulation, (b) generated by the PMU data generator, (c) estimated at the substation level, and (d) estimated at the control center all compare quite well 30 times a second for about eight seconds after a fault on the system. Many things can go wrong, however, as demonstrated in Figure 8 by introducing some jitter in the data delivery between the substation and the communication level, thus producing erroneous SE results at the control center.
A fast communication and computation system overlaying the power grid is a key enabler for applications taking advantage of PMUs and FACTS controllers to achieve the smart grid of the future. The tools needed to develop and test these new applications do not exist today, however. We have described such a tool—a simulation platform called GridSim—that can be used to develop and test wide area control and protection schemes.
We have developed this platform to simulate the power grid in real time for electromechanical dynamics and to generate and stream PMU data in standard format. It also includes the ability to deliver measurements and processed data over a high-bandwidth networked communication system called GridStat. Finally, we have used GridSim to simulate and test two new applications—oscillation monitoring and linear state estimation—that are quite different from each other but both utilize PMU streaming data in real time. We show that platforms such as GridSim can successfully and rapidly prototype new “smart” applications.
We should note that closed-loop control is not illustrated in this article. Both the OSM and the linear state estimator are real-time but open-loop applications, which means that the outputs are used by the operator to initiate manual control if necessary. Closed-loop control will be incorporated into GridSim, and the significant changes needed in the power system simulator to accomplish this are being developed.
We gratefully acknowledge the assistance of Powertech Labs and the Grid Protection Alliance in adapting their TSAT and openPDC products, respectively, for use in GridSim. This research was supported by a grant from the U.S. Department of Energy (Award #DE-OE0000032).
For Further Reading
Power Tech Labs. TSAT—Transient Security Assessment Tool. 2011.
D. Bakken, A. Bose, C. Hauser, D. Whitehead, and G. Zweigle, “Smart generation and transmission with coherent, real-time data,” Proc. IEEE (Special Issue on Smart Grids), vol. 99, no. 6, pp. 928–951, June 2011.
Grid Protection Alliance. The Open Source Phasor Data Concentrator. 2011.
G. Liu, V. M. Venkatasubramanian, and J. R. Carroll, “Oscillation monitoring system using synchrophasors,” in Proc. IEEE PES General Meeting, Calgary, Canada, July 2009, pp. 1–4.
T. Yang, H. B. Sun, and A. Bose, “Transition to a two-level linear state estimator, part I: Architecture, part II: Algorithm,” IEEE Trans. Power Syst., vol. 26, no. 1, pp. 46–62, Feb. 2011.
David Anderson is with Washington State University.
Chuanlin Zhao is with Washington State University.
Carl H. Hauser is with Washington State University.
Vaithianathan Venkatasubramanian is with Washington State University.
David E. Bakken is with Washington State University.
Anjan Bose is with Washington State University.