In the fall of 2001, Georgia System Operations Corp. (GSOC; Atlanta, Georgia, U.S.) faced a major business and technical challenge. GSOC's business environment was changing rapidly with the need to receive, process and deliver more information more quickly and from more parties than ever before. This required a strategic shift in business rules, business processes, technical strategy, skill set development and technical tools for GSOC to meet the challenge.
Two distinct business challenges required GSOC to re-evaluate traditional application development methodologies, architecture, software tools and implementation strategies. Due to the complex, critical nature of the two software applications described below, and the absolute need for the applications to be delivered on schedule, GSOC needed a more reliable system.
Electronic Energy Tagging
The external challenge was generated by the industry's need to processes electronic tags (E-Tags) more accurately and at a higher volume. Electronic energy tags reserve transmissions and document the energy transactions between buyers, sellers, control areas and transmission providers in the North American energy market. The North American Electric Reliability Council (NERC) issued an updated specification (1.7) for the electronic tag exchange. The new specification includes new constructs such as a new “market path” and implementation changes on how to handle adjustments to tags. The actual format of the data exchange changed from a comma separated variable (CSV) format to a full Simple Object Access Protocol (SOAP) using eXtensible Markup Language (XML).
Generation Resource Scheduling
The internal challenge was to provide the 39 Electric Membership Cooperative's (EMC) with the maximum flexibility to optimize their generation resources and allow the members to take advantage of changing market conditions. EMCs had different business needs based on their share of generation ownership, their risk profile and business strategy. Out of the 39 EMCs GSOC serves, four groups emerged with similar business needs. These groups all wanted to make their own choice of marketing agents and each selected a different party for handling their market transactions. In addition, the cooperatives still wanted to have the common benefit of the combined efficiency of operating the physical generation assets as a whole. The needs of the EMC group led to an application that would allow each of the four marketers to execute buy or sell transactions on behalf of their group and to submit psuedo schedules to GSOC for the remaining generation available. This model enables GSOC to know the total commitments (load, sales and reserves) and the total energy available (generation committed, purchases) to run the physical generation required. To prevent an imbalance in the total, each marketer was required to submit enough purchases and committed generation to meet the load responsibility of the EMCs in the group.
In analyzing the two-part problem, the GSOC Engineering and Energy Control Systems (ECS) departments realized that the total solution to both of these problems could not be purchased from outside due to the integration work that had to occur with other systems within the operation technical environment. The second conclusion was that each of the solutions would result in a software application that would be more complex than anything implemented before at GSOC and would require additional resources to implement in the schedule allowed.
E-Tag 1.7 Analysis
To get external input, GSOC brought in a software architecture firm to perform an assessment of the ETAG 1.7 challenge, GSOC skill sets, technical infrastructure and provide a recommended solution path. That engagement verified that the new E-Tag implementation using XML was not just a format change but an actual SOAP protocol implementation using XML as the data transfer format.
Given the short time frame, the consultant recommended GSOC purchase a partial solution from GSOC's E-Tag vendor, Open Access Technology International (OATI; Minneapolis, Minnesota, U.S.), and implement the new format. This would allow GSOC to focus on handling the filtering, presentation and system integration issues and not the transfer mechanism. The consultant also suggested GSOC consider a more robust architecture to implement the application and integration components, recommending the Java 2 Enterprise Environment (J2EE). J2EE is also a preferred architecture for other enterprise applications.
The new demands on generation resource scheduling burdened the installed system. The technology of the existing application was evaluated and the previous reliability reviewed. GSOC decided using the existing tools and infrastructure was too risky because of speed and reliability concerns. The application logic in stored procedures made the logic difficult to maintain and reuse, and it was difficult to modify the previous system architecture to provide high availability.
GSOC decided to choose J2EE as the architecture for application development and system integration, to begin to train and mentor the staff on the Java Language and to implement both applications with a single cost effective solution.
GSOC selected ClearNova Inc. (Roswell, Georgia) to act as technical project managers, software architects and mentors to the existing GSOC staff for the project. The time for implementation was short, so it was critical to get the programming resources started on implementing business logic as soon as possible. To avoid recreating J2EE software infrastructure, GSOC chose ClearNova's J2EE application development framework “ThinkCAP” to provide the common application components, including user authentication, user interface layout manager, content management, connection pooling, and relational to object mapping. It also provided a development environment and programming model for developing the business components.
GSOC needed a high-availability J2EE server and network configuration to implement the software development. These requirements led to a preferred architecture development (J2EE and ThinkCAP) and programming model (object-oriented Java using a model, view, controller programming model) that yielded a common application infrastructure able to host multiple mission critical applications.
GSOC took the following measures to put the applications in service.
Infrastructure and Training
The J2EE server infrastructure was implemented with redundant application servers running a commercial Java application server, redundant Apache Web servers located on a corporate DMZ network segment and a single Sybase RDBMS server with raid array. GSOC chose a Linux operating system for the Web servers and application servers, and purchased them pre-installed from Dell Computers Inc. (Round Rock, Texas, U.S.). The platform for the Sybase database was an IBM P Series RISC machine with an AIX operating system. Contributing to the high-availability environment were four layer-seven “application switches” manufactured by Radware Corp. The application switches provide load balancing and redirect the user session in the case an application or Web server fails. Once the applications are online, the configuration allows the system to be maintained with almost zero downtime by directing traffic to another application server while the other receives updates.
A ClearNova instructor introduced the GSOC programming staff to the Java programming language and trained them in the ThinkCAP development tool in a two-week period. This was done in parallel with GSOC and ClearNova staff installing the infrastructure and completing the high-level software design.
E-Tag 1.7 Implementation
The E-Tag 1.7 application implementation began with the purchase of the WebData database application. The WebData product from OATI solved the SOAP protocol problem by installing a server that handled the E-Tag/SOAP protocol and populated the E-Tag information into GSOC's SQL database. GSOC manages a mixture of firm contacts, purchased generation, and firm and non-firm daily purchases and sales that can generate hundreds of E-Tags an hour. To help manage the volume of E-Tags, the application had to verify the tags and check for syntax and completeness. The backend accounting systems also required accurate tagging information to resolve exception processing by the staff. The generation control system needed to have the net purchase and sale data calculated and sent as one component to drive the physical generation to the desired target.
GSOC decided not to use the WebData database stored procedures but to scan the database for changes and build similar tables in the GSOC database. This gave GSOC the flexibility to manage and maintain the data to meet its specific needs.
An unforeseen challenge was a parallel development effort by E-Tag vendors addressing the distribution of tags to meet the changing specification. As a result, good test data for verifying GSOC's application and integration was not available until a few weeks before the final cutover date.
The GSOC Operations Engineering department managed the E-Tag project with resources from the ECS department and software architects from ClearNova. The implementation was finished in time for the industry cutover on April 10, 2002, and consisted of a browser-based user interface to display the tags and allow filtering and reporting of tags. The resulting purchase and sales data was transferred to GSOC ECS, and the results of all tags and approvals were stored in GSOC's E-Tag database. The resulting application allowed the GSOC dispatchers to go into the 2002 summer operation with a tool that improved the quality of the operational data by presenting tags in a manner that allowed efficient processing and few errors.
Scheduling Member Implementation
The generation scheduling application was an enormous and challenging project called SchedMem. The new business rules were not finalized until early February 2002, and an implementation was required by contract to begin June 1, 2002. This project required communicating the physical constraints of each generating resource every two minutes to four marketing entities via a user interface and an FTP feed; processing schedules received via the user interface or from FTP; validating the schedules against a complex set of rules; accepting or rejecting the schedules to all parties; and presenting the amount of available generation to the system operators.
One marketing entity was more difficult because dynamic scheduling was implemented between GSOC and the marketing agent's control area allowing the marketer to respond to variations within the hour. Other complications involved accounting for the scheduling of a pumped hydro unit owned by the EMCs by tracking the pond water availability. The extensive changes and additions to the SchedMem application required the project team to create 85 user-interface screens and 165 Java components. In full operation, the database was processing 3 to 4 million transactions per hour. The system had internal users, external marketers and EMC members using the system as a true extranet application.
Benefits Gained/Lessons Learned/Next Steps
The success of the two software projects was largely due to the early recognition that the projects could not be completed using only in-house resources. The initial infusion of new tools and a new architecture were risky, but selecting outside expertise in the chosen technology to guide the implementation mitigated the risk.
The J2EE architecture is complex but excellent for building reliable enterprise-wide solutions. The architecture provides the flexibility to implement specific, highly scalable solutions. The complexities of the J2EE architecture were minimized by the use of a commercially available framework that allowed the business problems to be addressed quickly and leverage standard pre-built components, saving valuable implementation time. The ThinkCAP J2EE Development framework saved approximately six man-months of labor over a complete in-house development effort, easily paying for the cost of the framework on the first project. In addition, ThinkCAP saved several months of schedule time and reduced project risk by using proven software components.
The “build part, buy part” implementation strategy yielded a solid solution built on a solid software architecture using a consistent programming model. It also allowed the business specific logic to be custom tailored while maintaining schedule.
The system architecture and infrastructure for mission-critical applications requires close coordination between the software architect, network administrator and database analyst. It is critical to get the development, test and production infrastructure work done as soon as possible to avoid risk and to make the development team as productive as possible. Another lesson learned was that the correct development tools and development methodology are key to team productivity and team coordination. The proper tools are inexpensive compared to the labor that can be lost by an ill-coordinated or unproductive team.
With the right tools and the right guidance, an organization can retool an application development environment while still delivering critical applications on schedule. GSOC's real-time operations are now enabled with the tools, infrastructure and methodologies to adapt business processes to industry changes more quickly in a consistent supportable manner.
W. Neil Phinney is the manager of Energy Control Systems with Georgia System Operations Corp. (Atlanta, Georgia, U.S.). He has 32 years of diverse utility experience in generation, transmission and distribution with investor-owned, municipal and cooperative utilities. He has the BSEE and MBA degrees and is a registered professional engineer in the state of Florida.