A Practical Guide to Federal Service Oriented ArchitectureA Practical Guide to Federal Service Oriented ArchitecturePGFSOA_02a9f7ab-5303-4ca8-845a-77d7ae083730Based on the information in this document, chief architects and CIOs can become leaders capable of developing and effectively explaining the SOA business case for business stakeholders to inform and guide them in understanding and supporting the investment philosophy they are being asked to champion. Leaders should be able to develop plans for implementing and exploiting a successful roadmap for SOA adoption, and leverage and shape ongoing development, modernization, and enhancement (DME) activities within their agency. In addition, individuals should be able to leverage, support, and use cross-boundary initiatives working with their peers in other agencies, in local, state, and tribal governments, as well as private sector partners._ceb33a8f-4cce-4183-89e8-c5851de89a86To describe a target federal service oriented architecture vision and to provide guidance in the management and governance of enterprise-wide services ... to help federal chief architects and chief information officers in their efforts to adopt SOA best practices to further their organizations’ mission outcomes, meet increasingly demanding compliance requirements, and optimize their IT architectures_d0d23c5f-69c7-4c85-a4d0-fde9f2a56ddeImproved government responsivenessBy employing services to establish a flexible architecture centered on business and technology capabilities, the impact of change can be isolated and business processes can be more easily and rapidly modified to meet business and mission performance requirements.Simplified delivery of enhanced government servicesSOA and the “service” business model enable collaboration by simplifying access to services and streamlined value chains across organizational boundaries.More efficient governmentSOA facilitates mutually leveraged public and private sector investment; reuse of capability; elimination of undesirable redundancies; and a more focused model for on-going IT recapitalization.Information sharingSOA provides an effective and efficient approach to implementing reusable data exchanges - taking logical interoperability coming from multiple data modeling activities and rapidly evolving it into physical interoperability.Transparency, security, and resilienceSOA is predicated on a shared, standards-based infrastructure. This will enable consolidation, simplification, and optimization of IT Infrastructure, which in turn will enable greater transparency and audit-ability, as well as improved continuity of operations.EnterpriseImplement the Service-Oriented Enterprise_7c6644bd-6da0-4621-b6ca-de0be270492d4.1Establishing the service-oriented enterprise is perhaps the most difficult aspect of SOA because it challenges many of the conventional business practices employed today. This section highlights key enterprise changes necessary to enable the benefits from SOA. Recommendations for Service Oriented Enterprise: • Treat SOA as a change initiative. It must have strong executive buy-in, adequate resources, organizational visibility and sustained support. • Create a program plan with goals and objectives, and objective performance measures for the SOA initiative. • Establish a Center of Excellence (COE) to guide and manage the SOA initiative. • Adequately fund the COE and the SOA initiative. • Develop and sustain appropriate and viable Communities of Interest (COI) to help facilitate the development and use of shared standards, platforms and semantics. • Establish formal funding mechanisms to support the creation and delivery of services, coupled with appropriate charging mechanisms that are tied to service usage. • Establish a Federated Governance structure that includes a charter defining organizational structure and relationships, scope of responsibility, rules of behavior, conflict resolution processes and the authority and structure of the COE. • Adopt a twin-track SDLC that facilitates the incremental, innovative refresh of the IT assets on an on-going basis. • Create and use a services development, test and evaluation laboratory to meet enterprise requirements. • Adopt procurement policies and processes that encourage vendor competition around service models.Organizational ChangeTreat SOA Adoption as an Organizational Change Initiative_7adffbba-92d8-4ec7-ae8c-1f72c869da404.1.1The experience of the past few decades has shown that organizational change is difficult. For it to succeed, several critical factors must be in place: • There must be a compelling reason to adopt the change. That reason must be effectively articulated within the organization; • The executive leadership of the organization must be solidly behind the change initiative; • It must be treated seriously – create a program to oversee and manage its rollout - stakeholder objections must be addressed, etc.; • For a cross-cutting initiative such as SOA, often a central coordinating group, or center of excellence (COE) is required; and • Adequate resources to support the initiative must be allocated to sustain the change.Executive SupportObtain Executive Support_727dd241-f137-4731-bf11-852a184031c34.1.2Significant change initiatives require strong support from executive leadership. This typically only happens when there is a compelling need to make the change and the senior management of the organization understands the need and drives the change. Many federal agencies are already at this point. Executive management understands the need for greater responsiveness – as delivered through increased agility – and is frustrated by the reliance on outdated, rigid software applications. SOA offers the opportunity to incrementally reengineer the agency to achieve the desired flexibility. Change initiatives succeed when organizational passions are re-channeled to achieve well articulated goals consistent with those passions. Our advice is to downplay the SOA hype and buzzwords, and concentrate on the targeted business outcomes. Therefore, the task for senior management is to carefully define and articulate the goals and establish associated progress metrics that the rest of the organization can rally around. More importantly, senior management must sustain this level of support for the duration necessary to support the change. The chief architect is responsible for demonstrating how to apply architectural best practices to achieve the desired business outcomes. Having a service oriented Target Architecture enables the chief architect to define and articulate individual business solutions that clearly address today’s business challenges and business goals, while also incrementally enabling a strategic architectural vision. Critical Success Factors: • Understand today’s business challenges. • Conceive a service-oriented Target Architecture Vision for the enterprise. • Effectively, but rapidly, model business objectives, processes, and information that support the Target Architecture. • Architect and articulate solutions that solve today’s business challenges, while at the same time advancing the Target Architecture vision. The use of a SOA Adoption Roadmap will allow informed investment decision-making so that funding to support an enterprise architecture enhanced by SOA is allocated according to a logical sequencing plan.Plan and MetricsEstablish a Program Plan for SOA and Measure Results_ec335e60-be4f-44b2-845f-3b4d20af0e9a4.1.3Meaningful change requires a well thought out approach. Our recommendation is to treat SOA as a “program” – not in the sense of another stovepipe, but rather as a serious cross-organizational initiative – with a plan, a champion or manager, and defined results that can be measured. Specifically, the natural place for executive sponsorship is shared between the CIO and the mission or business owner associated with a specific high(est) priority initiative with enterprise scope. The program manager or champion should be the chief architect or chief technology officer, depending on specifics of the individuals, the organization of the IT function, and the ongoing IT planning and architecture activities and functions that are to be leveraged. This last point is key; we are not recommending an additional freestanding planning and architecture activity, we are recommending leveraging existing activities – enhance and extend - and focusing them within the guidance of this document. The establishment of a “program” entity does not reduce the power of collaboration at the individual organizational level that SOA facilitates, but it does help to provide strategic direction and focus to the SOA effort. We feel that the establishment of a program structure from the top, as described above, must be combined with the bottom-up efforts that are naturally occurring within the organization to insure an effective implementation. The focus of the program should be on enterprise and segment architecture, identifying and prioritizing services, and supporting governance activities that channel demand to reusable services. The program should leverage and align existing activities such as enterprise architecture or technology test and evaluation laboratories. More importantly, the program should work with existing DME (development, modernization, enhancement) programs and projects for solution architecture and implementation activities. A measurement strategy is an important component of any change management initiative, but is particularly important to the SOA implementation. We recommend that agencies use a goal-based measurement process that focuses on finding meaningful measures for monitoring the progress of SOA initiatives. Generally, there are three tiers of performance analysis: project, program, and enterprise. The project objectives are to develop, integrate, and deploy useful services that solve real business problems. While developing these tiers, keep in mind the following guidance: • Project performance outcomes are the inputs to program performance analysis. • The program objectives are to enhance productivity and efficiency by evaluating, procuring, and managing services shared across projects. • The program performance outcomes are the enterprise performance analysis inputs. • The enterprise objectives are to enhance productivity and efficiency by evaluating, selecting, and controlling services shared across programs. Agencies are constantly required to make IT investments to field new and better operational capabilities. Accordingly, they need meaningful metrics and benchmarks that allow them to monitor progress and make adjustments to ensure desired outcomes are achieved. A dashboard is a useful mechanism for presenting stakeholders a number of key indicators chosen to link SOA-enabled business and technology initiatives with business/mission performance targets.COEEstablish an SOA Center of Excellence (COE) to Oversee Adoption_dec1151c-b292-49f4-aa2f-299432f0ddec4.1.4For cross-organizational initiatives, a best practice is to establish a coordinating committee, or center of excellence (COE) of knowledgeable individuals from across the organization and augmented with experienced SOA practitioners. The purpose of the COE, which may be established within an agency’s existing EA program/governance structure (see Federated Governance discussion, below), is to set direction, draft policies, develop/adopt methods, and oversee the development/execution of the service portfolio plan. An additional purpose is to elicit participation and buy-in from the organizational and stakeholder components. Across the Federal space, maturity with EA implies organizations have architecture review boards and subordinate architecture working groups. The recommendation is to use that framework where it exists and extend it explicitly to SOA and the related concepts in this document. Further, agencies should be looking to participate in the AIC Services subcommittee, other like cross Federal organizations and entities, and external standards bodies to insure Government-wide alignment and adoption of external best practices.Change InitiativeAppropriately Plan and Fund the Change Initiative_65ffcb27-2196-460a-81f5-ade70ae6deb84.1.5In order to initiate and sustain any change initiative, sufficient on-going planning, funding and resource allocation must occur. Another challenge for an SOA-based implementation is funding the necessary infrastructure, which can represent a large expense that is not easily associated with a specific program. Organizations need to identify a means to align continuing enterprise infrastructure funding with the needs of the SOA implementation. Given the scope and importance of the targeted initiative, it is reasonable to build these incremental costs into its business case. The amount of funding necessary depends on the maturity of the organization, its strategy for implementing SOA and its overall objectives. In addition, while highlighting the full cost via an integrated business case, it is important to capture the benefits as outlined elsewhere in this document. For more mature organizations, SOA reuse and agility result in cost savings. Organizations need to plan proactively to identify and harvest the savings – for reinvestment in additional SOA capabilities, to enable additional mission or business performance improvements, or to reallocate resources outside IT. A significant integration of SOA into an agency environment can take a long time. It may be years before the point of cost savings through reusable services can be reached. Once it is reached, savings can grow exponentially. Our recommendation is to highlight the full scope of costs and benefits by leveraging the agency EA through segment architectures. This approach will highlight overall opportunities for reuse, information sharing, reduced duplication, and overall cost effectiveness as well as clearly delineate the full spectrum of investments required and how they interrelate. It will produce a full and proactive picture of the federated governance structures required, and how they relate to existing governance bodies and processes. Perhaps most importantly, by leveraging natural partitions evident via the architecture, the implementation challenges can be decomposed into a smaller number of lower risk steps that form a natural roadmap to the target outcomes.Processes and PlatformsBuild Community Processes and Collaborative Platforms_655eb503-61ca-4a1b-822a-95521292c5fa4.1.6Many of the benefits of SOA are derived from sharing – sharing information, sharing business processes, sharing reference architectures, and sharing services. As a result, a mechanism must be put into place to enable the collaboration that is necessary to establish standards for sharing. One potentially effective mechanism for collaboration in the federal government is the Community of Interest (COI) model where participants from a particular domain (horizontal or vertical) come together with sufficient resources to establish the basis for sharing. The typical issues dealt with include semantics, standards for exchange, platforms, etc. To be successful, a COI must have a governance mechanism, sufficient funding, and portfolio management oversight, and must provide a collaborative environment for authorized users. The COI should also have defined operating processes, including a process for resolving conflicts and arriving at collaborative decisions. This process may include agreements based on consensus, a majority vote of the members, or other procedures that are agreed to by the members.GovernanceEstablish Federated Governance_f8e6ef25-032a-4037-b036-83c241f8f8664.1.7Effective governance recognizes that it is not just about control, policing, and enforcement functions – it is also about providing essential services. Likewise, governance has jurisdictional boundaries, both within programs, at the enterprise level, and beyond. For this document, we adopt the following definition of (IT) governance: “Specifying the decision rights and accountability framework to encourage desirable behavior in the use of IT.” [Weill, 2004] Agencies must clearly establish governance charters, statements of scope, and areas of responsibility for specific organizational elements within the governance structure. In Section 3, the need for SOA governance was established and specifically the need for federated SOA governance. What does a federated governance model look like and how is it different from a non-federated model? Weill and Ross [Weill, 2004, p.61] define federated IT governance as “…coordinated decision making involving both the center [central authority] and the business units,” suggesting a two tiered vertical model that shares power in some manner. However, this concise definition does not accurately convey the principles of federalism based on a multi-tiered environment (see Exhibit 3-5). Within the context of the federal Establish Effective governance recognizes that it is not just about control, policing, and enforcement functions – it is also about providing essential services. Likewise, governance has jurisdictional boundaries, both within programs, at the enterprise level, and beyond. For this document, we adopt the following definition of (IT) governance: “Specifying the decision rights and accountability framework to encourage desirable behavior in the use of IT.” [Weill, 2004] Agencies must clearly establish governance charters, statements of scope, and areas ofgovernment, federated governance deals with semi-autonomous, but interconnected, organizations (at multiple levels) coordinating their efforts through a centralized mechanism. To be effective, the central authority should have the capabilities to effectively carry out the responsibilities delegated to it by the federation. This includes overseeing the establishment of standards and resolving conflicts, and providing the necessary resources, including funding and staff, to effectively operate. While the members of the federation retain their individual program authorities, they give up some control to the centralized authority to create the shared value that each seeks through the federation. At this point, the discussion has focused on actor relationships or management of the SOA environment. What technical aspects of the SOA need to be governed in a federated environment? The provider/consumer model introduced in the beginning of this document provides insight. The provider’s lifecycle involves service development consisting of these primary phases: requirements, design, implement, publish, manage/service, and retire. The consumer’s lifecycle is concerned with the manage/service phase of the provider’s lifecycle and includes: discovery, binding, using, and ending with disassociation. Within these two lifecycles, specific aspects of the SOA lifecycle would concern governance with respect to two or more actors and across multiple tiers. These include, but are not limited to: • Service requirements to include performance metrics • Service design specifically with respect to standards (i.e., interoperability) • Service versioning • Service funding • Service stewardship • Service elevation to enterprise-wide services • Service compensation for elevated services • Service monitoring and diagnosis • Service registration • Service publishing • Service discovery • Service consumption • Service security (i.e., trust channel mechanisms). Also of concern is the governance of federal SOA infrastructure (i.e., SOI) where enterprise-services, at any level, may reside. These may initially be hosted by an agency specific COE, but will evolve to a federated COE due to the joint requirements of the participants. These include, but are not limited to: • Federated SOA funding • Federated SOA requirements • Federated SOA design • Federated service repository • Federated monitoring and diagnosis • Federated SOA security • Federated consumption. Federated governance includes the agreed upon incentives and rules of behavior among peers that enable collaboration to occur across related domains. We offer the following suggestions for creating and leveraging federated governance: • Turn enterprise business/operational service level objectives into measures of effectiveness and associated mission level agreements, business/mission level agreements, and/or service level agreements. Use these to provide incentives instead of “mandates” wherever possible. Rigorously enforce compliance with the agreements. • Adopt commercial open standards and provide implementation guidance in the form of well documented reference implementations of those standards. • Develop a funded forum for representatives of all stakeholders to weigh in on architecture, process, and requirement development and prioritization. • Establish agreements (MOUs) to define and enable value-based interaction among the participants in the community. The US DoD Cross-domain Information Exchange Framework (CIEF) is an example that can be employed to define the terms of the agreements. • Balance enterprise concerns with program level objectives. • Establish overall service operational scope objectives within and across enterprises and gain executive support. • Guarantee enterprise service levels to program adopters and indemnify risk associated with using services provided by others. • Employ enterprise architecture tools and artifacts to identify significant information exchanges across domains of interest. From a practical standpoint, it is recommended that existing governance structures be leveraged whenever possible. For example, on alternative might be to leverage the existing e-Gov and LOB initiative communities. In general, we only recommend establishing new governance organizations when there is currently no existing governing body to absorb the new responsibilities.COI FundingProvide Communities of Interest (COI) with Sufficient Funding_57c1b5d4-8b63-4cf8-8f86-6ba444d065ae4.1.8The most successful communities of interest are those with the delegated authority to accomplish change and the means by which to do so. In most cases, this means having access to sufficient funding to operate effectively. Having access to the necessary budget to implement the services, solutions, or other enablers can make all the difference. Different communities will require different COI models; however it is likely that different COI models will evolve for different patterns of collaboration. For now, we recommend that agencies examine successful COI models (such as the National Information Exchange Model – NIEM – and the Gov Benefits E-Gov initiative) and that principal stakeholders be required to invest commensurate with the expected value of their return. The investments should be used to design, specify, and develop shared services, create service certification capabilities, or to accomplish the objectives of the COI. Coupled with enforceable Service Level Agreements (SLA) that incorporate objective service delivery metrics, this will establish the service value exchange that has been lacking in previous cross-agency collaborative efforts. To facilitate this, it is recommended that organizations develop an enforceable SLA model. (Note - not all stakeholders are equal. For constituencies whose participation is valuable or even vital but their return from participation is negligible, the barriers for entry must be made as low as possible.)DT&E LaboratoryCreate a Services Development, Test, and Evaluation (DT&E) Laboratory_a070a05c-fb90-4ff1-ab45-3a95e8badfaf4.1.9An SOA laboratory for collaborative DT&E, i.e., a federal community SOA space, is a combined design time, build time, and run time environment with the following characteristics: • Community brokerage to reconcile requirements and expertise across organizations. • A collaborative, distributed, service-oriented build time development environment. • A secure, shared, service-oriented runtime test environment where prototype capability bundles can be adaptively verified and validated against common government requirements. • A legal and intellectual property rights (IPR) regime to support open technology development (OTD). Use this community SOA space as the natural place to satisfy enterprise requirements. In other words, the physical SOA framework needn’t be restricted to just providing transport capabilities. It can also address broad enterprise requirements like information assurance, interoperability, test and evaluation, and information value chain management as well. While testing service performance in a federated environment is challenging within the Federal government, the benefits of a common testing capability for services are substantial.Funding and ChargesEstablish Service Funding and Charging Mechanisms_dd39e730-c4eb-4d8f-ae24-db4abfce4c534.1.10Effective funding and charging mechanisms that provide incentives for appropriate behavior among service providers and consumers are critical success factors for SOA in the longer term. Funding mechanisms deal with the investment required to acquire or provision services. In general, there are four types of service funding mechanisms: project, reciprocal arrangements, consortium, and central/corporate. While most organizations begin with a 'project-based funding' of services, project level investment usually leads to every program developing or acquiring its own business and infrastructure services built to its own specifications; thus obviating most of the potential benefit at the enterprise or inter-enterprise level of SOA. This is an immature approach to implementing SOA and can lead to "service anarchy." Reciprocal arrangements between projects have the same issues, but allow for some sharing of services on a bilateral basis. The consortium funding model aligns well with the COI recommendations above and includes the following: • The investment is shared across the stakeholders of the COI; • The model is appropriate for the provisioning of “core business services” for specific domains; and • The model requires a very clear and articulate value proposition for all players, the involvement of motivated individuals, and extreme patience and discipline. The central or corporate funding model is most appropriate for infrastructure or “utility” level services in the layered services architecture. In this case, the central authority (such as the OCIO) funds the development of services on behalf of the entire organization or community. Charging mechanisms are a potentially viable funding model, but have yet to be proven in the federal arena. Charging mechanisms are methods to recoup the costs for shared services across multiple user communities. The three basic charging mechanisms are: no charge, flat fee and usage charges. Although not charging is often an initial strategy, particularly for centrally-provided utility services, over the long term, not charging does not provide the proper incentive to provision and upgrade such services. In the short term, however, not charging may be a desirable approach because fees act as a tax or disincentive for projects to consume shared services – the opposite of the desired behavior. Flat fees are easier to administer, but usage charges are the most equitable method of covering the cost of provisioning and maintaining services. Alternatives within the usage charge approach include volume discounts and charging early users and new adopters on a different basis. Creating enabling services at the enterprise level therefore implies the need for some explicit budgeting above the individual program level and/or clear incentives to programs to make it worth their while to federate. Our recommendation is to adopt the mechanisms most appropriate for the program or enterprise given its stage of SOA maturity. As organizations become more mature, a portfolio of sharable services makes the decision of program managers to consume services easier. Eventually, as more programs contribute to the service portfolio, the model becomes self-sustaining.SDLCUse a Service-Based SDLC with Incremental Development Practices_07f78a59-6fff-491f-a66a-fc71b4d5bea94.1.11The service-based solution model is characterized by two primary roles: provider and consumer. Because the lifecycles, skills, and responsibilities for these two roles are significantly different, SOA methodologies separate the solution development lifecycle (SDLC) into two tracks (known as “twin track” development): service provisioning and solution assembly. An important step for agencies to take is to revise their SDLC to incorporate twin track development and recognize the different life cycles for services and (composite) applications. SOA implementations are by design, continuously improving environments that deliberately feed on the innovation of the various participants. Therefore, SOA implementations should not be designed with a single deliverable and a firm deadline. That is, SOA should not be fielded as a traditional major acquisition program via a long waterfall process. Rather, it should be fielded as the service-oriented capability via the lifecycle maintenance model, i.e., as a continuous and innovative technology refresh. Lifecycle maintenance of a modern computer network requires incremental innovation on a time scale consistent with Moore’s law and aimed at specific operational issues and opportunities. Agencies should define their services environment as a major resource that is already in place and that requires continuous improvement. The task is to enhance business agility through incremental re-capitalization. Think of SOA lifecycle maintenance as an innovative and evolving technology refresh rather than as a major acquisition.ProcurementPractice Service-Based Procurement_965d7acb-76ec-457c-808d-efa2f16f8db14.1.12Service-based procurement is the logical successor to the concept of incremental development and a service-oriented SDLC. While the current acquisition environment (based on the Federal Acquisition Regulations (FAR)) does not explicitly describe a service-based approach to procuring IT assets, we believe that all of the necessary acquisition steps required can be undertaken within the FAR framework. For example, the FAR does encourage procuring managed services in general when it is in the government’s best interest. As discussed above, agencies should adopt policies that encourage vendor competition around service models. Accordingly, contract incentives and penalties should revolve around SLAs that address both the frequency of software capability refresh and associated target business outcomes. Service-based procurement has the following elements: Procurement strategy: Services form the basic building blocks for solutions and should be procured separately from the integration of the services into solutions. From the service portfolio plan, which identifies all of the required services within a particular domain or segment, services with a common purpose should be bundled into defined solutions that can be described within procurement packages. Because the IT industry is in the midst of a natural progression of increasing commoditization, this should be factored into the procurement strategy. For commoditized capabilities, service procurement should be largely focused on Commercial-Off-The-Shelf (COTS) products or third party services. Agencies should only invest to develop a specialized capability when commercial solutions are unable to meet current government requirements. Requirements statement: Agencies should develop specifications for services based on use cases, desired outcomes, performance criteria, etc. Where applicable, agencies should reuse specifications and performance outcomes from industry, other programs, and other agencies. Requirements should be factored to reflect service types that already exist in industry or government. Specifications should not be over-engineered. Agencies should use test driven approaches to specification – define the tests that the successful service must pass and require vendors to demonstrate that their services pass the tests. Let the potential service providers innovate and compete around your use cases. Requirements should also address critical service management criteria such as quality of service adjustment over time, managing granularity, SLA auditing and transaction control with events, warnings and alert notification capabilities, policy enforcement, advanced security and authentication management capabilities. Government Furnished Equipment (GFE): License agreements for acquired services should be defined to allow appropriate consumption of services across the federal government or the transfer of the underlying software component to other agencies acting as service providers. In this manner, services can be made available to programs/contracts as government furnished equipment. Reference architectures and implementations for domains/segments should be made publicly available to industry and incorporated into RFPs. SOA, within the context of FEA, can help the federal government identify the intended scope of use for products and services. Integration services: Contract with a service-oriented systems integrator to integrate the services into business processes and composite applications. The integrator must possess the necessary skills to work within the agency’s technical environment, including any middleware products, such as an enterprise service bus (ESB). Certification of services: Establish a capability to test and certify services procured against the specifications and requirements of your technical environment and desired business outcomes. Write these objective certification requirements into your RFPs, contract SLAs, and task orders. This can be accomplished through the Services Development, Test and Evaluation Laboratory discussed above. Services source selection board: Include operational customers and representatives from independent industry expert bodies in addition to the representatives from the COI. This representation should be agreed upon by the COI members. Wrap legacy application transactions: Agencies should contract with operations and maintenance contractors when they have determined requirements for the service and the legacy application is best able to meet the requirement for the relevant timeframe. Services management: Agencies should consider contracting to provide services management, including configuration management, over the growing portfolio of services. Service execution management should include a real time monitoring capability (e.g., dashboard) similar to those used by network operation centers. Advance institutional knowledge and capture best practices: While the draw of SOA as a software paradigm is undeniable, each agency will travel this path at a different speed. Some agencies will embrace the approach and move quickly down the path – encountering challenges and overcoming obstacles earlier than others will. The spirit of SOA is sharing – not just sharing services: agencies should devote the effort to document their lessons learned, abstract from implementations to reference architectures, document discoveries and patterns, and capture best practices (or at least practices that have worked). Best practices are a set of actions that solve a problem critical to business success most often found within organizations that excel against business and mission objectives. The capture of a “best practice” is traditionally coupled with measuring via benchmarking. Benchmarking gauges performance against leaders and seeks to find and describe practices that most heavily contribute. Christopher Alexander, who originated the notion of patterns in the field of building (i.e., brick and mortar) architecture described patterns as a recurring solution to a common problem in a given context and system of forces [Alexander 1979]. By advocating and supporting the collection and dissemination of best practices and patterns, federal executives, architects, and developers can reduce the uncertainty and risk in determining when, where, and how to apply SOA. The Federal CIO Council and GSA have already created a vehicle for sharing best practices, services, architectures, templates and many other artifacts that can be used to give organizations a “jump start” in SOA. The Core.gov website is a collaboration environment designed to facilitate sharing and communication. Agencies should look to leverage this resource and in the spirit of sharing should contribute to it as well.ArchitectureImplementing a Service Oriented Architecture_999c884c-febc-45c8-a298-923533835c054.2Enterprise architecture has evolved as a discipline to take a view across programs and organizations – both horizontally and vertically, and is maturing rapidly in most federal government organizations. As a result, federal agencies have new insight into value chain inefficiencies and undesirable redundancies. At the same time, the Federal Enterprise Architecture is enabling government-wide consolidation and improvements, catalogued each year in the Federal Transformation Framework. The result is a need to interoperate and share information on a broader scope at every level of the Federal Government. To achieve the benefits of service orientation, they must be purposefully designed into the enterprise target architecture. Depending on the maturity of the organization the enterprise architecture may still be evolving, but architectures at the business unit or line of business level may be in place and more mature. Regardless, it is important to extract a working service portfolio from the target architecture(s) in order to link services effectively to business requirements. Ultimately, the most effective target service portfolio plan will be derived from enterprise business models that are developed based on real business activities. The service portfolio plan identifies the enduring collection of services required to support the automation of your business/mission and, through this automation, ultimately improve business outcomes. Once established, this enterprise service portfolio plan becomes the lynch pin of your SOA. The target service portfolio plan will: • Decouple business operations from the underlying current technology architecture. • Provide a holistic vision that can serve as the common context for all proposed IT solutions. • Enable restructuring of your IT portfolio to include both solutions and services. • Enable improved procurement practices and governance of outsourced IT capabilities. • Drive the restructuring of your legacy portfolio to reduce the cost and risk associated with strategic undertakings such as outsourcing. Recommendations for Service Oriented Architecture: • Develop, in collaboration with business units and IT, business models that help to align the EA with business objectives. • Develop an EA Target Architecture that is service based and focused on business priorities. • Based on the service-based Target Architecture, develop investment portfolios that are founded on services and solutions focused on fulfilling key business objectives. • Establish or adopt reference architectures and reference implementations that can be used by project implementation teams to jump start their SOA efforts. • Use the EA artifacts and the database of information developed during the EA process to mitigate the compliance burden placed on projects. • Assess all legacy assets in terms of their relationship to Target Architecture objectives, and factor this analysis into decisions for how to provision each service.Business ObjectivesUse EA and SOA to Align with Business Objectives_8eb1ae7c-1dac-4ea0-8c83-b225f276613d4.2.1Service-oriented architecture enables organizations to establish their context within complex value chains. Federal agencies should take an enterprise view of the value they provide to their constituencies – across programs or sub-organizations within their agencies, between agencies and often different levels of government, and at times between governments internationally. The provider/consumer model intrinsic to SOA allows each organization to focus on providing the services their constituents require while enabling them to request, with new clarity, the services they want. Critical Success Factors: • Business models are developed jointly by the business operators and IT, but are maintained by IT to insure that they are integrated within the overall enterprise architecture. • Up-to-date and accurate business models are maintained in your EA. Depth increases with maturity by incorporating detail required for each business solution. • Business models become the primary means for communicating changing business requirements.Enterprise ArchitectureIntroduce Services as a First-Order Concept in your EA_333460cf-ccda-4943-bd42-6926950ab4094.2.2As described in the previous section (the Target), the EA Target Architecture for federal agencies should be based on service-oriented concepts and built out based upon business priorities. EA should drive the development of the portfolio of services; and, portfolio management should be based on services and solutions rather than on applications. By reorienting portfolio management to focus on the services and solutions, agencies can establish the enterprise foundation for service orientation. An additional benefit from service based portfolio management is that the chief architect has a tool to rein in the multitude of disconnected SOA initiatives that have sprung up across many agencies. The service portfolio plan identifies the services needed to provide common capabilities across the agency and can be used to promote the collaboration that is necessary to implement enterprise services. The service portfolio plan also provides the basis for aligning these initiatives with the agency’s EA.Target ArchitectureEstablish a Service-Based Target Architecture_ebad8fe1-a770-488c-b30d-5d8b9274ff614.2.3The target architecture should incorporate a layered services architecture like the one presented in Exhibit 4-1 below. Layered models are well understood among architects. A layered service model is used to define and constrain the dependencies between services and to identify the set of policies that apply to each service layer. The figure below shows that if we zoom in on the Service Architecture, we should see a layered service model that allows us to allocate services to specific layers, depending on their characteristics and intended use. The layered model shown below accommodates the following layers: 1. Underlying Layer used to bring in resource APIs and provide access to legacy systems. 2. Utility Layer for highly reused services (this may include enterprise services provided by a parent service unit). 3. Core Business Services to transform and access business information. 4. Process Services to orchestrate an assembly of lower order services. 5. Solution Layer that includes composite applications.Models and PatternsAdopt Model-Based Architecture and Pattern-Based Design_d8f1fd1e-5016-4c98-ae8f-84974c2540ba4.2.4A best practice in the architecture discipline is to establish or adopt reference architectures and reference implementations that can be used by project implementation teams to jump start their efforts. Commonly used architecture patterns are a form of the reference architecture and reference implementation. • Pattern-based design incorporates both the concepts of reference architectures as well as model based approaches. Architecture patterns exist at the design, build and runtime levels of the architecture. • Model-based architecture (also referred to as model driven architecture, or MDA, by the Object Management Group) is an approach to bridging the divide between business requirements and technical solutions. Similar to moving down the rows of the Zachman Framework (The Zachman Institute for Framework Advancement (ZIFA), http://www.zifa.com), it involves a progressively more detailed and specific set of models. These models range from the most abstract depiction of the business, to a functionally complete, but platform- independent model, to a platform-specific specification from which code can be generated. The advantage of this approach is the linkage between the different levels and the automated mechanisms for moving from one level to the next. This provides traceability from the business requirements to the code and allows the impact of changes to be identified and managed. Combined with SOA which allows the impact of changes to be isolated to particular services, the model based approach holds great promise to enable flexibility in IT – the ability to quickly modify the code base necessary to implement new capabilities. Agencies should begin to adopt model based approaches to application development and integrate them into their service based SDLC. As they mature this capability, they should pursue pattern-based design techniques. However, the use of model based and pattern based design techniques should be closely linked to effective technical performance measures.Compliance and AlignmentEnable Automated Compliance and Alignment_5476df8e-ce72-4f19-afb6-2092e17272294.2.5Many EA efforts in the federal government are focused on compliance. It is not uncommon for organizations to resort to after-the-fact efforts to demonstrate compliance with the EA so that program funds will be approved. Enforcing the many compliance requirements that EA programs demand is a growing challenge; the default approach is to rely on teams of experts charged with sifting through inconsistent and often incorrect development lifecycle documentation of hundreds of projects. By adopting the SOA paradigm and the recommendations in this Guide, agencies should be able to use the EA to mitigate or even largely eliminate the compliance burden today’s EA programs place on projects. Since the move to SOA is an evolutionary sequence of steps, each program step will either take the agency closer to the target architecture or away from it. By placing more emphasis on architecting the solution and by employing model-driven architecture best practices in conjunction with the service portfolio planning technique discussed above, project teams can be provided with technologically compliant reference architectures that have service reuse requirements embedded in them. In this way, the EA becomes a benefit to programs rather than a burden. By using the portfolio approach described above, “core business services” can be developed for specific domains or verticals and “enterprise services” can be developed for common horizontal services. The Segment Architecture approach advocated by OMB is focused on this exact outcome of building out the EA incrementally in vertical and horizontal directions [OMB, 2007]. Developing services based on the service portfolio plan will ensure that the development efforts are automatically aligned with the EA and consistent with the principle of “architect, invest, implement” defined by OMB.” As a result, programs will be consumers and potentially providers of multiple services across the enterprise.Legacy AssetsLeverage Legacy Assets to Enable Evolutionary Progress_e048ba1e-640b-4b4a-aa22-01349a8ed7224.2.6One of the main advantages of adopting SOA as a modernization approach is the fact that it is an incremental or evolutionary approach, rather than a wholesale replacement approach. In the more mature states, an advantage of SOA is the ability to rapidly innovate, that is to experiment with and then adopt new business processes, much faster and more cheaply than with traditional IT systems. The challenge is how to reach that mature state in government agencies where legacy systems are prevalent and tightly coupled with business processes, data, and rules that support critical business and mission capabilities. Based on a services portfolio plan and with a clear understanding of your phased approach to achieving targeted incremental releases of capabilities, legacy assets must be assessed and factored into the analysis for service provisioning in each phase of the plan. In the early phases, there may be no choice: legacy transactions must be wrapped to expose the services. Over time, the four modernization alternatives—replace, consolidate, extend, and re-platform, should be considered for each legacy application. As an initial step, sharing can be achieved by wrapping legacy transactions with service interfaces, where practical. Then, examine service requirements over time to determine whether the wrapped transactions are able to meet future needs. Note that wrapped transactions are still subject to the rigidities of the underlying legacy application. Most applications have an inherent work flow definition; capture that first, then capture as many embedded business rules as possible. Many legacy systems do not have clear architectural delineation among data, business logic, and user interface. You may need to re-factor portions of applications to expose them as services. Business rules (logic) are typically embedded in the code and are not externalized to a separate component such as a business rules engine. Many use older generation database technologies such as hierarchical, networked, or file-based, where there is a lot of data redundancy, and little inherent documentation. Some legacy systems are too old and/or complex to be fully understood and therefore when the services are developed on top of such systems, rigidity, slow performance or maintenance problems may result. Legacy assets are key components of the existing enterprise architecture that can demonstrably add value to the enterprise’s strategic objectives. Consider the following advice. • Reuse existing and still relevant business rules. Use an analysis tool to better understand legacy systems and harvest these rules. Consider the several approaches to decoupling data from business logic and user interface in legacy systems: • For a large or complex application, use discovery tools that facilitate identification of business rules, business processes, and flows. Use tools to perform re-factoring of legacy code to decouple and minimize the risk of developing “rogue” services. • Use, if necessary, the alternative of custom adapters that interface with the legacy code. Use ESB adapters to access legacy code. • As a last resort, consider using standards such as Database Access Integration Services (DAIS) to access data in a database directly as a service [EPRI, 2001]. Top down and bottom up approaches should be used when producing services from legacy systems. Combine both approaches, balancing what the business needs with what the current IT organization is able to support. In the top-down approach, business processes and their flows drive what services are required that can be satisfied by legacy systems and how these services should be orchestrated. Allow business users or their counterparts in IT to drive this approach because of their familiarity with particular business domains. In the bottom-up approach, available services are identified by mining legacy applications and then determining what business processes are feasible to orchestrate. IT personnel should drive this approach. As services proliferate, ensure the utilization of adequate governance criteria in order to maximize the reuse of developed services without compromising security and efficiency. Applying and implementing governance from the beginning will pay rich dividends as the SOA efforts and consequently the service assets grow.InfrastructureImplement the Service Oriented Infrastructure_9f4a65bc-bd09-4535-bb6d-a23aedf048674.3The keys to implementing the Service Oriented Infrastructure focus on some of the most complex and demanding architectural concerns. Without the infrastructure that enables service-oriented architectures, the rest of these concerns are moot. It is critical to the growth of your SOA to understand the key infrastructure needs early in order to support the growth in services. Recommendations for Service Oriented Infrastructure: • Adopt a Federated approach to security and privacy for defined Communities of Interest (COI) that identifies common security and privacy solutions based on a risk/reward approach. • Determine the most effective approach to achieve semantic interoperability, either through processes like NIEM or through semantic technologies. • Incorporate run time and build time service management functionality to define, monitor, enforce, and adjust Service Level Agreements (SLA). • Utilize a Registry/Repository to discover needed services during application build and to insure that key criteria such as reliability, efficiency, dependency, and adherence to and/or violations of policies are fulfilled. • Adopt a trust model that addresses both security and privacy and also quality of service. • Establish a collaborative test/evaluation and certification/accreditation process for services. • Evaluate emerging technology against relevant test cases. Using better technology can lead to working smarter rather than just working harder. For example, Internet Business Logic is a kind of Wiki for applications written as business rules in open vocabulary, executable English. As befits a Wiki, shared use is free. It also works as an advanced SOA endpoint on the Web. See www.reengineeringllc.com.Security, Scalability, and InteroperabilityFocus on Enterprise Security, Scalability, and Interoperability_75c9ed84-3d50-4595-8b85-dc5b428fc60e4.3.1The flexibility of SOA comes with some costs in terms of complexity and the need to manage a greater number of moving parts. The issues are notionally similar to traditional environments. As a result, organizations need to pro-actively address the issues of security, scalability and interoperability in the design and implementation of their Service Oriented Infrastructure. Security: Due to the number of environments, domains, and platforms that potentially will be crossed in executing a business process based on SOA, a federated approach to security must be adopted. While work is on-going to produce government-wide security architectures and standards, defined communities of interest should perform pragmatic risk/reward analysis to define level of service requirements for common security issues. The fundamental security areas to address include the following: • Authentication and identity management across domains and environments • Authorization and confidentiality (access control) • Integrity (no inappropriate modifications are made) • Availability (reliable service, no denial of service) • Non-repudiation (positive identification and cannot deny providing or receiving services) • Audit and monitoring • Security administration and policy management Industry standards are under development and are rapidly improving, including WS-Security and Liberty Alliance specifications. However, while the standards available today are necessary, they are not yet sufficient for the most stringent government use cases. Scalability: With traditional applications, the number of users is typically known beforehand and the performance of the systems can be tuned to that user base. On the other hand, in an SOA environment, the number of users or consumers of the service is (almost intentionally) unknown. Loose coupling implies that the service is not aware of the number or types of ways in which it will be accessed. However, in reality, to be able to meet the terms of an SLA, the service provider must have some indication of the level of demand and develop mechanisms to scale up to accommodate spikes in demand. SOA Governance should provide this by having consumers register to consume a service from a provider. This allows providers to understand who its potential consumers are (and consumers to understand the SLA that the provider is offering) and to develop demand expectations prior to provisioning the service. Some operating environments (such as Java Enterprise Edition - JEE) have mechanisms built in to provide scalability; however, this must still be anticipated and provided for. Thus, as a best practice, service providers should have SLAs in place with potential users and use this information to design scalability into the service offering as appropriate. Interoperability: The concept of interoperability in an SOA enabled routable network is fundamentally different than in a traditional point to point information exchange architecture. In the latter, the job is to guarantee that one known system can synchronize with another. In the former, the job is to allow a virtually infinite number of unknown nodes supported by an unlimited number of known and unknown services to interoperate. Both traditionally and with SOA, use of standards across domains is a necessary, but not sufficient approach. Engineers also need pragmatic reference implementations. Reference implementations are examples of standard components bundled effectively to solve a critical problem. SOA engineers need these interoperable reference implementations for both semantic interoperability and run time infrastructure. There are two fundamentally different approaches to achieve semantic interoperability among disparate organizations, systems, domains, etc. The first approach is pre-instantiation – negotiating semantic consistency and then implementing it through common definitions (e.g., metadata schema). Achieving this commonality has proven extremely difficult. However, it has been approached by a few very focused COIs (e.g., the NIEM process between DOJ, DHS, and others) and has proven to be well worth the effort. Many efforts to negotiate semantic standards have fallen under their own weight, often as a result of participants resisting compromises for common definitions. The second approach to semantic interoperability is post- instantiation – using adaptors and translators to reconcile different data sources for common processing. Indeed, most middleware, including EAI (enterprise application integration), EII (enterprise information integration), and ESB (enterprise service bus) technologies contain capabilities to enable this. However the drawback is that in cases where the need for rigor is high, adapters are not capable of translating or aggregating disparate data sources. An emerging development in the second approach is the use of semantic technologies and ontologies to establish precise relationships among data that can be used with inference engines to uncover additional relationships and enable interoperability across domains. While these technologies are at an early stage, they appear to hold significant promise for the future. The increasing “openness”, granularity, and modularity in the service implementation infrastructure (e.g., Java Business Integration (JBI), or Service Component Architecture (SCA)) allows considerable cross enterprise interoperability at reasonable cost and time scales. Service Management: Service management becomes increasingly important as the number of services and collaborating organizations increases. Agencies should incorporate run time and build time service management functionality to define, monitor, enforce, and adjust the service level agreements (SLAs) between service providers and their consumers. The service management details (quality of service) should be rolled up to populate management dashboards for use at operational, tactical, and strategic levels in a manner similar to how network operation centers monitor network performance.Discovery and TrustEstablish Discovery and Trust Mechanisms_f1c307c6-67b4-4ad7-b182-cd9d039d1a774.3.2The SOI is responsible for enabling the operation and management of the service-oriented architecture, as well as providing the tools and environments to support the development, acquisition, and integration of services and service-based solutions. Discovery - Registries/Repositories: No SOA is complete without governance and visibility. Reuse of services, and for that matter, all assets is not possible without the ability to discover these assets as and when required. Registries and repositories are the safe houses where all assets can be located and managed. Providers use this capability to advertise their services and assets for others to use. Owners utilize the capabilities of the Registry/Repository to manage their assets through the various lifecycle stages from development through operations into retirement. Consumers use them to locate and identify assets that may meet their requirements. The current level of maturity in the industry regarding Service Discovery does not allow for run-time discovery and consumption. The current most effective discovery mechanisms use Registries/Repositories at build time to discover pre-existing services that can be consumed within a composite application. Upon evaluation and selection, the service is then designated using UDDI registries as the accepted service at run time. In evaluating a discovered service, additional criteria such as reliability, efficiency, dependency, adherence to and/or violations of policies, etc provide valuable information to aid in determining the suitability of a service. Registry/Repository provides the capability to store this type of metadata related to the assets (including services) and the relationships among them; thus forming the cornerstone for SOA Governance. Trust – Enterprise Security/Privacy and Level of Service: Discovery of valuable information or services across verticals is useless without a trust model that enables the discoverer to consume it with confidence; part of this relates to the need for enterprise security and privacy. There are various commercial security and privacy services on the market, but none has been shown to scale across a large federated enterprise. Further, none is certified and accredited for the most robust government applications. In fact, US Government policy insists that government network security services be developed and managed by the government. Any IT architecture must guarantee a level of service. Therefore, any SOA instantiation requires an ability to monitor and optimize quality of service with respect to discovery, security/privacy, and other vital functionality.Testing and CertificationEstablish an Adaptive and Collaborative Testing and Certification Environment_73a92524-cc6d-4fb6-a293-ddd7a424d88b4.3.3SOA allows an adaptive and collaborative approach to information processing. Hence, the SOA assessment method, i.e., validation and verification (V&V), test and evaluation (T&E), and/or certification and accreditation (C&A), should itself be an adaptive and collaborative process. This adaptive approach is as fundamentally different from traditional T&E methods as SOA is fundamentally different from previous computing paradigms. Making the transformation will require similar attention. An “SOA assessment” is a useful engineering service embedded throughout the software development and deployment process, based on measurable and testable parameters that can accomplish the following: • Reduce policy documents, and operational requirements to measurable and testable attributes coded in machine readable formats. • Bundle candidate information processing capabilities into reference implementations. • Assess both the adequacy of the service-oriented infrastructure and the value-add to business objectives. • Weigh risk and reward to scale the rigor of assessment as appropriate. • Assist developers of large complex systems to deploy their capability continuously and incrementally. • Rapidly assess information processing capability prior to its operational deployment. Follow up with operational V&V. • Reduce the barrier to entry for IT vendors and developers who don’t generally deal with the U.S. Government. SOA certification should be addressed in three broad categories: • Software performance and vulnerability, • Network performance and vulnerability, and • Demonstrated value added to enterprise objectives. Effective testing and certification to guarantee SLA terms and conditions is required to establish and sustain trust in shared services. Publishing services to a controlled test environment goes one step further by empowering government partners, business partners, IT vendors, and system integrators to enable compatibility and add value. An investment is required to develop the assessment tools and processes necessary to achieve these outcomes or outsource “testing as a service.”2008-06-302010-02-08http://smw.osera.gov/pgfsoa/images/a/a4/PGFSOA_v-1.1_Final_2008-06-30.pdfArthurColman (www.drybridge.com)colman@drybridge.com