Why All SOAs Need ESBs

I've recently asked one of my developers to research some integration and middleware technologies for a project we're working on. After spending a couple days on this, he said to me "these things are all part of ESBs now". I.e. all the integration and middleware vendors have pretty much taken these capabilities and bundled them into their ESB platforms. "Oh you need a message bus? That's part of our ESB now." "Oh you need this adapter? That's included in our ESB." In most cases when you're implementing an SOA, you will need some piece of middleware or integration technology. With just about every vendor making these things part of their ESB suites, you're probably gonna end up getting an ESB even if you don't need a whole freakin' ESB. That's why all SOAs need ESBs (regardless of whether or not you really need them)!

Service Reuse

The more I think about the phrase "service reuse" or any other variation of it, the more I don't like it. I feel like it's making me think about services from a perspective that is too technical and too low-level. Reuse is something you want to achieve with code but you shouldn't be thinking about services as just code. When you're designing and building your services, you need to think about it from a much higher and more business-oriented perspective. You shouldn't be building a service because you think it's a great piece of code that other applications may be able to reuse. You build a service because it represents some significant capability of your organization that needs to be offered up to other units within the organization or external partners so that the business can operate more effectively. It's about sharing business capabilities and information, not reusing software. When you keep this in mind, you'll be more likely to build services that end up getting "more reuse" because you're providing valuable business capabilities or information that others need to use.

Extending Reuse to the Last Mile

While services make it easier to share your data and functionality, some developer still has to write the code to consume your service and present the results to the end user. In many cases, that code to consume and present a visualization of the service can be reused by other applications. This has already exploded on the Web with Facebook widgets, embeddable YouTube videos, blog traffic counters, etc., so what I'm talking about here is nothing new.

However, within the enterprise the use of shared UI widgets is still not very prevalent. This is problematic because it still leaves the creation of useful applications out of those shared enterprise services within the hands of IT. This really does not make the enterprise that much more agile at delivering useful capabilities to the end users. Real agility is achieved when end users can create those capabilities themselves by building lightweight micro-apps that are composed of various shared widgets for consuming and visualizing those enterprise services. These micro-apps can themselves be packaged up as widgets so that they can be reused within other apps.

So the next thing to do after you're done building those enterprise services is to start building some UI widgets for those services so that you end up with a portfolio of widgets that can be shared alongside your portfolio of services. This will extend the ability to share (or reuse) those services all the way to the end user, i.e. the last mile. You can think of this as sort of a "widgets-based architecture" that is built on top of your services-based architecture.

Don't Go Cheap With the Service Interfaces

Vendors need to realize that in this day and age service interfaces for their products should be a core feature. Unfortunately many vendors view them as secondary requirements and thus don't invest the necessary resources into their development. Often they are built in as an afterthought and usually by some junior developer on the team. The results are poorly designed service interfaces into the products with inadequate levels of functionality and poor usability. If I'm going to buy a piece of enterprise software, I'm not looking only at what functionality it provides, but also how well it will integrate into my existing environment. Vendors that get this will win the sales, vendors that don't will fall to the wayside.

SOA Message Exchage Patterns

There are a variety message exchange patterns that can be used to implement an SOA. Know them and when to apply them. It can mean the difference between an SOA that scales and performs vs. one that doesn't. The traditional synchronous request/response may not always be the most appropriate in all scenarios. Here are some examples of other types of MEPs:

• Asynchronous request/response using a callback--the consumer sends a request and provides a reference to a callback service as part of the request; instead of responding synchronously to the request, the service responds asynchronously by sending the response to the callback service
• Publish/subscribe--consumers register their interest in receiving some data that the service provides by subscribing to the service; whenever the service gets or creates that data, it publishes it to all subscribed consumers. This is somewhat of an extension of the asynch req/resp with callback MEP. The difference is that the pub/sub MEP is used in more of a one-to-many type of scenario, i.e. the service has some data that may be applicable/consumed by multiple consumers; whereas the asynch req/resp with callback is used in more of a one-to-one scenario, i.e. the consumer is requesting some data or functionality that is specific to it (not applicable to other consumers) and doesn't want/need to wait for the response to return immediately
• Request and poll--the consumer sends a request to the service but doesn't want/need to wait for the response so the service returns some type of identifier that the consumer can use later to poll the service for the response. This MEP is useful for when you're trying to achieve the behavior of the asynch req/resp with callback but can't provide a callback service perhaps because firewall settings don't allow inbound messages.
• One-way push, aka "fire and forget"--the consumer sends the service a request and doesn't need a response so it just fires it off and forgets about it; perhaps it would be more accurate to call it a message instead of a request since the consumer's not expecting a response. Also it doesn't have to be the consumer firing off the message, it can be the service doing that as well. This is in fact the publish half of the pub/sub MEP for scenarios that don't have any type of guaranteed delivery requirements.

Barriers to Agility Differ Across Organizations

A lot of folks talk describe the main benefit of SOA is the increased agility that it brings to the organization; mainly describing the architectural and technical approaches and mechanisms that bring about increased agility. The problem with that is that the barriers to agility at many different organizations are not technical in nature. Take the government for example, the reason it takes them such a long time to roll out a change in their IT systems is due to the processes or bureacracy that exists not because of technical challenges. Technical challenges exist but they are not the long poles in the tent. Many of these organizations can gain increased agility by first streamlining their bureacratic processes--for example, security certification and accreditation processes that are required before a system can go live--these often require a lot of paper pushing but don't really do much to ensure that the system is really more secure.

Two Most Common SOA Mistakes

Two of the most common mistakes I've seen by inexperienced people trying to implement SOA are trying to make everything a service and building a huge data model and trying to force everyone to use it.


First of all, not every capability needs to or should be exposed as a service. There are overhead and security risks with exposing something as a service. If the capability is an internal capability that isn't used by anything outside your system, then don't make it a service. Another way to look at it is if it's a capability that isn't used or shared beyond the boundaries of your organization, then again it probably doesn't make much sense to create a service out of it. Of course there can always be exceptions, but you're better off by starting with those capabilities that are used beyond your system and organizational boundaries.

The second mistake is committed by those who spend months (or years) building huge, really detailed data models and accompanying XML schemas for their domains and then require anybody building services in that domain to use those schemas. Don't get me wrong, having common logical data models is a good thing and having some common schemas based on those models is also a good thing. The problem is that these guys are building these things like they're building relational database models, with little flexibility for customization and extension. Building data models and schemas for services is not the same as building a relational model for a system, even a very large system. It takes a very different approach guys. Take a look at NIEM and ebXML's Core Components, they were designed from the get go to support extension and customization. Their models and schemas are designed to be modular so that you can compose them into structures that match the specific needs of your service, yet they are all still derived from the same "core" such that you get the common understanding across all the services that use them. There's a lot more to talk about with this particular issue. I'll discuss more in a follow-up post.

More on EA, DODAF and SOA

In a previous post, I talked about how EA can be used to influence SOA to provide the strategic perspective. Here are some more thoughts along those lines, with some guidance on service-oriented analysis using DODAF views.

I mentioned that SOA should be aligned to the organization's strategic objectives and that EA can help drive this. So how?

The DODAF defines a view called the OV-1 (high level operational concept) that describes/depicts the key entities (i.e. operational nodes), their activities, and interactions in support of the organization's mission(s). So once the organization's strategic objectives are determined, how the organization will execute on those objectives can be modeled using the OV-1. The OV-1 is a simple high level view, but the creation of it forces you to think about how the different groups/sub-organizations will interact to execute the strategic objectives. If you can't figure this out, then you have some unachievable objectives!

With the OV-1 created, you now have the organization's strategic objectives codified in an architectural artifact. Scoping your SOA using the OV-1 and using it as the starting point for service-oriented analysis ensures that your SOA is aligned to the organization's strategic objectives.

As I mentioned earlier, the OV-1 depicts the operational nodes (groups, departments, etc.), their activities, and interactions in support of the execution of the organization's objectives. Since SOA's all about creating and sharing capabilities across different ownership domains, the key things to focus on first from the OV-1 are the operational nodes and their interactions (the activities within each operational node can be useful later for further decomposition). With these operational nodes and their interactions, you have your candidates for the service providers, service consumers, and services. From that point on, it's just further analysis and decomposition to identify the services that comprise your SOA and who should provide them.

So there you have it--step-by-step guidance on how to ensure your SOA is aligned to the strategic objectives of your organization. Of course, this is all easier said than done.

Stay tuned, in a future installment I will discuss how other DODAF views can be used in that further analysis and decomposition.

Red Herring: DODAF vs. SOA

Those of you who work with the DoD are probably very familiar with DODAF and lately have probably heard a lot about how DODAF is incompatible/inadequate for representing Service Oriented Architectures. While there may be some truth to that, I consider it to be somewhat of a red herring. DODAF is primarily used to create Enterprise Architectures (EA). The real problem (at least based on what I've come across) is that those working on EA's don't really understand SOA and vice versa. Those working on EAs are usually more strategy and management focused and don't understand (or aren't interested) in the principles of service orientation. On the flip side, those working on SOAs are usually very technically-focused and don't understand many of the management and strategic objectives of EA (or find them boring and don't want to be bothered by them). As a result the two efforts are usually separate when they need to be interdependent. So how are they interdependent? There are a couple ways to look at it.

One, SOA is an architectural style and this style can be applied in the creation of the EA. An EA models how an enterprise works using multiple layers/views, e.g. organizational, operational, technical. Using a service-oriented style, the enterprise can be modeled as a set of service providers/consumers and the services through which they interact. This approach can be taken through all the layers of the EA. This is how SOA can be used to influence the EA.

On the other hand, the EA effort can and should also influence the SOA effort. As mentioned earlier, those working on the SOA effort are very technical and implementation-focused, they look at it as "let's build an SOA". The SOA needs to be aligned with the strategic objectives or problems that the enterprise is trying to address. Those that are very technical and implementation-focused will usually not be able to identify those strategic objectives/problems. For example, which parts of the enterprise are very dynamic such that they would benefit from the agility that can be achieved through SOA (and can justify the cost and effort)? An EA provides a holistic view of the enterprise such that the strategic objectives/problems can be identified and prioritized which in turn should be used to scope and define a roadmap for the SOA.

So stop arguing about whether or not DODAF can be used to depict SOAs and start figuring out how to reconcile the EA and SOA efforts!

Woohoo! I Can Now Access Your Crap Data More Easily

With the growing popularity of enterprise mashups, more and more organizations are starting to service-enable their data sources, i.e. building web services on top of their data sources to enable access to their data. Unfortunately, most of them are not addressing their underlying data problems first before doing so. I've written about this before, talking about the need for an SOA data strategy. It still amazes me to see how many people out there still think that by slapping web services on top of their databases, amazing things will happen and their problems will be solved. All that will happen is that this will bring all their crappy data up front and center on some flashy AJAX-based mashup application. We've seen this before in the past with BI and reporting applications that used some of the more traditional data integration technologies. After the BI app is built, the end user realizes that she still can't do any real analysis because the data is crap.

soa, enterprise mashups, data strategy

Scenarios for RESTful Web Services

The RESTful approach is very attractive for large scale integration scenarios that cross many organizational boundaries. This is because the constraints imposed by the REST principles emphasize interoperability and scalability. The constraint of uniform interfaces supports those scenarios in which the consumer base for the services is so broad that it makes it difficult to create and maintain a large set of custom interfaces. In such cases, it makes more sense to apply a design in which a single interface can support all the required interactions. Unfortunately, modeling everything as a set of resources that are all exposed through a uniform interface is not always easy. Developers are accustomed to designing a specific interface for each piece of functionality or data that they wish to expose; forcing them to always use a uniform interface is antithetical to this. However, scenarios in which services are just providing access to data can easily support the uniform interface constraint. This is because any kind of data can be manipulated through the same set of create, read, update, and delete operations. Thus, scenarios in which it is primarily data that needs to be shared through web services make REST an easy choice.

soa, web services, REST, SOAP

Scenarios for SOAP WS-* Web Services

The SOAP WS-* approach provides a broad set of standards and specifications for quality of service features and also gives developers a lot of flexibility to define custom interfaces for the services that they wish to expose. This flexibility is useful for application-to-application integration scenarios internal to an organization. This is also useful in scenarios in which legacy applications need to be exposed to the rest of the enterprise. In either of these scenarios, the existing applications often constrain how the services may be exposed, so the flexibility to design service interfaces that can adapt to these constraints is important. Additionally, in these scenarios the services are often providing complex functionality and processes that may be difficult to model in a resource-oriented manner with uniform interfaces. It is also these types of scenarios that typically require many of the complex quality of service features that the SOAP WS-* approach has broad support for. Finally, these types of scenarios are more commonly found inside a single organization and less so across organizational boundaries. The SOAP WS-* approach typically results in large number of custom interfaces, but when this is occurring within a single organization, they are a lot easier to control and maintain than in scenarios where there are many organizations that are dependent on those interfaces.

soa, web services, REST, SOAP

What a F!#@ing Waste of Time!

Been doing some work with web services management lately and looking at the various standards and specs for that stuff, most notably WSDM and WS-Management. It's always frustrating when you have competing standards. So not only are WSDM and WS-Management competing standards but they are built on top of other competing standards, e.g. WS-Notification vs. WS-Eventing, WS-ResourceFramework vs. WS-Transfer, etc. So the vendors from both camps have decided that they should converge the two sets. After spending a couple years creating these competing standards, now they're going to spend some more time to converge them. So you screw the customers in the first place by creating the competing specs and now you screw them again because you're going to create a whole other set of specs and the previous set will eventually get deprecated. Not to mention the time you waste yourselves creating the competing sets only to get back together to converge them. Those of us in the customer community need to put an end to this madness.

web services, WSDM, ws-management

SOA = Service Oriented Application

I see a disturbing trend out there where people who are working on a single application say that they are creating an SOA. When they decompose the application into modules, instead of calling them subsystems or components as we use to, they call them services. Then they put a WSDL on them, register them in a registry, and bam they tell me they have a SOA. I guess that is appropriate if SOA to them means Service Oriented Application. I just hope they don't go around telling the business that now that they have an SOA everything's going to be much more agile, they're going to save a bunch of money from reuse, systems will be more interoperable, etc., etc. What are the chances that they're going to get any use from those services outside of that single application? Probably zero. So given that, how's the business going to be more agile? Beats me. Since those "services" and their WSDLs were designed from the perspective of that single application, what are the chances that they will be interoperable with others who might want to use them? Just because you're using WSDL/SOAP/HTTP doesn't guarantee you interoperability.

soa

Business Process Patterns

In my work with various clients, I'm starting to take note of the common scenarios I'm seeing of how people are using SOA to improve their business processes. I'm sure others are starting to notice some recurring scenarios as well. By capturing these scenarios, we can develop a set of patterns that can be used to address them. What sets these apart from the other types of patterns we see in software such as design patterns or implementation patterns are that these are analysis patterns. Years ago, Martin Fowler wrote a book called Analysis Patterns which I still find to be a very useful book today. The patterns in that book however are finer-grained and more domain specific. What I am referring to are a broader set of analysis patterns that help you to analyze and decompose complex business processes which can be applied to any domain. The relevance to SOA of course is that business process analysis is one of the critical initial steps to any SOA initiative. Having such a set of common process analysis patterns will help to create a much more definitive SOA methodology which I think we are still lacking.

Here's one example of a recurring scenario that I see: Multiple Variant Processes to Perform a Common Function. In a large organization with multiple subsidiaries, there are often common functions that each of those subsidiaries perform but with their own slightly unique processes for doing so. For example, a lot of large organizations have subsidiaries that have their own unique processes for procurement. Often times the parent organization will want to integrate with the processes of the subsidiaries or it will want to integrate the subsidiaries’ processes together. So how do you go about analyzing all the subsidiaries’ processes to figure what/where to integrate? This can be pretty overwhelming if there are a lot of subsidiaries and the processes are complex.

An effective way to analyze the variant processes is to abstract a common process out of them. This is analogous to extracting a trend line from a scatter plot graph. In a scatter plot, trying to look at the individual points to understand and draw relationships out of the data is very difficult, thus, you generate a common trend line out of those points.

With process analysis, it is very easy to get caught up in all the details of the individual processes and lose sight of the relationships across the processes that are important for integration. Thus, you should abstract a common process out of the individual processes and use that common process to direct your analysis of the individual processes.


soa, patterns, bpm

XML Compression is Not the Answer to Your SOA Performance Woes

Articles like this are misleading. XML compression will reduce the size of the message and thus reduce the time to transfer the message. But if you've ever done any performance testing and profiling on Web services, you will know that most of the times the performance bottlenecks are not in the message transfer but the XML processing, i.e. parsing the message. Thus, compressing the message will not do you much good. Some may say that in a bandwidth constrained environment, the compression does help. But if you're working in a bandwidth constrained environment, you probably shouldn't even be using such technologies in the first place. What does help tremendously is offloading the XML processing to dedicated XML appliances and the article does point that out.

soa, web services, xml

When Should You Use an ESB?

With all the hype around ESBs, a lot of people think that you need an ESB to build an SOA. That isn't always the case. I'm getting tired of seeing boilerplate SOA architecture diagrams with an ESB in the center. As with any other product, you should only use it if you require the functionality provided by it. Though there's no firm definition of what exactly is an ESB, you often see a core set of functionality that's provided by all ESB products. I talk about them in this posting. They are:

• Protocol Adaptors
• Data Transformation
• Routing
• Orchestration
• Synchronous/Asynchronous Messaging
• Endpoint virtualization

So when might you need such functionality? For example, if you're working in an environment with legacy systems and not all of them can support SOAP/HTTP. Or perhaps if you're building some type of automated workflow system or BPM. Or some event-driven system that requires asynchronous messaging and routing. If you just required some data transformation or endpoint virtualization I probably would not go with an ESB--it would be a bit overkill. Anyways, the point is look at your requirements and your environmental constraints first to see if you need the functionality that's provided by an ESB before you throw an ESB into your architecture.

soa, esb

A Couple Thoughts on SOA Governance Related Issues

Most large organizations have various security audit and certification processes that applications have to go through before they can go into production. These can usually take several weeks to complete. The holy grail of SOA of course is to be able to quickly compose new applications by orchestrating a set of existing services. However, you kind of defeat the purpose if your new SOA allows you to compose new applications in a few days or a couple weeks but then you have to wait several more weeks for the security certification before you can put it into production and start using it. Thus these security processes need to become more agile as well in order to support this new approach to building applications. This just re-emphasizes the point that SOA is not just about the technologies...the organizational processes have to support it as well.

The second thought has to do with the concept of portfolio management. This is the practice of treating your IT systems and applications as a portfolio, similar to how you treat your portfolio of investments. You regularly review your IT portfolio to see how it's aligned with your business objectives and other criteria you've established and decide how to fund (or invest in) them accordingly. Those systems that don't meet your criteria get reduced funding or get cut altogether. I contend that this practice needs to be applied to the services in your SOA as well. Those services need to be treated as a portfolio that get reviewed regularly to see whether they're being used, how much they're being used, how much is it costing to maintain them, how much cost savings are you getting as a result of reusing them, etc., etc.--you define the criteria that's important to your organization. With all the rush by everybody to stand up Web services and build their SOAs, I'm sure that there'll be plenty of services that actually don't ever get used. You gotta weed those suckers out. Your registry needs to be kept up to date and relevant, otherwise if there's too much junk in there, your developers will avoid it like the plague. Bottom line--include a portfolio management process for the services in your SOA. Perhaps start by conducting your reviews annually and then adjust the frequency of such reviews based on the volatility of your organization.

soa, governance, portfolio management

AOP and SOA Implementations

We are all familiar with the good old fashioned principle of separation of concerns and thanks to Thomas Erl we know how it's a fundamental principle behind Service Oriented Architectures. But we mostly think about separation of concerns at the architectural level. It's also a good idea to apply the separation of concerns at the implementation level, when coding the services that make up the SOA. This is where the techniques of Aspect Oriented Programming come in. While separation of concerns is a design principle, AOP allows you to carry it through to implementation. The Spring Framework is a nice lightweight Java framework with very good facilities for AOP. There's a companion security system for Spring called the Acegi Security System that allows you to secure your applications developed using Spring. Both of these frameworks provide very nice alternatives to JEE for developing Web services. I'm working on an article that shows you how to develop secure Web services using Apache Axis, Spring, and Acegi. Sure, there are plenty of ways to implement Web services but using Spring allows you to use its AOP facilities to drive the separation of concerns down into the implementation. Spring itself also has support for Web services, but since I'm more familiar with Axis, I decided to go with Axis for the example in the article. Acegi has some pretty nice security functionality but to be able to use it to secure Web services developed with Axis requires a bit of work to bridge the Web service security context from Axis into the Acegi security context. This is one of the things that I will show in the article. Stay tuned, I will post a link to the article once it becomes available.

soa, aop, spring framework, acegi, web services

BAM and SOA

Saw some talk (from David Linthicum and David Black) recently in the blogosphere on doing business activity monitoring (BAM) and business intelligence (BI) within SOAs. This is an idea I've been keen on for a while now. Wrote an article about it for DM Review last August.

The opportunity lays in the fact that SOAs have a holistic view of the business processes that are executing within them, thus they can provide the context that ties together the data collected from BAM or BI to provide actionable information at a business level versus just giving you the raw data that your analysts have to try to manually tie together. Combine this with the fact that you have standard XML messages from which to harvest the data from and it becomes a very attractive opportunity that organizations with SOAs should definitely take advantage of.

In his entry, Black goes a bit further to talk about how ESBs can provide a good platform for this stuff, which is essentially what I was detailing in my article as well. According to his blog, Black, who recently joined Cape Clear, will be working to help them add BAM capabilities to their ESB. This will be a great feature that will distinguish them from their competitors. I'll definitely be keeping my eye out for this feature to see when it becomes available.

soa, bam, bi, esb