Tieu Luu's Weblog

This Blog Has Moved

2009-09-08T22:27:00.002-04:00

I've moved this blog to my own domain. Here's the new URL:

http://tieuluu.com/blog

Widget Wars: OpenSocial vs. OpenAjax vs. W3C Widgets

2009-07-29T21:41:00.004-04:00

The popularity of widgets these days has brought to attention the need for interoperability, i.e. for widgets developed for one site or platform to be able to run in other sites and widgets developed by different people to be able to work with each other. So much so that I know of at least 3 somewhat competing specifications for widgets. There's the gadget portion of the OpenSocial specs which was adopted from the Google Gadgets work. Then there's OpenAjax which is more broadly focused on Ajax interoperability but has a lot of pieces geared towards widget interoperability. And finally there's the W3C's Widgets 1.0 Family of Specifications. Based on a preliminary analysis, the OpenAjax specs appear to be the most comprehensive for widget interoperability issues but OpenSocial seems to have gained more adoption. The W3C work appears to be moving very slowly (no surprise) and it's unclear to me how much adoption they have. Right now my preference is the OpenAjax specs based on how comprehensive it is but admittedly I have not done a thorough analysis and comparison of the three. Plus, a spec that is too comprehensive may not always be a good thing, e.g. tends to be more complex and may be trying to standardize more than what's necessary. Stay tuned, I'll post a more thorough analysis once I've had more time to parse through each of them in more detail.

Why All SOAs Need ESBs

2009-07-26T22:22:00.004-04:00

I've recently asked one of my developers to research some integration and middleware technologies for a project we're working on. After spending a couple days on this, he said to me "these things are all part of ESBs now". I.e. all the integration and middleware vendors have pretty much taken these capabilities and bundled them into their ESB platforms. "Oh you need a message bus? That's part of our ESB now." "Oh you need this adapter? That's included in our ESB." In most cases when you're implementing an SOA, you will need some piece of middleware or integration technology. With just about every vendor making these things part of their ESB suites, you're probably gonna end up getting an ESB even if you don't need a whole freakin' ESB. That's why all SOAs need ESBs (regardless of whether or not you really need them)!

Web Service Message Level Implementations

2008-08-13T01:12:00.006-04:00

If you're like me and prefer working with the raw XML in your web service implementations instead of dealing with the mess of generated data binding code, this article will show you how to do so using JAX-WS and XPath. In this example, we use this approach to implement a WS-Notification consumer service that receives WSDM-based messages for monitoring the health and performance of services.

JAX-WS has the Provider interface that you can implement to get access to the raw XML message. You have the choice of implementing Provider<Source>, Provider<DataSource>, or Provider<SOAPMessage>. We'll be using the Provider<Source> in this example. The invoke(Source) method is what gets called to process a service request. We'll break down the implementation of that method step-by-step.

First we send the source through a Transformer to get a DOM tree that we can use for XPath processing:


DOMResult dom = new DOMResult();
Transformer trans = TransformerFactory.newInstance().newTransformer();
trans.transform(source, dom);

Next, we instantiate an XPathFactory and create an XPath that we'll use to process the DOM tree against. The third line calls the setNamespaceContext() method and uses a custom class I created that provides a mapping of namespace URI's to prefixes and vice-versa. You can find more information on how to implement such a class here. We need this because we'll be using namespace prefixes in our expressions.


XPathFactory xpf = XPathFactory.newInstance();
XPath xp = xpf.newXPath();
xp.setNamespaceContext(MyNamespaceContext.getInstance());

Now we'll process our first expression to extract the NotificationMessage elements from the message, passing in the XPath expression for the NotificationMessage and our DOM tree to evaluate the expression against. A WS-Notification message may contain multiple NotificationMessages so we get a NodeList in return.

NodeList resultList = (NodeList)xp.evaluate("/wsnt:Notify/wsnt:NotificationMessage", dom.getNode(), XPathConstants.NODESET);

Then we'll iterate through each item in the NodeList, i.e. each NotificationMessage, and process another expression against it to extract the ResourceId element and print it out.

String exprPrefix = "";
int len = resultList.getLength();
for (int i=1;i<=len;i++) {      
   String expr1 = "/wsnt:Notify/wsnt:NotificationMessage[" + i
       + "]" + "/wsnt:Message/muws1:ManagementEvent"
       + "/muws1:SourceComponent/muws1:ResourceId";      
   String resourceId = xp.evaluate(expr1, dom.getNode());      
   System.out.println("Resource ID: " + resourceId);
}

The entire source code listing can be found here. You'll notice that the class has the following two annotations:

@WebServiceProvider()

@ServiceMode(value=Service.Mode.PAYLOAD)

The first one tells JAX-WS that this class is a Provider endpoint. The second one says that we only want to work with the message at the payload level, i.e. we don't need to access the message headers. In addition to these annotations, we have to customize the WSDL to mark the port as a Provider interface. This can be done by embedding the customizations directly within the WSDL or through a separate customization file. We'll do it using a separate provider-customize.xml file--you can find this file here.

Service Reuse

2008-06-25T21:10:00.006-04:00

The more I think about the phrase "service reuse" or any other variation of it, the more I don't like it. I feel like it's making me think about services from a perspective that is too technical and too low-level. Reuse is something you want to achieve with code but you shouldn't be thinking about services as just code. When you're designing and building your services, you need to think about it from a much higher and more business-oriented perspective. You shouldn't be building a service because you think it's a great piece of code that other applications may be able to reuse. You build a service because it represents some significant capability of your organization that needs to be offered up to other units within the organization or external partners so that the business can operate more effectively. It's about sharing business capabilities and information, not reusing software. When you keep this in mind, you'll be more likely to build services that end up getting "more reuse" because you're providing valuable business capabilities or information that others need to use.

Extending Reuse to the Last Mile

2008-06-03T21:03:00.007-04:00

While services make it easier to share your data and functionality, some developer still has to write the code to consume your service and present the results to the end user. In many cases, that code to consume and present a visualization of the service can be reused by other applications. This has already exploded on the Web with Facebook widgets, embeddable YouTube videos, blog traffic counters, etc., so what I'm talking about here is nothing new.

However, within the enterprise the use of shared UI widgets is still not very prevalent. This is problematic because it still leaves the creation of useful applications out of those shared enterprise services within the hands of IT. This really does not make the enterprise that much more agile at delivering useful capabilities to the end users. Real agility is achieved when end users can create those capabilities themselves by building lightweight micro-apps that are composed of various shared widgets for consuming and visualizing those enterprise services. These micro-apps can themselves be packaged up as widgets so that they can be reused within other apps.

So the next thing to do after you're done building those enterprise services is to start building some UI widgets for those services so that you end up with a portfolio of widgets that can be shared alongside your portfolio of services. This will extend the ability to share (or reuse) those services all the way to the end user, i.e. the last mile. You can think of this as sort of a "widgets-based architecture" that is built on top of your services-based architecture.

Don't Go Cheap With the Service Interfaces

2008-05-23T20:54:00.004-04:00

Vendors need to realize that in this day and age service interfaces for their products should be a core feature. Unfortunately many vendors view them as secondary requirements and thus don't invest the necessary resources into their development. Often they are built in as an afterthought and usually by some junior developer on the team. The results are poorly designed service interfaces into the products with inadequate levels of functionality and poor usability. If I'm going to buy a piece of enterprise software, I'm not looking only at what functionality it provides, but also how well it will integrate into my existing environment. Vendors that get this will win the sales, vendors that don't will fall to the wayside.

SOA Message Exchage Patterns

2008-04-01T22:30:00.001-04:00

There are a variety message exchange patterns that can be used to implement an SOA. Know them and when to apply them. It can mean the difference between an SOA that scales and performs vs. one that doesn't. The traditional synchronous request/response may not always be the most appropriate in all scenarios. Here are some examples of other types of MEPs:

Asynchronous request/response using a callback--the consumer sends a request and provides a reference to a callback service as part of the request; instead of responding synchronously to the request, the service responds asynchronously by sending the response to the callback service
Publish/subscribe--consumers register their interest in receiving some data that the service provides by subscribing to the service; whenever the service gets or creates that data, it publishes it to all subscribed consumers. This is somewhat of an extension of the asynch req/resp with callback MEP. The difference is that the pub/sub MEP is used in more of a one-to-many type of scenario, i.e. the service has some data that may be applicable/consumed by multiple consumers; whereas the asynch req/resp with callback is used in more of a one-to-one scenario, i.e. the consumer is requesting some data or functionality that is specific to it (not applicable to other consumers) and doesn't want/need to wait for the response to return immediately
Request and poll--the consumer sends a request to the service but doesn't want/need to wait for the response so the service returns some type of identifier that the consumer can use later to poll the service for the response. This MEP is useful for when you're trying to achieve the behavior of the asynch req/resp with callback but can't provide a callback service perhaps because firewall settings don't allow inbound messages.
One-way push, aka "fire and forget"--the consumer sends the service a request and doesn't need a response so it just fires it off and forgets about it; perhaps it would be more accurate to call it a message instead of a request since the consumer's not expecting a response. Also it doesn't have to be the consumer firing off the message, it can be the service doing that as well. This is in fact the publish half of the pub/sub MEP for scenarios that don't have any type of guaranteed delivery requirements.

Barriers to Agility Differ Across Organizations

2008-02-21T17:58:00.004-05:00

A lot of folks talk describe the main benefit of SOA is the increased agility that it brings to the organization; mainly describing the architectural and technical approaches and mechanisms that bring about increased agility. The problem with that is that the barriers to agility at many different organizations are not technical in nature. Take the government for example, the reason it takes them such a long time to roll out a change in their IT systems is due to the processes or bureacracy that exists not because of technical challenges. Technical challenges exist but they are not the long poles in the tent. Many of these organizations can gain increased agility by first streamlining their bureacratic processes--for example, security certification and accreditation processes that are required before a system can go live--these often require a lot of paper pushing but don't really do much to ensure that the system is really more secure.

Let's Get Real Guys ...

2008-02-04T21:21:00.001-05:00

I came across this article in SOA World today:

Bridging the Gap between Business & IT with BPMN & BPEL
— Because the role of IT organizations is to enable business managers to run their businesses better, there has been a constant need for aligning IT closer to business. We often hear business managers complain that a software solution isn't what the business needed. We have personal experience with this problem - in a previous life one of us worked for a consulting company that built turnkey applications for large enterprises.

It basically talks about the approach and tooling from Oracle to allow business users and IT to jointly create business process models to, as the title says "bridge the gap between business and IT". While I have no major objections to the approach they're describing in the article--I think it's actually a pretty good solution to address the standards divide, i.e. BPMN vs. BPEL, in the business process space, I do object to how they envision this being used. They describe essentially a process where a business user uses the graphical BPMN-based modeling tool to design the business process and IT takes this as a blueprint to create the BPEL-based executable process.

I don't know what kind of business users they have come across (or are envisioning), but I've never met any business user who's able or willing to use any type of modeling tool to layout their business process for you. They're too busy actually trying to run the business. The best you can expect to get from them is some type of document that roughly describes the process and it's up to us IT shmucks to try create some type of process model out of that.

Oracle has one of the better suites for BPM out there since their acquisition of Collaxa several years ago, but let's get real guys, it's not the business users that are using that, no matter how graphical and friendly the tool is--it's still us IT guys that are the ones solely using that stuff. These guys need to know who their customers are and how they're using the product and not make up some nirvana-like scenario of business users modeling their business process and IT taking that to create the executable for alignment between the business and IT.

SOAP/WS-* vs. REST and The Art of Politecture

2008-01-30T21:24:00.001-05:00

Those of you who've worked on very large scale enterprise-wide architectures know that it's always the people and politics that are the biggest hurdles. I came across the term "politecture" in Nick Malik's blog here. You can follow the link for the definition, but it's essentially an architecture that's been designed with the influence of political factors. Some may say this isn't right, but I think it's perfectly fine, even necessary when you're designing a large scale architecture that has to be adopted by many people and organizations.

So what's this have to do with the SOAP/WS-* vs. REST debate? I tend to agree that there are very many scenarios in large scale distributed systems in which a simpler RESTful approach is a technically superior solution. However, IBM, Microsoft, OASIS et al. have done such a good job touting the power of standards in the WS-* world that most folks within large enterprises are sold on the WS-*-based architecture. Whether or not those standards actually help you is a topic for another discussion, but the point is that within most large enterprises you're going to have a much better chance of getting people to adopt your WS-* "standards-based" architecture than you will with a "maverick" REST-based architecture. Is this a problem? Maybe, but from what I've seen usually not. So what if you end up with a technically more complex solution? I can solve the technical problems that result from the added complexity of WS-*. Like I said at the beginning of this post, it's the people and political issues that are the biggest hurdles. From my experience, it's usually the people and politics that cause projects to fail or be canceled. If I can't get people to adopt my architecture because I'm going with a "non-standards-based RESTful architecture", I'm not even going to have a project to worry about it failing because WS-* is more complex, less interoperable, etc.

As an architect, you're not implementing, but you need to get everybody to agree to what you've defined so that they will implement it. If this means making some compromises that perhaps result in a less technically superior solution, but will allow you to achieve consensus then go ahead and do it. As I've painfully realized in my transition from developer to architect over the past few years, things are not so black and white with architecture, i.e. you can't always just go with the solution that makes the most technical sense. If you fail to see this or cannot adapt to this, then you will not be a successful architect.

Two Most Common SOA Mistakes

2008-01-26T20:53:00.002-05:00

Two of the most common mistakes I've seen by inexperienced people trying to implement SOA are trying to make everything a service and building a huge data model and trying to force everyone to use it.

First of all, not every capability needs to or should be exposed as a service. There are overhead and security risks with exposing something as a service. If the capability is an internal capability that isn't used by anything outside your system, then don't make it a service. Another way to look at it is if it's a capability that isn't used or shared beyond the boundaries of your organization, then again it probably doesn't make much sense to create a service out of it. Of course there can always be exceptions, but you're better off by starting with those capabilities that are used beyond your system and organizational boundaries.

The second mistake is committed by those who spend months (or years) building huge, really detailed data models and accompanying XML schemas for their domains and then require anybody building services in that domain to use those schemas. Don't get me wrong, having common logical data models is a good thing and having some common schemas based on those models is also a good thing. The problem is that these guys are building these things like they're building relational database models, with little flexibility for customization and extension. Building data models and schemas for services is not the same as building a relational model for a system, even a very large system. It takes a very different approach guys. Take a look at NIEM and ebXML's Core Components, they were designed from the get go to support extension and customization. Their models and schemas are designed to be modular so that you can compose them into structures that match the specific needs of your service, yet they are all still derived from the same "core" such that you get the common understanding across all the services that use them. There's a lot more to talk about with this particular issue. I'll discuss more in a follow-up post.

More on EA, DODAF and SOA

2007-12-26T21:25:00.001-05:00

In a previous post, I talked about how EA can be used to influence SOA to provide the strategic perspective. Here are some more thoughts along those lines, with some guidance on service-oriented analysis using DODAF views.

I mentioned that SOA should be aligned to the organization's strategic objectives and that EA can help drive this. So how?

The DODAF defines a view called the OV-1 (high level operational concept) that describes/depicts the key entities (i.e. operational nodes), their activities, and interactions in support of the organization's mission(s). So once the organization's strategic objectives are determined, how the organization will execute on those objectives can be modeled using the OV-1. The OV-1 is a simple high level view, but the creation of it forces you to think about how the different groups/sub-organizations will interact to execute the strategic objectives. If you can't figure this out, then you have some unachievable objectives!

With the OV-1 created, you now have the organization's strategic objectives codified in an architectural artifact. Scoping your SOA using the OV-1 and using it as the starting point for service-oriented analysis ensures that your SOA is aligned to the organization's strategic objectives.

As I mentioned earlier, the OV-1 depicts the operational nodes (groups, departments, etc.), their activities, and interactions in support of the execution of the organization's objectives. Since SOA's all about creating and sharing capabilities across different ownership domains, the key things to focus on first from the OV-1 are the operational nodes and their interactions (the activities within each operational node can be useful later for further decomposition). With these operational nodes and their interactions, you have your candidates for the service providers, service consumers, and services. From that point on, it's just further analysis and decomposition to identify the services that comprise your SOA and who should provide them.

So there you have it--step-by-step guidance on how to ensure your SOA is aligned to the strategic objectives of your organization. Of course, this is all easier said than done.

Stay tuned, in a future installment I will discuss how other DODAF views can be used in that further analysis and decomposition.

Red Herring: DODAF vs. SOA

2007-10-20T20:54:00.001-04:00

Those of you who work with the DoD are probably very familiar with DODAF and lately have probably heard a lot about how DODAF is incompatible/inadequate for representing Service Oriented Architectures. While there may be some truth to that, I consider it to be somewhat of a red herring. DODAF is primarily used to create Enterprise Architectures (EA). The real problem (at least based on what I've come across) is that those working on EA's don't really understand SOA and vice versa. Those working on EAs are usually more strategy and management focused and don't understand (or aren't interested) in the principles of service orientation. On the flip side, those working on SOAs are usually very technically-focused and don't understand many of the management and strategic objectives of EA (or find them boring and don't want to be bothered by them). As a result the two efforts are usually separate when they need to be interdependent. So how are they interdependent? There are a couple ways to look at it.

One, SOA is an architectural style and this style can be applied in the creation of the EA. An EA models how an enterprise works using multiple layers/views, e.g. organizational, operational, technical. Using a service-oriented style, the enterprise can be modeled as a set of service providers/consumers and the services through which they interact. This approach can be taken through all the layers of the EA. This is how SOA can be used to influence the EA.

On the other hand, the EA effort can and should also influence the SOA effort. As mentioned earlier, those working on the SOA effort are very technical and implementation-focused, they look at it as "let's build an SOA". The SOA needs to be aligned with the strategic objectives or problems that the enterprise is trying to address. Those that are very technical and implementation-focused will usually not be able to identify those strategic objectives/problems. For example, which parts of the enterprise are very dynamic such that they would benefit from the agility that can be achieved through SOA (and can justify the cost and effort)? An EA provides a holistic view of the enterprise such that the strategic objectives/problems can be identified and prioritized which in turn should be used to scope and define a roadmap for the SOA.

So stop arguing about whether or not DODAF can be used to depict SOAs and start figuring out how to reconcile the EA and SOA efforts!

Woohoo! I Can Now Access Your Crap Data More Easily

2007-10-11T21:08:00.001-04:00

With the growing popularity of enterprise mashups, more and more organizations are starting to service-enable their data sources, i.e. building web services on top of their data sources to enable access to their data. Unfortunately, most of them are not addressing their underlying data problems first before doing so. I've written about this before, talking about the need for an SOA data strategy. It still amazes me to see how many people out there still think that by slapping web services on top of their databases, amazing things will happen and their problems will be solved. All that will happen is that this will bring all their crappy data up front and center on some flashy AJAX-based mashup application. We've seen this before in the past with BI and reporting applications that used some of the more traditional data integration technologies. After the BI app is built, the end user realizes that she still can't do any real analysis because the data is crap.

soa, enterprise mashups, data strategy

Scenarios for RESTful Web Services

2007-09-26T21:36:00.000-04:00

The RESTful approach is very attractive for large scale integration scenarios that cross many organizational boundaries. This is because the constraints imposed by the REST principles emphasize interoperability and scalability. The constraint of uniform interfaces supports those scenarios in which the consumer base for the services is so broad that it makes it difficult to create and maintain a large set of custom interfaces. In such cases, it makes more sense to apply a design in which a single interface can support all the required interactions. Unfortunately, modeling everything as a set of resources that are all exposed through a uniform interface is not always easy. Developers are accustomed to designing a specific interface for each piece of functionality or data that they wish to expose; forcing them to always use a uniform interface is antithetical to this. However, scenarios in which services are just providing access to data can easily support the uniform interface constraint. This is because any kind of data can be manipulated through the same set of create, read, update, and delete operations. Thus, scenarios in which it is primarily data that needs to be shared through web services make REST an easy choice.

soa, web services, REST, SOAP

Scenarios for SOAP WS-* Web Services

2007-09-26T21:32:00.000-04:00

The SOAP WS-* approach provides a broad set of standards and specifications for quality of service features and also gives developers a lot of flexibility to define custom interfaces for the services that they wish to expose. This flexibility is useful for application-to-application integration scenarios internal to an organization. This is also useful in scenarios in which legacy applications need to be exposed to the rest of the enterprise. In either of these scenarios, the existing applications often constrain how the services may be exposed, so the flexibility to design service interfaces that can adapt to these constraints is important. Additionally, in these scenarios the services are often providing complex functionality and processes that may be difficult to model in a resource-oriented manner with uniform interfaces. It is also these types of scenarios that typically require many of the complex quality of service features that the SOAP WS-* approach has broad support for. Finally, these types of scenarios are more commonly found inside a single organization and less so across organizational boundaries. The SOAP WS-* approach typically results in large number of custom interfaces, but when this is occurring within a single organization, they are a lot easier to control and maintain than in scenarios where there are many organizations that are dependent on those interfaces.

soa, web services, REST, SOAP

Dust Off Those Patterns Books

2007-09-22T23:02:00.000-04:00

With the growing popularity of Event-Driven Architectures (EDA) and Complex Event Processing (CEP), patterns such as the blackboard and observer patterns will become important again (not that they ever really ceased being important).

The blackboard pattern is used in systems that deal with problems that don't have a deterministic approach to reaching a solution, mainly AI systems. The pattern consists of a common space (the blackboard) where data is posted to be shared. Independent knowledge agents listen for data of interest from other agents and post partial solutions to the blackboard for consumption by other agents. The data posted to the blackboard represent the current state of the system. Control agents monitor the state of processing and determine when a solution is sufficient and processing can stop. The blackboard pattern was made popular when it was described in Buschman et al's book on software architecture patterns. A lot of EDA's today are implemented using messaging systems. The blackboard pattern can be applied to such architectures--the messaging bus becomes the blackboard and the knowledge and control agents are publishers and subscribers to the bus.

The observer pattern is a very simple pattern that's often used in GUI applications as part of an MVC framework. The pattern consists of a subject that may emit events and observers that are interested in those events. The observers register themselves with the subject for events that they are interested in and when those events occur, the subject notifies the registered observers. The observer pattern is the basis from which EDA's are built. This pattern is documented and described in many places but most people probably learned or read about it from the Gang of Four's design patterns book.

EDA, CEP, design patterns

What a F!#@ing Waste of Time!

2007-08-30T21:00:00.000-04:00

Been doing some work with web services management lately and looking at the various standards and specs for that stuff, most notably WSDM and WS-Management. It's always frustrating when you have competing standards. So not only are WSDM and WS-Management competing standards but they are built on top of other competing standards, e.g. WS-Notification vs. WS-Eventing, WS-ResourceFramework vs. WS-Transfer, etc. So the vendors from both camps have decided that they should converge the two sets. After spending a couple years creating these competing standards, now they're going to spend some more time to converge them. So you screw the customers in the first place by creating the competing specs and now you screw them again because you're going to create a whole other set of specs and the previous set will eventually get deprecated. Not to mention the time you waste yourselves creating the competing sets only to get back together to converge them. Those of us in the customer community need to put an end to this madness.

web services, WSDM, ws-management

Event Driven Architectures

2007-08-28T20:40:00.001-04:00

Came across a pretty cool open source product several weeks ago. It's an event stream processing engine called Esper developed by a couple guys who look like they have a lot of experience building systems for the financial sector. Makes sense since I can see a lot applications for EDA's in the financial domain. Esper defines its own EQL (event query language) that allows you to define SQL-like expressions to process against the stream of events coming in. Thus you can use EQL to define rules to correlate the events that are passing through. The thing I like about this product is its ability to take in XML messages as events. You can use XPath to define the properties of those XML-based events and then reference those events and properties in your EQL expressions. This has a lot of applicability in Web Services and ESB-based systems where you have a lot of XML messages being passed around. This could probably be used to implement the supply chain ESB scenario that I wrote about in this article.

EDA, esper

Guard Your Data Services Carefully

2007-04-20T21:53:00.000-04:00

A common thing to do these days is to create data access web services directly on top of your databases to open up access to that data to other services and applications, essentially creating a data services layer for your SOA. I've even recommended such an approach before. While conceptually this may be a good idea, this also presents an increased security risk.

To be widely usable (or reusable), these data services tend to need to support a general querying interface. A lot of EII products will in fact let you create data services that will take XQuery to query the data source. This is in effect analogous to widely opening up a SQL interface into your database. There's a reason why direct connections to databases are not made widely available--either someone can issue malicious queries to the database or some inept developer can create some massive query that will bring the database to its knees.

Besides not widely opening up direct database connections, RDBMSes also have pretty fine-grained access control so that you can specify what users have what kind of access to specific tables and columns. The data services in your SOA will also need such fine-grained access control policies. In this case, they will be specifying access control on the specific operations and XML elements and attributes. The key is to be able to specify and enforce the policies down to the specific elements and attributes.

Ideally, this would be done through a tight integration between the service's policy enforcement point and the XQuery processor (assuming you're supporting XQuery on your data service) so that if your XQuery references attributes and elements you don't have access to, the query wouldn't even be processed. I don't know of any products that do this, but there are some other ways to do it.

Assuming the source of your data service is an RDBMS, you could set up an access control policy internal to the RDBMS that maps to the access control policy of the data service so that once the XQuery is translated to SQL and executed in the RDBMS you let the RDBMS's security mechanisms enforce the access control.

If maintaining two sets of mapped access control policies like this sounds like a pain in the ass, the other approach is to have a post processing step where you can strip out data from the XML using XSLT for the parts that the client does not have access to. With this approach, you need to make sure that you've designed the XML schema such that once you've stripped out those parts you still have a valid XML document. Also this approach may not be the most efficient--bringing back all the data as XML and then applying XSLT to filter stuff out. It would be much more efficient to do that in the RDBMS.

And then lastly, similar to how direct access to databases are not made widely available, you're going to have to be more restrictive about who you grant access to your data services. This will also prevent any Joe Schmoe developer in your enterprise from issuing some crazy XQuery that will bring down your data service and the database behind it. Such restrictiveness may be in direct conflict with one of your reasons for creating data services in the first place, but it's probably a necessary compromise.

Flickr Getting More Structured Tags

2007-01-29T22:35:00.000-05:00

Link to blog entry about it ...

AJAX Will Become Less Cool

2007-01-24T22:23:00.000-05:00

This article discusses the various AJAX toolkits that are available today. With the wide array of toolkits that are available, it's becoming easier to develop web applications that use AJAX. This will take away some of the coolness factor of AJAX since part of that stems from the black magic that's required in JavaScript programming. Especially with the server-side and declarative programming models of the toolkits listed in the article, the developer won't even be exposed to any JavaScript at all! I'm not so sure I like the toolkits with the declarative programming models--building your UIs using their custom XML tags. I'm sure it makes things really easy and will allow for more adoption of AJAX, etc., etc.--but then it makes building AJAX apps like building ColdFusion apps! And that ain't cool! (Apologies in advance to any ColdFusion developers still out there).

ajax

BPM a la Depeche Mode

2007-01-20T17:39:00.000-05:00

Here's a good article on BPM and how it's supported in ALBPM. Those of you who are Depeche Mode fans should find the article title and section headings familiar-sounding--very creative.

Business Process Management (BPM) For The Masses by Mariano Benitez -- Companies generate value by executing on business processes. In this article, Mariano Benitez explains just what business process management is, and how a process-oriented approach to building software solutions can be a powerful business-transforming paradigm.

bpm

SaaS is Good for the DoD

2006-11-24T01:25:00.000-05:00

The Department of Defense, especially the Defense Information Systems Agency is starting to move to a software as a service model for its computing needs. I see this as a good thing but not everybody in the industry is happy about this as evidenced in this article. Not surprising as this will definitely start to hurt their cash cows--big contracts for crappy applications that no one likes to use. I've been working in the public sector for a couple years now and just in those two short years I've seen a lot of this. From a taxpayer's perspective I'm glad to see that the government is starting to wise up. The SaaS model will force the contractors to create good software that people will want to use since the government won't have to pay unless they're actually using it. Obviously some of the contractors are afraid, but those of us who understand this new model see a lot of opportunities here. The SaaS model will still require a lot of good engineering work (though not actually doing the implementation) as well as a strong understanding of the business/domain. Someone's still gotta do this and we all know how understaffed the government is.

SaaS

SOA = Service Oriented Application

2006-11-22T20:21:00.000-05:00

I see a disturbing trend out there where people who are working on a single application say that they are creating an SOA. When they decompose the application into modules, instead of calling them subsystems or components as we use to, they call them services. Then they put a WSDL on them, register them in a registry, and bam they tell me they have a SOA. I guess that is appropriate if SOA to them means Service Oriented Application. I just hope they don't go around telling the business that now that they have an SOA everything's going to be much more agile, they're going to save a bunch of money from reuse, systems will be more interoperable, etc., etc. What are the chances that they're going to get any use from those services outside of that single application? Probably zero. So given that, how's the business going to be more agile? Beats me. Since those "services" and their WSDLs were designed from the perspective of that single application, what are the chances that they will be interoperable with others who might want to use them? Just because you're using WSDL/SOAP/HTTP doesn't guarantee you interoperability.

soa

OpenDocument and Microformats

2006-11-04T00:18:00.000-05:00

Microformats allow you to embed chunks of semantic metadata into XHTML documents using simple extensions to the existing tags. Some simple examples are embedded event and people information using the hCalendar and hCard microformats. What's nice about this is that now machines can accurately parse and extract key entities out of web pages that embed microformat metadata. Microformats are really starting to take off now on the web, especially with the Web 2.0 sites. OpenDocument is an OASIS XML standard for storing and exchanging office documents (i.e. reports, spreadsheets, presentations, etc.). Given that most of the information being generated by organizations today are such office-type documents, if the OpenDocument format really takes off, it would be nice to be able to embed semantic metadata into these types of documents using Microformats. A lot of organizations are employing content-management type of solutions to be able to better manage and use their corporate information. A lot of these types of solutions employ sophisticated text mining components to extract key entities out of such documents so that they can be better categorized and searched against. Imagine how much more accurate such text mining could be if there were Microformat metadata embedded in the office documents. Not only would it improve the accuracy of the text mining, but it would open up all that corporate information to a whole new world of interesting things that could be done to it as we are seeing with information on the web.

microformats, opendocument, metadata

Business Process Patterns

2006-10-07T00:59:00.000-04:00

In my work with various clients, I'm starting to take note of the common scenarios I'm seeing of how people are using SOA to improve their business processes. I'm sure others are starting to notice some recurring scenarios as well. By capturing these scenarios, we can develop a set of patterns that can be used to address them. What sets these apart from the other types of patterns we see in software such as design patterns or implementation patterns are that these are analysis patterns. Years ago, Martin Fowler wrote a book called Analysis Patterns which I still find to be a very useful book today. The patterns in that book however are finer-grained and more domain specific. What I am referring to are a broader set of analysis patterns that help you to analyze and decompose complex business processes which can be applied to any domain. The relevance to SOA of course is that business process analysis is one of the critical initial steps to any SOA initiative. Having such a set of common process analysis patterns will help to create a much more definitive SOA methodology which I think we are still lacking.

Here's one example of a recurring scenario that I see: Multiple Variant Processes to Perform a Common Function. In a large organization with multiple subsidiaries, there are often common functions that each of those subsidiaries perform but with their own slightly unique processes for doing so. For example, a lot of large organizations have subsidiaries that have their own unique processes for procurement. Often times the parent organization will want to integrate with the processes of the subsidiaries or it will want to integrate the subsidiaries’ processes together. So how do you go about analyzing all the subsidiaries’ processes to figure what/where to integrate? This can be pretty overwhelming if there are a lot of subsidiaries and the processes are complex.

An effective way to analyze the variant processes is to abstract a common process out of them. This is analogous to extracting a trend line from a scatter plot graph. In a scatter plot, trying to look at the individual points to understand and draw relationships out of the data is very difficult, thus, you generate a common trend line out of those points.

With process analysis, it is very easy to get caught up in all the details of the individual processes and lose sight of the relationships across the processes that are important for integration. Thus, you should abstract a common process out of the individual processes and use that common process to direct your analysis of the individual processes.

soa, patterns, bpm

XML Compression is Not the Answer to Your SOA Performance Woes

2006-09-27T20:43:00.000-04:00

Articles like this are misleading. XML compression will reduce the size of the message and thus reduce the time to transfer the message. But if you've ever done any performance testing and profiling on Web services, you will know that most of the times the performance bottlenecks are not in the message transfer but the XML processing, i.e. parsing the message. Thus, compressing the message will not do you much good. Some may say that in a bandwidth constrained environment, the compression does help. But if you're working in a bandwidth constrained environment, you probably shouldn't even be using such technologies in the first place. What does help tremendously is offloading the XML processing to dedicated XML appliances and the article does point that out.

soa, web services, xml

Are You Really Building an SOA?

2006-09-23T20:38:00.000-04:00

I was looking at the OASIS SOA RM's definition of SOA today. It defines it as "... a paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains". It appears that a key part of that definition is "under the control of different ownership domains".

So this begs the question, are you building an SOA if what you're building is within the scope of a single application? (If you want to get nitpicky and parse the definition then the answer would always be no because it defines it as an architectural paradigm and you can't "build" an architectural paradigm. But in this case I'm referring to SOA as an instantiation of a system designed using that architectural paradigm.) A lot of times you hear people talk about taking a single existing application and modernizing it to create an SOA. Are they really creating an SOA in such cases? Perhaps the question should be if you're not trying to organize and utilize distributed capabilities that are under the control of multiple ownership domains, then why are you building an SOA?

Are there other scenarios that may be valid? I think so.

A few years ago I was working for a company where we were creating a satellite command and control application. It was a CORBA-based system and we were following all the principles and design tenets of SOA such as loose-coupling, separation of concerns, interface-based development, etc., etc. We were also using the CORBA Naming Service, which can be considered a registry. There were a set of core satellite command and control services that were used by multiple client apps that made up this application suite. It was a relatively large distributed system that allowed an organization to control a fleet of satellites. Looks and smells like an SOA, doesn't it? But this was a shrinkwrapped product and we never designed it to support the notion that the core services would be under the control of multiple ownership domains ...

soa

When Should You Use an ESB?

2006-09-16T00:14:00.000-04:00

With all the hype around ESBs, a lot of people think that you need an ESB to build an SOA. That isn't always the case. I'm getting tired of seeing boilerplate SOA architecture diagrams with an ESB in the center. As with any other product, you should only use it if you require the functionality provided by it. Though there's no firm definition of what exactly is an ESB, you often see a core set of functionality that's provided by all ESB products. I talk about them in this posting. They are:

Protocol Adaptors
Data Transformation
Routing
Orchestration
Synchronous/Asynchronous Messaging
Endpoint virtualization

So when might you need such functionality? For example, if you're working in an environment with legacy systems and not all of them can support SOAP/HTTP. Or perhaps if you're building some type of automated workflow system or BPM. Or some event-driven system that requires asynchronous messaging and routing. If you just required some data transformation or endpoint virtualization I probably would not go with an ESB--it would be a bit overkill. Anyways, the point is look at your requirements and your environmental constraints first to see if you need the functionality that's provided by an ESB before you throw an ESB into your architecture.

soa, esb

Separation of Concerns in Web Services Article

2006-09-06T21:12:00.000-04:00

My article on separation of concerns in Web service implementations is now available on the O'Reilly site onjava.com. Here's the link.

A Couple Thoughts on SOA Governance Related Issues

2006-08-29T21:13:00.000-04:00

Most large organizations have various security audit and certification processes that applications have to go through before they can go into production. These can usually take several weeks to complete. The holy grail of SOA of course is to be able to quickly compose new applications by orchestrating a set of existing services. However, you kind of defeat the purpose if your new SOA allows you to compose new applications in a few days or a couple weeks but then you have to wait several more weeks for the security certification before you can put it into production and start using it. Thus these security processes need to become more agile as well in order to support this new approach to building applications. This just re-emphasizes the point that SOA is not just about the technologies...the organizational processes have to support it as well.

The second thought has to do with the concept of portfolio management. This is the practice of treating your IT systems and applications as a portfolio, similar to how you treat your portfolio of investments. You regularly review your IT portfolio to see how it's aligned with your business objectives and other criteria you've established and decide how to fund (or invest in) them accordingly. Those systems that don't meet your criteria get reduced funding or get cut altogether. I contend that this practice needs to be applied to the services in your SOA as well. Those services need to be treated as a portfolio that get reviewed regularly to see whether they're being used, how much they're being used, how much is it costing to maintain them, how much cost savings are you getting as a result of reusing them, etc., etc.--you define the criteria that's important to your organization. With all the rush by everybody to stand up Web services and build their SOAs, I'm sure that there'll be plenty of services that actually don't ever get used. You gotta weed those suckers out. Your registry needs to be kept up to date and relevant, otherwise if there's too much junk in there, your developers will avoid it like the plague. Bottom line--include a portfolio management process for the services in your SOA. Perhaps start by conducting your reviews annually and then adjust the frequency of such reviews based on the volatility of your organization.

soa, governance, portfolio management

AOP and SOA Implementations

2006-07-30T17:01:00.000-04:00

We are all familiar with the good old fashioned principle of separation of concerns and thanks to Thomas Erl we know how it's a fundamental principle behind Service Oriented Architectures. But we mostly think about separation of concerns at the architectural level. It's also a good idea to apply the separation of concerns at the implementation level, when coding the services that make up the SOA. This is where the techniques of Aspect Oriented Programming come in. While separation of concerns is a design principle, AOP allows you to carry it through to implementation. The Spring Framework is a nice lightweight Java framework with very good facilities for AOP. There's a companion security system for Spring called the Acegi Security System that allows you to secure your applications developed using Spring. Both of these frameworks provide very nice alternatives to JEE for developing Web services. I'm working on an article that shows you how to develop secure Web services using Apache Axis, Spring, and Acegi. Sure, there are plenty of ways to implement Web services but using Spring allows you to use its AOP facilities to drive the separation of concerns down into the implementation. Spring itself also has support for Web services, but since I'm more familiar with Axis, I decided to go with Axis for the example in the article. Acegi has some pretty nice security functionality but to be able to use it to secure Web services developed with Axis requires a bit of work to bridge the Web service security context from Axis into the Acegi security context. This is one of the things that I will show in the article. Stay tuned, I will post a link to the article once it becomes available.

soa, aop, spring framework, acegi, web services

BAM and SOA

2006-07-06T09:50:00.000-04:00

Saw some talk (from David Linthicum and David Black) recently in the blogosphere on doing business activity monitoring (BAM) and business intelligence (BI) within SOAs. This is an idea I've been keen on for a while now. Wrote an article about it for DM Review last August.

The opportunity lays in the fact that SOAs have a holistic view of the business processes that are executing within them, thus they can provide the context that ties together the data collected from BAM or BI to provide actionable information at a business level versus just giving you the raw data that your analysts have to try to manually tie together. Combine this with the fact that you have standard XML messages from which to harvest the data from and it becomes a very attractive opportunity that organizations with SOAs should definitely take advantage of.

In his entry, Black goes a bit further to talk about how ESBs can provide a good platform for this stuff, which is essentially what I was detailing in my article as well. According to his blog, Black, who recently joined Cape Clear, will be working to help them add BAM capabilities to their ESB. This will be a great feature that will distinguish them from their competitors. I'll definitely be keeping my eye out for this feature to see when it becomes available.

soa, bam, bi, esb

SOA Data Strategy Article

2006-06-15T18:19:00.000-04:00

Just a follow-up to an earlier post about SOA Data Strategy. The article we wrote is now available on the SOA Web Services Journal here.

soa

Java SDK for Web Services Security

2006-06-05T22:49:00.000-04:00

Came across this announcement from Forum Systems on the JDJ website about their release of a Java SDK for Web Services security:

Forum Systems Releases Java Web Services Security
— Forum released the Forum Java Web Services Security (JWSS) Software Development Kit (SDK) version 1.0 offering developers a comprehensive library of application programming interfaces (API's) to leverage in coding Web Services applications. The JWSS SDK v1.0 addresses the need for security to be enforced within the application itself in order to ensure privacy and integrity of Web Services and SOA applications.

Haven't had a chance to actually check out the SDK but there's definitely a need for those types of tools. Sun has their own set with the XWS Security that's part of the Java Web Services Developer Pack.

While I'm all for separation of concerns and not mixing security code in with the business logic code, there are definitely certain scenarios where you can't avoid it. Although I think they could have come up with a better example to justify when you may need to mix security logic in with your business logic. The example they used is to verify the origin of messages at the point of consumption to ensure that they were not physically intercepted. If that's what I needed to do, I would do that in a request interceptor that is deployed within the endpoint. Most web services toolkits allow you to set up request and response interceptors to do pre and post processing such as this. No need to mix this type of security code in with your business logic.

One scenario I have come across is when you have to make an authorization decision based on certain characteristics of the data that is being processed. In such cases, only the business logic understands the semantics of the data enough to make such a decision. So to do this you either have to put the security code into the business logic or you have to add business logic into your security code so that it understands the data semantics.

So the important questions for the guys at Forum Systems about this SDK are:

What standards/specifications is this SDK built on top of/compliant with?
What kind of interoperability tests have been conducted for Web Services that are secured using this SDK?

soa, web services security, java

Old School Guys Starting to Move into SaaS

2006-05-24T12:11:00.000-04:00

You can tell that SaaS is really taking off when you start to see the old school software guys starting to offer the capabilities of their products through on-demand services. This article in InformationWeek talks about Informatica (a long time vendor of ETL products) offering data integration as an on-demand service to integrate internal enterprise applications with other SaaS vendors such as salesforce.com. This won't be available until third quarter this year. Seems to me like they're a little late to the game. Companies like Bridgewerx have been offering such capabilities for a while now. It will interesting to see what kind of capabilities Informatica's services will offer. Given their ETL background, they'll probably have some pretty good data transformation and cleansing capabilities which could be a competive edge for them over some of the other companies offering this stuff.

SaaS, ETL,
data integration

Popularity of UDDI in Federal Agencies

2006-05-07T21:55:00.000-04:00

Not sure how popular UDDI is in the private sector with the Fortune companies (as I've been working in the government sector for almost a couple years now), but UDDI is alive and well in the federal government, especially in the Department of Defense. The DoD is so large and the amount of money they spend on this stuff is so much that even if they were the only ones using UDDI, that would be enough to sustain it for a very long time. The various Agencies and Services in the DoD already or are planning to have UDDI registries.

In fact so many different organizations are rolling out UDDI registries that they need to start looking at how to federate all these registries. To that regard, there's been a lot of talk that the UDDI specs don't adequately support federation. Admittedly, I'm no expert in UDDI federation but I did take a quick look at the UDDI 3.0 specs the other day and to me it looked like there was decent support for federation in there. You can create a federation by setting up a root registry and a bunch of afiliate registries that get blocks of keys allocated to them from the root registry so that there are no ID collisions in the registry. Then there are also the Replication and Subscription APIs. So if you don't like the affiliates model they have, then you can set up your own federation model and use these APIs to synchronize the data among all the instances. So what else is missing?

soa, uddi

Security Data Conflicts in Large Scale SOAs

2006-04-28T11:41:00.000-04:00

Most large scale SOAs will use some type of role-based or attribute-based security mechanisms to control access to the data and services. In most organizations, this will involve integrating multiple siloes of security domains containing user profiles, policies, access control lists, etc. Each system can in fact be viewed as a siloed security domain since most systems have their security models and infrastructures designed and developed independently of other systems. When you look at it, this is really just another integration scenario. Just as with other integration scenarios, you will run into conflicts with the data. Here, you're dealing with security data--such as attributes about the users (or principals) and attributes about the resources. For an organization that's using attribute-based access control, here are some examples of some of the types of conflicts that might be encountered:

The names of the attributes that describe the principals may not match across the different security domains. You'll probably have security policies that say something like "grant access to this resource if attribute ABC of the user equals XYZ". With multiple security domains, there's no guarantee that that attribute about a user will always be called ABC. If that's the case, how does the service deciding whether or not to grant access evalute the policy and make a decision?
The attributes that describe a principal may not exist in all domains. In other words, some domains may use more attributes to describe their principals and hence reference these attributes in their policies while in other domains principals aren't even described with such attributes. Using the example from #1, how can you evaluate the policy if the user doesnt' even have an attribute ABC?
The semantics of the attributes vary across different domains. This one's actually pretty common. There are a lot of access control rules out there that say "grant access to this resource if the user is an admin". But the concept of admin can vary greatly from one domain to another. For example you can have system admins and HR admins. Does that mean you should grant some system admin access to some HR service that deals with personnel files?

Some of these issues get into the areas of federated identity management and some of the standards out there such as SAML, WS-Federation, WS-Trust can help. Also there are products out there by companies such as Vordel and Layer 7 based on some of these standards that helps you to mediate some of the conflicts. If you have to deal with some of the issues I've described here, it's worthwhile to take a look at what they have to offer. But at the end of the day it will still require some coordinated effort to sync up all the participating parties so that there is some commonality in the security language used to describe the principals, resources, and access control rules.

soa, security, web services security, SAML, ws-trust, ws-federation

Would You Build an SOA Using CORBA Today?

2006-04-18T22:46:00.000-04:00

Had an interesting discussion with some colleagues today. If you were building an SOA today and didn't have to worry about any legacy issues, would you always go with web services or would you still consider CORBA in some cases? My position was that in some cases it may still make sense to go with CORBA instead of web services. For example if you didn't have wide interoperability requirements (a lot of external partners to integrate with) and performance is a major concern, then you may be better off using CORBA. I think the main advantage of web services over CORBA is the improved interoperability due to the use of XML and http, but if you're working strictly within the organization then perhaps you don't need all that. Maybe you go with a hybrid approach where within the organization the services use CORBA and any services that are exposed externally use web services. Bottom line is don't rule out CORBA just yet, there are many situations in which it still makes sense.

soa, CORBA, web services

XML Parser that Uses Pointers to Locations in the File

2006-04-17T09:57:00.000-04:00

Reminds me of something I did for Mercator back when I was working on their transformation engine that was part of their integration broker. Can't get into too much detail since that product's still in existence although under a different company and different product name. The idea is good when dealing with large XML files since you don't have to load the whole document into memory. I can certainly see how this can improve memory requirements but I'm a bit skeptical about the performance, especially if you have to do file i/o whenever you need to get some attribute or element. Take a look at the Xerces-C code. If it hasn't changed too drastically since I last looked at it (about 3 years ago), it's not that difficult to build some mods that hook into their scanner classes to do this as well. Maybe that way you can get the improved memory usage as well as conformance to the W3C XML specs.

XML

Excel--The End User Friendly Rich Client for Web Services?

2006-04-13T08:53:00.000-04:00

StrikeIron recently announced the ability to integrate web services from their Business Network into Excel. Web services have been lacking an end user-friendly client for a while. Will this announcement from StrikeIron push Excel to fill that gap? We all know that Excel has always been the tool of choice for business analysts. Now that they can directly tap into reusable web services from Excel, maybe we'll really start to see the use of web services as true business services that are reusable by people on the business side instead of just developers.

web services, SaaS, rich clients

Using an ESB to Connect the Enterprise to Mobile Users

2006-04-12T12:40:00.000-04:00

Years ago, I worked for a company that built products for mobile and wireless applications. We had a platform for developing systems that connected the enterprise applications with users in the field through their handhelds, PDA's, smartphones, etc. I was thinking about it the other day and it struck me how similar that platform was to what we call ESBs today. Thus, it struck me that besides being SOA infrastructure, ESBs are also good for extending the enterprise to the edge--the mobile users. Here are some reasons why:

Enterprise apps are available as a set of loosely-coupled services that can be selectively exposed to the mobile devices. This is not a direct result of using an ESB, but if an organization is using an ESB, then they are most likely building (or already have) an SOA--so their enterprise apps will be available as a set of loosely-coupled services.

The asynchronous messaging, guaranteed messaging, and store and forward capabilities of most ESBs are perfect for occasionally connected devices.

The data transformation capabilities of ESBs can be used to transform data for optimized presentation on the devices. For example, converting from XML, HTML to WML.

The protocol mediation capabilities allow for converting to/from different protocols supported by the devices, e.g. WAP-->HTTP-->SOAP. Note some type of gateway will still be needed to convert between the various wireless protocols such as GSM, CDMA, CDPD, Bluetooth, etc. to the TCP/IP based protocols that most of the enterprise apps are built on top of. But once all that's in place, it will abstract complexities of wireless communications protocols from the enterprise developers, and vice-versa--developers of the mobile apps don't have to deal with the enteprise issues.

Here's a rough sketch of what it may look like.

ESB, mobile computing

SOA Data Strategy

2006-04-10T22:18:00.000-04:00

Typically when organizations are thinking about migrating to an SOA, the data environment is not one of the top things on their mind. They are more focused on modeling their business processes and figuring out how to create reusable services out of them. That is rather unfortunate because the data plays a big part in driving the proper execution of those business processes. Additionally, the data environment of the typical large organization has a lot of issues that will only become more problematic in an SOA environment due to the more real-time and integrated nature of SOA-based applications. In some cases, trying to move to an SOA without first addressing the data issues is akin to trying to build a high-rise on top of a landfill--you're trying to build a complex structure on top of a weak foundation.

A couple of colleagues and I are writing an article for the June issue of SOA Web Services Journal that talks about creating a data strategy for an SOA to ensure that you have a proper data environment before jumping into your SOA transformation. This strategy addresses issues such as data governance, enterprise data models and standards, security, data quality, and data services. Keep an eye out for it. There's a lot of useful information in there. A lot of it is based on experiences from projects our firm's done with the large government agencies. I will post a link to the article here when they make it available online.

SOA, data integration

Where are the Coads, Fowlers, Rumbaughs, Boochs, Jacobsons of the SOA world?

2006-04-07T16:47:00.000-04:00

Back in the 90's when someone wanted to be a better OO developer, they would pick up some books from these fellows. With SOA being all the hype these days, developers need similar guidance on how to do good Service Oriented Analysis and Design (SOAD). As SOA is relatively immature (though some claim they’ve been doing it for years), there is less guidance and accepted principles on how to come up with good service models (analogous to object models in OO) such that they are loosely-coupled, reusable, agile, etc. Really, the only advice that people give today is to make services coarse-grained. In the SOA world, there is currently a lack of commonly-accepted structured methodologies that prescribe the proper approaches for coming up with good service models. IBM has started to do some work in this area but they themselves state this is just a beginning and there is much work that still needs to be done. Another interesting area to keep an eye on is the upcoming book by Lublinsky and Manolescu on SOA patterns.

SOA, object-oriented

How Robust is Your ESB?

2006-04-07T15:31:00.000-04:00

Organizations are increasingly looking at using an Enterprise Service Bus as the technical foundation for building SOAs these days. With the ESB playing such a critical role in your SOA infrastructure, it is important to understand the characteristics by which to measure the robustness of your ESB. There are various definitions of an ESB out in industry, so the best way to look at them is by the capabilities that people commonly think of when they refer to an ESB.

Although the functionality of an ESB can vary greatly from vendor to vendor, you will find that common to all of them is a core set of functionality that includes the following:

Protocol Adaptors
Transformation
Routing
Orchestration
Synchronous/Asynchronous Messaging
Endpoint virtualization

Based this reference set of capabilities and an understanding of how this type of functionality is typically implemented, one can get an idea of where there may be potential performance bottlenecks, fragile points, and other robustness issues that can plague an ESB. Knowing what these are can help you to make a more informed decision when choosing your ESB vendor and product.

Protocol Adapters
One of the core functionalities of ESBs is protocol adaptation--that is connecting services that run on one type of protocol to services that run on another type of protocol. For example, adapting from SOAP to JMS, FTP to SOAP, SOAP to IIOP, etc. ESBs that are built on top of a messaging-oriented middleware will typically use adapters that connect different protocols into the messaging bus. This is similar to how some products accomplish data transformation using a canonical data model but here the ESBs use a canonical protocol internally based on messaging. Adapters then just transform other protocols to/from this format, while internally components communicate using this normalized protocol. Other ESBs that are not built on top of a message-oriented middleware will typically use a series of interceptors, similar to a pipes and filters architecture in which messages flow through a sequence of interceptors. Usually this type of architecture will perform both protocol adaptation and data transformation as the messages flow through the interceptors.

Transformation
Data transformation is very resource intensive. All the parsing that is required makes it very CPU intensive. The resulting parse trees are loaded into memory for manipulation, making it also very memory intensive. For very large datasets, not everything can be loaded into memory, making the operations also very I/O intensive. Because of these characteristics, it is very important to look for an ESB with a high quality transformation engine. The transformation engine can be evaluated using different types of datasets and transformation. For example, evaluate how it handles large, complex source and target schemas. How does it handle complex transformations? How does it handle large datasets? Run tests using these scenarios and observe CPU, memory, and I/O usage for the transformation engine.

Routing
Services communicate with each other by placing messages onto the bus and the ESB determines how to route those messages to the correct destinations. There are a couple different ways in which this functionality may be implemement. Some implementations may use a generic set of dispatchers that listen on a set of generic incoming channels and dispatches the messages to their destinations. Another approach uses specific channels for the different services. With this approach the service consumer just places the message onto the specific channel of its intended service and a dispatcher specific to that service will dispatch the message to the destination.

Some implementations route messages by having different queues for different destinations. Observe the CPU and memory usage of the message bus processes to see if the message router is saturating the message bus. What is the performance of message router for different types of routes, multiple hops, etc.? Parsing of the message itinerary can also be performance intensive. Message routers often use multiple threads to listen to incoming messages and route them to their destinations.

Orchestration
Orchestration allows developers to hook together multiple services using complex logic to form a larger integrated process. Most ESBs provide orchestration capabilities based on some type of BPEL (or equivalent) orchestration engine. Orchestration is a complex capability and there are many issues you need to consider when evaluating an ESB’s orchestration capabilities. Does it pre-compile the orchestration instructions to improve performance or does it process the instructions on the fly? If the instructions are pre-compiled, then it will be more difficult to dynamically update the process to respond to new events. On the other hand, if the instructions are processed on the fly, then there may be a performance hit due to that. A process is typically long running and thus maintains state and context. This means that the orchestration engine is usually a stateful service. Being stateful makes it more difficult to distribute for failover and scalability. For example, if the ESB containing the orchestration engine in which the process is running fails, how will it migrate that process and its state to a backup instance? The statefulness also creates an affinity with a particular ESB instance which means the messages have to flow through that particular instance—making load distribution across multiple instances difficult, if not impossible.
Keep an eye on the memory usage of the orchestration engine since a lot of implementations build an entire object graph of the process. How many processes can it run concurrently?

Synchronous/Asynchronous Messaging
The messaging backbone is what allows the highly distributed nature of an ESB. One of the results of this is that the capabilities of the ESB can then be distributed across multiple nodes--one of the primary distinctions between an ESB and some of the more traditional EAI hubs. Some of the vendors whose products don't have a messaging backbone will argue that one is not necessary to achieve this highly distributed characteristic of ESBs. In any case, regardless of how the capability is implemented, most ESBs support highly distributed synchronous and asynchronous messaging. Thus, it is important to understand what characteristics should be of concern when evaluating this capability. Important things to consider--throughput of the messaging, size of the messages, latency of message transfer, reliability of message delivery. Observe how these metrics vary across different messaging patterns:

Small, chatty messages
Large, bursty messages
How many inbound messages can it handle
How many outbound messages can it handle

Endpoint Virtualization
Clients are abstracted away from the actual physical network location of a service. They invoke services by using logical addresses for the service endpoints provided by the ESB. This allows for such things as multiple instances of that service for load-balancing and failover. One of the concerns here is whether or not a particular ESB's implementation of endpoint virtualization is interoperable with the WS-Addressing standards.

SOA, ESB