How to configure a URL Alias for a REST Resource in webMethods Integration Server

After I created my first REST Resource in webMethods Integration Server, I immediately saw potential for improvement regarding the URL under which it was made accessible:

http://localhost:5555/rest/ExamplePackage/Resources/Session

The default URL always starts with rest followed by the fully qualified name of the REST Resource. What bugs me is the latter: the path exposes the internal package structure to the outside. And if – for whatever reason – we decide to change this structure later, all clients need to be changed accordingly. And due to REST’s loose coupling you may not be aware of all consumers of your service and only find out late in the process that you missed to update a caller.

So, what we want is a logical URL instead of a physical one that represents our package structure, e.g.:

http://localhost:5555/User/Session

This logical URL should never change due to an internal restructuring.

URL Aliases

Integration Server provides an easy way of creating such logical URLs: URL Aliases.

You can find the dialog for creating URL Aliases in Integration Server’s web interface under Settings -> URL Aliases:

Creating a URL Alias in webMethods Integration Server

  • Alias
    This is the logical URL you want to define for your REST resource, e.g. User/Session.
  • URL Path
    The Alias above gets expanded to this fully qualified path, e.g. rest/ExamplePackage/Resources/Session.
  • Package
    The Alias will be saved to this package. It resides in an XML file called config/urlalias.cnf in the package’s folder in the file system and can be versioned along with the rest of the package’s contents.

Now I can call the REST Resource via the URL Alias:

curl^
      -u Administrator:manage^
      -X POST^
      -H "Accept: application/json"^
      -H "Content-Type: application/json"^
      -d "{\"Username\":\"Stefan\",\"Password\":\"secr23et\"}"^
      http://localhost:5555/User/Session

Calling a REST Resource via a URL Alias in webMethods Integration Server

Partial Matching

As stated in the REST Developer’s Guide, you may need to activate Partial Matching for your URL Aliases to work. See Integration Server Administrator’s Guide for details.

All you need to do is add this Extended Setting and restart IS:

watt.server.url.alias.partialMatching=true

How to create a REST service in webMethods Integration Server

A while ago I created my first REST service in webMethods Integration Server. It’s quite easy to do and works correctly out of the box. Software AG managed to implement the standards quite well and I didn’t have to go through hours of configuring (like I had to with SOAP webservices).

How to create a REST service in webMethods Integration Server

First of all, you don’t actually create a REST service, but a REST resource. A resource represents a business entity for wich you can call the basic CRUD operations through (multiple) services, that respond to the common HTTP methods (GET, POST, PUT, DELETE). So, a REST resource uses multiple Flow services that implement the logic. These services get called automatically by IS, when the corresponding HTTP call is made by the client.

My example implements a common scenario: Authenticating a user based on its username and password.

The first – and most important – question was, how to name the REST resource. I couldn’t name it User, as one might think at first, because this name doesn’t fit the notion of REST: I don’t create or read a user. The user already has to be there and simply gets authenticated. In a SOAP context I would create a Flow service like AuthenticateUser with Username and Password as parameters. But in REST we can only use the basic CRUD operations. And there is no HTTP method called AUTHENTICATE. So I had to find another name for the resource.

What I ended up with was the concept of a Session. When a user gets authenticated, a Session is created for him. When he logs off, the Session gets destroyed. I can also read the Session to find out, whether the user is already logged in. So it all seemed to make sense now.

After the nomenclature problem is solved, here’s a quick step-by-step guide on how I implemented the REST resource in IS.

Step-by-step guide

  • Create a Flow service that implements the logic you need.
    In my case, I created AuthenticateUser as described above (see Best practices for structuring packages in webMethods Integration Server for details on the package’s folder structure).
    Flow service AuthenticateUser
    As the REST resource is merely an additional access layer on top of existing logic (like a Web Service Descriptor), it’s good practice to keep the two concerns – business logic and access logic – separated. Another advantage of this approach is the fact, that you can simply re-use your existing Flow services and only have to add the REST part on top of it.
  • Make sure the Flow service works as expected.
    This step might seem obvious, but if you introduce another layer of complexity in your system, you better make sure the underlying parts already work correctly. Otherwise, the search for potential errors might take very long and you can’t be sure, if it’s really the REST part that goes wrong.
    Successful result of a call to AuthenticateUser
  • Create a REST Resource.
    This is a easy as selecting New -> REST Resource from Designer’s context menu, giving the resource a name and selecting the needed HTTP methods.
    Creating a REST Resource in webMethods Integration Server - Step 1
    Creating a REST Resource in webMethods Integration Server - Step 2
    Here, I used POST, because I want to create a new user session. If you are not exactly sure, which HTTP methods to use, here’s a quick mapping: GET -> read, POST -> create, PUT -> update, DELETE -> delete. Also see Using HTTP Methods for RESTful Services and When should we use PUT and when should we use POST?.
    Creating a REST Resource in webMethods Integration Server - Step 3
    Voilá, we created our first “REST Service”! 🙂
    Creating a REST Resource in webMethods Integration Server - Step 4
  • Define the parameters for the REST services.
    The newly created service _post has no parameters yet (aside from two technical parameters $resourceID and $path, see REST Developer’s Guide for an explanation of those). So you need to add them.
    Adding paramters to the REST service
    In my case, I simply used the same parameters from AuthenticateUser. But they also can be different.
  • Implement the REST service.
    Implementing the _post service is quite simple, because the main logic is already implemented in AuthenticateUser. Therefore, _post can simply call AuthenticateUser and only needs to take care of REST specialties, like setting a meaningful response code. Otherwise, IS returns 200 (see List of HTTP status codes for additional information) even if the authentication was not successful. Fortunately, there is an HTTP code that describes almost exactly what I need: 401 Unauthorized. You can set the response code with the IS service pub.flow:setResponseCode.
    Implementation of the _post service
  • Test the REST service.
    You can now call the _post service directly from Designer and make sure it works.
    Call _post
    Successful response from _post
    Unsuccessful response from _post
  • Test the REST service from outside IS.
    After you made sure, that the _post service works as expected, you can try and call it from outside IS. My tool of choice for this task is curl (which comes with the installation of Git for Windows, by the way). All REST resources are available in IS under the following URL without any further configuration:

    http://server:port/rest/FullyQualifiedResource
    

    Or translated to my example:

    http://localhost:5555/rest/ExamplePackage/Resources/Session
    

    You can call the REST service with this curl command on Windows (on Linux you need to replace ^ with \, see Windows: How to specify multiline command on command prompt?):

    curl^
      -u Administrator:manage^
      -X POST^
      -H "Accept: application/json"^
      -H "Content-Type: application/json"^
      -d "{\"Username\":\"Stefan\",\"Password\":\"secret\"}"^
      http://localhost:5555/rest/ExamplePackage/Resources/Session
    

    IS automatically unwraps the JSON payload and converts it to IS documents. You only need to provide the same document structure and field names as defined in your _post parameters.
    Result of curl call to _post service
    By the way: I had to use double quotes for the JSON payload and needed to escape the quotes inside the JSON data to make it work on Windows. When I used single quotes (e.g. -d '{"Username":"Stefan","Password":"secret"}'), I got this nice error:

    org.codehaus.jackson.JsonParseException:Unexpected character (''' (code 39)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: com.wm.net.HttpInputStream@74356f93; line: 1, column: 2]
    

And this leaves you with a fully functional REST service in webMethods Integration Server! 🙂

Official Documentation

These documents helped me along the way:

Recommended reading

For a very good introduction to REST I recommend the Pluralsight course REST Fundamentals:


REST Fundamentals – $29.00

REST is an overloaded, and thus misunderstood term in architectural circles these days. This course attempts to clear up some of the misunderstandings about REST as well as provide a more practical approach for designing RESTful solutions – both clients and services. Additionally, the course looks at REST from the perspective of the cloud and describes how REST is well-suited to meet the demands that the cloud brings to bear on a modern architecture.

Best practices for structuring packages in webMethods Integration Server

When I started out implementing services in webMethods Integration Server, I didn’t really know how to structure my packages. I lacked the experience in what should go where and which resources are best grouped together. So I ended up with a different folder structure for all my packages. At this point, finding the resource I wanted to edit was very cumbersome.

So I asked a Software AG consultant for his recommendation regarding the folder structure of my packages. And this is what we came up with:

Best practice for webMethods Integration Server package structure

  • PackageTemplate
    • Adapters
      • EntireX
      • JDBC
      • NET
    • Documents
    • Internal
      • Documents
      • Services
    • Resources
    • Schemas
    • Services
    • Triggers
    • Webservices

I’ve used this structure for quite some time now and I’m pretty happy with it so far. Let me go into the details on the different folders.

Root Folder

The root folder should be named exactly like the package (hence PackageTemplate). Although this might seem like redundancy, you need to be aware of the fact, that the package name is not part of the service namespace! To be able to uniquely identify a service, you need some kind of fully qualified name. And the package name is not part of that. So it’s best practice to create a folder with the same name as the package as the root folder for the package contents. You can then qualify a service like this: PackageTemplate.MyService.

Public API

The folders Services and Documents represent the package’s public API, i.e. the resources that are publicly available to every other package and every external consumer. Think of them like public attributes and methods in Java. This API needs to be stable, because changing it is expensive and error-prone. Once you publish a service, you’ll never know who calls it. Therefore, changing the interface later may lead to errors in the consumers.

Private API

The folder Internal contains the private API of the package. This API can be modified as you like, because there are no external dependencies on it. Think of the services and documents in their respective subfolders als private methods and attributes in Java. You can hide the internals of your package by putting them in this structure.

This does not mean, that other packages or consumers are not able to use these resources. They are not technically hidden from the outside. It’s merely a convention that every package has to follow and every developer needs to respect (and every architect needs to enforce 😉 ).

Webservices and REST Resources

Webservices (SOAP and REST) form an additional layer on top of a package’s documents and services. They only provide another way of accessing the resources through common interfaces. And they might look quite different from the underlying services (e.g. a POST-Method might get translated to a simple service call) and follow their own lifecycle. Therefore, they should be put into their own folders. Webservices contains only Web Service Descriptors and all REST Resources and their services (_get etc.) are put into Resources.

Schemas and Triggers

The folders Schemas and Triggers contain their respective resources. Schemas can define the data structures for multiple services and Triggers implement asynchronous sequences. These resources play a central role in many packages and need to be taken care of separately.

Adapters

The folder Adapters contains the package’s different adapter services. For example, in our case there are adapter services for EntireX, JDBC and .NET. And they are all quite different from your Flow services, as they need additional Integration Server adapters installed and integrate external systems into Integration Server. It’s useful to separate those special services from your own.


What best practices do you follow when structuring your Integration Server packages? Do you have anything to add to my list?

Avoid Redundancy! – Lessons Learned from SOA-fying a Monolith

One of the two main lessons I learned while SOA-fying our monolithic application was:

Avoid redundancy.

Avoid redundancy

We started out implementing the data structures for our services by hand. This means, I had to create the same data structure over and over again in different systems. For example, a simple Webservice for reading a person from our system led to at least three different implementations of the same data:

  • An Integration Server document
  • A Java class
  • A Natural Parameter Data Area

You can see an example of these three data definitions below.

Redundant data definitions

Perhaps you can imagine how much fun I had when a parameter needed to be added or changed. Especially in Integration Server changing an interface literally hurts due to the hundreds of lines – the simple person example above already contains about 70 attributes – I had to draw. Often I even had to draw these lines multiple times due to transformers that needed to be added between the mappings (e.g. Natural does not have the concept of null, so I had to transform empty String fields to null).

DSL to the rescue!

What we ended up doing was to implement a Domain Specific Language for defining our data model. You can see an example for a person below.

Domain Specific Language for data models

Based on this single source of truth now all the other representations of the data model get generated automatically. For example, there’s a code generator for Natural PDAs, Java classes, IS documents, XML files for our output management system etc. If a system gets added to our infrastructure and needs to represent the data in a special way, we simply need to add another code generator to the list.

By the way, we use Xtext and Xtend for all our DSLs. If you’re interested in the topic and understand German, you can take a look at the project documentation of my former apprentice here: Projektdokumentation von Markus Amshove (mit 100% bewertet).

To sum up: If you do the same thing over and over again, lean back and think about it for a minute. Do you really need to model the same data in different systems? Or could there be a way to automate the process or even generate the needed artifacts? If there is, follow the DRY principle and implement it.

SOAP is dead – Lessons Learned from SOA-fying a Monolith

I’ll continue my series of blog posts regarding the lessons we learned while SOA-fying our monolithic Adabas/Natural application with a more technical lesson:

SOAP is dead.

SOAP is dead

This may be a harsh statement, taking into account that we started out with Webservices based on SOAP and at the moment our whole infrastructure is based on it. However, while we were still searching for the right solution to the communication problem two major programming languages stopped supporting SOAP out of the box: Groovy and Ruby. And we used both of them.

If you take a closer look at the current notions in architecture and distributed systems, you’ll find plenty of conference talks and frameworks for providing and consuming REST services. However, SOAP seems to be legacy technology already.

To be honest, we don’t have a single REST service in production, yet. But after working with SOAP Webservices for quite some time now, I can definitely see the advantages of a more loosely coupled approach like REST. The interface design for SOAP services can take quite a lot of time, because you have to define individual operations and data types. And changing the interface – e.g. adding a parameter – leads to a required re-build of all systems using it. With REST you only need simple CRUD operations and can use JSON as a very flexible data format that can be added to later on.

I’m glad that Integration Server provides an easy means to publish REST services. You can even put them on top of existing services and simply provide a more flexible interface to the consumers. This is definitely a solution I want to try out in the near future.

So, if you start with an SOA today, take a closer look at REST and don’t blindly use the default implementation SOAP as it may not be supported any more in the near future.

Test everything! – Lessons Learned from SOA-fying a Monolith

Another lesson we learned while making our legacy application ready for a service-oriented architecture, is this:

Test everything.

Test everything

When I started out writing Flow services in webMethods Integration Server (IS), there was no (nice) way of automatically testing them. Although we were told multiple times by consultants, that there would be a test framework for IS, we never got the actual code. So, I had to develop a test framework myself, simply to be sure that everything still worked as before after a deployment of IS.

The result of my development effort are two small Java frameworks:

I wanted to be able to test IS services (Flow, Java, etc.) with the established tool we already used in our projects: JUnit. However, Integration Server’s Java interface relies on IData, the internal representation of IS’s pipeline. And working with it can get pretty annoying, because it’s nothing more than a big hash table with its own API. So you would have to deconstruct your Java objects into the structure of IData every time you call a service and compose them back together when you get the results, only to be able to call an assertion on it.

ao-idata-converter

My solution for this problem is a project called ao-idata-converter. It takes any Plain Old Java Object (POJO) and converts it to IData with a bit of reflection magic. You can even use beans with Getters and Setters and map attribute names to different fields in the pipeline. So, with ao-idata-converter you can get rid of all the converting from and to IData in your code.

IData convertedObject =
    new ObjectConverter().convertToIData("address", addressObject);

ao-integrationserver

The next problem I faced, was the need to use lots of boilerplate code to be able to call an IS service. If you generate a Java client for a given IS service, the resulting class contains all the setup, authentication, input etc. you need to call the service. However, it’s not reusable and you’ll end up with lots of duplication if you want to call multiple services (which would be the default behaviour, I guess).

Therefore, I abstracted all the needed boilerplate code into a separate framework, ao-integrationserver, that provides an easy API for calling IS services on different endpoints with authentication and input/output parameters represented by simple POJOs. If you follow a certain naming convention for your Java packages and classes, you’ll be able to call an IS service by creating a single class with only a few lines of code. So, adding a new service to your Java library takes only a few minutes at most.

public class max extends Service<max.Input, max.Output>
{
    public static class Input extends ServiceInput
    {
        public String[] numList;
    }

    public static class Output extends ServiceOutput
    {
        public String maxValue;
    }
}

Our test suite

Below you can see a screenshot of our IS test suite in Eclipse. The test suite automatically runs on our Jenkins build server every time we deploy IS and we can point it to any stage (development, test, production) within a matter of seconds to make sure that all services still work as expected.

Screenshot of our test suite for Integration Server

If you would like to know more about the two frameworks or our deployment pipeline, feel free to contact me any time. If you would like to participate in the development of the frameworks, I would also love to hear from you (e.g. via a Pull Request).

Make use of Diversity – Lessons Learned from SOA-fying a Monolith

One of the lessons we learned while SOA-fying our legacy application (an Adabas/Natural monolith, that is almost 20 years old) is:

Make use of diversity.

Make use of diversity!

When we designed the central interface for our service modules, we made sure that developers with different backgrounds worked together on the design team. For example, we had older and younger developers work together with Natural and Java developers. They all worked towards the same goal: creating an interface that all of our systems could use to communicate with each other.

We put together ideas from the “old” world – procedural data processing – and ideas from the “new” world – object and service orientation – and merged them into a flexible interface that combines the best concepts of both worlds. The more experienced developers brought their domain knowledge to the table and the younger developers added their new ways of thinking about the technical problems.

What we ended up with was an interface that could easily be provided and consumed from all the platforms we use. Because we put the interface to the test right away and made sure that problems would become visible quite quickly, we only needed a few iterations before the final design was ready.

And every developer that was involved in the process knew the interface already and could actively promote its usage to the rest of the team. And each of those deveopers could use the language they were already familiar with and didn’t have to teach new concepts – which isn’t that easy for both Natural and Java developers by the way.

So, my advice to you, if you plan to design a central interface, data model, or business process, is: Make sure to integrate as many different views and backgrounds as possible. The coordination might take a bit longer (at first), but you’ll probably end up with a much better solution.

If you make use of the individual strengths of your developers, the end result will be a well rounded solution.

SOA-fying a Monolith – Innovation World 2015

My talk for Software AG’s Innovation World 2015 in Las Vegas got accepted and is already visible on the agenda: Lessons Learned from SOA-fying a Monolithic Legacy Application.

Logo Innovation World 2015

How do you modernize a monolithic legacy application to meet the requirements of today’s service-oriented world? In this talk, Stefan Macke shares his insights from SOA-fying a 15-year-old Adabas & Natural insurance application with the help of webMethods Integration Server, a bunch of unit tests and a domain-specific language for creating a canonical data model. He presents technical, architectural as well as organizational lessons learned from the modernization project.

How to test JMS processing in webMethods/Terracotta Universal Messaging and Integration Server with SoapUI and HermesJMS

Universal Messaging is the default messaging component used by webMethods Integration Server. Here is a short tutorial how you can test JMS processing using SoapUI and HermesJMS.

  • SOAPUIDIR points to the installation directory of SoapUI, e.g. C:\Program Files\SmartBear\SoapUI-5.1.3.
  • NIRVANADIR points to the installation directory of Nirvana, e.g. C:\SoftwareAG\nirvana.

Setup Universal Messaging

  • First of all you need to create the needed artifacts in your Universal Messaging realm. Start with the JNDI Provider URL and click Apply.
  • Then add a Connection Factory, Topic Connection Factory, and Topic: Create artifacts in Universal Messaging

Setup Integration Server

  • Create a JNDI Provider Alias for Universal Messaging under Settings > Messaging > JNDI Settings: Create JNDI Alias in Integration Server
  • You can now test the alias and should see the artifacts you created in Universal Messaging: Test JNDI Alias in Integration Server
  • Create a JMS Connection Alias for the JNDI Alias under Settings > Messaging > JMS Settings. Use the corresponding values from Universal Messaging for JNDI Provider Alias Name and Connection Factory Lookup Name: Create JMS Connection Alias in Integration Server
  • Enable the Connection Alias: Test JMS Connection Alias in Integration Server

Setup SoapUI/HermesJMS

  • Copy the following JARs to SOAPUIDIR\hermesJMS\lib: NIRVANADIR\lib\jndi.jar, NIRVANADIR\lib\nClient.jar, NIRVANADIR\lib\nJ2EE.jar, NIRVANADIR\lib\nJMS.jar.
  • Create a new session named IS and add all above JARs to a new classpath group named IS: Add provider JARs for Universal Messaging to HermesJMS
    Click Apply and restart HermesJMS.
  • You should now be able to select IS under Loader and hermes.JNDIConnectionFactory under Class: Configure session in HermesJMS
    Add the properties host, port, initialContextFactory, providerURL, and binding under Connection Factory and configure them according to your environment. You can find the needed values in the JNDI settings of Universal Messaging: JNDI properties in HermesJMS
  • You should now be able to discover queues and topics for the session: Discover queues and topics with HermesJMS
  • To test the subscription to a topic, you can now browse the topic: Browse topic with HermesJMS
  • If you send a test message with Software AG Designer, you should see the message in HermesJMS: Send a JMS test message with Integration Server

    Browse the JMS messages in HermesJMS

Possible errors

  • hermes.HermesException: The binding property to locate the ConnectionFactory in the Context is not set: hermes.HermesException: The binding property to locate the ConnectionFactory in the Context is not set
    Add the property binding under Connection Factory in the session preferences (right-click on the session and Edit).

Additional Links

How to create a new webMethods Integration Server instance

In newer versions of the webMethods suite, you can install multiple instances of Integration Server into a single installation. Some central packages will be re-used in every instance. However, administration of the instances may be a bit harder, as these packages have to be updated manually, e.g. in case of a version update or fix installation.

Here is how to setup a new Integration Server instance (ISDIR points to the Integration Server directory, e.g. C:\SoftwareAG\IntegrationServer):

  1. Run ISDIR\instances\is_instance.bat create -Dinstance.name=testIS1 -Dprimary.port=5550 -Ddiagnostic.port=9990 -Djmx.port=8077 -Dlicense.file=C:\license.xml
    This will start an Ant build that creates the new IS instance under instances\testIS1. Of course, you may need to adjust the parameters according to your needs.
  2. If the build finishes successfully, the instance can be started with ISDIR\instances\testIS1\bin\startup.bat. You should now be able to connect to localhost:5550 and see the administration page for your new instance:
    Administration page for the newly created Integration Server instance
  3. If you want to install the new instance as a Windows service, you can run ISDIR\instances\testIS1\support\win32\installSvc.bat:
    Create a new service for the newly created Integration Server instance
    You should now see another IS service:
    The service for the the newly created Integration Server instance
  4. You should also see the new instance in Command Central. A refresh or a restart of the platform manager may be needed.
    The newly created Integration Server instance in Command Central