Avoid Redundancy! – Lessons Learned from SOA-fying a Monolith

One of the two main lessons I learned while SOA-fying our monolithic application was:

Avoid redundancy.

Avoid redundancy

We started out implementing the data structures for our services by hand. This means, I had to create the same data structure over and over again in different systems. For example, a simple Webservice for reading a person from our system led to at least three different implementations of the same data:

  • An Integration Server document
  • A Java class
  • A Natural Parameter Data Area

You can see an example of these three data definitions below.

Redundant data definitions

Perhaps you can imagine how much fun I had when a parameter needed to be added or changed. Especially in Integration Server changing an interface literally hurts due to the hundreds of lines – the simple person example above already contains about 70 attributes – I had to draw. Often I even had to draw these lines multiple times due to transformers that needed to be added between the mappings (e.g. Natural does not have the concept of null, so I had to transform empty String fields to null).

DSL to the rescue!

What we ended up doing was to implement a Domain Specific Language for defining our data model. You can see an example for a person below.

Domain Specific Language for data models

Based on this single source of truth now all the other representations of the data model get generated automatically. For example, there’s a code generator for Natural PDAs, Java classes, IS documents, XML files for our output management system etc. If a system gets added to our infrastructure and needs to represent the data in a special way, we simply need to add another code generator to the list.

By the way, we use Xtext and Xtend for all our DSLs. If you’re interested in the topic and understand German, you can take a look at the project documentation of my former apprentice here: Projektdokumentation von Markus Amshove (mit 100% bewertet).

To sum up: If you do the same thing over and over again, lean back and think about it for a minute. Do you really need to model the same data in different systems? Or could there be a way to automate the process or even generate the needed artifacts? If there is, follow the DRY principle and implement it.

SOAP is dead – Lessons Learned from SOA-fying a Monolith

I’ll continue my series of blog posts regarding the lessons we learned while SOA-fying our monolithic Adabas/Natural application with a more technical lesson:

SOAP is dead.

SOAP is dead

This may be a harsh statement, taking into account that we started out with Webservices based on SOAP and at the moment our whole infrastructure is based on it. However, while we were still searching for the right solution to the communication problem two major programming languages stopped supporting SOAP out of the box: Groovy and Ruby. And we used both of them.

If you take a closer look at the current notions in architecture and distributed systems, you’ll find plenty of conference talks and frameworks for providing and consuming REST services. However, SOAP seems to be legacy technology already.

To be honest, we don’t have a single REST service in production, yet. But after working with SOAP Webservices for quite some time now, I can definitely see the advantages of a more loosely coupled approach like REST. The interface design for SOAP services can take quite a lot of time, because you have to define individual operations and data types. And changing the interface – e.g. adding a parameter – leads to a required re-build of all systems using it. With REST you only need simple CRUD operations and can use JSON as a very flexible data format that can be added to later on.

I’m glad that Integration Server provides an easy means to publish REST services. You can even put them on top of existing services and simply provide a more flexible interface to the consumers. This is definitely a solution I want to try out in the near future.

So, if you start with an SOA today, take a closer look at REST and don’t blindly use the default implementation SOAP as it may not be supported any more in the near future.

Test everything! – Lessons Learned from SOA-fying a Monolith

Another lesson we learned while making our legacy application ready for a service-oriented architecture, is this:

Test everything.

Test everything

When I started out writing Flow services in webMethods Integration Server (IS), there was no (nice) way of automatically testing them. Although we were told multiple times by consultants, that there would be a test framework for IS, we never got the actual code. So, I had to develop a test framework myself, simply to be sure that everything still worked as before after a deployment of IS.

The result of my development effort are two small Java frameworks:

I wanted to be able to test IS services (Flow, Java, etc.) with the established tool we already used in our projects: JUnit. However, Integration Server’s Java interface relies on IData, the internal representation of IS’s pipeline. And working with it can get pretty annoying, because it’s nothing more than a big hash table with its own API. So you would have to deconstruct your Java objects into the structure of IData every time you call a service and compose them back together when you get the results, only to be able to call an assertion on it.

ao-idata-converter

My solution for this problem is a project called ao-idata-converter. It takes any Plain Old Java Object (POJO) and converts it to IData with a bit of reflection magic. You can even use beans with Getters and Setters and map attribute names to different fields in the pipeline. So, with ao-idata-converter you can get rid of all the converting from and to IData in your code.

IData convertedObject =
    new ObjectConverter().convertToIData("address", addressObject);

ao-integrationserver

The next problem I faced, was the need to use lots of boilerplate code to be able to call an IS service. If you generate a Java client for a given IS service, the resulting class contains all the setup, authentication, input etc. you need to call the service. However, it’s not reusable and you’ll end up with lots of duplication if you want to call multiple services (which would be the default behaviour, I guess).

Therefore, I abstracted all the needed boilerplate code into a separate framework, ao-integrationserver, that provides an easy API for calling IS services on different endpoints with authentication and input/output parameters represented by simple POJOs. If you follow a certain naming convention for your Java packages and classes, you’ll be able to call an IS service by creating a single class with only a few lines of code. So, adding a new service to your Java library takes only a few minutes at most.

public class max extends Service<max.Input, max.Output>
{
    public static class Input extends ServiceInput
    {
        public String[] numList;
    }

    public static class Output extends ServiceOutput
    {
        public String maxValue;
    }
}

Our test suite

Below you can see a screenshot of our IS test suite in Eclipse. The test suite automatically runs on our Jenkins build server every time we deploy IS and we can point it to any stage (development, test, production) within a matter of seconds to make sure that all services still work as expected.

Screenshot of our test suite for Integration Server

If you would like to know more about the two frameworks or our deployment pipeline, feel free to contact me any time. If you would like to participate in the development of the frameworks, I would also love to hear from you (e.g. via a Pull Request).

Make use of Diversity – Lessons Learned from SOA-fying a Monolith

One of the lessons we learned while SOA-fying our legacy application (an Adabas/Natural monolith, that is almost 20 years old) is:

Make use of diversity.

Make use of diversity!

When we designed the central interface for our service modules, we made sure that developers with different backgrounds worked together on the design team. For example, we had older and younger developers work together with Natural and Java developers. They all worked towards the same goal: creating an interface that all of our systems could use to communicate with each other.

We put together ideas from the “old” world – procedural data processing – and ideas from the “new” world – object and service orientation – and merged them into a flexible interface that combines the best concepts of both worlds. The more experienced developers brought their domain knowledge to the table and the younger developers added their new ways of thinking about the technical problems.

What we ended up with was an interface that could easily be provided and consumed from all the platforms we use. Because we put the interface to the test right away and made sure that problems would become visible quite quickly, we only needed a few iterations before the final design was ready.

And every developer that was involved in the process knew the interface already and could actively promote its usage to the rest of the team. And each of those deveopers could use the language they were already familiar with and didn’t have to teach new concepts – which isn’t that easy for both Natural and Java developers by the way.

So, my advice to you, if you plan to design a central interface, data model, or business process, is: Make sure to integrate as many different views and backgrounds as possible. The coordination might take a bit longer (at first), but you’ll probably end up with a much better solution.

If you make use of the individual strengths of your developers, the end result will be a well rounded solution.