Continuous Deployment with Natural – Legacy Coder Podcast #6

Continuous Deployment with Software AG's Adabas/Natural

After you have automated the build process for your application based on Software AG’s Adabas and Natural it’s time to take the next step and also deploy the changes to production after each push to Git! I’ll tell you how in the sixth episode of the Legacy Coder Podcast.

Recap: Automating your build process with Continuous Integration

  • Listen to Episode 2 of the Legacy Coder Podcast for an introduction on why you should automate everything from compilation to deployment.
  • What is Continuous Integration?
    • CI is a software development practice. Every developer integrates (i.e. merges) at least daily. Each integration is built and tested automatically.
    • This leads to less integration problems and faster deployment cycles.
  • What can go wrong when manually deploying to production (or the next stage)?
    • Finding all the modules that you need to deploy.
    • NAT0082 Invalid command, or Subprogram does not exist
    • NAT0936 Format/length conflict in parameter
    • NAT0933 GDA time-stamp conflict
      The infamous GDA Timestamp Conflict in Software AG's Natural
    • Separating different features in the same modules from each other.
      Manually merging features in changed Natural modules with WinMerge
    • Nested functions or other dependent modules can’t be compiled.
    • Compile errors nobody cares about (anymore).

Continuous Deployment with Blue/Green Deployments

  • Basic Definitions
    • What is Continuous Delivery?
      • CD builds on top of CI and makes sure that the software can be released at any time. The deployment process has to be fully automated. You just need to push a button to deploy your software to the target environment.
    • What is Continuous Deployment?
      • CD goes even one step further and automatically deploys the software to production if all steps of the deployment pipeline pass. Finally, no more “release days”!
    • What is a Deployment Pipeline?
      • The deployment pipeline defines all the steps (called “stages”) necessary to build and deploy a new version of the software. Usually, there are stages for compiling, testing, packaging and deploying the application.
        Deployment Pipeline for Natural
    • What is Pipeline as Code?
      • Instead of manually creating the deployment pipeline in a CI server like Jenkins you can define all the necessary steps in a domain specific language that the CI server can execute. All the information needed to build and deploy the application are now contained inside the repository.
    • What are Blue/Green Deployments?
      • To be able to deploy to production while users and batch jobs are still using the environment you need another target environment for the deployment process. If everything works as expected you can simply switch to the new environment.
        Blue/Green Deployments for Natural with different FUsers
  • Advantages of a fully automated deployment process with blue/green deployments
    • No more missing files, GDA timestamp conflicts, or format/length conflicts.
    • Fast feedback (“Fail Fast”) for the developers, problems are visible immediately.
    • You can reliably and easily roll back to a previous state if something breaks.
    • Interactive tests in the production environment are possible.
    • Running processes and interactive session won’t be disturbed.
    • Deployments don’t need to be done in the evenings or on weekends.
    • No more annoying tasks for the (expensive) software developers.

Continuous Deployment for Software AG’s Natural

  • These are the tools we use for our CI/CD process (most are Open Source).
    • Git for version control.
    • Gradle as a wrapper for SAG’s Ant script.
    • Jenkins as a CI server.
    • A custom Java program for compiling Natural on the target environment.
    • NatUnit for unit testing.
    • NaturalONE with additional plugins for NatUnit and staging.
    • Redmine for documentation and communication with subject matter experts.
    • Our former apprentice Jonas built a build lamp with a Raspberry Pi that shows the current state of our Natural build.
      Build lamp for Natural above the coffee machine
  • Implementation at AO
    • AO doesn’t have a single pipeline but 3 different ones for each individual branch: DEV, QA, and PROD. The core stages are identical, but they contain individual steps, e.g. for testing and deployment.
      • DEV: only one target FUser that’s overwritten every time, QA: Blue/Green Deployment with instant switch, PROD: Blue/Green Deployment with nightly switch so the users aren’t disturbed.
        Continuous Delivery Pipelines for Software AG's Natural at ALTE OLDENBURGER
    • Before the pipeline can do its job, the developer needs to decide which modules to stage. This is done with Redmine where each changed module is assigned to an issue.
      • An Eclipse plugin developed by ALTE OLDENBURGER searches for all modules assigned to a given issue number and stages them. After a push to Git the pipeline starts.
        Eclipse Workflow Plugin for staging Natural modules with the help of Redmine
    • The first stage of the pipeline checks out the latest version of the Natural sources from Git.
    • After that the target Fuser gets erased and recreated from scratch.
    • We use SAG’s Ant script to upload the sources to the target server.
    • We do a full build every time, not an incremental build.
      • Compile errors are formatted as JUnit results so they can be displayed inside Jenkins.
      • We don’t use SAG’s script for that because it doesn’t follow the StepLib chain (for whatever reason).
    • After the compilation additional test data for testing legacy modules that rely on a certain database state are imported into Adabas.
    • Now the unit and integration tests are executed.
    • If the tests pass, the application is deployed into the next “free” FUser.
    • After the deployment the current FUser may be set to the new one. But this depends on the environment.
    • After the release is done there are a few additional steps required, e.g. restarting the Natural RPC servers against the new FUser.
    • After each production release a tag in Git is created.
    • And finally a changelog is generated from these Git tags.

How to start with Continuous Deployment for Natural

  • First of all: use Git (or any other version control system, but I would prefer Git). Make your current production sources the new master branch and add development sources on top of that into a new branch.
  • Switch to NaturalONE. Not only do you get a modern IDE with features like Code Completion, but more importantly you get the possibility to use version control from within NaturalONE. Your repository is now the source of truth (and not the server).
  • Make your codebase compile! Don‘t allow any compilation error whatsoever!
  • Add an additional FUser for the automated deployment after each push to Git.
  • Generate the deployment script from within NaturalONE and put it into a Jenkins build. You don‘t need to create a Jenkinsfile yet. Start small!
  • Start writing automated tests, e.g. with NatUnit. By the way: It’s open source! Feel free to use and modify it. https://sourceforge.net/projects/natunit/

Recommended reading

Links

Automating Your Natural Build Process – Legacy Coder Podcast #2

Automating Your Natural Build Process

How can you automate the build process for your application based on Software AG’s Adabas and Natural? I talk about our journey towards a completely automated build from scratch after each push to Git in the second episode of the Legacy Coder Podcast.

How to automate the build process for an Adabas/Natural application

  • Why would I want to automate the build process?
    • What hurts should be done more often!
    • No more long integration phases before each release!
    • You’ll get a faster time to market for new features.
    • You can get rid of knowledge silos concerning the release of your application.
  • What steps may need to be automated?
    • Upload (transfering the Natural source code to the server)
    • Compile (CATALOG, STOW, CATALL etc.)
    • Unit Tests (e.g. with NatUnit, see Unit Testing Natural Applications)
    • Deployment (the actual release of the new version on the target stage)
    • Restarting RPC servers or whatever individual steps might be necessary in your environment
  • What tools are available for automating the Natural build process?
    • It all starts with Git or any other versioning control system (Subversion, TFS, CVS).
    • You can use Jenkins for Continuous Integration.
    • Software AG’s command line tools (ftouch etc.) can be scripted.
    • Even the CATALL can be scripted by calling into Natural BATCHMODE.
    • NaturalONE’s ant scripts for uploading and compiling Natural sources.
    • Any additional tool that you might need (Bash scripts etc.)
  • How can I get started?
    • Make your project fully compilable. There can be no compile errors whatsoever.
    • Start with a single step, e.g. automate only the compile step first.
    • Script everything (e.g. with Bash) right from the beginning. You can easily call these scripts from Jenkins later on.
    • Hook Jenkins up to your source code repository. It provides plugins for every major system (e.g. Git, Subversion).
    • Try to automate the build with Jenkins. It’s completely free and easy to setup. Perhaps ask a Java colleague if he can help you.
  • How can you go even further than what’s possible today?
    • Try to automate imports of Adabas data that’s important for your application.
    • Try to automate Adabas schema migrations.
    • Implement feature toggles for more fine grained control over your application’s functionality at runtime.

Recommended reading

Links

Using Gradle wrapper behind a proxy server with self-signed SSL certificates

Today, I wanted to add a Gradle Wrapper to my Java project but had a few issues. I am behind a proxy and it changes the SSL certificates to be able to scan traffic for viruses.

My first attempt to start gradlew build resulted in:

    Exception in thread "main" java.net.UnknownHostException: services.gradle.org
            at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
            at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
            at java.net.Socket.connect(Socket.java:589)
            at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
            at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
            ...

Gradle didn’t use the proxy server and tried to connect to the internet directly. This was solved by setting the proxy server in %GRADLE_USER_HOME%\gradle.properties (see Gradlew behind a proxy):

    systemProp.http.proxyHost=192.168.1.1
    systemProp.http.proxyPort=80
    systemProp.http.proxyUser=userid
    systemProp.http.proxyPassword=password
    systemProp.https.proxyHost=192.168.1.1
    systemProp.https.proxyPort=80
    systemProp.https.proxyUser=userid
    systemProp.https.proxyPassword=password

The next try lead to:

    Downloading https://services.gradle.org/distributions/gradle-2.11-bin.zip

    Exception in thread "main" javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
            at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
            at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
            at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
            at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
            at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
            ....

The reason for the SSLHandshakeException were the proxy’s selft-signed certificates, that could not be validated. I had to add them to the Java keystore (see Java: Ignore/Trust an invalid SSL cert for https communication and Cacerts default password? -> the default password for the Java keystore is changeit):

    "%JAVA_HOME%\bin\keytool" -import -trustcacerts -alias MY_ALIAS -file MY_CERT.crt -keystore "%JAVA_HOME%\jre\lib\security\cacerts"

Now, Gradle was able to connect to gradle.org to download the distribution. However, the proxy server would not let the ZIP file through:

    Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://downloads.gradle.org/distributions/gradle-2.11-bin.zip
            at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1840)
            at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
            at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
            ...

So I configured Gradle to “download” the ZIP file from the local hard drive in %GRADLE_USER_HOME%\gradle.properties (see How to use gradle zip in local system without downloading when using gradle-wrapper):

    distributionBase=GRADLE_USER_HOME
    distributionPath=wrapper/dists
    zipStoreBase=GRADLE_USER_HOME
    zipStorePath=wrapper/dists
    distributionUrl=gradle-2.11-bin.zip

I manually downloaded the distribution file and put it into %GRADLE_USER_HOME%\wrapper\dists\gradle-2.11-bin\[SOME_HASH]\.

And finally the build was successful! 😀

    D:\MY_PROJECT>gradlew build
    Unzipping D:\GradleUserHome\wrapper\dists\gradle-2.11-bin\452syho4l32rlk2s8ivdjogs8\gradle-2.11-bin.zip to D:\GradleUserHome\wrapper\dists\gradle-2.11-bin\452syho4l32rlk2s8ivdjogs8
    Starting a new Gradle Daemon for this build (subsequent builds will be faster).
    Parallel execution with configuration on demand is an incubating feature.
    :compileJava UP-TO-DATE
    ...