Automatically reconnect to a database after a failure in JBoss EAP 7

My JBoss EAP 7 server couldn’t cope with a failing database. After a database restart or failure, e.g. due to maintenance, it simply would not connect to the database again automatically. The application simply stopped working as soon as the database was unavailable for a short period of time.

JBoss’s server.log was full of (not very helpful) error messages like this:

2017-03-01 12:05:18,175 ERROR [it.macke.repository.UserRepository] (default task-17) Error reading user: org.hibernate.exception.JDBCConnectionException: could not prepare statement: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not prepare statement
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
...
Caused by: org.hibernate.exception.JDBCConnectionException: could not prepare statement
    at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
    at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
...
Caused by: java.sql.SQLNonTransientConnectionException: Connection is close
    at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)

To make the application work again, I had to restart JBoss and every now and then even the whole Windows server on which it runs.

As it turns out, this is a common problem with JBoss. However, the solution is quite easy. You only need to configure JBoss to validate the database connections. Here’s how this would look in standalone.xml (take a look at the content of element validation):

<datasource jndi-name="java:jboss/jdbc/MyDS" pool-name="MyDS">
    <connection-url>jdbc:mariadb://localhost:3306/login</connection-url>
    <driver-class>org.mariadb.jdbc.Driver</driver-class>
    <driver>mariadb</driver>
    <security>
        <user-name>user</user-name>
        <password>secret</password>
    </security>
    <validation>
        <check-valid-connection-sql>select 1</check-valid-connection-sql>
        <validate-on-match>true</validate-on-match>
        <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter"/>
    </validation>
</datasource>

And if you want to script the setting, here’s the code for the CLI:

/subsystem=datasources/data-source=MyDS:write-attribute(name=validate-on-match,value=true)
/subsystem=datasources/data-source=MyDS:write-attribute(name=check-valid-connection-sql,value="select 1")
/subsystem=datasources/data-source=MyDS:write-attribute(name=exception-sorter-class-name,value="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter")

The above settings make JBoss validate the connection on every new request (validate-on-match). But you can also have it do the checking in the background, e.g. every few seconds. And you need to consider the correct setting for your database. I use MariaDB/MySQL for my application and the exception-sorter and check-valid-connection-sql might be different depending on your database (e.g. org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker and select 1 from dual for Oracle). Take a look at the available parameters here: Database Connection Validation JBoss EAP 7 – Configuration Guide

How to enable access logging (accesslog) in JBoss EAP 7

Configuring JBoss EAP 7 to write an access log (e.g. like Apache webserver) is quite easy with the CLI:

/subsystem=undertow/server=default-server/host=default-host/setting=access-log:add

If you need any additional configuration, take a look at this: Wildfly 10.0.0.Final Model Reference.

For example, to change the prefix of the log’s file name:

/subsystem=undertow/server=default-server/host=default-host/setting=access-log:write-attribute(name=prefix, value="my_access")

Alternatively, you could change the configuration in the config XML (e.g. standalone.xml):

<subsystem xmlns="urn:jboss:domain:undertow:3.1">
    ...
    <server name="default-server">
        ...
        <host name="default-host" alias="localhost">
            ...
            <access-log prefix="my_access" />
            ...
        </host>
    </server>
    ...
</subsystem>

JBoss will now write every access to the application to accesslog in the log directory (e.g. JBOSS_HOME\standalone\log):

192.168.1.1 - - [23/Jun/2016:13:29:24 +0200] "GET /MyApp/MyPath/MyFile.xhtml HTTP/1.1" 200 5738

How to import SSL certificates into webMethods Integration Server

In this article I described how you can generate a self-signed SSL certificate to enable HTTPS in webMethods Integration Server: How to create a self-signed SSL certificate for webMethods Integration Server with OpenSSL. Now it’s time to import a real certificate.

If you have received the signed certificate from your Certificate Authority, you can follow these steps to import it into Integration Server. I’m using OpenSSL on a Linux machine and Java’s keytool on my Windows workstation for the command line work.

Prepare the certificate

  • The private key has to be in PEM format and needs to be BASE64 encoded. At least in my case OpenSSL wasn’t able to handle it otherwise.
  • First of all, you need to protect your private key with a password, if you haven’t already done so.

    openssl rsa -des3 -in integrationserver.key -out integrationserver.key
    
  • If the certificate is in format DER (in my case it was and the file had the ending cer), it has to be converted to PEM:

    openssl x509 -in integrationserver.cer -inform DER -out integrationserver.crt -outform PEM
    
  • Now the keystore for Integration Server can be created:

    openssl pkcs12 -export -des3 -in integrationserver.crt -inkey integrationserver.key -out integrationserver.p12
    
  • Now we need to create a Truststore containing the issuing certificates of our certificate. You need to download the required certificates for the whole certificate chain and add them to a Truststore:

    keytool -import -alias rootCA -keystore integrationserver.jks -file rootCA.crt
    

    You need to repeat this command for each certificate of the chain with a unique alias.

Import the certificate into Integration Server

  • Create a Truststore Alias under Security -> Keystore -> Create Truststore Alias.
    Create a Truststore Alias in webMethods Integration Server
  • Create a Keystore Alias under Security -> Keystore -> Create Keystore Alias.
    Create a Keystore Alias in webMethods Integration Server
  • Create an HTTPS Port Security -> Ports -> Add Port.
    Create an HTTPS Port in webMethods Integration Server
  • Enable access through the new port.
    Enable access through an HTTPS Port in webMethods Integration Server
  • Test your new HTTPS connection in a browser:
    https://YOUR-SERVERNAME:5443/

Links

How to create a new webMethods Integration Server instance

In newer versions of the webMethods suite, you can install multiple instances of Integration Server into a single installation. Some central packages will be re-used in every instance. However, administration of the instances may be a bit harder, as these packages have to be updated manually, e.g. in case of a version update or fix installation.

Here is how to setup a new Integration Server instance (ISDIR points to the Integration Server directory, e.g. C:\SoftwareAG\IntegrationServer):

  1. Run ISDIR\instances\is_instance.bat create -Dinstance.name=testIS1 -Dprimary.port=5550 -Ddiagnostic.port=9990 -Djmx.port=8077 -Dlicense.file=C:\license.xml
    This will start an Ant build that creates the new IS instance under instances\testIS1. Of course, you may need to adjust the parameters according to your needs.
  2. If the build finishes successfully, the instance can be started with ISDIR\instances\testIS1\bin\startup.bat. You should now be able to connect to localhost:5550 and see the administration page for your new instance:
    Administration page for the newly created Integration Server instance
  3. If you want to install the new instance as a Windows service, you can run ISDIR\instances\testIS1\support\win32\installSvc.bat:
    Create a new service for the newly created Integration Server instance
    You should now see another IS service:
    The service for the the newly created Integration Server instance
  4. You should also see the new instance in Command Central. A refresh or a restart of the platform manager may be needed.
    The newly created Integration Server instance in Command Central