Automatically reconnect to a database after a failure in JBoss EAP 7

My JBoss EAP 7 server couldn’t cope with a failing database. After a database restart or failure, e.g. due to maintenance, it simply would not connect to the database again automatically. The application simply stopped working as soon as the database was unavailable for a short period of time.

JBoss’s server.log was full of (not very helpful) error messages like this:

2017-03-01 12:05:18,175 ERROR [it.macke.repository.UserRepository] (default task-17) Error reading user: org.hibernate.exception.JDBCConnectionException: could not prepare statement: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not prepare statement
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
...
Caused by: org.hibernate.exception.JDBCConnectionException: could not prepare statement
    at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
    at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
...
Caused by: java.sql.SQLNonTransientConnectionException: Connection is close
    at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)

To make the application work again, I had to restart JBoss and every now and then even the whole Windows server on which it runs.

As it turns out, this is a common problem with JBoss. However, the solution is quite easy. You only need to configure JBoss to validate the database connections. Here’s how this would look in standalone.xml (take a look at the content of element validation):

<datasource jndi-name="java:jboss/jdbc/MyDS" pool-name="MyDS">
    <connection-url>jdbc:mariadb://localhost:3306/login</connection-url>
    <driver-class>org.mariadb.jdbc.Driver</driver-class>
    <driver>mariadb</driver>
    <security>
        <user-name>user</user-name>
        <password>secret</password>
    </security>
    <validation>
        <check-valid-connection-sql>select 1</check-valid-connection-sql>
        <validate-on-match>true</validate-on-match>
        <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter"/>
    </validation>
</datasource>

And if you want to script the setting, here’s the code for the CLI:

/subsystem=datasources/data-source=MyDS:write-attribute(name=validate-on-match,value=true)
/subsystem=datasources/data-source=MyDS:write-attribute(name=check-valid-connection-sql,value="select 1")
/subsystem=datasources/data-source=MyDS:write-attribute(name=exception-sorter-class-name,value="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter")

The above settings make JBoss validate the connection on every new request (validate-on-match). But you can also have it do the checking in the background, e.g. every few seconds. And you need to consider the correct setting for your database. I use MariaDB/MySQL for my application and the exception-sorter and check-valid-connection-sql might be different depending on your database (e.g. org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker and select 1 from dual for Oracle). Take a look at the available parameters here: Database Connection Validation JBoss EAP 7 – Configuration Guide

How to enable access logging (accesslog) in JBoss EAP 7

Configuring JBoss EAP 7 to write an access log (e.g. like Apache webserver) is quite easy with the CLI:

/subsystem=undertow/server=default-server/host=default-host/setting=access-log:add

If you need any additional configuration, take a look at this: Wildfly 10.0.0.Final Model Reference.

For example, to change the prefix of the log’s file name:

/subsystem=undertow/server=default-server/host=default-host/setting=access-log:write-attribute(name=prefix, value="my_access")

Alternatively, you could change the configuration in the config XML (e.g. standalone.xml):

<subsystem xmlns="urn:jboss:domain:undertow:3.1">
    ...
    <server name="default-server">
        ...
        <host name="default-host" alias="localhost">
            ...
            <access-log prefix="my_access" />
            ...
        </host>
    </server>
    ...
</subsystem>

JBoss will now write every access to the application to accesslog in the log directory (e.g. JBOSS_HOME\standalone\log):

192.168.1.1 - - [23/Jun/2016:13:29:24 +0200] "GET /MyApp/MyPath/MyFile.xhtml HTTP/1.1" 200 5738

How to test a REST API with Arquillian

Testing a REST API on a real application server with Arquillian is easy – if you know what you need to do 😉

I have this simple REST service, that authenticates a user:

@POST
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public Response authenticateUser(final UserData userData)
{
    ...
    return Response
        .status(401)
        .entity(false)
        .build();
}

Let’s write a test for this REST API, that uses a real application server and calls the interface from the outside.

Arquillian dependencies

The REST extensions for Arquillian make it fairly easy to test a deployed web application. Let’s start with their dependency in gradle.build:

// main BOM for Arquillian
'org.jboss.arquillian:arquillian-bom:1.1.11.Final',
// JUnit container
'org.jboss.arquillian.junit:arquillian-junit-container:1.1.11.Final',
// Chameleon (or any other specific container, e.g. JBoss, Wildfly etc.)
'org.arquillian.container:arquillian-container-chameleon:1.0.0.Alpha6',
// REST extensions
'org.jboss.arquillian.extension:arquillian-rest-client-impl-jersey:1.0.0.Alpha4'

Arquillian test

I can now write a test against the REST interface on the server like this:

@Deployment
public static WebArchive createDeployment()
{
    return ShrinkWrap
        .create(WebArchive.class)
        .addPackages(true, Filters.exclude(".*Test.*"),
            SessionResource.class.getPackage(),
            ...)
        .addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml");
}

@Test
@RunAsClient
public void authenticateUser(
    @ArquillianResteasyResource final WebTarget webTarget)
{
    final Response response = webTarget
        .path("/sessions")
        .request(MediaType.APPLICATION_JSON)
        .post(Entity.json(new UserData(
            "myuser",
            "mypassword")));
    assertEquals(true, response.readEntity(Boolean.class));
}

Note the @RunAsClient. This tells Arquillian to run the test as a client against the remote server and not within the remote server. The test class isn’t even deployed to the application server in this case (see Filters.exclude(".*Test.*")). That’s exactly what I needed, because I wanted to test the REST API from the outside, to make sure it’s available to its clients. Alternatively, you could set the whole deployment to @Deployment(testable = false), instead of configuring each test individually.

The REST extensions for Arquillian now call the test method with a WebTarget as a parameter. It points directly to the URL of the deployed web application, e.g. http://1.2.3.4:8080/044f5571-3b1a-4976-9baa-a64b56f2eec1/rest. So you don’t have to manually create the target URL everytime.

JBoss server configuration

It took me a while to find out how the target server – JBoss EAP 7 in my case – needs to be configured, to make this test pass, because the initial error message after my first attempt to run the test wasn’t very helpful at all:

javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused: connect
    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:287)
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:255)
    ...
Caused by: java.net.ConnectException: Connection refused: connect
    at java.net.DualStackPlainSocketImpl.connect0(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)

I could see in the server’s logs, that the deployment of the WAR file into JBoss was successful, so it had to be the REST client, that couldn’t connect to the API. A simple System.out.println(webTarget.getUri()); showed the problem: http://0.0.0.0:8080/044f5571-3b1a-4976-9baa-a64b56f2eec1/rest. Arquillian uses the IP configuration of the target server and I had configured JBoss to listen on any of its interfaces (hence 0.0.0.0).

To fix this problem I made JBoss listen on the exact public IP address of the server by adding this to \standalone\configuration\standalone.xml. Be sure to use the real IP address and not 0.0.0.0 or <any-address />.

<interfaces>
    <interface name="management">
        <inet-address value="1.2.3.4"/>
    </interface>
    <interface name="public">
        <inet-address value="1.2.3.4"/>
    </interface>
</interfaces>

How to test JBoss EAP 7 with Arquillian

It took me quite some time to get my Arquillian tests running against a remote JBoss EAP 7.0.0.Beta1 application server, so I thought I’d share my configuration.

At the time of this writing, there was no Arquillian container adapter for JBoss EAP 7 available. So I had to use the Arquillian Chameleon Container. However, it turns out, that Chameleon is even easier to configure than a specific container, because it more or less configures itself. Chameleon automatically downloads the needed container for you, if you tell it what application server you want to use.

Java project

Arquillian dependencies

Let’s start with the dependencies in appsgradle.build:

// main BOM for Arquillian
'org.jboss.arquillian:arquillian-bom:1.1.11.Final',
// JUnit container
'org.jboss.arquillian.junit:arquillian-junit-container:1.1.11.Final',
// Chameleon
'org.arquillian.container:arquillian-container-chameleon:1.0.0.Alpha6'

arquillian.xml

My arquillian.xml looks like this.

<arquillian xmlns="http://jboss.org/schema/arquillian"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://jboss.org/schema/arquillian http://jboss.org/schema/arquillian/arquillian_1_0.xsd">
    <container qualifier="chameleon" default="true">
        <configuration>
            <property name="chameleonTarget">jboss eap:7.0.0.Beta:remote</property>
            <property name="managementAddress">THE_SERVERNAME_OR_IP</property>
            <property name="managementPort">THE_PORT_IF_NOT_9990</property>
            <property name="username">THE_USER</property>
            <property name="password">THE_PASSWORD</property>
        </configuration>
    </container>
</arquillian>

The single line containing chameleonTarget defines the target server to be a remote JBoss EAP 7. That means, JBoss has to run on the target machine at the specified address or hostname and port. I chose this style of testing over the managed alternative, because it reduces the overhead of starting and stopping the server during tests. In addition, I think it’s a more realistic test, as the application gets deployed to a “real” server.

Arquillian test

I can now write a test against the server like this:

@Deployment
public static WebArchive createDeployment()
{
    return ShrinkWrap
        .create(WebArchive.class)
        .addPackage(MyClass.class.getPackage())
        .addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml")
        .addAsResource("arquillian/persistence.xml", "META-INF/persistence.xml");
}

@Test
public void authenticateUser()
{
    // asserts
}

persistence.xml

My persistence.xml for the Arquillian test simply uses JBoss’s default data source for the test. Hibernate creates and drops the tables before and after each test run.

<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.1"
    xmlns="http://xmlns.jcp.org/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/persistence http://xmlns.jcp.org/xml/ns/persistence/persistence_2_1.xsd">
    <persistence-unit name="myPU">
        <jta-data-source>java:jboss/datasources/ExampleDS</jta-data-source>
        <properties>
            <property name="hibernate.hbm2ddl.auto" value="create-drop" />
        </properties>
    </persistence-unit>
</persistence>

Server configuration

To successfully run the test, quite some configuration on the server side was needed. Here’s what I had to do.

Add a management user

If you want to deploy to a remote JBoss server, you need a management user. If you test against a JBoss server on your local machine, this is not the case. The credentials have to be configured in arquillian.xml (see above).

C:\jboss-eap-7.0.0.Beta\bin>add-user.bat

What type of user do you wish to add?
 a) Management User (mgmt-users.properties)
 b) Application User (application-users.properties)
(a): a

Enter the details of the new user to add.
Using realm 'ManagementRealm' as discovered from the existing property files.
Username : remote
Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.
 - The password should be different from the username
 - The password should not be one of the following restricted values {root, admin, administrator}
 - The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s)
Password :
WFLYDM0098: The password should be different from the username
Are you sure you want to use the password entered yes/no? yes
Re-enter Password :
What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[  ]:
About to add user 'remote' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'remote' to file 'C:\jboss-eap-7.0.0.Beta\standalone\configuration\mgmt-users.properties'
Added user 'remote' to file 'C:\jboss-eap-7.0.0.Beta\domain\configuration\mgmt-users.properties'
Added user 'remote' with groups  to file 'C:\jboss-eap-7.0.0.Beta\standalone\configuration\mgmt-groups.properties'
Added user 'remote' with groups  to file 'C:\jboss-eap-7.0.0.Beta\domain\configuration\mgmt-groups.properties'
Is this new user going to be used for one AS process to connect to another AS process?
e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.
yes/no? no

Bind to the public IP address

To make JBoss listen on the public IP address of the server, add this to \standalone\configuration\standalone.xml. The default configuration only allows connections on 127.0.0.1, which of course won’t be available from the outside.

<interfaces>
    <interface name="management">
        <any-address />
    </interface>
    <interface name="public">
        <any-address />
    </interface>
</interfaces>

Local repository

If you use a local repository like Artifactory (perhaps because you are behind a proxy server like me), you need to configure Maven to use this repository. Arquillian Chameleon downloads its needed artifacts directly during the test (and not during the build, where your individual repository would be configured in the build file) and uses whatever repository is configured as central. In my case, Chameleon could not connect to repo1.maven.org to download its dependencies, so I had to configure the local repository in the central Maven configuration file ~\.m2\settings.xml like so:

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd">
    <profiles>
        <profile>
            <id>artifactory</id>
            <repositories>
                <repository>
                    <id>central</id>
                    <url>http://artifactory.intranet/java-repos</url>
                    <snapshots>
                        <enabled>false</enabled>
                    </snapshots>
                </repository>
            </repositories>
        </profile>
    </profiles>
    <activeProfiles>
        <activeProfile>artifactory</activeProfile>
    </activeProfiles>
</settings>

How to create and deploy a web service with Java EE 7 and Wildfly 9

Apparently, creating and deploying an web service with Java EE 7 is just as easy as it was with Java EE 6. This maybe due to the fact, that support for web services and SOAP in JAX-WS hasn’t changed much since Java EE 6, as Adam Bien points out here: The State of SOAP / JAX-WS. However, just for the record, here’s the basic code needed to provide a simple web service:

package [...];

import javax.jws.WebMethod;
import javax.jws.WebService;

@WebService
public class Hello
{
    private final String message = new String("Hello, ");

    public Hello()
    {}

    @WebMethod
    public String sayHello(final String name)
    {
        return message + name + ".";
    }
}

Simply compile and deploy the WAR file (e.g. YourApplicationName.war), e.g. into \standalone\deployments of the standalone Wildfly installation. You should now be able to access the generated WSDL under http://localhost:8080/YourApplicationName/Hello?wsdl.