WFLYCTL0348: TimeoutException while running Keycloak in a Docker container with an external database (MariaDB)

I tried to start Keycloak in a Docker container with an external database (MariaDB) as described in the documentation of jboss/keycloak:

docker run --rm --name=keycloak \
  -e KEYCLOAK_USER=admin \
  -e KEYCLOAK_PASSWORD=admin123 \
  -e DB_VENDOR=mariadb \
  -e DB_ADDR=192.168.178.143 \
  -e DB_PORT=3306 \
  -e DB_DATABASE=keycloak \
  -e DB_USER=keycloak \
  -e DB_PASSWORD=keycloak \
  jboss/keycloak

Unfortunately, I got this exception after 300 seconds (5 minutes) and the container stopped: WFLYCTL0348: Timeout after [300] seconds waiting for service container stability.

05:39:48,036 INFO  [org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider] (ServerService Thread Pool -- 71) Initializing database schema. Using changelog META-INF/jpa-changelog-master.xml
05:44:35,971 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
    ("core-service" => "management"),
    ("management-interface" => "http-interface")
]'
05:44:40,975 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@e99e54 for operation add-deployer-chains at address [] failed -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:523)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1518)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1472)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1445)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.AbstractOperationContext$Step.access$400(AbstractOperationContext.java:1319)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:876)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:726)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1413)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:527)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:515)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:477)
        at org.jboss.as.server@12.0.3.Final//org.jboss.as.server.ServerService.boot(ServerService.java:451)
        at org.jboss.as.server@12.0.3.Final//org.jboss.as.server.ServerService.boot(ServerService.java:404)
        at org.jboss.as.controller@12.0.3.Final//org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:416)
        at java.base/java.lang.Thread.run(Thread.java:834)

05:44:40,976 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@e99e54 for operation add-deployer-chains at address [] failed -- java.util.concurrent.TimeoutException

Some tables were created in the target MariaDB database but the database migration (Liquibase) wasn’t fully completed. The database migration couldn’t be finished in 5 minutes and Wildfly’s timeout aborted the transaction.

After a few hours of trial and error I found three different settings regarding deployment and transaction timeout: jboss.as.management.blocking.timeout, deployment-timeout, and default-timeout. Because they all have to be set differently (e.g. via environment variables or Java parameters) I created a Dockerfile that contains all the needed settings and imports them via JBoss CLI. Finally, this setup solves the timeout problem and Keycloak starts (after about 10 minutes of database migration).

Dockerfile

You can build your custom Keycloak Docker image from the following Dockerfile with this command:

docker build -t keycloak .

The file contains all needed settings for the remote database and imports the timeout settings from config.batch into Keycloak’s Wildfly server.

Dockerfile

FROM jboss/keycloak

COPY --chown=jboss ./config.batch /tmp/config.batch

RUN cd $JBOSS_HOME \
  && ./bin/jboss-cli.sh --file=/tmp/config.batch \
  && rm -rf $JBOSS_HOME/standalone/configuration/standalone_xml_history \
  && rm -rf $JBOSS_HOME/standalone/data \
  && rm -rf $JBOSS_HOME/standalone/tmp \
  && rm -rf /tmp/config.batch

ENV KEYCLOAK_USER admin
ENV KEYCLOAK_PASSWORD admin123
ENV DB_VENDOR mariadb
ENV DB_ADDR 192.168.178.143
ENV DB_PORT 3306
ENV DB_DATABASE keycloak
ENV DB_USER keycloak
ENV DB_PASSWORD keycloak

After the image has beed built successfully, you can simply start a Keycloak container with this command:

docker run --it --rm -p 8080:8080 --name=keycloak keycloak

The initial database migration might take some time (more than 10 minutes in my case), but the second time the container starts it doesn’t need to migrate the database anymore, so it should start in a few seconds.

config.batch

The following file configures Wildfly’s timeouts so that Keycloak can finish the database migration during startup. I’ve set the timeout to 900 seconds (15 minutes). Feel free to adjust it to your need.

Important: Make sure to use standalone-ha.xml instead of standalone.xml.

config.batch

embed-server --std-out=echo --server-config=standalone-ha.xml
batch
/system-property=jboss.as.management.blocking.timeout:add(value=900)
/subsystem=deployment-scanner/scanner=default:write-attribute(name=deployment-timeout,value=900)
/subsystem=transactions:write-attribute(name=default-timeout,value=900)
run-batch
stop-embedded-server

Leave a Comment

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax