I am setting up a Keycloak Cluster, with 3 instances running in standalone clustered mode. I have set up 3 mysql instances on each machine with group replication.
I am now unable to use those databases to start the Keycloak servers. I get:
FATAL [org.keycloak.services] (ServerService Thread Pool -- 61) java.lang.RuntimeException: Failed to update database
Without any other hints as to why anything is failing. The debug log just before the failure looks like this:
onnections.jpa.updater.liquibase.conn.DefaultLiquibaseConnectionProvider] (ServerService Thread Pool
-- 61) Foreign key constraint added to SCOPE_MAPPING (ROLE_ID)
15:49:08,749 DEBUG [org.keycloak.connections.jpa.updater.liquibase.conn.DefaultLiquibaseConnectionProvider] (ServerService Thread Pool -- 61) Foreign key constraint added to CLIENT (REALM_ID)
15:49:08,775 DEBUG [org.keycloak.connections.jpa.updater.liquibase.conn.DefaultLiquibaseConnectionProvider] (ServerService Thread Pool -- 61) Foreign key constraint added to CREDENTIAL (USER_ID)
15:49:08,792 DEBUG [org.keycloak.connections.jpa.updater.liquibase.conn.DefaultLiquibaseConnectionProvider] (ServerService Thread Pool -- 61) Foreign key constraint added to KEYCLOAK_ROLE (APPLICATION)
15:49:08,807 DEBUG [org.keycloak.connections.jpa.updater.liquibase.conn.DefaultLiquibaseConnectionProvider] (ServerService Thread Pool -- 61) Foreign key constraint added to REALM (MASTER_ADMIN_APP)
15:49:08,819 DEBUG [org.keycloak.connections.jpa.updater.liquibase.conn.DefaultLiquibaseConnectionProvider] (ServerService Thread Pool -- 61) Foreign key constraint added to REALM_SOCIAL_CONFIG (REALM_ID)
15:49:08,835 DEBUG [org.keycloak.connections.jpa.updater.liquibase.conn.DefaultLiquibaseConnectionProvider] (ServerService Thread Pool -- 61) Foreign key constraint added to USER_FEDERATION_CONFIG (USER_FEDERATION_PROVIDER_ID)
15:49:08,836 DEBUG [org.keycloak.connections.jpa.updater.liquibase.conn.DefaultLiquibaseConnectionProvider] (ServerService Thread Pool -- 61) ChangeSet META-INF/jpa-changelog-1.0.0.Final.xml::1.0.0.Final-KEYCLOAK-5461::sthorger@redhat.com ran successfully in 1166ms,843 DEBUG [org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider] (ServerService Thread Pool -- 61) Going to release database lock namespace=DATABASE
15:49:08,843 DEBUG [org.keycloak.connections.jpa.updater.liquibase.lock.CustomLockService] (ServerService Thread Pool -- 61) Going to release database lock
15:49:08,843 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (ServerService Thread Pool -- 61) JtaTransactionWrapper rollback
15:49:08,847 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (ServerService Thread Pool -- 61) JtaTransactionWrapper end
15:49:08,848 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 61) MySqlDS: returnConnection(30c5c766, false) [0/10]
15:49:08,848 DEBUG [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (ServerService Thread Pool -- 61) Unfinished local transaction was rolled back.org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@62fbc82a[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1ae8b47b connection handles=0 lastReturned=1590911345304 lastValidated=1590911345304 lastCheckedOut=1590911345304 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@7015a6fb mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@138e4ff2[pool=MySqlDS] xaResource=LocalXAResourceImpl@182e1df6[connectionListener=62fbc82a connectionManager=741b5364 warned=false currentXid=null productName=MySQL productVersion=5.7.30-log jndiName=java:jboss/datasources/MySqlDS] txSync=null]er] (ServerService Thread Pool -- 61) JtaTransactionWrapper end
15:49:08,849 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (ServerService Thread Pool -- 61) JtaTransactionWrapper resuming suspended
15:49:08,849 DEBUG [org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider] (ServerService Thread Pool -- 61) Going to release database lock namespace=KEYCLOAK_BOOT
15:49:08,849 DEBUG [org.keycloak.connections.jpa.updater.liquibase.lock.CustomLockService] (ServerService Thread Pool -- 61) Going to release database lock
15:49:08,855 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (ServerService Thread Pool -- 61) JtaTransactionWrapper rollback
15:49:08,855 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (ServerService Thread Pool -- 61) JtaTransactionWrapper end
15:49:08,855 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 61) MySqlDS: returnConnection(6667c0b5, false) [0/10]
Then JDBC_PING Starts failing due to:
ERROR [org.jgroups.protocols.JDBC_PING] (ServerService Thread Pool -- 60) JGRP000115: Could not open connection to database: java.sql.SQLException: javax.resource.ResourceException: IJ000470: You are trying to use a connection factory that has been shut down: java:jboss/datasources/MySqlDS
Also a lot of other errors start popping up like:
WFLYCTL0195: Interrupted awaiting transaction commit or rollback
the databse connection seems to be working, as the database contains the keycloak created tables after all of this. On server restart, same thing happens, except instead of creating the database it complains that tables already exist.
Using mysql 5.7.3
module.xml:
<?xml version="1.0" ?>
<module xmlns="urn:jboss:module:1.3" name="com.mysql">
<resources>
<resource-root path="mysql-connector-java-5.1.48-bin.jar"/>
</resources>
<dependencies>
<module name="javax.api"/>
<module name="javax.transaction.api"/>
</dependencies>
</module>
standalone-ha.xml config:
<datasource jndi-name="java:jboss/datasources/MySqlDS" pool-name="MySqlDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
<connection-url>jdbc:mysql://localhost:3306/keycloak?useSSL=false</connection-url>
<driver>mysql</driver>
<pool>
<max-pool-size>10</max-pool-size>
</pool>
<security>
<user-name>usr</user-name>
<password>pwd</password>
</security>
</datasource>
...
<driver name="mysql" module="com.mysql">
<xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>
</driver>
...
<spi name="connectionsJpa">
<provider name="default" enabled="true">
<properties>
<property name="dataSource" value="java:jboss/datasources/MySqlDS"/>
<property name="initializeEmpty" value="true"/>
<property name="migrationStrategy" value="update"/>
<property name="migrationExport" value="${jboss.home.dir}/keycloak-database-update.sql"/>
</properties>
</provider>
</spi>