WebLogic: Connection has been administratively disabled

Recently one of ADF applications deployed to Weblogic Managed server throwing following error messages in logs when tried to perform any database operations. Provided root cause for these errors and solution to handle this.

Error Message 1:
oracle.jbo.JboException: JBO-29114 ADFContext is not setup to process messages for this exception. Use the exception stack trace and error code to investigate the root cause of this exception. Root cause error code is JBO-29000. Error message parameters are {0=java.sql.SQLRecoverableException, 1=IO Error: Connection reset}
        Truncated. see log file for complete stacktrace

Caused By: java.sql.SQLRecoverableException: IO Error: Connection reset
        at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:962)
        at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1283)

Caused By: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:730)

Error Message 2:
oracle.jbo.JboException: JBO-29114 ADFContext is not setup to process messages for this exception. Use the exception stack trace and error code to investigate the root cause of this exception. Root cause error code is JBO-29000. Error message parameters are {0=java.sql.SQLRecoverableException, 1=Connection has been administratively disabled. Try later.}
        Truncated. see log file for complete stacktrace

Caused By: java.sql.SQLRecoverableException: Connection has been administratively disabled. Try later.
        at weblogic.jdbc.common.internal.ConnectionEnv.checkIfEnabled(ConnectionEnv.java:1025)
        at weblogic.jdbc.wrapper.PoolConnection.checkConnection(PoolConnection.java:63)
        at weblogic.jdbc.wrapper.Connection.preInvocationHandler(Connection.java:100)

Cause:
There could be several reasons. Review Admin/Managed server logs to find the exact error message.

Most of the times, it could be due to unavailability of Database. If DBMS becomes and remains unavailable, the data source will test and try to replace dead connections while trying to satisfy connection requests. This behavior is beneficial because it enables the data source to react immediately when the database becomes available. However, testing a dead database connection can take as long as the network timeout, and can cause a long delay for clients.

To minimize this delay, the WLS data sources include logic that disables the data source after 2 consecutive failures to replace a dead connection. When an application requests a connection from a disabled data source, WebLogic Server throws a PoolDisabledSQLException immediately to notify the client that a connection is not available.

Fix:
Immediate fix:
You need to restart your WebLogic Admin or Managed Server(based on data source targets) to recreate new JDBC connections. All the current transactions are lost, as the database was shutdown.

Permanent fix:
1. High availability of your RDBMS is required to minimize this issue.

2. The WebLogic Server team recommends making sure that "Test on Connection Reserve" is selected in the datasource --> connection pool configuration (under the advanced section). This might have a small performance impact (as each datasource connection request will ping the database to check whether it is active or not before giving the connection to application). However, enabling this would ensure that only a valid connection will be given to the application, and there would not be any stale connection in the environment (another possible cause for - administratively disabled).

The following connection pool configs can be found in Admin console > Services > Data Sources > click on the datasource > Configuration tab > Advanced.
Below shows how the configurations would look appear the domain's datasource file, e.g.: <path_to_domain>/config/jdbc/<datasource_name>-xxxx-jdbc.xml.

The fix has been confirmed working with the following settings:

<test-frequency-seconds>1</test-frequency-seconds>
<test-connections-on-reserve>true</test-connections-on-reserve>
<test-table-name>SQL SELECT 1 FROM DUAL</test-table-name>
<seconds-to-trust-an-idle-pool-connection>1</seconds-to-trust-an-idle-pool-connection>