[WebLogic] Solution to JVM crash caused by WebLogic 2023 third quarter update

Affected versions:

  • Oracle WebLogic 12c (12.2.1.4.0)
  • Oracle WebLogic 14c (14.1.1.0.0)

Problem description:

The latest version of OPatch (13.9.4.2.13) officially released by Oracle in July 2023 has a new bug (which will cause the following GDR-70005 error when installing WebLogic PSU:

Caused by: com.oracle.cie.gdr.utils.LocalizedGdrException: GDR-70005: Failed to apply inventory patching to home.
GDR-70005: A failure occurred while attempting to apply inventory patches to the home <ORACLE_HOME>
GDR-70005: Contact customer support.

In order to solve the above problem, the official has provided a temporary patch. After upgrading OPatch, you can install this temporary patch (PatchID: 23335292). The installation record is as follows:

[2023/10/08 10:19:06] Repair Bug 35619659 before apply wlserver patches.
Archive: p23335292_13940015_Generic.zip
   creating: 23335292/
   creating: 23335292/etc/
   creating: 23335292/etc/config/
  inflating: 23335292/etc/config/actions.xml
  inflating: 23335292/etc/config/inventory.xml
  inflating: 23335292/etc/config/patchdeploy.xml
   creating: 23335292/files/
   creating: 23335292/files/oracle.nginst.common/
   creating: 23335292/files/oracle.nginst.common/13.9.4.0.0/
   creating: 23335292/files/oracle.nginst.common/13.9.4.0.0/oracle.nginst.common.symbol/
   creating: 23335292/files/oracle.nginst.common/13.9.4.0.0/oracle.nginst.common.symbol/modules/
  inflating: 23335292/files/oracle.nginst.common/13.9.4.0.0/oracle.nginst.common.symbol/modules/com.oracle.cie.gdr_1.8.4.0.jar.23335292
   creating: 23335292/files/inventory/
   creating: 23335292/files/inventory/Components/
   creating: 23335292/files/inventory/Components/oracle.nginst.common/
   creating: 23335292/files/inventory/Components/oracle.nginst.common/13.9.4.0.0/
   creating: 23335292/files/inventory/Components/oracle.nginst.common/13.9.4.0.0/patches/
   creating: 23335292/files/inventory/Components/oracle.nginst.common/13.9.4.0.0/patches/23335292/
  inflating: 23335292/files/inventory/Components/oracle.nginst.common/13.9.4.0.0/patches/23335292/compDef.xml
  inflating: 23335292/README.txt
Oracle Interim Patch Installer Version 13.9.4.2.13
Copyright (c) 2023, Oracle Corporation. all rights reserved.


Oracle home directory: /weblogic/Oracle/Middleware/Oracle_Home
Master product inventory: /weblogic/oraInventory
   From:/weblogic/Oracle/Middleware/Oracle_Home/oraInst.loc
OPatch version: 13.9.4.2.13
OUI version: 13.9.4.0.0
Log file location:/weblogic/Oracle/Middleware/Oracle_Home/cfgtoollogs/opatch/opatch2023-10-08_10-19-07am_1.log


OPatch detects the Middleware Home as "/weblogic/Oracle/Middleware/Oracle_Home"

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 23335292

Continue? [y|n]
Y (auto-answered by -silent)
User responded with: Y
All checks passed.

Please shut down the Oracle instance running outside this ORACLE_HOME on the local system.
(Oracle Home = '/weblogic/Oracle/Middleware/Oracle_Home')


Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User responded with: Y
Backing up files...
Applying interim patch '23335292' to OH '/weblogic/Oracle/Middleware/Oracle_Home'

Patching component oracle.nginst.common, 13.9.4.0.0...

Patching component oracle.nginst.common, 13.9.4.0.0...
Patch 23335292 successfully applied.
N-Apply process is complete. No bug fixes are lost.
Log file location: /weblogic/Oracle/Middleware/Oracle_Home/cfgtoollogs/opatch/opatch2023-10-08_10-19-07AM_1.log
Executing: com.oracle.cie.gdr.utils.RuntimeExecShutdownHook$Exec@5716d05c

Then, during the process of installing this temporary patch, I discovered new problems here. If this patch is installed successfully (actually there is still a script executing in the background, and you need to wait 15 to 30 seconds to end), within these 15-30 seconds, if you install a new patch immediately, the following JVM Crash will appear Error reported:

Start to apply the patch, id:33093748
 
Oracle Interim Patch Installer Version 13.9.4.2.13
Copyright (c) 2023, Oracle Corporation. all rights reserved.

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007fc4627371d0, pid=104444, tid=0x00007fc4650bc700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_381) (build 1.8.0_381-b32)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.381-b32 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libzip.so + 0x121d0] newEntry.isra.4 + 0x60
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /weblogic/tools/bugrepair/hs_err_pid104444.log
Compiled method (nm) 4178 113 n 0 java.util.zip.ZipFile::getEntry (native)
 total in heap [0x00007fc44d145550,0x00007fc44d1458c0] = 880
 relocation [0x00007fc44d145678,0x00007fc44d1456c0] = 72
 main code [0x00007fc44d1456c0,0x00007fc44d1458c0] = 512
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
/weblogic/Oracle/Middleware/Oracle_Home/OPatch/opatch: Line 1364: 104444 Abandoned (vomit core) $JAVA $otherOpt $JAVA_VM_OPTION $JRE_MEMORY_OPTIONS -cp $opatchClassPath -DOPatch.ORACLE_HOME="$OH" -DOPatch.DEBUG=" $DEBUGVAL" -DOPatch.MAKE="$MAKEVAL" -DOPatch.RUNNING_DIR="$BASE" -DOPatch.MW_HOME="$MWH" -DOPatch.WL_HOME="$WL_HOME" -DOPatch.COMMON_COMPONENTS_HOME="$COMMON_COMPONENTS_HOME" -DOPatch .OUI_LOCATION="$OUI_LOCATION" -DOPatch.FMW_COMPONENT_HOME="$FMW_COMPONENT_HOME" -DOPatch.OPATCH_CLASSPATH="$CLASSPATH" -DOPatch.WEBLOGIC_CLASSPATH="$WEBLOGIC_CLASSPATH" -DOPatch.SKIP_OUI_VERSION_CHECK="$SKIP_OUI_VERSION_CHECK" -DOPatch atch.NEXTGEN_HOME_CHECK="$ IS_NEXTGEN_HOME" -DOPatch.PARALLEL_ON_FMW_OH=$PARALLEL_ON_FMW_OH $FUSION_TRUST_OPTION ${oracleOcmService} ${_bootClassPath} oracle/opatch/OPatch $args $invptr

OPatch failed with error code 134

In order to solve this problem, you need to set a sleep time in the script after installing this temporary patch, such as

sleep 30s

Then install other patches and the above problem will not occur.

Reference:

https://support.oracle.com/epmos/faces/DocumentDisplay?id=2806740.2

https://support.oracle.com/epmos/faces/DocumentDisplay?id=2962593.1

https://support.oracle.com/epmos/faces/SrDetail?srNumber=3-34497119531