Resolving Conflicts While Patching the EM Agent

Unless you’re in a cave asleep, you’ve seen the recent Oracle PSUs and patches that were released in January.  This has many of my customers patching their agents, and a few have noticed a problem with some previously applied patches.  Thanks to Brian for pointing this out!

If you applied the recommended Java patches sometime last year (18502187, 18721761), and you go to apply the 12.1.0.4.5 agent patch 20282974 (or an earlier bundle), the analyze step will fail as the Java patches are included in the agent bundle, but for some reason it can’t “ignore” them.    Checking the detailed results of the failed Analyze job will show which step failed.   In this specific case, the step PrerequisiteCheckForApply will be marked Failed and the error will look like this:

ap1

To remedy this with EM, you can create a patch plan with the Java patches (18502187, 18721761) and select Rollback in the Deployment Options.  This will run the rollback for these patches.

ap2

Once rolled back, re-analyze your 12.1.0.4.5 patch plan and it should be successful and allow you to deploy!

When you update Agents, keep these things in mind:

  • Be sure to update any Agent clones/gold images with the newly updated patch and plug-ins if they’ve been upgraded or patched.
  • If you staged patches for auto-deployment in the OMS $ORACLE_HOME/install/oneoffs, remove the old patches and you should just need the 12.1.0.4.5 patch now, or any current Discovery patches.
  • Update your procedural documents to reflect the current patches required.

For more details on how to patch your agent using EM, see the Administrator’s guide.

 

Enterprise Manager Patching FAQ

Every time a new plug-in, patch or PSU comes out for Enterprise Manager, I get a series of questions.   I’m going to quickly address a few of the ones I’ve gotten this month with the release of the new plug-ins and the PSU patches to try and help you in updating your EM site. I wrote a detailed blog EM Patching 101 about the various patches, and that’s still relevant so read if you haven’t (or read it again)!

What’s the difference between the Plug-in release (released 1/5/2015) and the Plug-in system bundle (released 12/31/2014)?  

The plug-in system bundle will come out every month on the last day of the month.  The next one is scheduled for 1/31/2015.   This is a cumulative compilation of monthly plug-in bundle patches.   So it will look at what plug-ins you have installed on the OMS, and it will deploy the latest plug-in patch bundle to that plug-in.  It will skip what you don’t have installed.   These patches will include fixes and will be indicated by the 5th digit in the version.  For example, the latest Database plug-in patch was 12.1.0.6.7.

The plug-in updates released on 1/5 were full plug-in version upgrades.  These need to be deployed to the OMS by using Extensibility -> Plug-ins and deploy to OMS.  They may require downtime.  Plug-in updates for Database, Fusion Middleware, Storage Management and Cloud were released.  These updates include new features and fixes.   For example, the Database plug-in is now 12.1.0.7.0.

Which one should I apply?

My recommendation would be to test and apply both.   The plug-in updates for Database and other components you use have the latest features and code that will enable new features or improvements.   You will also want to apply the latest system bundle patch (whether it’s December’s or January’s) to patch the other plug-ins that did not get updated (MOS for example).

Should I deploy the new Plug-ins first or apply patches first?

My suggestion is to apply the Plug-in updates first.  The plug-ins released in January include Database, Fusion Middleware, Cloud and Storage Management plug-ins.  When you download and apply these to the OMS, and Agents if required, they will not need the patches from the December 31st patch bundle, and you’ll have the most recent update. After deploying the plug-in updates, when you apply the Plug-in System patch (20188140) you will see that the updated plug-ins (DB, MW, etc) are skipped.

How often should I patch EM?   Agents?

I recommend to most my customers to stick to a quarterly patching cycle, and this typically follows the PSU cycle.  So start looking at the patches that are released in the January PSU and testing them.  At that time, grab all the recommended or latest plug-in and agent patches, and apply those as well.   The agent and plug-in patches will come out monthly, and if you need a patch for a particular bug or issue, then you should apply.  However, patching EM monthly is not feasible for most customers.

What patches do I need to apply as of now (January 2015)?

As mentioned in the previous Patching 101 blog, there’s multiple components involved that each have their own patch. So here’s what I’d recommend if you were my customer deploying or updating this month:

  • Latest Database PSU Patch (1/15) for your repository version
  • OMS 12.1.0.4.2 PSU Patch – 19830994
  • OMS 12.1.0.4.7 Plug-in System Patch – 20188140
  • Weblogic 10.3.0.6.0.10 PSU Patch – 19637463
  • WebLogic Server one-off recommended – 16420963
  • OPMN Patch (CVE-2014-4212) – 19345576
  • Oracle Help Patch (CVE-2015-0426) – 20075252
  • Agent 12.1.0.4.4 Bundle – 20109357
  • Agent Plugin Patches for non-Updated plug-ins as necessary – From Doc ID 1900943.1  (Siebel, OVI, Exadata)

When I apply Weblogic PSU it has a conflict, can I roll it back and how do I do it?

When you follow instructions to apply the PSU, you may receive a warning that there’s a conflict and patches are mutually exclusive such as the following:

$ ./bsu.sh -install -patch_download_dir=<MW_HOME>/utils/bsu/cache_dir -patchlist=12UV -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch 12UV is mutually exclusive and cannot coexist with patch(es): FSR2

You must first deinstall the conflict patch first, and install the new PSU.  The patches are cumulative.

$ ./bsu.sh -remove -patchlist=FSR2 -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Removing Patch ID: FSR2..
Result: Success

Now you can install the new PSU:

$ ./bsu.sh -install -patch_download_dir=<MW_HOME>/utils/bsu/cache_dir -patchlist=12UV -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Installing Patch ID: 12UV..
Result: Success

How do I apply the Oracle Help Patch (20075252)?

The January PSU calls for 17617649 on the Oracle Help product but this conflicts with a pre-existing EM patch.   There is a merge patch 20075252 that is available for this.  The Oracle Help patch is applied to the <MW_HOME?/oracle_common and uses the OPatch from that directory as well.    Read the readme.txt for full patching steps, but some tips are listed here in applying.

emctl stop oms -all
export MW_HOME=<MW_HOME>
export ORACLE_HOME=$MW_HOME/oracle_common
export PATH=$ORACLE_HOME/OPatch:$PATH

You should be able to do an opatch lsinventory and get the output of the ORACLE_HOME and the opatch apply should be similar to below:

How do I apply the OPMN Patch (19345576)?

Following the readme.txt it wants you to set Oracle Home to the  Classic/WebTier home.  For EM, this is the MW_HOME/Oracle_WT directory.

emctl stop oms -all
export MW_HOME=<MW_HOME>
export ORACLE_HOME=$MW_HOME/Oracle_WT
export PATH=$MW_HOME/oracle_common/OPatch:$PATH

You should be able to do an opatch lsinventory and get the output of the ORACLE_HOME and the opatch apply should be similar to below:

Let me know if you have any other patching questions and I’ll add them here!

Simplified Agent and Plug-in Deployment

On your site of hundreds or thousands of hosts have you had to patch agents immediately as they get deployed?  For this reason I’ve always been a big fan of cloning an agent that has the required plug-ins and all the recommended core agent and plug-in patches, then using that clone for all new agent deployments. With EM 12c this got even easier as you can now clone the agent using the console “Add Host” method. You still have to rely on the EM users to use the clone.The one problem I have with cloning is that you have to have a reference target for each platform that you support. If you have a consolidated environment and only have Linux x64, this may not be a problem. If you are managing a typical data center with a mixture of platforms, it can become quite the maintenance nightmare just to maintain your golden images.You must update golden image agents whenever you get a new patch (generic or platform specific) for the agent or plug-in, and recreate the clone for each platform. Typically, I find people create a clone for their most common platforms, and forget about the rest. That means, maybe 80% of their agents meet their standard patch requirements and plug-ins upon deployment, but the other 20% have to be patched post-deploy, or worse – never get patched!

While deployed agents and plug-ins can be patched easily using EM Patches & Updates, but what about the agents still getting deployed or upgraded? Wouldn’t it be nice if they got patched as part of the deployment or upgrade? This article will show you two new features in EM 12.1.0.3 (12cR3) that will help you deploy the most current agent and plug-in versions. Whether you have 100s or 1000s of agents to manage, reducing maintenance and keeping the agents up to date is an important task, and being able to deploy or upgrade to a fully patched agent will save you a lot of time and effort.

Read original post here.

Patching 101 – The User Friendly Guide to Understanding EM Patches

There was a conversation on twitter last week about available patches for Enterprise Manager (EM) 12.1.0.4, and it got a little deeper than 140 characters will allow.  I’ve written this blog to give a quick Patching 101 on the types of EM patches you might see and the details around how they can be applied.

Read original post here