EM 12cR5 – What’s the Scoop?

In case you missed the announcement, EM has been announced.  See https://blogs.oracle.com/oem/entry/new_enterprise_manager_release_allows for details.

Many of my customers have heard about the release of EM last week and wondering what are the new features and whether they should plan to update right away.

The main new feature in is the Hybrid Cloud, which essentially enables a customer to use their on-premise Enterprise Manager to view their Oracle  Cloud targets.  This has been a long requested feature allowing the customers to manage and monitor their Oracle Cloud targets alongside their on-premise targets.   This also enables additional features, such as easy cloning to and from the Oracle Cloud.

The Database Plug-in has been updated to with the main features also enhancing the Hybrid Cloud experience.   For Exadata customers, this plug-in will enable thin cloning options on Exadata ACFS.

The Middleware Plug-in has also been updated to and will enable additional MW cloud options such as allowing users to deploy JVMD agents to Oracle Cloud virtual hosts,

Additional updates have been made to the Cloud, Virtual Infrastructure and Chargeback and Consolidation Plug-ins to enhance and enable the Hybrid Cloud functions.
In summary, if you are an Oracle Cloud customer, then this release is a must upgrade!   For full details about the new features enabled in the upgrade see the documentation here.


Join me at Collaborate 2015!

In a few weeks I’ll be heading to Las Vegas for a few days of relaxation and fun.  Well, really I’ll be working, but it will still be fun!

As always, I’ve got a busy schedule planned.  Here’s where you can find me over the week if you’re attending (there will be more but this is what I know now!):

Sunday 4/12

9-12p – Everything I Needed to Know About Enterprise Manager I Learned at COLLABORATE – Hands-on Lab

This should be fun.  Three hours w/ myself, Kellyn and Werner.  We may not stay on topic, but I can promise you’ll learn something new!

Monday 4/13

9:15-10:15 – Zero to Manageability in One Hour: Build a Solid Foundation for Oracle Enterprise Manager 12c

Learn how to properly plan, architect and implement EM12c, and how to setup the monitoring framework (roles, users, groups, templates, etc).

5:30-8:00 – Welcome Reception in Exhibit Hall at EM booth

Tuesday 4/14

5:30-7:00 – Happy Hour in Exhibit Hall at EM booth

Thursday 4/16

11:00-12:00 – Under the Hood of Enterprise Manager – A Troubleshooting Primer

Really, this is Werner’s session not mine.  But if you’ve never heard him talk about EM, you’re missing out.  Learn how to troubleshoot your EM system and find out what it’s doing.

12:15-1:15 – Smarter Monitoring with Adaptive Thresholds and Time Based Metrics

Then just stay seated for my talk on advancing your monitoring with adaptive and time based thresholds, metric extensions and more.

There are many more EM sessions and I’ll be trying to hit as many as I can!   Here’s the full list and hours for the EM demo booth.

If you can’t make it to Las Vegas, take a look at Collaborates Plug-in to Vegas option!  For a registration fee of $395, you get access to sessions from the top 3 tracks – Cloud Computing, Database Performance and Manageability.  Sessions are broadcast live online. If you’re interested in EM12c, there’s quite a line up of sessions available, including mine!

Who Needs to be Super?

In the days past, everybody in EM used to end up with Super Administrator privileges due to lack of granularity in permissions.  Not any more!  Now we have more permissions than you know what to do with, but that’s another blog topic all together!

Here’s a quick list of activities that one might still need Super Administrator for – note these are all considered EM administration activities and most are accessed through the Setup menu:

  • Create Administrators – can get around this by using LDAP integration and auto-provisioning
  • Access EM Audit
  • Access Security Console
  • License Management
  • Connector Setup
  • Data Exchange
  • Add/Edit Registration Password
  • Configure Notification Mail Server
  • Configure Notification Mail Customization
  • Configure SNMP
  • Configure Global Repeat Notifications
  • Setup Privilege Delegation Templates (sudo/pbrun)
  • Decommission Agent (bug fix 19430853 will allow users with Full privileges to perform again)

If your user is not responsible for users and security, and doesn’t get paged when EM stops working, then they have no business with Super Administrator privileges or SYSMAN.

Preventing Alerts on OS Audit File Size when Upgrading DB Plug-in

In January, the DB Plug-in was released for EM 12c.   Not long after, my friend Brian found they added a new metric with default thresholds.   The new metric group is Operating System Audit Files and the metric alerts on Size of Audit Files.  Depending on the size and agent of your environment, you may immediately start getting notifications or pages as the default thresholds are 10MB/20MB, which can be quite small.

An example of the notification you might receive:

 Target type=Database Instance 
 Target name=emrep 
 Message=35.39 MB of Audit Trail files collected (.aud: 35.39, .xml: 0, .bin: 0) 
 Event reported time=Feb 6, 2015 6:53:56 AM PST 
 Target Lifecycle Status=Production 
 Operating System=Linux
 Associated Incident Id=2103 
 Associated Incident Status=New 
 Associated Incident Owner= 
 Associated Incident Acknowledged By Owner=No 
 Associated Incident Priority=None 
 Associated Incident Escalation Level=0 
 Event Type=Metric Alert 
 Event name=sizeOfOSAuditFiles:FILE_SIZE 
 Metric Group=Operating System Audit Records
 Metric=Size of Audit Files
 Metric value=35.39
 Key Value= 
 Rule Name=DBA_Incident_Rule,Create incident for critical metric alerts 
 Rule Owner=SYSMAN 
 Update Details:
 3.39 MB of Audit Trail files collected (.aud: 3.39, .xml: 0, .bin: 0)
 Incident created by rule (Name = DBA_Incident_Rule, Create incident for critical metric alerts; Owner = SYSMAN).

So if you’re planning to upgrade 1000 agents with the new Database Plugin, you might start getting a little nervous about receiving all of these pages.   Since the metric didn’t exist before, it’s not included in your templates to be disabled.  Even if it were in the templates, it would likely alert before you could reapply templates.

Luckily, Incident Rules provide a method to exclude a particular event when evaluating an Incident Rule.

From Setup -> Incidents -> Incident Rules, you’ll want to edit your defined Incident Rule.  If you haven’t customized an Incident Rule, you can select the default Incident Ruleset and do a Create Like to clone and be able to edit.


Select the Metric Alert rule, and click Edit.


On this first screen, you’ll see the Advanced Selection Options.  If you expand this you’ll see an option for Event name.  This is where you can exclude a specific metric event by select Not Equals and enter the event name.   In the case of this metric, the event name is sizeOfOSAuditFiles:FILE_SIZE.



Click Next and Continue until you finally get to Save.   To validate, you can use the Simulate Rules or trigger an alert to see if it sends the email.

This concept can be applied to help filter out other events, or categories of metrics as needed.


Resolving Conflicts While Patching the EM Agent

Unless you’re in a cave asleep, you’ve seen the recent Oracle PSUs and patches that were released in January.  This has many of my customers patching their agents, and a few have noticed a problem with some previously applied patches.  Thanks to Brian for pointing this out!

If you applied the recommended Java patches sometime last year (18502187, 18721761), and you go to apply the agent patch 20282974 (or an earlier bundle), the analyze step will fail as the Java patches are included in the agent bundle, but for some reason it can’t “ignore” them.    Checking the detailed results of the failed Analyze job will show which step failed.   In this specific case, the step PrerequisiteCheckForApply will be marked Failed and the error will look like this:


To remedy this with EM, you can create a patch plan with the Java patches (18502187, 18721761) and select Rollback in the Deployment Options.  This will run the rollback for these patches.


Once rolled back, re-analyze your patch plan and it should be successful and allow you to deploy!

When you update Agents, keep these things in mind:

  • Be sure to update any Agent clones/gold images with the newly updated patch and plug-ins if they’ve been upgraded or patched.
  • If you staged patches for auto-deployment in the OMS $ORACLE_HOME/install/oneoffs, remove the old patches and you should just need the patch now, or any current Discovery patches.
  • Update your procedural documents to reflect the current patches required.

For more details on how to patch your agent using EM, see the Administrator’s guide.


Enterprise Manager Presentations at RMOUG Training Days 2015

RMOUG Training Days – Feb 17-19 (Denver)

This is a great regional Oracle User’s Group conference with some of the best speakers you’ll find on multiple subjects!  Something enticing about being in freezing cold temperatures during ski season I think?  My colleague Kellyn Pot’vin-Gorman does a great job promoting it and making sure people get a great educational experience.   If you’re in the area, it’s a great 2 day conference!   See the agenda and register now!

I will be presenting two sessions on Thursday: Smarter Monitoring with Adaptive Thresholds and Time-Based Metrics and  Enterprise Manager 12c: Extending Beyond the DBA.  Several of my colleagues will also be there presenting and running Deep Dives covering Enterprise Manager, check out the line up of Oracle speakers presenting on #EM12c:

Tuesday 2/17
1:00 – 3:00 Hands-On: Delivering Schema as a Service with Enterprise Manager (Pete Sharman)

3:15 – 5:15 Hands-On: Delivering Pluggable Database as a Service with Enterprise Manager (Pete Sharman)

Wednesday 2/18

8:30-9:30 A Day in the Life of an Enterprise Manager Agent (Andrew Bulloch)

12:15-1:15 SIG Meetings – Enterprise Manager 12c

2:30-3:30 Enterprise Manager Snap Clone: Snapshot Your Data without Snapping Your Storage (Pete Sharman)

Thursday 2/19

9:45-10:45 Smarter Monitoring with Adaptive Thresholds and Time-Based Metrics (Courtney Llamas)

11:15-12:15 Enterprise Manager 12c: Extending Beyond the DBA (Courtney Llamas)

2:45-3:45 Under the Hood of Enterprise Manager: A Troubleshooting Primer (Werner De Gruyter)

4:00-5:00 The Power of the AWR Warehouse (Kellyn Gorman)

4:00-5:00 The High Availability Enterprise Manager Roundtable (Werner De Gruyter)

Enterprise Manager Patching FAQ

Every time a new plug-in, patch or PSU comes out for Enterprise Manager, I get a series of questions.   I’m going to quickly address a few of the ones I’ve gotten this month with the release of the new plug-ins and the PSU patches to try and help you in updating your EM site. I wrote a detailed blog EM Patching 101 about the various patches, and that’s still relevant so read if you haven’t (or read it again)!

What’s the difference between the Plug-in release (released 1/5/2015) and the Plug-in system bundle (released 12/31/2014)?  

The plug-in system bundle will come out every month on the last day of the month.  The next one is scheduled for 1/31/2015.   This is a cumulative compilation of monthly plug-in bundle patches.   So it will look at what plug-ins you have installed on the OMS, and it will deploy the latest plug-in patch bundle to that plug-in.  It will skip what you don’t have installed.   These patches will include fixes and will be indicated by the 5th digit in the version.  For example, the latest Database plug-in patch was

The plug-in updates released on 1/5 were full plug-in version upgrades.  These need to be deployed to the OMS by using Extensibility -> Plug-ins and deploy to OMS.  They may require downtime.  Plug-in updates for Database, Fusion Middleware, Storage Management and Cloud were released.  These updates include new features and fixes.   For example, the Database plug-in is now

Which one should I apply?

My recommendation would be to test and apply both.   The plug-in updates for Database and other components you use have the latest features and code that will enable new features or improvements.   You will also want to apply the latest system bundle patch (whether it’s December’s or January’s) to patch the other plug-ins that did not get updated (MOS for example).

Should I deploy the new Plug-ins first or apply patches first?

My suggestion is to apply the Plug-in updates first.  The plug-ins released in January include Database, Fusion Middleware, Cloud and Storage Management plug-ins.  When you download and apply these to the OMS, and Agents if required, they will not need the patches from the December 31st patch bundle, and you’ll have the most recent update. After deploying the plug-in updates, when you apply the Plug-in System patch (20188140) you will see that the updated plug-ins (DB, MW, etc) are skipped.

How often should I patch EM?   Agents?

I recommend to most my customers to stick to a quarterly patching cycle, and this typically follows the PSU cycle.  So start looking at the patches that are released in the January PSU and testing them.  At that time, grab all the recommended or latest plug-in and agent patches, and apply those as well.   The agent and plug-in patches will come out monthly, and if you need a patch for a particular bug or issue, then you should apply.  However, patching EM monthly is not feasible for most customers.

What patches do I need to apply as of now (January 2015)?

As mentioned in the previous Patching 101 blog, there’s multiple components involved that each have their own patch. So here’s what I’d recommend if you were my customer deploying or updating this month:

  • Latest Database PSU Patch (1/15) for your repository version
  • OMS PSU Patch – 19830994
  • OMS Plug-in System Patch – 20188140
  • Weblogic PSU Patch – 19637463
  • WebLogic Server one-off recommended – 16420963
  • OPMN Patch (CVE-2014-4212) – 19345576
  • Oracle Help Patch (CVE-2015-0426) – 20075252
  • Agent Bundle – 20109357
  • Agent Plugin Patches for non-Updated plug-ins as necessary – From Doc ID 1900943.1  (Siebel, OVI, Exadata)

When I apply Weblogic PSU it has a conflict, can I roll it back and how do I do it?

When you follow instructions to apply the PSU, you may receive a warning that there’s a conflict and patches are mutually exclusive such as the following:

$ ./bsu.sh -install -patch_download_dir=<MW_HOME>/utils/bsu/cache_dir -patchlist=12UV -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch 12UV is mutually exclusive and cannot coexist with patch(es): FSR2

You must first deinstall the conflict patch first, and install the new PSU.  The patches are cumulative.

$ ./bsu.sh -remove -patchlist=FSR2 -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Removing Patch ID: FSR2..
Result: Success

Now you can install the new PSU:

$ ./bsu.sh -install -patch_download_dir=<MW_HOME>/utils/bsu/cache_dir -patchlist=12UV -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Installing Patch ID: 12UV..
Result: Success

How do I apply the Oracle Help Patch (20075252)?

The January PSU calls for 17617649 on the Oracle Help product but this conflicts with a pre-existing EM patch.   There is a merge patch 20075252 that is available for this.  The Oracle Help patch is applied to the <MW_HOME?/oracle_common and uses the OPatch from that directory as well.    Read the readme.txt for full patching steps, but some tips are listed here in applying.

emctl stop oms -all
export MW_HOME=<MW_HOME>
export ORACLE_HOME=$MW_HOME/oracle_common

You should be able to do an opatch lsinventory and get the output of the ORACLE_HOME and the opatch apply should be similar to below:

How do I apply the OPMN Patch (19345576)?

Following the readme.txt it wants you to set Oracle Home to the  Classic/WebTier home.  For EM, this is the MW_HOME/Oracle_WT directory.

emctl stop oms -all
export MW_HOME=<MW_HOME>
export PATH=$MW_HOME/oracle_common/OPatch:$PATH

You should be able to do an opatch lsinventory and get the output of the ORACLE_HOME and the opatch apply should be similar to below:

Let me know if you have any other patching questions and I’ll add them here!

On the Books for 2015…

As always, starting 2015 with a full plate! Several presentations and papers in the works for the next few months… now if my kids could stay healthy and get back to school so I can get some real work done!   One webinar, 2 presentations, 1 whitepaper and a hands on lab in progress… oh and finding time to blog on my list of topics that keeps growing…

IOUG Webinar – Jan 21 at 12:00pm CDT – Online

I’m excited to do another webinar for IOUG this year, covering EM install and setup.  Be sure to register!

Zero to Manageability in One Hour: Build a Solid Foundation for Oracle Enterprise Manager 12c

The goal in every Oracle Manager 12c rollout is to take it from zero to manageability in the shortest time possible. This presentation will show you how to accomplish this feat.

Speakers will demonstrate how to properly architect and deploy Oracle Enterprise Manager 12c implementation, including designing for high availability and scalability. Through this demonstration a list of essential techniques and tips compiled from Oracle Enterprise Manager Development’s Strategic Customer Programs team will also be shared.   Topics such as users, roles, groups, templates, and incidents will be discussed, plus key architectural decisions.

By attending this webinar, attendees will learn how to:
• Organize targets, notifications and users properly
• Configure for best practices after the install is complete
• Properly plan and architect an EM 12c environment

RMOUG Training Days – Feb 17-19 – Denver, CO

This is a great regional Oracle User’s Group conference with some of the best speakers you’ll find on multiple subjects!  Something enticing about being in freezing cold temperatures during ski season I think?    I  don’t ski, but I love this conference because I can get a little personal benefit from it.  I grew up in Fort Collins, CO and proud to be a CSU Ram.  I still have many friends and family there so when I get back I try to meet up with someone!

My colleague Kellyn Pot’vin does a great job setting up the agenda and making sure people get a great educational experience.   If you’re in the area, it’s a great 2 day conference at reasonable rates.   See the agenda here and register.

I will be presenting Enterprise Manager 12c: Extending Beyond the DBA on Thursday at 11:15.    Several of my colleagues will also be there presenting and providing Hand on Labs covering Enterprise Manager and we’ll also be holding an EM Roundtable to wrap up the conference.

Collaborate – April 12-16 – Las Vegas, NV

Last year was my first year at Collaborate and I was surprised at how big and diverse this conference was for a User Group sponsored event.  They’ve done a great job making it a full week of great sessions, exhibits and labs!  If you haven’t gone before, maybe this is the year to check it out?   You can find out about registration here.

I don’t have the schedule for my sessions, but I do know it’s going to be busy!   I have two sessions scheduled so far – Zero to Manageability in One Hour: Build a Solid Foundation for Oracle Enterprise Manager 12c and Smarter Monitoring with Adaptive Thresholds and Time Based Metrics.  I’m also co-presenting with Werner  on Under the Hood of Enterprise Manager – a Troubleshooting Primer.  We’re hoping to get a Hand on Lab set up for EM as well, still waiting for word on that!

Besides, who doesn’t love a few days in Las Vegas?   This year the conference is at Mandalay Bay, which is where my husband and I got married almost 12 years ago. Hoping to get him to join me for a few days if we can line up a sitter!

That should be enough for this Spring right??  At least for me it is!

Notifications for Expiring DBSNMP Passwords

Most user accounts these days have a password profile on them that automatically expires the password after a set number of days.   Depending on your company’s security requirements, this may be as little as 30 days or as long as 365 days, although typically it falls between 60-90 days. For a normal user, this can cause a small interruption in your day as you have to go get your password reset by an admin. When this happens to privileged accounts, such as the DBSNMP account that is responsible for monitoring database availability, it can cause bigger problems.

In Oracle Enterprise Manager 12c you may notice the error message “ORA-28002: the password will expire within 5 days” when you connect to a target, or worse you may get “ORA-28001: the password has expired”. If you wait too long, your monitoring will fail because the password is locked out. Wouldn’t it be nice if we could get an alert 10 days before our DBSNMP password expired? Thanks to Oracle Enterprise Manager 12c Metric Extensions (ME), you can! See the Oracle Enterprise Manager Cloud Control Administrator’s Guide for more information on Metric Extensions.

Read more here

Organizing Your Oracle Enterprise Manager Targets

If you’re monitoring more than a handful of servers or databases in your Enterprise Manager 12c (EM), you have probably started creating groups to manage many targets together.   If you haven’t, this is one of the most critical aspects of setting up your EM to properly monitor and manage targets.  There are several use cases where you will want to perform a single action on multiple targets.

  • Setting monitoring thresholds
  • Granting privileges
  • Sending notifications
  • Applying compliance rules
  • Viewing dashboards
  • Running jobs, upgrades, backups
  • Creating reports

Continue reading “Organizing Your Oracle Enterprise Manager Targets”