Who Needs to be Super?

In the days past, everybody in EM used to end up with Super Administrator privileges due to lack of granularity in permissions.  Not any more!  Now we have more permissions than you know what to do with, but that’s another blog topic all together!

Here’s a quick list of activities that one might still need Super Administrator for – note these are all considered EM administration activities and most are accessed through the Setup menu:

  • Create Administrators – can get around this by using LDAP integration and auto-provisioning
  • Access EM Audit
  • Access Security Console
  • License Management
  • Connector Setup
  • Data Exchange
  • Add/Edit Registration Password
  • Configure Notification Mail Server
  • Configure Notification Mail Customization
  • Configure SNMP
  • Configure Global Repeat Notifications
  • Setup Privilege Delegation Templates (sudo/pbrun)
  • Decommission Agent (bug fix 19430853 will allow users with Full privileges to perform again)

If your user is not responsible for users and security, and doesn’t get paged when EM stops working, then they have no business with Super Administrator privileges or SYSMAN.

Preventing Alerts on OS Audit File Size when Upgrading DB Plug-in

In January, the DB Plug-in was released for EM 12c.   Not long after, my friend Brian found they added a new metric with default thresholds.   The new metric group is Operating System Audit Files and the metric alerts on Size of Audit Files.  Depending on the size and agent of your environment, you may immediately start getting notifications or pages as the default thresholds are 10MB/20MB, which can be quite small.

An example of the notification you might receive:

 Target type=Database Instance 
 Target name=emrep 
 Message=35.39 MB of Audit Trail files collected (.aud: 35.39, .xml: 0, .bin: 0) 
 Event reported time=Feb 6, 2015 6:53:56 AM PST 
 Target Lifecycle Status=Production 
 Operating System=Linux
 Associated Incident Id=2103 
 Associated Incident Status=New 
 Associated Incident Owner= 
 Associated Incident Acknowledged By Owner=No 
 Associated Incident Priority=None 
 Associated Incident Escalation Level=0 
 Event Type=Metric Alert 
 Event name=sizeOfOSAuditFiles:FILE_SIZE 
 Metric Group=Operating System Audit Records
 Metric=Size of Audit Files
 Metric value=35.39
 Key Value= 
 Rule Name=DBA_Incident_Rule,Create incident for critical metric alerts 
 Rule Owner=SYSMAN 
 Update Details:
 3.39 MB of Audit Trail files collected (.aud: 3.39, .xml: 0, .bin: 0)
 Incident created by rule (Name = DBA_Incident_Rule, Create incident for critical metric alerts; Owner = SYSMAN).

So if you’re planning to upgrade 1000 agents with the new Database Plugin, you might start getting a little nervous about receiving all of these pages.   Since the metric didn’t exist before, it’s not included in your templates to be disabled.  Even if it were in the templates, it would likely alert before you could reapply templates.

Luckily, Incident Rules provide a method to exclude a particular event when evaluating an Incident Rule.

From Setup -> Incidents -> Incident Rules, you’ll want to edit your defined Incident Rule.  If you haven’t customized an Incident Rule, you can select the default Incident Ruleset and do a Create Like to clone and be able to edit.


Select the Metric Alert rule, and click Edit.


On this first screen, you’ll see the Advanced Selection Options.  If you expand this you’ll see an option for Event name.  This is where you can exclude a specific metric event by select Not Equals and enter the event name.   In the case of this metric, the event name is sizeOfOSAuditFiles:FILE_SIZE.



Click Next and Continue until you finally get to Save.   To validate, you can use the Simulate Rules or trigger an alert to see if it sends the email.

This concept can be applied to help filter out other events, or categories of metrics as needed.


Resolving Conflicts While Patching the EM Agent

Unless you’re in a cave asleep, you’ve seen the recent Oracle PSUs and patches that were released in January.  This has many of my customers patching their agents, and a few have noticed a problem with some previously applied patches.  Thanks to Brian for pointing this out!

If you applied the recommended Java patches sometime last year (18502187, 18721761), and you go to apply the agent patch 20282974 (or an earlier bundle), the analyze step will fail as the Java patches are included in the agent bundle, but for some reason it can’t “ignore” them.    Checking the detailed results of the failed Analyze job will show which step failed.   In this specific case, the step PrerequisiteCheckForApply will be marked Failed and the error will look like this:


To remedy this with EM, you can create a patch plan with the Java patches (18502187, 18721761) and select Rollback in the Deployment Options.  This will run the rollback for these patches.


Once rolled back, re-analyze your patch plan and it should be successful and allow you to deploy!

When you update Agents, keep these things in mind:

  • Be sure to update any Agent clones/gold images with the newly updated patch and plug-ins if they’ve been upgraded or patched.
  • If you staged patches for auto-deployment in the OMS $ORACLE_HOME/install/oneoffs, remove the old patches and you should just need the patch now, or any current Discovery patches.
  • Update your procedural documents to reflect the current patches required.

For more details on how to patch your agent using EM, see the Administrator’s guide.


Enterprise Manager Presentations at RMOUG Training Days 2015

RMOUG Training Days – Feb 17-19 (Denver)

This is a great regional Oracle User’s Group conference with some of the best speakers you’ll find on multiple subjects!  Something enticing about being in freezing cold temperatures during ski season I think?  My colleague Kellyn Pot’vin-Gorman does a great job promoting it and making sure people get a great educational experience.   If you’re in the area, it’s a great 2 day conference!   See the agenda and register now!

I will be presenting two sessions on Thursday: Smarter Monitoring with Adaptive Thresholds and Time-Based Metrics and  Enterprise Manager 12c: Extending Beyond the DBA.  Several of my colleagues will also be there presenting and running Deep Dives covering Enterprise Manager, check out the line up of Oracle speakers presenting on #EM12c:

Tuesday 2/17
1:00 – 3:00 Hands-On: Delivering Schema as a Service with Enterprise Manager (Pete Sharman)

3:15 – 5:15 Hands-On: Delivering Pluggable Database as a Service with Enterprise Manager (Pete Sharman)

Wednesday 2/18

8:30-9:30 A Day in the Life of an Enterprise Manager Agent (Andrew Bulloch)

12:15-1:15 SIG Meetings – Enterprise Manager 12c

2:30-3:30 Enterprise Manager Snap Clone: Snapshot Your Data without Snapping Your Storage (Pete Sharman)

Thursday 2/19

9:45-10:45 Smarter Monitoring with Adaptive Thresholds and Time-Based Metrics (Courtney Llamas)

11:15-12:15 Enterprise Manager 12c: Extending Beyond the DBA (Courtney Llamas)

2:45-3:45 Under the Hood of Enterprise Manager: A Troubleshooting Primer (Werner De Gruyter)

4:00-5:00 The Power of the AWR Warehouse (Kellyn Gorman)

4:00-5:00 The High Availability Enterprise Manager Roundtable (Werner De Gruyter)

Enterprise Manager Patching FAQ

Every time a new plug-in, patch or PSU comes out for Enterprise Manager, I get a series of questions.   I’m going to quickly address a few of the ones I’ve gotten this month with the release of the new plug-ins and the PSU patches to try and help you in updating your EM site. I wrote a detailed blog EM Patching 101 about the various patches, and that’s still relevant so read if you haven’t (or read it again)!

What’s the difference between the Plug-in release (released 1/5/2015) and the Plug-in system bundle (released 12/31/2014)?  

The plug-in system bundle will come out every month on the last day of the month.  The next one is scheduled for 1/31/2015.   This is a cumulative compilation of monthly plug-in bundle patches.   So it will look at what plug-ins you have installed on the OMS, and it will deploy the latest plug-in patch bundle to that plug-in.  It will skip what you don’t have installed.   These patches will include fixes and will be indicated by the 5th digit in the version.  For example, the latest Database plug-in patch was

The plug-in updates released on 1/5 were full plug-in version upgrades.  These need to be deployed to the OMS by using Extensibility -> Plug-ins and deploy to OMS.  They may require downtime.  Plug-in updates for Database, Fusion Middleware, Storage Management and Cloud were released.  These updates include new features and fixes.   For example, the Database plug-in is now

Which one should I apply?

My recommendation would be to test and apply both.   The plug-in updates for Database and other components you use have the latest features and code that will enable new features or improvements.   You will also want to apply the latest system bundle patch (whether it’s December’s or January’s) to patch the other plug-ins that did not get updated (MOS for example).

Should I deploy the new Plug-ins first or apply patches first?

My suggestion is to apply the Plug-in updates first.  The plug-ins released in January include Database, Fusion Middleware, Cloud and Storage Management plug-ins.  When you download and apply these to the OMS, and Agents if required, they will not need the patches from the December 31st patch bundle, and you’ll have the most recent update. After deploying the plug-in updates, when you apply the Plug-in System patch (20188140) you will see that the updated plug-ins (DB, MW, etc) are skipped.

How often should I patch EM?   Agents?

I recommend to most my customers to stick to a quarterly patching cycle, and this typically follows the PSU cycle.  So start looking at the patches that are released in the January PSU and testing them.  At that time, grab all the recommended or latest plug-in and agent patches, and apply those as well.   The agent and plug-in patches will come out monthly, and if you need a patch for a particular bug or issue, then you should apply.  However, patching EM monthly is not feasible for most customers.

What patches do I need to apply as of now (January 2015)?

As mentioned in the previous Patching 101 blog, there’s multiple components involved that each have their own patch. So here’s what I’d recommend if you were my customer deploying or updating this month:

  • Latest Database PSU Patch (1/15) for your repository version
  • OMS PSU Patch – 19830994
  • OMS Plug-in System Patch – 20188140
  • Weblogic PSU Patch – 19637463
  • WebLogic Server one-off recommended – 16420963
  • OPMN Patch (CVE-2014-4212) – 19345576
  • Oracle Help Patch (CVE-2015-0426) – 20075252
  • Agent Bundle – 20109357
  • Agent Plugin Patches for non-Updated plug-ins as necessary – From Doc ID 1900943.1  (Siebel, OVI, Exadata)

When I apply Weblogic PSU it has a conflict, can I roll it back and how do I do it?

When you follow instructions to apply the PSU, you may receive a warning that there’s a conflict and patches are mutually exclusive such as the following:

$ ./bsu.sh -install -patch_download_dir=<MW_HOME>/utils/bsu/cache_dir -patchlist=12UV -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch 12UV is mutually exclusive and cannot coexist with patch(es): FSR2

You must first deinstall the conflict patch first, and install the new PSU.  The patches are cumulative.

$ ./bsu.sh -remove -patchlist=FSR2 -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Removing Patch ID: FSR2..
Result: Success

Now you can install the new PSU:

$ ./bsu.sh -install -patch_download_dir=<MW_HOME>/utils/bsu/cache_dir -patchlist=12UV -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Installing Patch ID: 12UV..
Result: Success

How do I apply the Oracle Help Patch (20075252)?

The January PSU calls for 17617649 on the Oracle Help product but this conflicts with a pre-existing EM patch.   There is a merge patch 20075252 that is available for this.  The Oracle Help patch is applied to the <MW_HOME?/oracle_common and uses the OPatch from that directory as well.    Read the readme.txt for full patching steps, but some tips are listed here in applying.

emctl stop oms -all
export MW_HOME=<MW_HOME>
export ORACLE_HOME=$MW_HOME/oracle_common

You should be able to do an opatch lsinventory and get the output of the ORACLE_HOME and the opatch apply should be similar to below:

How do I apply the OPMN Patch (19345576)?

Following the readme.txt it wants you to set Oracle Home to the  Classic/WebTier home.  For EM, this is the MW_HOME/Oracle_WT directory.

emctl stop oms -all
export MW_HOME=<MW_HOME>
export PATH=$MW_HOME/oracle_common/OPatch:$PATH

You should be able to do an opatch lsinventory and get the output of the ORACLE_HOME and the opatch apply should be similar to below:

Let me know if you have any other patching questions and I’ll add them here!

On the Books for 2015…

As always, starting 2015 with a full plate! Several presentations and papers in the works for the next few months… now if my kids could stay healthy and get back to school so I can get some real work done!   One webinar, 2 presentations, 1 whitepaper and a hands on lab in progress… oh and finding time to blog on my list of topics that keeps growing…

IOUG Webinar – Jan 21 at 12:00pm CDT – Online

I’m excited to do another webinar for IOUG this year, covering EM install and setup.  Be sure to register!

Zero to Manageability in One Hour: Build a Solid Foundation for Oracle Enterprise Manager 12c

The goal in every Oracle Manager 12c rollout is to take it from zero to manageability in the shortest time possible. This presentation will show you how to accomplish this feat.

Speakers will demonstrate how to properly architect and deploy Oracle Enterprise Manager 12c implementation, including designing for high availability and scalability. Through this demonstration a list of essential techniques and tips compiled from Oracle Enterprise Manager Development’s Strategic Customer Programs team will also be shared.   Topics such as users, roles, groups, templates, and incidents will be discussed, plus key architectural decisions.

By attending this webinar, attendees will learn how to:
• Organize targets, notifications and users properly
• Configure for best practices after the install is complete
• Properly plan and architect an EM 12c environment

RMOUG Training Days – Feb 17-19 – Denver, CO

This is a great regional Oracle User’s Group conference with some of the best speakers you’ll find on multiple subjects!  Something enticing about being in freezing cold temperatures during ski season I think?    I  don’t ski, but I love this conference because I can get a little personal benefit from it.  I grew up in Fort Collins, CO and proud to be a CSU Ram.  I still have many friends and family there so when I get back I try to meet up with someone!

My colleague Kellyn Pot’vin does a great job setting up the agenda and making sure people get a great educational experience.   If you’re in the area, it’s a great 2 day conference at reasonable rates.   See the agenda here and register.

I will be presenting Enterprise Manager 12c: Extending Beyond the DBA on Thursday at 11:15.    Several of my colleagues will also be there presenting and providing Hand on Labs covering Enterprise Manager and we’ll also be holding an EM Roundtable to wrap up the conference.

Collaborate – April 12-16 – Las Vegas, NV

Last year was my first year at Collaborate and I was surprised at how big and diverse this conference was for a User Group sponsored event.  They’ve done a great job making it a full week of great sessions, exhibits and labs!  If you haven’t gone before, maybe this is the year to check it out?   You can find out about registration here.

I don’t have the schedule for my sessions, but I do know it’s going to be busy!   I have two sessions scheduled so far – Zero to Manageability in One Hour: Build a Solid Foundation for Oracle Enterprise Manager 12c and Smarter Monitoring with Adaptive Thresholds and Time Based Metrics.  I’m also co-presenting with Werner  on Under the Hood of Enterprise Manager – a Troubleshooting Primer.  We’re hoping to get a Hand on Lab set up for EM as well, still waiting for word on that!

Besides, who doesn’t love a few days in Las Vegas?   This year the conference is at Mandalay Bay, which is where my husband and I got married almost 12 years ago. Hoping to get him to join me for a few days if we can line up a sitter!

That should be enough for this Spring right??  At least for me it is!

Creating EM Users for Performance Tuning

One of the key features for Database administration in Enterprise Manager 12c is the Performance pages utilizing the Diagnostic and Tuning packs. The DBAs are very familiar with these. Many customers are starting to ask how to let their Developers access the performance data without allowing full access to production.

Database Authentication

Even if a user has Super Admin permissions in EM, it doesn’t grant them access to the database target at all. With View Target permission on a database target, a user can get to the main target screen, as seen below:


As soon as they try to dig further into the SQL or Schema, they will be presented with a login box.


The EM user must have Connect/Connect Target Read-only privilege to get a login prompt to a target.   You can either use a Named credential, or create a New credential.  If the EM user has access to Named credentials, they will be able to save the credential and set that as a Preferred credential for this target.  If the Preferred credential is set, the login will be automatic from now on.

From here, what the user can do is completely dependent on what that login was.   Do you have sys as sysdba logins?  If so, you can do just about everything.   Or do you have a restricted read-only database account that limits your actions?   Whatever database account permissions were set are still enforced through EM.   So the question is… what permissions do you need?

Database Authorization

As with any user account in an Oracle database, you have to specify explicit privileges or roles to be able to access certain pieces. Just as you would have to grant select on X table, you have to grant access to specific performance views and functions.

Performance Views

For basic view permissions on the Performance pages, the user needs at a minimum create session and select any dictionary.  If you were able to login, then you already have create session.   So let’s grant select any dictionary.

SQL> grant select any dictionary to scott;

Now when we go to the Performance Home page, we will see the data as below:



That works great for the Performance pages, but  let’s say you want to run an AWR report, can we do that?


Nope.  As you can see, we get an error that execute privilege on DBMS_WORKLOAD_REPOSITORY is required for this function.  So let’s grant that:

SQL> grant execute on dbms_workload_repository to scott;


Now you can successfully run ASH and AWR reports from EM.

SQL Access Advisor

SQL Access Advisor has additional permissions at the database level that are required.  SQL Access Advisor also submits a DBMS Scheduler job on the database, so we also need access to create jobs.  Without this persmision you’ll get an error as follows:


SQL> grant oem_advisor to scott;
SQL> grant create job to scott;

Now when we submit the SQL Access job, we get a Confirmation message that links to the Scheduler Job :



 SQL Tuning Advisor

For SQL Tuning Advisor, you’ll need to create a SQL Tuning Set and have permissions to dbms_workload_repository and administer the SQL Tuning sets.

SQL> grant execute on dbms_workload_repository to scott;
SQL> grant administer sql tuning set to scott;

Once you have these additional permissions, you’ll be able to successfully create a SQL Tuning Advisor job and receive the results.



These are some of the more common performance tuning privileges required to use the features through EM, however there are a lot of other features that could be accessed.   Depending on what functions the user requires, you may need additional permissions at the database level.    For additional privileges required, see Where to Find Information About Performance Related Features (Doc ID 1361401.1).

Hopefully this short blog will help you start to think about what roles and permissions your database users might need in order to make full use out of the Oracle Enterprise Manager Performance pages!

Notifications for Expiring DBSNMP Passwords

Most user accounts these days have a password profile on them that automatically expires the password after a set number of days.   Depending on your company’s security requirements, this may be as little as 30 days or as long as 365 days, although typically it falls between 60-90 days. For a normal user, this can cause a small interruption in your day as you have to go get your password reset by an admin. When this happens to privileged accounts, such as the DBSNMP account that is responsible for monitoring database availability, it can cause bigger problems.

In Oracle Enterprise Manager 12c you may notice the error message “ORA-28002: the password will expire within 5 days” when you connect to a target, or worse you may get “ORA-28001: the password has expired”. If you wait too long, your monitoring will fail because the password is locked out. Wouldn’t it be nice if we could get an alert 10 days before our DBSNMP password expired? Thanks to Oracle Enterprise Manager 12c Metric Extensions (ME), you can! See the Oracle Enterprise Manager Cloud Control Administrator’s Guide for more information on Metric Extensions.

Read more here

Getting Ready for OpenWorld 2014

This is going to be a busy month as I finish up presentations for multiple sessions at OpenWorld 2014.  It’s always such a great event, but a lot of work in September!

There’s a lot of valuable EM sessions that you can find here, but I’m going to highlight a few of the ones I’m working on with my team.

Zero to Manageability in One Hour: Build a Solid Foundation for Oracle Enterprise Manager 12c [CON8134]

Wednesday, Oct 1, 12:45 PM – 1:30 PM - Moscone South – 303
Speakers: Courtney Llamas (Oracle), Kellyn Pot’Vin (Oracle),  Dan Brint (SUNY)

We’re calling this our EM 101.  While we will have to skip a lot of the basic information due to the shorten sessions, this is geared towards those who are installing or upgrading to EM 12c and would like to learn from our experience with our “Fast Track” Methodology.   Even if you’ve already implemented or upgraded, you might learn a thing or two about organizing targets or managing your EM system.  We will also save some time for one of our customers (State University of New York) to share their experience in this process.

Abstract: Learn how to properly architect and deploy your Oracle Enterprise Manager 12c implementation, including designing for high availability and scalability. This session focuses on the essential tasks an implementation team must do to ensure Oracle Enterprise Manager 12c rollouts occur in a timely fashion and deliver a solid foundation for exploiting the marquee features of Oracle Enterprise Manager 12c. Topics such as users, roles, groups, templates, and incidents are discussed, plus key architectural decisions. The Strategic Customer Programs team in Oracle Enterprise Manager Development works with customers worldwide to guide them to success with their implementations and has compiled some essential techniques and tips for getting from zero to manageability in the shortest time possible. 

Under the Hood: Diagnosing and Troubleshooting Oracle Enterprise Manager 12c Release 4 [CON8225]

Monday, Sep 29, 1:30 PM – 2:15 PM - Moscone South – 302
Speakers:  Werner DeGruyter (Oracle), Andrew Bulloch (Oracle)

This is our “graduate level” class.  For the people responsible for EM day by day, who have to troubleshoot performance issues or agent issues, this session is for you!  Don’t miss out.

Abstract:  This session equips attendees with the knowledge they need in order to understand the health and status of Oracle Enterprise Manager. It covers essential background information on the Oracle Enterprise Manager infrastructure (such as the repository and the agent) and selected managed targets. Building on this knowledge, the session then looks a little deeper and discusses techniques and tips for monitoring the performance of Oracle Enterprise Manager and tuning any bottlenecks that are commonly identified in Oracle Enterprise Manager. It also explores the essential parts of the Oracle Enterprise Manager UI every Oracle Enterprise Manager administrator should be familiar with and discusses some techniques for logging and tracing to assist in diagnosing problems.

Wait, there’s more!

In addition to the sessions above, I’ll be co-presenting with a few customers on their sessions, so come see how real customers are using EM to manage and monitor their targets!

Advanced Diagnostics and Monitoring with Oracle Enterprise Manager 12c [CON4114]

Thursday, Oct 2, 10:45 AM – 11:30 AM - Moscone South – 301
Speakers:  Tyler Sharp (Cerner), Aaron Rimel (Cerner), Courtney Llamas

Abstract:  This session covers how Cerner Corporation, a leading healthcare IT company, leverages Oracle Enterprise Manager 12c for database-specific thresholds, creating proactive monitoring of 26,000+ targets. Learn how to scrape Automatic Workload Repository data combined with Oracle Enterprise Manager repository data, exploiting common script issues in a homogeneous environment or personalized thresholds with historical data to proactively alarm, hours or even days before a potential outage. Report on compliance to feed management data on complexity and compliance of your data centers. With metric extensions, Cerner realized a 40 percent reduction in patient hours affected. Increase returns on your Oracle Enterprise Manager investment, and move your organization to proactive smart alerting.

Using Oracle Enterprise Manager to Deliver Multitenant DBaaS on Oracle Exadata: Lessons Learned [CON5875]

Tuesday, Sep 30, 5:00 PM – 5:45 PM - Moscone South – 301
Speakers:  Manish Shah (Hartford), Brian Bennett (Hartford), Courtney Llamas

Abstract: Have you migrated to or are you planning to migrate to Oracle Enterprise Manager 12c? Want to know how to automate repetitive tasks and categorize and auto-assign targets to customized monitoring and alerting groups based on bronze, silver, and gold definitions in your DBaaS catalog? This session shows how to build an Oracle Enterprise Manager 12c framework aligned with your organizational model to manage targets on various platforms such as Oracle Database 12c pluggable and container databases. You’ll also learn how to optimize these targets on Oracle Exadata with a combination of administrative and privilege propagating groups, target properties, monitoring templates, and enterprise rule sets, resulting in a flexible yet scalable operational setup for your DBaaS offerings.

Full Visibility into Oracle WebLogic/Java Diagnostics with Oracle Enterprise Manager 12c [CON5983]

Monday, Sep 29, 5:15 PM – 6:00 PM - Moscone South – 200
Speakers:  Mark Consoles (Omgeo), Eldad Ganin (Omgeo), Avi Huber (Oracle)

Abstract: In this session, hear how Omgeo’s application team has leveraged Oracle Enterprise Manager Cloud Control 12c to streamline monitoring and management of its Oracle WebLogic Servers, JVMs, Oracle iPlanet Web Servers, and more to ensure that it can maximize the visibility and reliability of the company’s application systems built on Oracle technology to their fullest extent. By leveraging JVM Diagnostics and Oracle WebLogic monitoring to pinpoint and isolate problems in product environments without having to reproduce them in a test environment, Omgeo has been able to cut down its root cause analysis by orders of magnitude


Enterprise Manager, Oracle and other bits of life.