EM 12cR5 – What’s the Scoop?

In case you missed the announcement, EM 12.1.0.5 has been announced.  See https://blogs.oracle.com/oem/entry/new_enterprise_manager_release_allows for details.

Many of my customers have heard about the release of EM 12.1.0.5 last week and wondering what are the new features and whether they should plan to update right away.

The main new feature in 12.1.0.5 is the Hybrid Cloud, which essentially enables a customer to use their on-premise Enterprise Manager to view their Oracle  Cloud targets.  This has been a long requested feature allowing the customers to manage and monitor their Oracle Cloud targets alongside their on-premise targets.   This also enables additional features, such as easy cloning to and from the Oracle Cloud.

The Database Plug-in has been updated to 12.1.0.8 with the main features also enhancing the Hybrid Cloud experience.   For Exadata customers, this plug-in will enable thin cloning options on Exadata ACFS.

The Middleware Plug-in has also been updated to 12.1.0.8 and will enable additional MW cloud options such as allowing users to deploy JVMD agents to Oracle Cloud virtual hosts,

Additional updates have been made to the Cloud, Virtual Infrastructure and Chargeback and Consolidation Plug-ins to enhance and enable the Hybrid Cloud functions.
In summary, if you are an Oracle Cloud customer, then this release is a must upgrade!   For full details about the new features enabled in the 12.1.0.5 upgrade see the documentation here.

 

Join me at Collaborate 2015!

In a few weeks I’ll be heading to Las Vegas for a few days of relaxation and fun.  Well, really I’ll be working, but it will still be fun!

As always, I’ve got a busy schedule planned.  Here’s where you can find me over the week if you’re attending (there will be more but this is what I know now!):

Sunday 4/12

9-12p – Everything I Needed to Know About Enterprise Manager I Learned at COLLABORATE – Hands-on Lab

This should be fun.  Three hours w/ myself, Kellyn and Werner.  We may not stay on topic, but I can promise you’ll learn something new!

Monday 4/13

9:15-10:15 – Zero to Manageability in One Hour: Build a Solid Foundation for Oracle Enterprise Manager 12c

Learn how to properly plan, architect and implement EM12c, and how to setup the monitoring framework (roles, users, groups, templates, etc).

5:30-8:00 – Welcome Reception in Exhibit Hall at EM booth

Tuesday 4/14

5:30-7:00 – Happy Hour in Exhibit Hall at EM booth

Thursday 4/16

11:00-12:00 – Under the Hood of Enterprise Manager – A Troubleshooting Primer

Really, this is Werner’s session not mine.  But if you’ve never heard him talk about EM, you’re missing out.  Learn how to troubleshoot your EM system and find out what it’s doing.

12:15-1:15 – Smarter Monitoring with Adaptive Thresholds and Time Based Metrics

Then just stay seated for my talk on advancing your monitoring with adaptive and time based thresholds, metric extensions and more.

There are many more EM sessions and I’ll be trying to hit as many as I can!   Here’s the full list and hours for the EM demo booth.

If you can’t make it to Las Vegas, take a look at Collaborates Plug-in to Vegas option!  For a registration fee of $395, you get access to sessions from the top 3 tracks – Cloud Computing, Database Performance and Manageability.  Sessions are broadcast live online. If you’re interested in EM12c, there’s quite a line up of sessions available, including mine!

Who Needs to be Super?

In the days past, everybody in EM used to end up with Super Administrator privileges due to lack of granularity in permissions.  Not any more!  Now we have more permissions than you know what to do with, but that’s another blog topic all together!

Here’s a quick list of activities that one might still need Super Administrator for – note these are all considered EM administration activities and most are accessed through the Setup menu:

  • Create Administrators – can get around this by using LDAP integration and auto-provisioning
  • Access EM Audit
  • Access Security Console
  • License Management
  • Connector Setup
  • Data Exchange
  • Add/Edit Registration Password
  • Configure Notification Mail Server
  • Configure Notification Mail Customization
  • Configure SNMP
  • Configure Global Repeat Notifications
  • Setup Privilege Delegation Templates (sudo/pbrun)
  • Decommission Agent (bug fix 19430853 will allow users with Full privileges to perform again)

If your user is not responsible for users and security, and doesn’t get paged when EM stops working, then they have no business with Super Administrator privileges or SYSMAN.

Preventing Alerts on OS Audit File Size when Upgrading DB Plug-in

In January, the DB Plug-in 12.1.0.7.0 was released for EM 12c.   Not long after, my friend Brian found they added a new metric with default thresholds.   The new metric group is Operating System Audit Files and the metric alerts on Size of Audit Files.  Depending on the size and agent of your environment, you may immediately start getting notifications or pages as the default thresholds are 10MB/20MB, which can be quite small.

An example of the notification you might receive:

 Host=xxxxxx.us.oracle.com 
 Target type=Database Instance 
 Target name=emrep 
 Categories=Capacity 
 Message=35.39 MB of Audit Trail files collected (.aud: 35.39, .xml: 0, .bin: 0) 
 Severity=Critical 
 Event reported time=Feb 6, 2015 6:53:56 AM PST 
 Target Lifecycle Status=Production 
 Operating System=Linux
 Platform=x86_64
 Department=DBA
 Associated Incident Id=2103 
 Associated Incident Status=New 
 Associated Incident Owner= 
 Associated Incident Acknowledged By Owner=No 
 Associated Incident Priority=None 
 Associated Incident Escalation Level=0 
 Event Type=Metric Alert 
 Event name=sizeOfOSAuditFiles:FILE_SIZE 
 Metric Group=Operating System Audit Records
 Metric=Size of Audit Files
 Metric value=35.39
 Key Value= 
 Rule Name=DBA_Incident_Rule,Create incident for critical metric alerts 
 Rule Owner=SYSMAN 
 Update Details:
 3.39 MB of Audit Trail files collected (.aud: 3.39, .xml: 0, .bin: 0)
 Incident created by rule (Name = DBA_Incident_Rule, Create incident for critical metric alerts; Owner = SYSMAN).


So if you’re planning to upgrade 1000 agents with the new Database Plugin, you might start getting a little nervous about receiving all of these pages.   Since the metric didn’t exist before, it’s not included in your templates to be disabled.  Even if it were in the templates, it would likely alert before you could reapply templates.

Luckily, Incident Rules provide a method to exclude a particular event when evaluating an Incident Rule.

From Setup -> Incidents -> Incident Rules, you’ll want to edit your defined Incident Rule.  If you haven’t customized an Incident Rule, you can select the default Incident Ruleset and do a Create Like to clone and be able to edit.

inc1

Select the Metric Alert rule, and click Edit.

inc2

On this first screen, you’ll see the Advanced Selection Options.  If you expand this you’ll see an option for Event name.  This is where you can exclude a specific metric event by select Not Equals and enter the event name.   In the case of this metric, the event name is sizeOfOSAuditFiles:FILE_SIZE.

inc3

 

Click Next and Continue until you finally get to Save.   To validate, you can use the Simulate Rules or trigger an alert to see if it sends the email.

This concept can be applied to help filter out other events, or categories of metrics as needed.

 

Resolving Conflicts While Patching the EM Agent

Unless you’re in a cave asleep, you’ve seen the recent Oracle PSUs and patches that were released in January.  This has many of my customers patching their agents, and a few have noticed a problem with some previously applied patches.  Thanks to Brian for pointing this out!

If you applied the recommended Java patches sometime last year (18502187, 18721761), and you go to apply the 12.1.0.4.5 agent patch 20282974 (or an earlier bundle), the analyze step will fail as the Java patches are included in the agent bundle, but for some reason it can’t “ignore” them.    Checking the detailed results of the failed Analyze job will show which step failed.   In this specific case, the step PrerequisiteCheckForApply will be marked Failed and the error will look like this:

ap1

To remedy this with EM, you can create a patch plan with the Java patches (18502187, 18721761) and select Rollback in the Deployment Options.  This will run the rollback for these patches.

ap2

Once rolled back, re-analyze your 12.1.0.4.5 patch plan and it should be successful and allow you to deploy!

When you update Agents, keep these things in mind:

  • Be sure to update any Agent clones/gold images with the newly updated patch and plug-ins if they’ve been upgraded or patched.
  • If you staged patches for auto-deployment in the OMS $ORACLE_HOME/install/oneoffs, remove the old patches and you should just need the 12.1.0.4.5 patch now, or any current Discovery patches.
  • Update your procedural documents to reflect the current patches required.

For more details on how to patch your agent using EM, see the Administrator’s guide.

 

Enterprise Manager Presentations at RMOUG Training Days 2015

RMOUG Training Days – Feb 17-19 (Denver)

This is a great regional Oracle User’s Group conference with some of the best speakers you’ll find on multiple subjects!  Something enticing about being in freezing cold temperatures during ski season I think?  My colleague Kellyn Pot’vin-Gorman does a great job promoting it and making sure people get a great educational experience.   If you’re in the area, it’s a great 2 day conference!   See the agenda and register now!

I will be presenting two sessions on Thursday: Smarter Monitoring with Adaptive Thresholds and Time-Based Metrics and  Enterprise Manager 12c: Extending Beyond the DBA.  Several of my colleagues will also be there presenting and running Deep Dives covering Enterprise Manager, check out the line up of Oracle speakers presenting on #EM12c:

Tuesday 2/17
1:00 – 3:00 Hands-On: Delivering Schema as a Service with Enterprise Manager (Pete Sharman)

3:15 – 5:15 Hands-On: Delivering Pluggable Database as a Service with Enterprise Manager (Pete Sharman)

Wednesday 2/18

8:30-9:30 A Day in the Life of an Enterprise Manager Agent (Andrew Bulloch)

12:15-1:15 SIG Meetings – Enterprise Manager 12c

2:30-3:30 Enterprise Manager Snap Clone: Snapshot Your Data without Snapping Your Storage (Pete Sharman)

Thursday 2/19

9:45-10:45 Smarter Monitoring with Adaptive Thresholds and Time-Based Metrics (Courtney Llamas)

11:15-12:15 Enterprise Manager 12c: Extending Beyond the DBA (Courtney Llamas)

2:45-3:45 Under the Hood of Enterprise Manager: A Troubleshooting Primer (Werner De Gruyter)

4:00-5:00 The Power of the AWR Warehouse (Kellyn Gorman)

4:00-5:00 The High Availability Enterprise Manager Roundtable (Werner De Gruyter)

Enterprise Manager Patching FAQ

Every time a new plug-in, patch or PSU comes out for Enterprise Manager, I get a series of questions.   I’m going to quickly address a few of the ones I’ve gotten this month with the release of the new plug-ins and the PSU patches to try and help you in updating your EM site. I wrote a detailed blog EM Patching 101 about the various patches, and that’s still relevant so read if you haven’t (or read it again)!

What’s the difference between the Plug-in release (released 1/5/2015) and the Plug-in system bundle (released 12/31/2014)?  

The plug-in system bundle will come out every month on the last day of the month.  The next one is scheduled for 1/31/2015.   This is a cumulative compilation of monthly plug-in bundle patches.   So it will look at what plug-ins you have installed on the OMS, and it will deploy the latest plug-in patch bundle to that plug-in.  It will skip what you don’t have installed.   These patches will include fixes and will be indicated by the 5th digit in the version.  For example, the latest Database plug-in patch was 12.1.0.6.7.

The plug-in updates released on 1/5 were full plug-in version upgrades.  These need to be deployed to the OMS by using Extensibility -> Plug-ins and deploy to OMS.  They may require downtime.  Plug-in updates for Database, Fusion Middleware, Storage Management and Cloud were released.  These updates include new features and fixes.   For example, the Database plug-in is now 12.1.0.7.0.

Which one should I apply?

My recommendation would be to test and apply both.   The plug-in updates for Database and other components you use have the latest features and code that will enable new features or improvements.   You will also want to apply the latest system bundle patch (whether it’s December’s or January’s) to patch the other plug-ins that did not get updated (MOS for example).

Should I deploy the new Plug-ins first or apply patches first?

My suggestion is to apply the Plug-in updates first.  The plug-ins released in January include Database, Fusion Middleware, Cloud and Storage Management plug-ins.  When you download and apply these to the OMS, and Agents if required, they will not need the patches from the December 31st patch bundle, and you’ll have the most recent update. After deploying the plug-in updates, when you apply the Plug-in System patch (20188140) you will see that the updated plug-ins (DB, MW, etc) are skipped.

How often should I patch EM?   Agents?

I recommend to most my customers to stick to a quarterly patching cycle, and this typically follows the PSU cycle.  So start looking at the patches that are released in the January PSU and testing them.  At that time, grab all the recommended or latest plug-in and agent patches, and apply those as well.   The agent and plug-in patches will come out monthly, and if you need a patch for a particular bug or issue, then you should apply.  However, patching EM monthly is not feasible for most customers.

What patches do I need to apply as of now (January 2015)?

As mentioned in the previous Patching 101 blog, there’s multiple components involved that each have their own patch. So here’s what I’d recommend if you were my customer deploying or updating this month:

  • Latest Database PSU Patch (1/15) for your repository version
  • OMS 12.1.0.4.2 PSU Patch – 19830994
  • OMS 12.1.0.4.7 Plug-in System Patch – 20188140
  • Weblogic 10.3.0.6.0.10 PSU Patch – 19637463
  • WebLogic Server one-off recommended – 16420963
  • OPMN Patch (CVE-2014-4212) – 19345576
  • Oracle Help Patch (CVE-2015-0426) – 20075252
  • Agent 12.1.0.4.4 Bundle – 20109357
  • Agent Plugin Patches for non-Updated plug-ins as necessary – From Doc ID 1900943.1  (Siebel, OVI, Exadata)

When I apply Weblogic PSU it has a conflict, can I roll it back and how do I do it?

When you follow instructions to apply the PSU, you may receive a warning that there’s a conflict and patches are mutually exclusive such as the following:

$ ./bsu.sh -install -patch_download_dir=<MW_HOME>/utils/bsu/cache_dir -patchlist=12UV -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch 12UV is mutually exclusive and cannot coexist with patch(es): FSR2

You must first deinstall the conflict patch first, and install the new PSU.  The patches are cumulative.

$ ./bsu.sh -remove -patchlist=FSR2 -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Removing Patch ID: FSR2..
Result: Success

Now you can install the new PSU:

$ ./bsu.sh -install -patch_download_dir=<MW_HOME>/utils/bsu/cache_dir -patchlist=12UV -prod_dir=<MW_HOME>/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Installing Patch ID: 12UV..
Result: Success

How do I apply the Oracle Help Patch (20075252)?

The January PSU calls for 17617649 on the Oracle Help product but this conflicts with a pre-existing EM patch.   There is a merge patch 20075252 that is available for this.  The Oracle Help patch is applied to the <MW_HOME?/oracle_common and uses the OPatch from that directory as well.    Read the readme.txt for full patching steps, but some tips are listed here in applying.

emctl stop oms -all
export MW_HOME=<MW_HOME>
export ORACLE_HOME=$MW_HOME/oracle_common
export PATH=$ORACLE_HOME/OPatch:$PATH

You should be able to do an opatch lsinventory and get the output of the ORACLE_HOME and the opatch apply should be similar to below:

How do I apply the OPMN Patch (19345576)?

Following the readme.txt it wants you to set Oracle Home to the  Classic/WebTier home.  For EM, this is the MW_HOME/Oracle_WT directory.

emctl stop oms -all
export MW_HOME=<MW_HOME>
export ORACLE_HOME=$MW_HOME/Oracle_WT
export PATH=$MW_HOME/oracle_common/OPatch:$PATH

You should be able to do an opatch lsinventory and get the output of the ORACLE_HOME and the opatch apply should be similar to below:

Let me know if you have any other patching questions and I’ll add them here!

On the Books for 2015…

As always, starting 2015 with a full plate! Several presentations and papers in the works for the next few months… now if my kids could stay healthy and get back to school so I can get some real work done!   One webinar, 2 presentations, 1 whitepaper and a hands on lab in progress… oh and finding time to blog on my list of topics that keeps growing…

IOUG Webinar – Jan 21 at 12:00pm CDT – Online

I’m excited to do another webinar for IOUG this year, covering EM install and setup.  Be sure to register!

Zero to Manageability in One Hour: Build a Solid Foundation for Oracle Enterprise Manager 12c

The goal in every Oracle Manager 12c rollout is to take it from zero to manageability in the shortest time possible. This presentation will show you how to accomplish this feat.

Speakers will demonstrate how to properly architect and deploy Oracle Enterprise Manager 12c implementation, including designing for high availability and scalability. Through this demonstration a list of essential techniques and tips compiled from Oracle Enterprise Manager Development’s Strategic Customer Programs team will also be shared.   Topics such as users, roles, groups, templates, and incidents will be discussed, plus key architectural decisions.

By attending this webinar, attendees will learn how to:
• Organize targets, notifications and users properly
• Configure for best practices after the install is complete
• Properly plan and architect an EM 12c environment

RMOUG Training Days – Feb 17-19 – Denver, CO

This is a great regional Oracle User’s Group conference with some of the best speakers you’ll find on multiple subjects!  Something enticing about being in freezing cold temperatures during ski season I think?    I  don’t ski, but I love this conference because I can get a little personal benefit from it.  I grew up in Fort Collins, CO and proud to be a CSU Ram.  I still have many friends and family there so when I get back I try to meet up with someone!

My colleague Kellyn Pot’vin does a great job setting up the agenda and making sure people get a great educational experience.   If you’re in the area, it’s a great 2 day conference at reasonable rates.   See the agenda here and register.

I will be presenting Enterprise Manager 12c: Extending Beyond the DBA on Thursday at 11:15.    Several of my colleagues will also be there presenting and providing Hand on Labs covering Enterprise Manager and we’ll also be holding an EM Roundtable to wrap up the conference.

Collaborate – April 12-16 – Las Vegas, NV

Last year was my first year at Collaborate and I was surprised at how big and diverse this conference was for a User Group sponsored event.  They’ve done a great job making it a full week of great sessions, exhibits and labs!  If you haven’t gone before, maybe this is the year to check it out?   You can find out about registration here.

I don’t have the schedule for my sessions, but I do know it’s going to be busy!   I have two sessions scheduled so far – Zero to Manageability in One Hour: Build a Solid Foundation for Oracle Enterprise Manager 12c and Smarter Monitoring with Adaptive Thresholds and Time Based Metrics.  I’m also co-presenting with Werner  on Under the Hood of Enterprise Manager – a Troubleshooting Primer.  We’re hoping to get a Hand on Lab set up for EM as well, still waiting for word on that!

Besides, who doesn’t love a few days in Las Vegas?   This year the conference is at Mandalay Bay, which is where my husband and I got married almost 12 years ago. Hoping to get him to join me for a few days if we can line up a sitter!

That should be enough for this Spring right??  At least for me it is!

Creating EM Users for Performance Tuning

One of the key features for Database administration in Enterprise Manager 12c is the Performance pages utilizing the Diagnostic and Tuning packs. The DBAs are very familiar with these. Many customers are starting to ask how to let their Developers access the performance data without allowing full access to production.

Database Authentication

Even if a user has Super Admin permissions in EM, it doesn’t grant them access to the database target at all. With View Target permission on a database target, a user can get to the main target screen, as seen below:

dbuser1

As soon as they try to dig further into the SQL or Schema, they will be presented with a login box.

dbuser2

The EM user must have Connect/Connect Target Read-only privilege to get a login prompt to a target.   You can either use a Named credential, or create a New credential.  If the EM user has access to Named credentials, they will be able to save the credential and set that as a Preferred credential for this target.  If the Preferred credential is set, the login will be automatic from now on.

From here, what the user can do is completely dependent on what that login was.   Do you have sys as sysdba logins?  If so, you can do just about everything.   Or do you have a restricted read-only database account that limits your actions?   Whatever database account permissions were set are still enforced through EM.   So the question is… what permissions do you need?

Database Authorization

As with any user account in an Oracle database, you have to specify explicit privileges or roles to be able to access certain pieces. Just as you would have to grant select on X table, you have to grant access to specific performance views and functions.

Performance Views

For basic view permissions on the Performance pages, the user needs at a minimum create session and select any dictionary.  If you were able to login, then you already have create session.   So let’s grant select any dictionary.

SQL> grant select any dictionary to scott;

Now when we go to the Performance Home page, we will see the data as below:

dbuser6

AWR

That works great for the Performance pages, but  let’s say you want to run an AWR report, can we do that?

dbuser3

Nope.  As you can see, we get an error that execute privilege on DBMS_WORKLOAD_REPOSITORY is required for this function.  So let’s grant that:

SQL> grant execute on dbms_workload_repository to scott;

dbuser4

Now you can successfully run ASH and AWR reports from EM.

SQL Access Advisor

SQL Access Advisor has additional permissions at the database level that are required.  SQL Access Advisor also submits a DBMS Scheduler job on the database, so we also need access to create jobs.  Without this persmision you’ll get an error as follows:

dbuser5

SQL> grant oem_advisor to scott;
SQL> grant create job to scott;

Now when we submit the SQL Access job, we get a Confirmation message that links to the Scheduler Job :

dbuser7

 

 SQL Tuning Advisor

For SQL Tuning Advisor, you’ll need to create a SQL Tuning Set and have permissions to dbms_workload_repository and administer the SQL Tuning sets.

SQL> grant execute on dbms_workload_repository to scott;
SQL> grant administer sql tuning set to scott;

Once you have these additional permissions, you’ll be able to successfully create a SQL Tuning Advisor job and receive the results.

dbuser9

 

These are some of the more common performance tuning privileges required to use the features through EM, however there are a lot of other features that could be accessed.   Depending on what functions the user requires, you may need additional permissions at the database level.    For additional privileges required, see Where to Find Information About Performance Related Features (Doc ID 1361401.1).

Hopefully this short blog will help you start to think about what roles and permissions your database users might need in order to make full use out of the Oracle Enterprise Manager Performance pages!

Enterprise Manager, Oracle and other bits of life.