Category: Enterprise Manager

Deinstalling Oracle Enterprise Manager 13c in One Step

Not that you’d want to deinstall Oracle Enterprise Manager (OEM) 13c, but I just have to tell you how easy it is now!  This was one of my  pet peeves in OEM 12c.  Deinstalling had a list of about 20 steps, and the order was very particular, as was the alignment of the sun and moon.  Not anymore, in OEM 13c, it’s one command and it’s beautiful.   Really, it’s the simple things that make me happy!

This may not be a common concern for most people, because once they install they just continue to upgrade and move forward.  However when you do a lot of testing (especially destructive testing) and you have to rebuild your environment frequently, this is a big deal.

Want to see how easy it is?  First, be sure to start with your environment up (repository database and OMS).

Copy the deinstall script to a temporary location…

$cp /u01/oracle/m13c/oraclehome/sysman/install/EMDeinstall.pl /home/cllamas

Then run it.  Be sure you have your passwords and Oracle Home handy.

$/u01/oracle/em13/perl/bin/perl /home/cllamas/EMDeinstall.pl -mwHome /u01/oracle/em13 -stageLoc /home/cllamas

That’s it.  All done.  Agent, OMS (including BI Publisher), and the schema from the repository database.   The database is not removed, just cleaned up and ready to start a new install 😉

Again.  It’s the simple things that make me happy.   For more information checkout the deinstall docs, there’s additional ways to deinstall certain components, but when you’re looking for a clean sweep this is the trick.

Output of deinstall:
Refer to /home/cllamas/deinstall_2016-08-01_08-42-48.log for deinstall log
This is a First OMS install. So, this deinstalls the OMS , Repository and Agent. Confirm (y/n)y
User confirmed for deinstallation.

Enter the SYS Password :
Enter the sysman Password :
Enter the Admin Server password :
dbConnectStr – (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=abc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))

The command is /u01/oracle/em13/bin/emctl stop oms

Stopping oms………… Wait for the completion of the execution.
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Stopping Oracle Management Server…
Oracle Management Server Already Stopped
Oracle Management Server is Down
JVMD Engine is Down
return value is : 0

The command is /u01/oracle/em13/bin/omsca delete -full -OMSNAME EMGC_OMS1 -AS_USERNAME weblogic -REP_CONN_STR “(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=abc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))”

The oms delete will take sometime. Wait for the completion of the execution. Don’t abort the execution.

Oracle Enterprise Manager Cloud Control 13c Release 13.1.0.0.0
Copyright (c) 1996, 2015, Oracle. All rights reserved. Do You really want to delete the OMS (Y|N):Deleting BI Publisher Server Named “BIP” Enter Administration Server user password:
Processing command line ….
Repository Creation Utility – Checking Prerequisites
Checking Global Prerequisites
Repository Creation Utility – Checking Prerequisites
Checking Component Prerequisites
Repository Creation Utility – Drop
Repository Drop in progress.
Percent Complete: 22
Percent Complete: 47
Percent Complete: 49
Percent Complete: 100
Repository Creation Utility: Drop – Completion Summary
Database details:
Connect Descriptor : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=abc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))
Connected As : sys
Prefix for (prefixable) Schema Owners : SYSMAN
RCU Logfile : /u01/oracle/em13/cfgtoollogs/cfgfw/emsecrepmgr.log
Component schemas dropped:
Component Status Logfile
Oracle Platform Security Services Success /u01/oracle/em13/cfgtoollogs/cfgfw/opss.log

Repository Creation Utility – Drop : Operation Completed
OMS Deleted successfully

return value is : 0
Running the command /u01/oracle/em13/sysman/admin/emdrep/bin/RepManager -action drop -dbUser sys -dbRole sysdba -connectString “(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=abc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))” -mwHome /u01/oracle/em13 -mwOraHome /u01/oracle/em13 -oracleHome /u01/oracle/em13

Repository drop will take sometime. Wait for the completion of the execution. Don’t abort the execution.

stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
processing arguments
compiling arguments for validation
Enter sys user password :
Verify :
Performing PreDropAll action…
Enter password for: sys process_id:1p5c6aeconlq4
null
Done PreDropAll action…
Dropping BIP schema…
Enter sys user password :
Verify :
Action on BIP schema succeed.
Dropping APM schema… Component is already dropped Enter sysdba password:
Dropping OPSS schema… Component is already dropped Enter sysdba password:
[01-08-2016 08:45:11] executing query : “select count(*) from all_users where username=’SYSMANUPGR_OPSS'”
[01-08-2016 08:45:12] User SYSMANUPGR_OPSS Does Not Exist
Dropping STB schema… Processing command line …. Enter sysdba password:
Repository Creation Utility – Checking Prerequisites
Checking Global Prerequisites
Repository Creation Utility – Checking Prerequisites
Checking Component Prerequisites
Repository Creation Utility – Drop
Repository Drop in progress.
Percent Complete: 9
Percent Complete: 9
Percent Complete: 9
Percent Complete: 46
Percent Complete: 100
Repository Creation Utility: Drop – Completion Summary
Database details:
Connect Descriptor : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=abc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))
Connected As : sys
Prefix for (prefixable) Schema Owners : SYSMAN
RCU Logfile : /u01/oracle/em13/cfgtoollogs/cfgfw/emsecrepmgr.log
Component schemas dropped:
Component Status Logfile
Common Infrastructure Services Success /u01/oracle/em13/cfgtoollogs/cfgfw/stb.log

Repository Creation Utility – Drop : Operation Completed
Successfully dropped schema
Dropping MDS schema…
Enter sys user password :
Verify :
Action on MDS schema succeed.
Enter password for: sys process_id:1udv4t5atkf3v
Processing command line ….
Repository Creation Utility – Checking Prerequisites
Checking Global Prerequisites
Repository Creation Utility – Checking Prerequisites
Checking Component Prerequisites
Repository Creation Utility – Drop
Repository Drop in progress.
Repository Creation Utility: Drop – Completion Summary
Database details:
Connect Descriptor : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=abc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))
Connected As : sys
RCU Logfile : /u01/oracle/em13/sysman/log/schemamanager/m_080116_0845_AM/m_080116_0845_AM.DROP/rcu.log
Component schemas dropped:
Component Status Logfile
EM Repository Drop Success /u01/oracle/em13/sysman/log/schemamanager/m_080116_0845_AM/m_080116_0845_AM.DROP/em_repos_drop.log

Repository Creation Utility – Drop : Operation Completed
Performing PostDropAll action…
Enter password for: sys process_id:22ol5ee7e7oc
drop completed successfully
return value is : 0
The command executed is /u01/oracle/em13/oui/bin/detachHome.sh
Launcher log file is /tmp/OraInstall2016-08-01_09-18-50AM/launcher2016-08-01_09-18-50AM.log.
Starting Oracle Universal Installer

Checking swap space: must be greater than 500 MB. Actual 15455 MB Passed
Checking if this platform requires a 64-bit JVM. Actual 64 Passed (64-bit not required)
‘detachHome’ was successful.
Logs successfully copied to /u01/oraInventory/logs.
return value is : 0

Deleting the instance home
Deleting the em home
The deinstallation of OMS is successful.

The location of the file is : /etc/oragchomelist

 

 

 

Viva Las Vegas – Collaborate 2016

It’s that time of year again, time to spend a few days in Las Vegas at Collaborate with some of the top Oracle experts and exercise your brain (and your feet)! I will be heading out in a few days and have a great schedule lined up!

Collaborate 16

This year will be a little different as I’m representing two products.   For a while now I’ve been working more on Oracle Management Cloud, which is our new cloud services for Log Analytics, IT Analytics and Application Performance Monitoring.   These services are complimentary to your existing Oracle Enterprise Manager solution.  From a DBA perspective, I’m especially excited about IT Analytics, as it’s an answer to many of the questions we get asked about long-term performance, capacity and resource data that Oracle Enterprise Manager collects.  We will have a demo booth, as well as a session on Monday at 10:30 in Palm B – Oracle Management Cloud: Next-Generation Monitoring, Management and Analytics.   Be sure to attend this session to hear about what Oracle Management Cloud has to offer!   You can also learn more about the Oracle Management Cloud services here.

As always, there’s a lot of excellent Oracle Enterprise Manager sessions this year!  If you’re in Las Vegas on Sunday morning, be sure to register for the Hands-on Lab: Everything I Needed to Know About Enterprise Manager I Learned at COLLABORATE.   We’ll start at 9am with an overview of EM 13c and work on new features in target properties, dynamic groups, creating gold agent images, tablespace corrective actions, and more!  You won’t want to miss out on this one!  Be sure to pre-register as space is limited.

Here’s where you’ll find me:

Sunday – 9-1 – Hands-on Lab: Everything I Needed to Know About Enterprise Manager I Learned at COLLABORATE

Monday – 10:30am Palm B – Oracle Management Cloud: Next-Generation Monitoring, Management and Analytics – Learn about the latest offering in IT Operations Management – Log Analytics, IT Analytics and Application Performance Monitoring.

Tuesday – 4:45pm Palm B – Building a High-Available Enterprise Manager system with Werner de Gruyter – This session is a must attend for anybody who needs to build or maintain a highly available Enterprise Manager.

Wednesday – 8:00am Palm B – Oracle Enterprise Manager Security: a Practitioners Guide – Rise and shine with my session on how to make Enterprise Manager security work for your company in more ways than one!

For a full list of Oracle Management Cloud and Oracle Enterprise Manager demos, labs, sessions and SIG meetings be sure to save or print this handy schedule!   Don’t worry, if you’re not heading to Las Vegas, you can still catch my session by registering for the IOUG Virtual Forum!

I will also be on the Oracle demo grounds either at the Oracle Enterprise Manager booth or Oracle Management Cloud booth.   If you follow me on twitter or read my blog and we haven’t met in person, stop by and say hi!

EM 13c – Enhancements to DB Optimizer Statistics Console

Optimizer Statistics Console

There’s a lot of new features in Enterprise Manager 13c (EM 13c) that the DBAs will love.   One of the things I’d heard customers complain about was the lack of visibility into the statistics jobs that run in the target database using DBMS_SCHEDULER. The enhanced DB Optimizer Statistics Console provides a central place to manage database statistics, view a summary of all object status, and the status and performance of the statistics jobs that have run.   To access the console, from a database target select Performance / SQL / Optimizer Statistics.

The Operations, Configure and Status sections haven’t changed much.  In the Operations section you can Gather, Lock, Unlock, View, Restore and Delete statistics.  From the Configure section you can easily adjust Global Statistics Gathering Options, Object Level or make changes to the Auto Task and view SPA Validation Results. em13db3

The new notable new features here are the breakdown of Statistics Gathering tasks and the Jobs List.  From the chart, you can quickly see if you have job failures that might be affecting your performance.  In the status chart, you can identify if you have State statistics that need attention.

em13db1

Drilling down into the Statistics Gathering Job List or Auto Tasks will bring up a detailed report of the job run.

em13db2

Not a huge change, but hopefully one that will make the DBA’s life easier and provide better insight when evaluating database statistics!

 

EM 13c – Exciting Updates to Target Properties

It’s here, it’s finally here!  I know most of you have already downloaded the binaries and started installing or upgrading your test environment.   It’s just too tempting not to, right?  One question I’ve heard over and over since Oracle Enterprise Manager 12c came out… Can I use User Defined Target Properties in my Dynamic Group and Administration Group?  Sadly the answer has always been no.   Until now.  Now, the answer is proudly YES!

User Defined Target Properties

One of the small but powerful new features in EM 13c is the ability to use your custom target properties to define the Dynamic and Administration groups!  This will work with global target properties, the ones you set as target_type=”*”.  The target specific properties won’t show up in the select list.  Small compromise I think!

First, create your custom target property with emcli command.

$ ./emcli add_target_property -target_type=”*” -property=”Owner”
Property “Owner” added successfully

Next, create a Dynamic Group and select the Define Membership Criteria button.

group6

You’ll see a list of the default target properties. Click the Add/Remove Target Properties button.

target properties

In this list, you will now see the Owner target property that I created earlier.  Select the box and click Select. 
 target properties

Now, you need to set which values of this property you want to be added to this group by clicking the magnifying glass next to Owner.

group7

Since this is Jill’s group, we’re going to select Jill, click Move and then Select.

target properties

Now we see, that this group is going to contain any targets owned by Jill.

group4

Final step is to review membership and click OK.

target properties

Now that the group has been created, if Jill own’s any targets, we’ll see them listed in her group.

target properties

You will also see the global target property in the selection for the Administration Groups as shown here:

Administration Group with User Defined Target Property

 

Target Property List of Values

Another big enhancement is the ability to create a list of values to more accurately store your target properties.  Say your Line of Business has DBA, MW, and App.   However, admins keep entering the wrong values.  These won’t get used in Dynamic or Administration groups because the values were not expected.

To enable a Target Property to use a Master List of Values:

$ ./emcli use_target_properties_master_list -property_name=orcl_gtp_location -enable

Targets exists with values set for this property. Run the same command with -copy_from_targets flag to copy all values to the master list.

If your targets are already using this property, you’ll get the error message above.  Update your emcli command to include the -copy_from_targets flag.
$ ./emcli use_target_properties_master_list -property_name=orcl_gtp_location -enable -copy_from_targets
Successfully migrated property values

To see the target properties, on any target go to the target menu, then Target Setup / Properties.  Click Edit to update properties.

target properties

As you can see, there are no values listed for Location target property.

target properties

$ ./emcli add_to_target_properties_master_list -property_name=”orcl_gtp_location” -property_value=”Houston” -property_value=”Austin”
Successfully set 2 value(s) for property: orcl_gtp_location

Now under the edit Target Properties you’ll find the correct values listed:

target properties

If you added the wrong value, or you need to remove a value, you use the delete_from_target_properties_master_list command:

$ ./emcli delete_from_target_properties_master_list -property_name=”orcl_gtp_location” -property_value=”Houston, Austin”
Successfully deleted property-value

To see the valid values, you can use the list_target_properties_master_list_values command.

$ ./emcli list_target_properties_master_list_values -property_name=orcl_gtp_location
Target Properties Master list of values for property : orcl_gtp_location
Austin
Houston

 

For more on what you can do with Target Properties, you can see my previous post here.   I think with these two enhancements to target properties, EM administrators everywhere will smile a little brighter tonight.  Enjoy!

 

Enterprise Manager 13c – What’s New and What You Should Know!

Enterprise Manager 13c – What’s new and What You Should Know!

You may have heard by now that Enterprise Manager (EM) 13c has been  released and you can’t wait to get your hands on the new updates and see what it’s all about!   Here’s a few things you should know!

What’s With the New UI?

Just when you got used to the changes in EM 12c, they go and change it all up again right?  Well, change can be hard, but in this case, I think you’ll appreciate a few things.   The overall theme has been updated to match that of other Oracle Products.  Focus is on presentation of the data, so more charts, callouts of number of incidents/errors and tiles that help you get to important information quickly.

em13c
Enterprise Manager 13c

Where did my menu’s go?

Instead of a menu on the left for general use, and admin features on the right, the menu bar has been consolidated to the right side of the page.   Once everyone gets used to this, I think that it’s going to be a lot easier to navigate and more consistent.  Even I had a hard time remembering, was it on the left menu or the right?  This will take a little to get used to but I think it’s much more user friendly.

em13b

From left to right:

  • em13enterprise (Enterprise) – Monitoring, Jobs, Reports, Patching, etc.
  • em13target(Target) – All Targets, Databases, Exadata, etc.
  • em13favorites(Favorites) – Save your favorite targets for quick access
  • em13history(History) – View your last 10 pages
  • em13setupSetup – Security, incident rules, add targets, etc.

Next you’ll see the Search icon, when you click on the magnifying glass, a search box will appear.  This can be used to search for any target.

em13a

The em13notification is new, this will take you to the Notification Center, one of the new features we’ll discuss more later.

Finally you have the User Menu em13user where you’ll find user preferences and logout.

What versions of WebLogic and Java are installed?

The infrastructure stack for WebLogic is now 12.1.3.0.0, and Java is version 1.7.0.    Both are deployed during the Enterprise Manager install.

What’s the upgrade path?

Direct upgrade from Enterprise Manager 12.1.0.3 and higher will be supported.  The Repository database needs to be upgraded to DB 12.1.0.2 first.

If you’re still on EM 10g or 11g (we need to talk), you need to upgrade to EM 12.1.0.5 first before you can upgrade to 13c.

Is there a 2-system upgrade?

No, unfortunately, there is no 2-system upgrade so a full downtime will be required.

Can I use my 12c Agents?

Yes, if you’re agents are 12.1.0.3 or higher, they are compatible with 13.1.  However, you should plan to upgrade the Agents as soon as possible to take advantage of new features.

A Few New Features

Always On Monitoring Service – A separate service used in planned downtime, will receive target availability and send limited notifications.

Agent Gold Images – Create a standard agent and mass deploy updates with ease.  This will cover provisioning, upgrading and updating agents.    I will be posting about this separately, in detail.

Corrective Actions – Support for corrective actions on all event types (beyond just metrics), and out-of-the box customizable Tablespace corrective action.

Target Properties Master List of Values – Define a list of values for your target properties.

Incident Manager  –  New dashboards and export incident rules. Enough said.

That’s a quick summary, but I’ll be blogging about more features in detail.  In the meantime, takes sometime to review the New Features documentation.

 

 

 

All About That Agent…

For Enterprise Manager, the Agent plays a big role.  The Agent is in charge of gathering all information from the targets and performing administrative tasks and jobs.  To do it’s job, the Agent has to be healthy.  In this post, we’re going to take a look at some checks that you can perform to make sure that your Agents and Targets are monitored and reporting properly.

Agent Page

Let’s take a look at what we can find in the Agents page.   In my opinion, this is a page the main EM Administrator should be checking every day.  They should know what targets aren’t uploading, what Agents are down or have been blacked out for months.  From Setup / Manage Cloud Control / Agents we get the view below:

agents

First we have the ability to filter our view by Status.   We can view All, Up, Down, Blackout or Unreachable.  We can also look for agents that are Blocked or Misconfigured.  agents2

Agents are secured during install by default, so if there’s any No’s in the Secure Upload column, they should be resecured.

agents4

There are various statuses that you’ll find here.  The most common needing investigation will be Unreachable or Blocked.

agents5

Unreachable agents will have an additional icon for Diagnostic Analysis.  This will run some checks to help determine why an agent is unreachable.

agents7

The Last Successful Load can tell us a lot about an agent and it’s targets.   If you sort by this column, and find agents that haven’t uploaded for months or maybe even years, it’s a sign that nobody’s paying attention, and possibly that the target has been decommissioned.  If you can’t find a contact for these, then maybe a blackout is in order.agents6

I think the next two columns get overlooked often.

agents3

The first is monitored targets.  This is telling us how many targets (including host and agent) are being managed.  Now, every agent will have 3 at a minimum (agent, host, agent Oracle Home).  So if you see agents that have 2 or 3, that’s a big clue to start investigating just what are you monitoring on that server.  Is it new, or has it been decommissioned and not cleaned up properly?   The Broken Targets column is another key to diagnosing issues.  If there’s anything but a 0 in this column, it deserves some attention to find out exactly what is broken and why.

Metrics

Just like any other target in EM, you’ll find a wealth of information is collected about the agent target in Monitoring / All Metrics.  Each category can have multiple metrics collected.   You will find information here on certificate expiration, disk usage, memory usage, cpu usage,

agent1

When you click on the Category, the right-hand pane will show real-time data.   This can be very helpful in troubleshooting metric issues.  You will see the last upload time.  Metrics are uploaded every 15 minutes, so if it’s older than 15-20 min, the agent is having a problem.

agent2

Some of the metrics are very helpful, specifically in the EMD… categories.  In these metrics you’ll find information about agent and OMS communication (heartbeat, ping, and upload).    Take some time to review the metrics that are collected and available on your agents.

agent4

Reports

There are  a couple of Information Publisher reports that can be useful in identifying problems on Agents and other targets.   From Enterprise / Reports / Information Publisher, the first reports are under the Enterprise Manager Health category.

agentreports

Agent Clock Synchronization Offset will give you the agents whose system clock may be offset from the Repository system clock based on the timestamp of the last heartbeat recorded.  As many pieces of EM rely on the timestamp, an offset more than a few minutes can cause problems.

Agents in Questionable State shows Agents which are in Metric Error state, Agent Down state and Pending/Unknown state for the last 24 hours.

Agents Restarted shows the Agents which have restarted in the last 24 hours. If an Agent is consistently on this report, it’s a sign that it’s crashing.

Broken Targets is another view of the targets we looked at earlier, where they are misconfigured or not monitored for some reason. This is a good report to have emailed out regularly, to be sure you know what targets are having problems.

Targets Not Uploading Data is another one that EM Administrators should regularly view.   This is the report of the targets not uploading consistently, and the last upload date.

Another set of reports under the Target Status Diagnostics category, will help in diagnosing issues with specific targets.  Agent-based targets are going to be your Host, Database Instance, Listener.  Repository-based targets will be Cluster, Cluster Database, System, etc.  In a future post, I’ll break down these reports further.

agentreports2

In part 2 of this blog, we’ll take a look at using EMDIAG’s AGTVFY scripts, as well as looking at some of the Agent log files and common errors.   Stay tuned!

 

SMS Notifications From Enterprise Manager

Starting with Enterprise Manager (EM) 12c, you have the choice to send SMS notifications to a cell phone, or a pager (does anybody still use pagers?).  There’s been a couple questions on the forums about this so I thought I’d write it up since it appears to be a bit confusing as to how this works.

First, be sure that your Mail Server is setup in Setup / Notifications / Notification Methods and you can receive the test e-mail.

sms7

Next, create an EM administrator user, then login as that user to update your e-mail address and SMS/Pager information by clicking on the Username drop down menu and selecting Enterprise Manager Password & Email.

sms6

For the SMS/Pager, you need the text based address.  So if your provider was Verizon, it would be 8885555555@vtext.com.    Select the Email Type Pager for SMS messages as they are shorter than the Email format.   It’s important to note, that you will not see the multiple lines in the Setup / Security / Administrators view.  You can enter multiple e-mails separated by commas, but the E-mail Type option will not be available.

sms5

By default, both email and pager will be enabled in your Notification Schedule, you may adjust this as necessary by going to Setup / Notifications / My Notification Schedule.   You can receive notifications by both e-mail/page, just e-mail, or just page depending on what you configure here.

Next, you need to create an Incident Rule Set, or edit an existing one.   From Setup / Incidents / Incident Rules, select a rule set and rule to edit.  Once you get to the Action for the Rule, in the Basic Notifications section, select the EM Administrator in the Page box.  Save all changes.

sms8

In 12cR4, you can test the Incident Rule Set by selecting your Rule Set and clicking the Simulate Rules button (Setup / Incidents /  Incident Rules).  You will need to select a Target, Event Type and find an alert to simulate.  Then you will get a list of Actions that the Incident Rule Set will perform for this alert.

To test my notifications, I dropped the warning threshold on the tablespace metric to 15, something I know would trigger immediately.    Here’s the messages I received on my phone.

sms4

I also received the long format in my e-mail.  When you’re done testing, don’t forget to set your thresholds back!

 

Automating the Mundane with Corrective Actions and Oracle Enterprise Manager

In my opinion, one of the most under-utilized features of Oracle Enterprise Manager is the Corrective Actions that can be triggered when a metric alert threshold is crossed.  I think one reason it’s under-utilized is it’s very hard to think about where to start and what can be automated.  My advice from previously implementing, is to look at the alerts that are generated, or tickets, and determine which ones are most frequent and mundane.  The one that always comes to mind is Archive Destination.

Archive was the first CA we implemented at my previous company, because we got at least 20 per day.  Since our backups were controlled by a different team, all we could do was cut a ticket to them, and possibly kick off an archive backup hoping it would complete this time.  So we put this in a Corrective Action.  The script checked for hung backups sessions, checked that a backup wasn’t already running, looked through a config file to get the right information, and then kicked off an archive backup job.   Then it sent an email to a ticketing queue for the backup team to generate the ticket so they could investigate why it was failing.    We set the CA to run on Warning, with a fairly low threshold so we had plenty of time to react if it got to Critical.  This was such a success that we went on to write more, including automating tablespace adds and sending notifications to the application teams when process/sessions was exceeded.

A friend of mine, Tyler Sharp, recently started implementing Corrective Actions and has found tremendous time savings.    He recently had the idea to automate oradebug steps so they would always have the required debug when working with Oracle Support, instead of having to go through the process manually the next time around.   The CA is triggered when > 4 active sessions waiting on concurrency, or over 900 seconds db blocking time.  He was kind enough to share the script they’ve implemented below:

conn / as sysdba
set serverout on

DECLARE
   trace_name   VARCHAR2 (1000) := NULL;
   alter_session   VARCHAR2 (1000) := NULL;

BEGIN
   SELECT to_char (SYSDATE, ‘MMDDYY_HHMISS’) INTO trace_name FROM DUAL;
   alter_session :=
         ‘alter session set tracefile_identifier=”’||trace_name||’_AUTO_HANGANALYZE”’;
   DBMS_OUTPUT.PUT_LINE (alter_session);
   EXECUTE IMMEDIATE alter_session;
END;
/

oradebug setmypid;
oradebug unlimit;
oradebug dump ashdumpseconds 5
oradebug hanganalyze 3
execute sys.dbms_lock.sleep(180);
oradebug hanganalyze 3

As you can see, they set a trace file name, take an ashdump, then do the required hanganalyze twice with a  sleep in between.   Now the DBA can skip these steps when working an issue, and collect the files that were created at the right time, not 10 minutes later.   You’ll need to be sure to have a credential that has sysdba access to run this properly.

The great thing about Corrective Actions, is you can use them in a template so you can push them to all servers to keep your resolutions standard.    The Corrective Action is triggered for either Warning or Critical, or both.  Then you have the choice to get notified of that alert right away, or bypass notifications unless the Corrective Action failed.   This allows you a fall back in case the script or job has a problem fixing the issue.

To learn more about Corrective Actions, check the Oracle Enterprise Manager 12c Cloud Control Administrator’s Guide and check out the following blog posts for more ideas!

What are the Corrective Actions you’ve implemented, or would like to implement in your environment?

Enterprise Manager at Oracle OpenWorld 2015

You might have heard by now, there’s this little meeting coming up in 9 days called Oracle Open World.  If you happen to be heading to San Francisco, drop by and see a few friends, or 50,000.

Be sure to register for my session!  This year will be a little different as I’ll be running a panel with 4 Enterprise Manager Champions!

Using Oracle Enterprise Manager Effectively to Become an Oracle Enterprise Manager Champion [CON9711]

Joseph Kopilash, Director, Database Administration, Epsilon Data Management LLC
Steve Meredith, Boeing Oracle Enterprise Manager Service Manager, Boeing
Tyler Sharp, Technology Architect, Cerner
Eric Siglin, Senior Database Administrator, Electric Reliability Council of Texas

In this panel discussion, customers share their experiences with Oracle Enterprise Manager Cloud Control and discuss how they’ve implemented features leading to significant cost savings and operational efficiency. Get started with deployment and management of agents and establishing initial thresholds for alerting. Simplify administration of users with active directory integration. Maintain security standards and increased productivity by patching fleets of databases and stay current with critical patch updates. Learn from customers how they’ve made use of out-of-the-box and custom reports to manage engineered systems, database, and middleware targets by exception. Benefit from these top customer dos and don’ts.

Wednesday, Oct 28, 11:00 a.m. | Moscone South—104

When not presenting or meeting with customers, you’ll find me at the Engineered Systems Showcase or Hybrid Cloud Management: Single Pane of Glass for Complete Management—On-Premises and Public Cloud (SLD-026) booths.  Be sure to stop by and say hi!

For the full list of EM related sessions, you can refer to the Focus On docs.    Here’s some highlights of other sessions you’ll find interesting!
Sunday 

For those of you who arrive on on Saturday, or early Sunday… and aren’t drawn away by the beautiful city of San Francisco, check out the IOUG SIG Sunday sessions that are going on all day Sunday!   Watch out, they’re no longer restricted to Moscone West rooms!  Here’s a few I’m going to try to get to!

8am – Alfredo Krieg delivers Monitor Engineered Systems from a Single Pane of Glass: Oracle Enterprise Manager 12c [UGF10288] in Moscone South 270

10am – Erik Benner/Wassim Kayrala deliver Database Cloud in a Box—DBaaS on Oracle Database Appliance [UGF10279] in Moscone South 270

12pm – Rene Antunez delivers Private Cloud Provisioning Using Oracle Enterprise Manager 12c [UGF9959] in Moscone South 305

1:30pm – Ray Smith delivers You’ve Got It—Flaunt It: Oracle Enterprise Manager Extensibility [UGF9930] in Moscone South 305

Monday

12:15pm – Oracle Enterprise Manager: The Complete Solution and Oracle’s Best-Kept Secrets [CON9715] in Moscone South 300

Hear from Amit Ganesh, VP of Development for EM.

1:30pm – Managing at Hyper-Scale: Oracle Enterprise Manager as the Nerve Center of Oracle Cloud [CON9710] in Moscone South 300

Hear about how the Oracle Cloud relies on Oracle Enterprise Manager, and learn best practices for any implementation!

5-6pm – Join your fellow EM enthusiasts in the OTN Lounge (Moscone South) for the IOUG EM SIG.   Don’t worry if you’re not a member yet, stop by and get to know some of the folks you see on Twitter or hear speaking!

Tuesday

11am – General Session: Oracle Management Cloud—Real-Time Monitoring, Log, and IT Operations Analytics [GEN9778] in Moscone South 102

Definitely a must attend event!  Learn about Oracle’s newest offering Oracle Management Cloud.

Wednesday

After my session at 11am, there’s a lot going on at 12:15pm but if you’re interested in monitoring, I’d recommend  Way Beyond the Basics: Oracle Enterprise Manager Monitoring Best Practices [CON9721] in Moscone South 300.

https://events.rainfocus.com/oow15/catalog/focusOnDoc.jsp?focusId=18408

Thursday

https://events.rainfocus.com/oow15/catalog/focusOnDoc.jsp?focusId=18408

ECO 2015 Presentation

Had a great few days in Raleigh presenting and attending at the East Coast Oracle Users Group!    We ran out of time in the hand on lab, so if you’re looking for the metric and monitoring labs, you can find those in this post.    I’ve added my slides to slide share below