All About That Agent…

For Enterprise Manager, the Agent plays a big role.  The Agent is in charge of gathering all information from the targets and performing administrative tasks and jobs.  To do it’s job, the Agent has to be healthy.  In this post, we’re going to take a look at some checks that you can perform to make sure that your Agents and Targets are monitored and reporting properly.

Agent Page

Let’s take a look at what we can find in the Agents page.   In my opinion, this is a page the main EM Administrator should be checking every day.  They should know what targets aren’t uploading, what Agents are down or have been blacked out for months.  From Setup / Manage Cloud Control / Agents we get the view below:

agents

First we have the ability to filter our view by Status.   We can view All, Up, Down, Blackout or Unreachable.  We can also look for agents that are Blocked or Misconfigured.  agents2

Agents are secured during install by default, so if there’s any No’s in the Secure Upload column, they should be resecured.

agents4

There are various statuses that you’ll find here.  The most common needing investigation will be Unreachable or Blocked.

agents5

Unreachable agents will have an additional icon for Diagnostic Analysis.  This will run some checks to help determine why an agent is unreachable.

agents7

The Last Successful Load can tell us a lot about an agent and it’s targets.   If you sort by this column, and find agents that haven’t uploaded for months or maybe even years, it’s a sign that nobody’s paying attention, and possibly that the target has been decommissioned.  If you can’t find a contact for these, then maybe a blackout is in order.agents6

I think the next two columns get overlooked often.

agents3

The first is monitored targets.  This is telling us how many targets (including host and agent) are being managed.  Now, every agent will have 3 at a minimum (agent, host, agent Oracle Home).  So if you see agents that have 2 or 3, that’s a big clue to start investigating just what are you monitoring on that server.  Is it new, or has it been decommissioned and not cleaned up properly?   The Broken Targets column is another key to diagnosing issues.  If there’s anything but a 0 in this column, it deserves some attention to find out exactly what is broken and why.

Metrics

Just like any other target in EM, you’ll find a wealth of information is collected about the agent target in Monitoring / All Metrics.  Each category can have multiple metrics collected.   You will find information here on certificate expiration, disk usage, memory usage, cpu usage,

agent1

When you click on the Category, the right-hand pane will show real-time data.   This can be very helpful in troubleshooting metric issues.  You will see the last upload time.  Metrics are uploaded every 15 minutes, so if it’s older than 15-20 min, the agent is having a problem.

agent2

Some of the metrics are very helpful, specifically in the EMD… categories.  In these metrics you’ll find information about agent and OMS communication (heartbeat, ping, and upload).    Take some time to review the metrics that are collected and available on your agents.

agent4

Reports

There are  a couple of Information Publisher reports that can be useful in identifying problems on Agents and other targets.   From Enterprise / Reports / Information Publisher, the first reports are under the Enterprise Manager Health category.

agentreports

Agent Clock Synchronization Offset will give you the agents whose system clock may be offset from the Repository system clock based on the timestamp of the last heartbeat recorded.  As many pieces of EM rely on the timestamp, an offset more than a few minutes can cause problems.

Agents in Questionable State shows Agents which are in Metric Error state, Agent Down state and Pending/Unknown state for the last 24 hours.

Agents Restarted shows the Agents which have restarted in the last 24 hours. If an Agent is consistently on this report, it’s a sign that it’s crashing.

Broken Targets is another view of the targets we looked at earlier, where they are misconfigured or not monitored for some reason. This is a good report to have emailed out regularly, to be sure you know what targets are having problems.

Targets Not Uploading Data is another one that EM Administrators should regularly view.   This is the report of the targets not uploading consistently, and the last upload date.

Another set of reports under the Target Status Diagnostics category, will help in diagnosing issues with specific targets.  Agent-based targets are going to be your Host, Database Instance, Listener.  Repository-based targets will be Cluster, Cluster Database, System, etc.  In a future post, I’ll break down these reports further.

agentreports2

In part 2 of this blog, we’ll take a look at using EMDIAG’s AGTVFY scripts, as well as looking at some of the Agent log files and common errors.   Stay tuned!

 

Getting to Know EMDIAG: repvfy execute optimize

In my group, we work with a lot of customers with very large EM environments.  On the range of 2000+ agents.  So as you can imagine there’s a little bit of optimizing that needs to get done to account for these numbers.

A few of these standard tweaks have been put into the repvfy execute optimize command.    You can make all these changes individually, but if you want to get them all done at once, optimize is your tool.

There’s 3 categories of optimization that is handled at this point:  Internal Tasks, Repository Settings and Target system.    The script will first evaluate the size of your repository based on the number of agents, and from there determine what optimizations need to be done or recommended for future implementation.

Internal Task Tuning

Enterprise Manager uses short and long workers, depending on the task activity.  We typically recommend 2 workers for each for most larger systems, so in repvfy execute optimize this is what gets set. Smaller systems are usually sufficient with the default settings of 1 each.    You can view the configuration in EM on Manage Cloud Control -> Repository page.   Here you can also configure the short workers, but not the long.  If you see a high collection backlog, this is an indication that your in need of additional task workers.

shortworkers

The next step is to evaluate the current settings of the job system and ensure that there are enough connections available for the job system.  This change is not implemented automatically, but is printed out for you to change with emctl, as it will require a restart to take effect.   Recommendations for Large Job System Load can be found in the Sizing chapter of Advanced Installation Guide.  Increasing the number of connections may require an increase in database processes value.

Repository Settings Tuning

EM tracks system errors in one of it’s tables.   In larger systems, the MGMT_SYSTEM_ERROR_LOG table can become quite large over the 31 day default retention.   The optimize script reduces log retention to 7 days for normal operating.

There are also various levels of tracing enabled by default, this can generate a lot of extra activity during normal operations if you’re not utilizing the traces.    Tracing is turned off by the optimize command.  It can be enabled at any time by using the repvfy send start_trace -name <name>  and repvfy send start_repotrace commands.

Finally this step looks for any invalid SYSMAN objects and validates them, then checks for stale optimizer statistics and makes a recommendation as needed.

System Tuning

After an EM outage or downtime, all the agents will attempt to upload and update their status (or heartbeat) with the OMS.  There’s a grace period in which no alerts are sent.  In larger systems, this grace period may not be long enough to get all agents updated before alerts start going out.   This can be adjusted by increasing that grace period.

In 12.1.0.3 and higher, you can also increase the number of threads that perform the ping heartbeat tasks.  This should be done if you have more than 2000 agents per OMS.  The optimize command will make this calculation for you and recommend the appropriate emctl command to set the heartbeatPingRecorderThreads property.  Recommendations for Large Number of Agents can be found in the Sizing chapter of Advanced Installation Guide.

The optimize command will only output those items that require attention, so not every item will appear in the output on every site.
The recommended values reported in the output are specific for THAT environment  and should not be copied over to another environment just like that.  To tune another EM environment, run the optimize script on that environment.

Sample output from a small EM system:

bash-4.1$ ./repvfy execute optimize

Please enter the SYSMAN password:
SQL*Plus: Release 11.1.0.7.0 – Production on Thu Jul 9 07:59:35 2015

Copyright (c) 1982, 2008, Oracle. All rights reserved.

SQL> Connected.

Session altered.
Session altered.

========== ========== ========== ========== ========== ========== ==========
== Internal task system tuning ==
========== ========== ========== ========== ========== ========== ==========

– Setting the number of short workers to 2 (1->2)
– Setting the number of long workers to 2 (1->2)
========== ========== ========== ========== ========== ========== ==========
========== ========== ========== ========== ========== ========== ==========
== Job system tuning ==
========== ========== ========== ========== ========== ========== ==========

– On each OMS, run this command:
  $ emctl set property -name oracle.sysman.core.conn.maxConnForJobWorkers -value 72 -module emoms
  This change will require a bounce of the OMS

========== ========== ========== ========== ========== ========== ==========
========== ========== ========== ========== ========== ========== ==========
== Repository tuning ==
========== ========== ========== ========== ========== ========== ==========
– Setting retention for MGMT_SYSTEM_ERROR_LOG table to 7 days (31->7)

– Disabling PL/SQL tracing for module (EM.GDS)
– Disabling PL/SQL tracing for module (EM_DBM)

– Disabling repository metric tracing for ID (1234)

– Recompiling invalid object (foo,TRIGGER)
– Recompiling invalid object (bar,CONSTRAINT)

– Stale CBO statistics in the repository. Gather statistics for the SYSMAN schema
  Command to use:
  $ repvfy send gather_stats
  Or:
  SQL> exec emd_maintenance.gather_sysman_stats_job(p_gather_all=>’YES’);

========== ========== ========== ========== ========== ========== ==========
========== ========== ========== ========== ========== ========== ==========
== Target system tuning ==
========== ========== ========== ========== ========== ========== ==========

– Setting the PING grace period to (90) (60->90)

– Set the parameter oracle.sysman.core.omsAgentComm.ping.heartbeatPingRecorderThreads to 3
  $ emctl set property -module emoms -name oracle.sysman.core.omsAgentComm.ping.heartbeatPingRecorderThreads -value 3

========== ========== ========== ========== ========== ========== ==========
not spooling currently

Getting to Know EMDIAG – repvfy show score_card

Continuing on with a series of EMDIAG commands that I find useful, today we’re going to look at the repvfy show score_card.   You may also want to review the previous post on repvfy diag all.

One of the main functions of repvfy is the verify function (i.e. repvfy verify -details -level 9).  This goes through a long list of checks to identify areas you might need to investigate.  This could be anything from Agents with Clock Skew to Expired User accounts.

Now there’s a scorecard (show score_card) that reads the output of the verify log and will summarize the category of errors that were found, and a generate a score based on the weight, number of tests and number of violations.  This can help you in tracking improvements as you work through any cleanup and issues.

repvfy show score_card (also try score_card_details)

Category                  Score  #Tests #Viol
————————- —— —— ——
Best Practice              81.59      6    780
Configuration              21.64     20   2298
Data Integrity             25.09     29   2793
Monitoring/Operations      60.71     23    886
————————- —— —— ——

The output of show score_card_details will give the breakdown by category (Best Practice, Configuration, Data Integrity, Monitoring/Operations) and Module (Targets, Agents, etc) along with the test to run (i.e. repvfy verify targets -test 6002).

Note: The sample output below has been modified and shortened to fit the blog.

$ repvfy show score_card_details

Category        Rank Module          ID   Test               Score  #Viol
BP  1 TARGETS 6002 OMS mediated targets without backup Agent   10.0  639
BP  2 AGENTS  6006 Deployed Agent plugins lower than OMS plugin 6.6  106
BP  3 REPO  6039 Newer version avail for deployed OMS plugin    1.1    9
BP  4 REPO  6005 Tables with locked statistics                  0.3   23
BP  5 JOBS  6006 Job steps running more than two hours          0.3    2

Using the repvfy verify -details will help you identify targets to investigate.  Some tests have automated fixes that can be run with repvfy verify <module> -test <test#> -fix.     Other issues may have manual steps or suggestions, some may require opening an SR with Oracle Support.  After fixing issues, rerun the full repvfy verify -level 9 and regenerate the score card to track your progress!

Getting to Know EMDIAG – repvfy diag all

If you’ve worked with me, or called me about a problem with your Enterprise Manager, or even attended any of my sessions, you’ve probably heard me talk about EMDIAG.   One of the most popular components of EMDIAG is the Repvfy tool. This is basically a series of scripts and queries that will provide data from the repository to help diagnose configuration and data issues.  You can get more details on downloading and installing in EMDIAG Troubleshooting Kits Master Index (Doc ID 421053.1).

There are 3 components that make up EMDIAG:  repvfy, omsvfy and agtvfy.   Today, one of the features I am introducing you to is in repvfy.  This is the component that pulls data from the EM repository.

repvfy diag all  

This is my go to these days.  Instead of tell the customer I need X, Y, Z and A, B, C, I get this.   The diag all runs through various EMDIAG reports that are frequently used in troubleshooting issues with support and development.  It runs the different reports and zips them into a file that can then be uploaded easily.    There’s also a shorter version repvfy diag core.

adding: advisor_day_2015_07_06_084304.log (deflated 81%)
adding: advisors_2015_07_06_084304.log (deflated 83%)
adding: agent_health_2015_07_02_120925.log (deflated 83%)
adding: analyze.log (deflated 83%)
adding: backlog_2015_07_06_084304.log (deflated 84%)
adding: body1.log (stored 0%)
adding: body2.log (stored 0%)
adding: body3.log (stored 0%)
adding: cursor_2015_07_06_084304.log (deflated 84%)
adding: custom_2015_07_06_084304.log (deflated 86%)
adding: deinstall.log (deflated 79%)
adding: details_2015_07_02_082451.log (deflated 82%)
adding: details_2015_07_02_082451.sql (deflated 68%)
adding: details_2015_07_06_084304.log (deflated 87%)
adding: details_2015_07_06_084304.sql (deflated 74%)
adding: errors_2015_07_06_084304.log (deflated 83%)
adding: install.log (deflated 80%)
adding: job_health_2015_07_06_084304.log (deflated 83%)
adding: loader_health_2015_07_06_084304.log (deflated 91%)
adding: metric_stats_2015_07_06_084304.log (deflated 92%)
adding: mtm_2015_07_06_084304.log (deflated 89%)
adding: notif_health_2015_07_06_084304.log (deflated 85%)
adding: performance_2015_07_06_084304.log (deflated 88%)
adding: ping_health_2015_07_06_084304.log (deflated 76%)
adding: pkg.log (deflated 62%)
adding: space_2015_07_06_084304.log (deflated 91%)
adding: system_2015_07_06_084304.log (deflated 86%)
adding: task_health_2015_07_06_084304.log (deflated 83%)
adding: upgrade2.log (stored 0%)
adding: verify.log (deflated 89%)
adding: verify_2015_07_02_082451.log (deflated 49%)
adding: verify_2015_07_06_084304.log (deflated 45%)
adding: views.log (deflated 82%)

File created: /u01/oracle/em12r5/oms/emdiag/tmp/repvfy_2015_07_06_084304.zip

So just what does it gather information about?   Here’s a one line summary of each report:

advisors  – ADDM, ASH and AWR reports from the repository database
agent_health – summary of deployed agents, plugins and targets as well as availability and ping statistics
backlog – statistics from dbms_scheduler, loader subsystem, job subsystem, notification subsystem and the task/worker subsystem
cursor – cursor parameters and statistics for EM SQL
custom – summary of EM customizations done
errors – targets, agents, plugins, metrics, collections, jobs in error
job_health – summary of job subsystem configuration, statistics and performance
loader_health – summary of loader subsystem configuration, statistics and performance
metric_stats – performance summary of repository, loader subsystem, purge policies and metrics including top targets and metrics
mtm – summary of Repository and OMS configuration, housekeeping jobs, agent and plugin deployments
notif_health – summary of notification subsystem configuration, statistics and performance
performance – performance summary of repository, OMS, agents and internal subsystems.
ping_health – summary of agent ping jobs and communication
space – summary of schema statistics collections, table/index sizes and fragmentation
system – full configuration summary
task_health – summary of task subsystem configuration, statistics and performance
verify/details – the standard verification checks with detailed output

So depending on the issue you’re seeing, I will typically look at various reports.    If you have problems with notifications, I’m obviously going to go through the notif_health and probably the backlog and job_health reports.   If I’m just trying to get a good understanding of how your system is built, what targets you manage and what you’re doing with them, I’d start with the system and custom reports.

In future posts, we’ll break down some of these reports in detail, but that’s it for today!