All About That Agent…

For Enterprise Manager, the Agent plays a big role.  The Agent is in charge of gathering all information from the targets and performing administrative tasks and jobs.  To do it’s job, the Agent has to be healthy.  In this post, we’re going to take a look at some checks that you can perform to make sure that your Agents and Targets are monitored and reporting properly.

Agent Page

Let’s take a look at what we can find in the Agents page.   In my opinion, this is a page the main EM Administrator should be checking every day.  They should know what targets aren’t uploading, what Agents are down or have been blacked out for months.  From Setup / Manage Cloud Control / Agents we get the view below:

agents

First we have the ability to filter our view by Status.   We can view All, Up, Down, Blackout or Unreachable.  We can also look for agents that are Blocked or Misconfigured.  agents2

Agents are secured during install by default, so if there’s any No’s in the Secure Upload column, they should be resecured.

agents4

There are various statuses that you’ll find here.  The most common needing investigation will be Unreachable or Blocked.

agents5

Unreachable agents will have an additional icon for Diagnostic Analysis.  This will run some checks to help determine why an agent is unreachable.

agents7

The Last Successful Load can tell us a lot about an agent and it’s targets.   If you sort by this column, and find agents that haven’t uploaded for months or maybe even years, it’s a sign that nobody’s paying attention, and possibly that the target has been decommissioned.  If you can’t find a contact for these, then maybe a blackout is in order.agents6

I think the next two columns get overlooked often.

agents3

The first is monitored targets.  This is telling us how many targets (including host and agent) are being managed.  Now, every agent will have 3 at a minimum (agent, host, agent Oracle Home).  So if you see agents that have 2 or 3, that’s a big clue to start investigating just what are you monitoring on that server.  Is it new, or has it been decommissioned and not cleaned up properly?   The Broken Targets column is another key to diagnosing issues.  If there’s anything but a 0 in this column, it deserves some attention to find out exactly what is broken and why.

Metrics

Just like any other target in EM, you’ll find a wealth of information is collected about the agent target in Monitoring / All Metrics.  Each category can have multiple metrics collected.   You will find information here on certificate expiration, disk usage, memory usage, cpu usage,

agent1

When you click on the Category, the right-hand pane will show real-time data.   This can be very helpful in troubleshooting metric issues.  You will see the last upload time.  Metrics are uploaded every 15 minutes, so if it’s older than 15-20 min, the agent is having a problem.

agent2

Some of the metrics are very helpful, specifically in the EMD… categories.  In these metrics you’ll find information about agent and OMS communication (heartbeat, ping, and upload).    Take some time to review the metrics that are collected and available on your agents.

agent4

Reports

There are  a couple of Information Publisher reports that can be useful in identifying problems on Agents and other targets.   From Enterprise / Reports / Information Publisher, the first reports are under the Enterprise Manager Health category.

agentreports

Agent Clock Synchronization Offset will give you the agents whose system clock may be offset from the Repository system clock based on the timestamp of the last heartbeat recorded.  As many pieces of EM rely on the timestamp, an offset more than a few minutes can cause problems.

Agents in Questionable State shows Agents which are in Metric Error state, Agent Down state and Pending/Unknown state for the last 24 hours.

Agents Restarted shows the Agents which have restarted in the last 24 hours. If an Agent is consistently on this report, it’s a sign that it’s crashing.

Broken Targets is another view of the targets we looked at earlier, where they are misconfigured or not monitored for some reason. This is a good report to have emailed out regularly, to be sure you know what targets are having problems.

Targets Not Uploading Data is another one that EM Administrators should regularly view.   This is the report of the targets not uploading consistently, and the last upload date.

Another set of reports under the Target Status Diagnostics category, will help in diagnosing issues with specific targets.  Agent-based targets are going to be your Host, Database Instance, Listener.  Repository-based targets will be Cluster, Cluster Database, System, etc.  In a future post, I’ll break down these reports further.

agentreports2

In part 2 of this blog, we’ll take a look at using EMDIAG’s AGTVFY scripts, as well as looking at some of the Agent log files and common errors.   Stay tuned!

 

Changing OMS or Agent Properties in OEM Console

There was a thread going on earlier this week about how to change a value for an OMS property setting.   This is typically done when working with support to adjust a timing or enable debug or tracing.

The most common way is using emctl set property at command level.

$ emctl set property -name oracle.sysman.eml.maxInactiveTime -value 60 

To get the current setting there’s a emctl get property command:

$ emctl get property -name oracle.sysman.eml.maxInactiveTime

That usually works great if support has just given you the exact command to run, or you’re using a MOS note to reference the exact syntax.   However, if you’re getting older like me, and syntax is just one of those things that you tend to put in the way back corners of your brain… you tend to forget was it oracle.sysman.eml.maxInactiveTime or was it oracle.em.sysman.maxTimeout or…   There’s just too many to remember.   Lucky for us, there is now a place to view and set these properties in the EM  console.   You can find it under Setup / Manage Cloud Control / Management Services.

properties_1

Then under the Management Servers menu select Configuration properties.

properties_2

From here you’ll get a window that lists the non-default properties.  Understand, that some properties will not show up, that doesn’t mean they are not set, but that they just have the system default value.

properties_3

 

By switching the Show view to All, you’ll see a larger list of properties.  Not all of them are modifiable, as indicated by the lock icon.   If you want to view more information about a property, or modify it, click on the Name.    This will bring up a new window, with the ability to modify the value and save.  This view will also tell you whether the property is Dynamic (can be changed without OMS restart).   If you expand the Change History, you can also view the previous changes for this parameter.

properties_4

Of course, not all properties are OMS based, so there’s an equivalent option on the Agent side.  From the Agent home page, click on Agent menu then select Properties.

properties_9

You will get a list of properties, some which you can edit, some you can’t.   By default the basic properties are shown.

properties_10

Select Advanced Properties to see additional agent properties such as dynamicPropsComputeTimeout which is often adjusted on very large servers.

properties_11

This is great right?  But what if you want to change the same property on 1000 agents?   Well, that’s in here too!  Click on Setup / Manage Cloud Control / Agents.

properties_5

From there, click on the agents (or just one for now) and click Properties.properties_6

This will start a job wizard in which you can add additional agents by clicking Add in the Targets section.  Then click on the Parameters tab.

properties_7

Now you can set the parameter value that you wish to push out to all selected agents.

properties_8

The caveat — use with caution and common sense.  There’s a lot of parameters in here, and very little are documented, some should not be changed unless directed by Oracle Support.   So don’t go cowboy on us and start tweaking them all just to see what they do!

Resolving Conflicts While Patching the EM Agent

Unless you’re in a cave asleep, you’ve seen the recent Oracle PSUs and patches that were released in January.  This has many of my customers patching their agents, and a few have noticed a problem with some previously applied patches.  Thanks to Brian for pointing this out!

If you applied the recommended Java patches sometime last year (18502187, 18721761), and you go to apply the 12.1.0.4.5 agent patch 20282974 (or an earlier bundle), the analyze step will fail as the Java patches are included in the agent bundle, but for some reason it can’t “ignore” them.    Checking the detailed results of the failed Analyze job will show which step failed.   In this specific case, the step PrerequisiteCheckForApply will be marked Failed and the error will look like this:

ap1

To remedy this with EM, you can create a patch plan with the Java patches (18502187, 18721761) and select Rollback in the Deployment Options.  This will run the rollback for these patches.

ap2

Once rolled back, re-analyze your 12.1.0.4.5 patch plan and it should be successful and allow you to deploy!

When you update Agents, keep these things in mind:

  • Be sure to update any Agent clones/gold images with the newly updated patch and plug-ins if they’ve been upgraded or patched.
  • If you staged patches for auto-deployment in the OMS $ORACLE_HOME/install/oneoffs, remove the old patches and you should just need the 12.1.0.4.5 patch now, or any current Discovery patches.
  • Update your procedural documents to reflect the current patches required.

For more details on how to patch your agent using EM, see the Administrator’s guide.

 

Simplified Agent and Plug-in Deployment

On your site of hundreds or thousands of hosts have you had to patch agents immediately as they get deployed?  For this reason I’ve always been a big fan of cloning an agent that has the required plug-ins and all the recommended core agent and plug-in patches, then using that clone for all new agent deployments. With EM 12c this got even easier as you can now clone the agent using the console “Add Host” method. You still have to rely on the EM users to use the clone.The one problem I have with cloning is that you have to have a reference target for each platform that you support. If you have a consolidated environment and only have Linux x64, this may not be a problem. If you are managing a typical data center with a mixture of platforms, it can become quite the maintenance nightmare just to maintain your golden images.You must update golden image agents whenever you get a new patch (generic or platform specific) for the agent or plug-in, and recreate the clone for each platform. Typically, I find people create a clone for their most common platforms, and forget about the rest. That means, maybe 80% of their agents meet their standard patch requirements and plug-ins upon deployment, but the other 20% have to be patched post-deploy, or worse – never get patched!

While deployed agents and plug-ins can be patched easily using EM Patches & Updates, but what about the agents still getting deployed or upgraded? Wouldn’t it be nice if they got patched as part of the deployment or upgrade? This article will show you two new features in EM 12.1.0.3 (12cR3) that will help you deploy the most current agent and plug-in versions. Whether you have 100s or 1000s of agents to manage, reducing maintenance and keeping the agents up to date is an important task, and being able to deploy or upgrade to a fully patched agent will save you a lot of time and effort.

Read original post here.