Category: Oracle Management Cloud

Deploying Oracle Management Cloud Agents

When working with Oracle Management Cloud, you’ll likely need to deploy an agent at some point. In this example we’ll deploy a Gateway to facilitate communication from my on-premise systems to the Oracle Cloud, and a Cloud Agent which will upload metric and log data. For details on agent and network requirements for communicating to Oracle Management Cloud, see this previous post.

Deploying a Gateway

The Gateway agent is typically the first agent I deploy when working with customers. While it is an optional component, it adds a lot of simplicity to a deployment, especially when monitoring systems on-premise. The Gateway acts as a funnel, so all data from your on-premise systems is sent to the Gateway, and then bundled up to be sent to Oracle Cloud. This allows us to only open outbound access from the Gateway servers to Oracle. All other communication is done internally.

Considerations before deploying:

  • Server must have ability to communicate from server to Oracle Cloud tenant over 443
    • Directly or with proxy configuration
  • Install location must have sufficient disk space. This can be 10+gb depending on the size of your deployment. The more space available the longer data can be buffered in event of communication error.
  • Multiple Gateway agents can be deployed for high availability and load balancing.
  • To validate prerequisites before installing you can use ./AgentInstall.sh EXECUTE_PREREQ=true

Steps to deploy:

  1. Validate common prerequisites and gateway prerequisites
  2. Download gateway install file and transfer to server
  3. Obtain registration key
  4. Unzip file on stage location
  5. Edit gateway.rsp file with required parameters:
    1. TENANT_NAME
    2. AGENT_REGISTRATION_KEY
    3. AGENT_BASE_DIRECTORY
    4. OMC_URL
  6. Run ./AgentInstall.sh
  7. Validate by running omcli status agent cmd from AGENT_BASE_DIRECTORY
  8. Configure for automatic startup

Sample gateway.rsp file using a Proxy server for communication to Oracle Cloud:

TENANT_NAME=6e8c51034a19b42cd823f6fd518k24bvi
AGENT_REGISTRATION_KEY=Rt4ekdnMs69W-C1koW8vtR99tT 
AGENT_BASE_DIRECTORY=/u01/oracle/omc/gw  
OMC_URL=https://myomc-mycloud.uscom-east-1.oraclecloud.com/  
OMC_PROXYHOST=www.myproxy.com 
OMC_PROXYPORT=80

Validating the gateway is up and running:

$ cd /u01/oracle/omc/gw/agent_inst/bin
$ ./omcli status agent

Oracle Management Cloud Gateway

Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.

Version : 1.37.0
State Home : /u01/oracle/omc/gw/agent_inst
Log Directory : /u01/oracle/omc/gw/agent_inst/sysman/log
Binaries Location : /u01/oracle/omc/gw/GATEWAY_LINUX.X64_181218.1850/core/1.37.0
Process ID : 26365
Parent Process ID : 25985
URL : https://mygwhost.cloud.com:4459/emd/main/
Started at : 2019-01-07 06:21:02
Started by user : cllamas
Operating System : Linux version 3.8.13-118.24.2.el6uek.x86_64 (amd64)
Registered entities : 4
Sender Status : FUNCTIONAL
Data Receiver Upload Status : FUNCTIONAL
Last successful upload : 2019-01-07 08:01:52
Last attempted upload : 2019-01-07 08:01:51
Gateway Pending Files (MB) : 0.13
Gateway Pending Files : 16
Backoff Expiration : (none)

Agent is Running and Ready

Deploying Cloud Agents

Cloud Agents are responsible for the bulk of work in Oracle Management Cloud. They are used to collect configuration and metric data, run orchestration jobs and compliance checks, as well as upload log files. They can be deployed on any supported host or vm.

Considerations before deploying:

  • Server must have ability to communicate from server to Oracle Cloud tenant over 443 or to gateway server over deployed port
  • Install location must have sufficient disk space. Minimum is 1gb however we recommend 4-8gb dedicated space. The more space available the longer data can be buffered in event of communication error.
  • To validate prerequisites before installing you can use ./AgentInstall.sh EXECUTE_PREREQ=true

Steps to deploy:

  1. Validate common prerequisites and cloud agent prerequisites
  2. Download cloud agent install file and transfer to server
  3. Obtain registration key (previous key can be used)
  4. Unzip file on stage location
  5. Edit agent.rsp file with required parameters:
    1. TENANT_NAME
    2. AGENT_REGISTRATION_KEY
    3. AGENT_BASE_DIRECTORY
    4. OMC_URL
  6. Run ./AgentInstall.sh
  7. Validate by running omcli status agent cmd from AGENT_BASE_DIRECTORY
  8. Configure for automatic startup

Sample agent.rsp file using a gateway server for communication to Oracle Cloud:

TENANT_NAME=6e8c51034a19b42cd823f6fd518k24bvi
AGENT_REGISTRATION_KEY=Rt4ekdnMs69W-C1koW8vtR99tT 
AGENT_BASE_DIRECTORY=/u01/oracle/omc/cloud  
OMC_URL=https://myomc-mycloud.uscom-east-1.oraclecloud.com/  
GATEWAY_HOST=mygwhost.cloud.com
GATEWAY_PORT=4459

Validating the Cloud Agent is up and running:

$ cd /u01/oracle/omc/cloud/agent_inst/bin
$ ./omcli status agent
Oracle Management Cloud Agent

Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.

Version : 1.37.0
State Home : /u01/oracle/omc/cloud/agent_inst
Log Directory : /u01/oracle/omc/cloud/agent_inst/sysman/log
Binaries Location : /u01/oracle/omc/cloud/LAMA_LINUX.X64_181218.1850/core/1.37.0
Process ID : 8677
Parent Process ID : 8583
URL : https://mycloudhost.cloud.com:4460/emd/main/
Started at : 2019-01-07 06:29:14
Started by user : cllamas
Operating System : Linux version 3.8.13-118.24.2.el6uek.x86_64 (amd64)
Data Collector enabled : false
Sender Status : FUNCTIONAL
Gateway Upload Status : FUNCTIONAL
Last successful upload : 2019-01-07 07:56:55
Last attempted upload : 2019-01-07 07:56:55
Pending Files (MB) : 0.02
Pending Files : 8
Backoff Expiration : (none)

Agent is Running and Ready

Validating in Oracle Management Cloud

After installing the Gateway and the Cloud Agent, you should be able to see data in Oracle Management Cloud. If you go to Administration > Agents, you’ll see the newly installed Gateway. Validate the time since last check in in the highlighted column. This indicates the last known communication from the agent.

Click on the Cloud Agents tab and you should see your Cloud Agent. Validate the time since last check in is current.

Summary

Gateway and Cloud agents can be deployed on premises or in your Cloud targets. Basically anywhere you can transfer the file and run the install command. Once deployed, you can begin monitoring hosts or supported entities, as well as collect log files from that host. I’ll also discuss integrating with Oracle Enterprise Manager in a future post to show you how to leverage your existing monitoring and extend with OMC.

Resolving Log Analytics File Permissions

If you’ve just started using Oracle Management Cloud Log Analytics service, you might have noticed some files that you expect to see are not being loaded into the cloud.  Most likely this is because your agent does not have the right permissions to read the file.

Identifying Permission Warnings

In the Log Analytics dashboard, the bell icon indicates Warnings or notifications for the administrator.

laerrors1

If you click on this icon, you’ll get the breakdown of files that the Cloud Agent either can’t find or can’t read.   The file warnings are broken down by Top Targets and Top Sources.

laerrors2
Log Analytics Warnings

We’re going to take a look at the error that has to do with permissions, as this is most likely what you’ll find when setting up your log sources.  The agent owner needs to have read permissions on a file.  For some of the root level files, such as syslog and messages, you may need to grant permissions explicitly to that agent user.  You can see in the image below, the messages files exist, but I don’t have permissions to read them.

laerrors3

Resolving Permissions Errors

ACLs allow for a much granular method of access privileges then your standard chown/chmod does. If you don’t have the setfacl command available, you might need to install the acl package.  You can do that by issuing the following command as a root user:  yum install acl.    If that is not an option, the standard chmod method of granting read access will work as well.

As a root user use the setfacl command to add read permissions to the agent owner for these files. In my example, the agent owner is “cllamas”.

laerrors5

Now when I switch back to the agent owner, I can successfully read the file.

laerrors6

Refreshing my dashboard automatically clears the warning message and updates the Top Targets charts.

laerrors7

Since I don’t have Apache HTTP running I can Suppress the other errors.  I can fix this permanently by disassociating the Apache log source under Configuration.

laerrors9

 

Sending Data to Oracle Management Cloud

Sending Data to Oracle Management Cloud

Updating and reposting to reflect current state as of today for agent install and tenants.

Lately I’ve been busy helping customers deploy the new Oracle Management Cloud (OMC) services and discover what it can do for them.   One of the most frequent questions I get when talking with new customers about OMC is how do we collect the data and send it to Oracle’s Cloud?  Using Agents installed on-premises or in the Cloud, we can collect various points of data to be used in the many services that make up Oracle Management Cloud.  The goal of this blog is to introduce you to our points of communication and explain where we get data and how it’s used.

It’s important to note, all communication is outbound from Agents to Oracle Cloud.   There is no inbound communication.  All agent communication is over HTTPS.  If your systems have direct access to the internet, they can communicate directly.  Most enterprise customers will likely configure a proxy or firewall/ACL changes to allow communication over certain servers and ports.

Each tenant will have a unique URL that the agents would access, typically it would look like:

https://<instance>-<tenant>.<datacenter>.oraclecloud.com/*

For example:   https://omc-tenant.uscom-east-1.oraclecloud.com

You must be able to reach this URL.  Whether it’s via proxy server, or firewall, the agent does not care so much. Just that it can communicate.   To test this, you can use curl to see if your particular server can reach the tenant.

curl -I –tlsv1.2 https://omc-tenant.uscom-east-1.oraclecloud.com

The rest of this blog post will discuss the various agent components and how they communicate with each other in a typical Oracle Management Cloud deployment.

Agents

As with any systems monitoring tool, there are agents that help with the collection and processing of data.     Each of the agents will be discussed below in regards to what services

Oracle Management Cloud architecture

Gateway

The Gateway is an optional component and is used to buffer and send all data from Cloud Agents, APM Agents and Data Collectors to the Oracle Cloud.   This way, only one server has to have internet access.  Due to the amount of activity for buffering data, this may require a standalone machine as the CPU utilization can be up to 20% for large environments depending on the number of agents and volume of data being sent.    All communication from Gateway agent to Oracle Cloud is over HTTPS (port 443) and is outbound only.

Cloud Agent

The Cloud Agent is responsible for collecting log, metrics, performance, configuration, orchestration jobs and compliance checks to be used in the various services (Log Analytics, IT Analytics, Infrastructure Monitoring, Configuration & Compliance, etc).  It is also responsible for executing remediation actions or instructions per Orchestration service.  This agent will sit on the target host, whether on-premises or in a Cloud.  The agent can communicate to OMC directly, via a proxy, or the Gateway.   The Cloud agent communicates to the Cloud over HTTPS (port 443) directly, or to the Gateway agent over the port the Gateway was installed on (default is 4459).

APM Agent

When using Application Performance Monitoring (APM), you’ll be using the APM agent which is installed on the server where the application runs. Whether the application is Java, .Net or Node.js, the agent collects and sends performance data about the application to OMC.  This agent can communicate directly, via a proxy, or through the Gateway (over HTTPS on Gateway port).  As APM also collects End User Metric data from the application users browser, that data is sent directly to OMC.

Data Collector

There is also an optional Data Collector component for users who already have Oracle Enterprise Manager (OEM) configured.  The Data Collector extracts target properties, associations, metrics, incidents, version and additional configuration data about the targets in OEM and shares this with OMC.   The Data Collector talks to OMC directly over HTTPS or via the Gateway (over HTTPS on Gateway port).  You can have multiple Data Collectors if you have more than one OEM configured.  This is a great opportunity to consolidate data into one place if your OEM is separated by organization, geography or lifecycle status.

Deployment

Each agent has a zip file that can be downloaded from a static link or via the OMC Console.   Installation parameters are set in an agent.rsp file and then an install script is called:

$ ./AgentInstall.sh agent.rsp

As each agent has a different purpose, their install parameters are slightly different.   Details can be found in the installation docs referenced at the end of this blog.

Maintaining Agents

Since Oracle Management Cloud is an agile product with monthly releases, the agents may be updated monthly as well.  This does not mean you have to upgrade your agents monthly, but upgrading quarterly will keep your agents in top shape.    For Gateway, Cloud and Data Collector agents, when you’re ready to upgrade simply go to the Agents administration page and select Upgrade.  This will instruct your agent to download the latest agent and perform an out-of-place upgrade seamlessly.  If the upgrade fails for some reason, your original agent will remain active.

APM agents must be upgraded manually as they’re tightly integrated with the application.   These should be built into your application update cycles.

Summary

As you can see, communication to Oracle Management Cloud has a couple of options to meet different needs.  All communication will go over https and can communicate directly to the cloud or through a centralized gateway.  If you’re looking for additional information, you may find these sources helpful:

An Introduction to Oracle Management Cloud Log Analytics

I know it’s been a while since I’ve blogged about anything, but I have good reason!  I’ve been busy working with a new service offering that Oracle has launched, called Oracle Management Cloud.   I gave a quick glimpse earlier this year of the three services that were launched – Log Analytics, Application Performance Monitoring and IT Analytics.   Since then we’ve also added Infrastructure Monitoring.   Right now, I’m going to talk a little more about Log Analytics and how you can get started using it.

Log Analytics

From application managers to system admins to DBAs, we all have to look at log files. If you have multiple servers, it can take hours to gather all the relevant logs and comb through them, correlating times and searching for error messages.  Log Analytics can make this process simpler and automated.  By loading all related log files to the Oracle Cloud, you can search through logs from multiple targets, and store terabytes of data. Gone are the days of being approached by someone weeks after an issue happened and asked for another log file only to find out that the logs have rolled over already.

The beauty of Log Analytics is that instead of making you create all your own parsers and log sources, Oracle has provided some very common parsers and sources out-of-the-box.  This list continues to grow every  month.  You can get started immediately for Oracle Database (including audit, trace alert, listener, asm), WebLogic, Tomcat, Windows Events, Linux, Solaris, Apache and more.  But what if you have another product, like LifeRay portal supporting your PeopleSoft application that you want to collect logs for?  Easy.   Log Analytics is very flexible and allows you to create a new parser specific to any kind of log you want to read and pull that data in alongside your other logs for quick analysis and troubleshooting.

Getting to Know Log Analytics

Let’s breakdown the Log Analytics page and take a look at what we can do. At the top we have our search bar which allows us to search using keywords and phrases.  It also shows you the recent searches when you click in the search bar, so you can select one from your history and rerun it.   To the right, you have options to Save, Open and Configure.  Save allows you to save your search as a custom widget.  This can be added in a dashboard, we’ll talk more about that later. Open allows you to open a previously saved search.  Configure takes you to the configuration options for log sources and parsers. We’ll talk about this more in another blog.  The Run button executes your search.

Log Analytics Search Bar

Also in this section you have the time selector.  This window expands to give you many pre-defined time windows, or allow for a custom time window.

Log Analytics Time Selector

The Data panel on the left has all the properties and fields.

log10

We can filter on any of these fields to find a specific entity type, entity, severity or error ID.  You can also filter on Groups or Systems (created within OMC or imported from Oracle Enterprise Manager). When you click on one of the fields, you’ll be presented with the list of entries, as well as a chart that shows the trend of logs associated with that entry over the time period you have selected.

entity

The Visualize panel allows us specify which fields and how we want to view the data.  Pie chart, records with a histogram, heat map, tile, sunburst… An option for any type of query.

log4

Then you have your chart.  By default you’ll see a pie chart of logs by log source over the last 60 minutes.

log11

Putting it to Work

Now that we’ve talked about the layout and features of Log Analytics, let’s take a look at some of the ways we can use our log files.  If we select a certain type of logs, let’s say PeopleSoft Application Tuxedo Access Logs (shown here), and with a couple of clicks drill down, we’re presented with the histogram view of this particular log type, and a time sequence of the entire set of associated log files.  From here I can continue to drill down or filter based on specific fields or time.

log12

One thing that’s important to note in the picture above, is that all timestamps are normalized to UTC.  This can be a great help when you’re consolidating logs from different applications and servers across the globe that have different timezones.  With all log files automatically consolidated and aligned in time-order, I can now see for example, when someone logged in and changed permissions or my transaction rate dropped.

One of my favorite LA features is what I call the needle in the haystack finder, aka clustering.  Let’s say we take a look at our Database Listener logs for the last 7 days.  Looking at the chart below, you can see we have 7m+ entries.

cluster1a

Now that’s a lot of logs to look through!  In fact, I really can’t remember the last time I found anything good in the Listener logs.  Always hated those.  Wouldn’t it be great if we could see what was clogging up the logs, or if there’s really anything important in there?  A lot of time, you’ll see connection errors sporadically and they don’t mean anything.  But how do you know if they are safe to ignore or if they’re repeating frequently?

I’m  going to use the Cluster command to show OMCs powerful machine learning.  I can do that by just adding |cluster to the search string, or selecting Cluster from the Visualize dropdown.  Now I see that my 7 million entries comprises of 25 unique message signatures.  This is much easier to digest and identify problems.  You see that the top error in this case is TNS-12514 error, and it happens to coincide (as seen in the Trend chart) with the connection established from the exaboard user.  Now I have something to go on.  In just a few clicks I’ve found a problem with the service_name being used by this user.

cluster2

Again, this can be applied to all types of log files.   Database, Middleware, Host, Application…  just to name a few.  I’ve seen customers identify changes to filesystems, invalid user attempts and more with just a few clicks.

Hopefully you’ve enjoyed this quick introduction to Log Analytics.  I’ll be back with more details on how to use this in everyday tasks, as well as how to create custom log sources and parsers.

To learn more about Oracle Management Cloud go to https://cloud.oracle.com/management or contact me here.

 

 

 

Viva Las Vegas – Collaborate 2016

It’s that time of year again, time to spend a few days in Las Vegas at Collaborate with some of the top Oracle experts and exercise your brain (and your feet)! I will be heading out in a few days and have a great schedule lined up!

Collaborate 16

This year will be a little different as I’m representing two products.   For a while now I’ve been working more on Oracle Management Cloud, which is our new cloud services for Log Analytics, IT Analytics and Application Performance Monitoring.   These services are complimentary to your existing Oracle Enterprise Manager solution.  From a DBA perspective, I’m especially excited about IT Analytics, as it’s an answer to many of the questions we get asked about long-term performance, capacity and resource data that Oracle Enterprise Manager collects.  We will have a demo booth, as well as a session on Monday at 10:30 in Palm B – Oracle Management Cloud: Next-Generation Monitoring, Management and Analytics.   Be sure to attend this session to hear about what Oracle Management Cloud has to offer!   You can also learn more about the Oracle Management Cloud services here.

As always, there’s a lot of excellent Oracle Enterprise Manager sessions this year!  If you’re in Las Vegas on Sunday morning, be sure to register for the Hands-on Lab: Everything I Needed to Know About Enterprise Manager I Learned at COLLABORATE.   We’ll start at 9am with an overview of EM 13c and work on new features in target properties, dynamic groups, creating gold agent images, tablespace corrective actions, and more!  You won’t want to miss out on this one!  Be sure to pre-register as space is limited.

Here’s where you’ll find me:

Sunday – 9-1 – Hands-on Lab: Everything I Needed to Know About Enterprise Manager I Learned at COLLABORATE

Monday – 10:30am Palm B – Oracle Management Cloud: Next-Generation Monitoring, Management and Analytics – Learn about the latest offering in IT Operations Management – Log Analytics, IT Analytics and Application Performance Monitoring.

Tuesday – 4:45pm Palm B – Building a High-Available Enterprise Manager system with Werner de Gruyter – This session is a must attend for anybody who needs to build or maintain a highly available Enterprise Manager.

Wednesday – 8:00am Palm B – Oracle Enterprise Manager Security: a Practitioners Guide – Rise and shine with my session on how to make Enterprise Manager security work for your company in more ways than one!

For a full list of Oracle Management Cloud and Oracle Enterprise Manager demos, labs, sessions and SIG meetings be sure to save or print this handy schedule!   Don’t worry, if you’re not heading to Las Vegas, you can still catch my session by registering for the IOUG Virtual Forum!

I will also be on the Oracle demo grounds either at the Oracle Enterprise Manager booth or Oracle Management Cloud booth.   If you follow me on twitter or read my blog and we haven’t met in person, stop by and say hi!

Introducing Oracle Management Cloud…

Oracle Management Cloud

If you were at Oracle OpenWorld this year you might have had the chance to preview Oracle’s latest cloud service – Oracle Management Cloud (OMC).   The three OMC services launched this month are Application Performance Monitoring (APM), Log Analytics (LA) and IT Analytics (ITA).    The goal for OMC is to bring together different types of operational data for use by both businesses and IT.  All data is stored in a unified data platform that allows you to navigate from one service to another while working on specific use cases.   Here’s a quick overview of each service to get you up to speed.

Application Performance Monitoring

This service gives you insight into the end user performance and experience, from  browser statistics to  AJAX calls.  With statistics on page load times and errors, you can drill down to find errors, see the request calls, and review memory usage and garbage collection.

omcla4

The integration with Log Analytics allows the user to drill down to server and application logs related to the poor performance time periods.  By using saved searches and creating custom dashboards you can see the information that’s important to your application in one view.

omsla

Log Analytics

Upload all your logs to the cloud – database, application, middleware, server, infrastructure – then search and explore to identify problems or resolve issues.   Troubleshoot problems by exploring logs in context of the application using topology-aware log exploration. Then utilize the cluster feature to identify outliers or frequently occurring patterns.  There’s always a lot of noise in log files, so this allows you to filter out the noise and get to the real errors that you need to see.

la

Another key feature is the ability to correlate other logs within a time period.  Let’s say you found an error at 1:00, you can then see the log entries 1 minute before and after on different systems to identify any correlated log entries.    LA also includes the ability to save queries and use them in a dashboard so you can replicate your searches with ease.  Not to mention storing logs in the Oracle cloud instead of on your servers for long periods of time.

IT Analytics

Finally a way to look at IT resources holistically and see how targets in your environment compare to one another.  Resource Analytics will allow you to view current utilization as well as forecast things like storage and CPU across targets, or groups of targets. Answer questions like how much storage will we need in 6 months?  How about what databases are consuming the most CPU and how much have they increased recently?   itacpu1

With Performance Analytics you can identify bottlenecks across Database or Middleware targets.  Have you ever wondered what the worst SQL in your environment is?  I had a manager once who really liked to point out the worst performing SQLs across applications, how easy this will be now!

Probably the most exciting feature is the Data Explorer which allows you to create custom queries and save them as a dashboard that you can create for your unique requirements.   Below I’ve searched for database instances based on their DB Wait time, hovering over the chart you get a popup with the target, type of wait and value.  I can now save this widget to a dashboard if I like.itawaits

Learn More

More information on OMC can be found in this ebook from Oracle and on the Oracle Cloud website.   You can also watch the launch video here.   I will be posting more about each of the services over the next few weeks, as well as sharing my experiences with our first customer implementations, so stay tuned!