Result! I have finally been able to gather a complete RDA (Oracle Remote Diagnostic Agent) output on my 2 node RAC system. After consulting the relevant documentation on MOS-which is spread over at least 42 Doc IDs-I found them not to be very helpful to the degree that some of what I read is actually wrong or contradicting. I put together a short note, primarily to myself, to remind me of the process. I hope you find it useful, too.
The RDA version I used for this post is 8.14.x from MOS March 4th 2017. My RAC nodes are based on Oracle Linux 7.3/UEK 4.
Starting the data collection
I will be speaking about the following topics in Rocky Mountain Oracle User group Training days (RMOUG, Denver) February 7-9, 2017.
Come to my presentations and say Hi to me
I have just realised that the number of posts about RAC 12c Release 1 on this blog is rather too small. And since I’m a great fan of RAC this has to change :) In this mini-series I am going to share my notes about creating a Data Guard setup on my 2 node 18.104.22.168.161018 RAC primary + identical 2 node RAC standby system in the lab.
NOTE: As always, this is just a demonstration using VMs in my lab, based on my notes. Your system is most likely different, so in real-life you might take a different approach. The techniques I am using here were suitable for me, and my own small scale testing. I tried to make sure they are valid, but you may want to allocate more resources in your environment. Test, test, test on your own environment on test kit first!
The lab Environment
My environment consists of the following entities:
I was testing an application performance in 12c, and one job was constantly running slower than 11g. This post is to detail the steps. I hope the steps would be useful if you encounter similar issue.
In an one hour period, over 90% of the DB time spent on waiting for library cache lock waits. Upon investigation, one statement was suffering from excessive waits for ‘library cache lock’ event. We recreated the problem and investigated it further to understand the issue.
Following is the output of wait_details_rac.sql script (that I will upload here) and there are many PX query servers are waiting for ‘library cache lock’ wait event.
UPDATE 28-JUL-2016: the problem – as stated by Anil in the comments section – seems to be fixed. I applied the Proactive Bundle Patch 23273686 and OJVM patch 23177536 to the Grid and RDBMS home. After a successful execution of datapatch the errors in the CRSD agent log I reported in this post went away, and I can also see the settings in the database alert.log.
Scroll to the bottom of the post for more detail.
For those not yet on that patch level-the original post
Over the last many years, some of you have invited me to attend conferences in India, and talk about Oracle RAC and performance. I have not had an opportunity to make it to conferences in India, until now
Thanks for coming to my presentations in RAC day at Dublin, Ohio. Please find the presentation files below. Hopefully, I will get video files and upload that here too.
md5 checksum of the zip file is:
$md5sum ooug_2015_pdf.zip df8bdcbc02926e5bbd721514b473bf16 ooug_2015_pdf.zip
I had a recent conversation at Oracle OpenWorld 2015 about a locking anomaly in a 3-node RAC system which was causing unexpected deadlocks. Coincidentally, this conversation came about shortly after I had been listening to Martin Widlake talking about using the procedure dbms_stats.set_table_prefs() to adjust the way that Oracle calculates the clustering_factor for indexes. The juxtaposition of these two topics made me realise that the advice I had given in “Cost Based Oracle – Fundamentals” 10 years ago was (probably) incomplete, and needed some verification. The sticking point was RAC.
I will be talking about RAC and performance in-depth, with lots of demos, in a RAC day training with Ohio Oracle User group on Nov 16,2015 Monday. Venue for the presentation is Dublin, Ohio.
Agenda for the day:
08:00a – 09:00: Registration / Breakfast 09:00a – 09:15: Announcements -Introduction of the speaker 09:15a – 10:30: Underpinning for Oracle RAC and Clusterware 10:30a – 10:45: Break 10:45a – 11:45: RAC cache fusion internals 11:45a – 01:00: Lunch 01:00p – 02:00: RAC Performance tuning Part 1 – Wait events and object tuning 02:00p – 02:15: Break 02:15p – 03:30: RAC performance tuning Part 2 – locks, library cache locks etc. 03:30p – 03:45: Member Announcements, Gift Drawings
Please RSVP to the co-ordinators so that you will have a seat
Following on from the last post, I’ve brought my NFS RAC stuff up to date also.
I noticed I had not done a RAC install using NFS on Oracle Linux 6, so I threw that in for good measure too.