aix

I am often working on systems with large number of LUNs, going into the hundreds. Trying to keep track is difficult at best. Sometimes though you might want to limit yourself to a specific number of disks-I knew about the dskfilt option in collectl but only today learned about a similar feature in nmon. The following is a very greatly simplified example of course, but it gives you the idea.

Assume you are interested in 4 disks-sd{a,b,c,d}. Let’s further assume that your disks are used for 2 ASM disk groups, DATA and RECO. You could create a file “disks.dat” with the following contents:

DATA sda sdb
RECO sdc sdd

Then pass that file to nmon with the -g flag. When you type “g” now, you are shown IO stats for those disks.

I just applied PSU 3 for 11.2.0.1 on AIX 5.3 TL11. That would have been more straight forward, had  there not been a host of errors when relinking oracle. There were loads of these:

SEVERE:OPatch invoked as follows: 'apply '
INFO:
Oracle Home       : /u01/app/oracle/product/11.2.0.1
Central Inventory : /u01/app/oracle/product/oraInventory
 from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0.1/oui
Log file location : /u01/app/oracle/product/11.2.0.1/cfgtoollogs/opatch/opatch2010-11-01_09-28-22AM.log
...
[more output]
...
INFO:Running make for target ioracle
INFO:Start invoking 'make' at Mon Nov 01 09:29:26 GMT 2010Mon Nov 01 09:29:26 GMT 2010
INFO:Finish invoking 'make' at Mon Nov 01 09:29:40 GMT 2010
WARNING:OUI-67215:
OPatch found the word "error" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
ld: 0711-415 WARNING: Symbol ldxdts is already exported.
ld: 0711-415 WARNING: Symbol ldxsto is already exported.
ld: 0711-415 WARNING: Symbol lnxadd is already exported.
...
ld: 0711-319 WARNING: Exported symbol not defined: cout__3std
...
ld: 0711-224 WARNING: Duplicate symbol: .kotgscid
...
ld: 0711-783 WARNING: TOC overflow. TOC size: 219488    Maximum size: 65536
 Extra instructions are being generated for each reference to a TOC
 symbol if the symbol is in the TOC overflow area.

Wow, that looked like a completely corrupted installation of Oracle now, and I got ready to roll the change back. However, MOS had an answer to this!

These errors can be ignored as per this MOS document: Relinking causes many warning on AIX [ID 1189533.1]. See also MOS note Note 245372.1 TOC overflow Warning Can Safely Be Ignored and BUG:9828407 – TOO MANY WARNING MESSAGES WHEN INSTALLING ORACLE 11.2 for more information.

The HP-UX Itanium and AIX (PPC64) ports of Oracle Database 11g Release 2 can now be downloaded from OTN. Happy Holidays!!! Tweet This Post

Recently, my client deployed a new application and had this intermittent “Deadlock Storm” …

A trace file was sent and I was able to pinpoint the cause of the deadlock and the session that caused it.
The deadlock was a TX enqueue with mode of 4 (S – share) which could be verified by looking at the following lines of the Process State dump:

   last wait for 'enq: TX - row lock contention' blocking sess=0x 7000000cb239d60 seq=7849 wait_time=2929705 seconds since wait started=3
            name|mode=54580004, usn<<16 | slot=a0028, sequence=283f2

the “enqueue and lock mode” is explained as:
mode=54580004 (see above)
5458 (hex) = TX (ascii)
0004 (hex) = mode 4 (S – share)

Recently I encountered a performance problem scenario where a simple sqlplus “/ as sysdba” took about 2minutes to finish, this is critical to the client’s business because they have a local C program that loads Call Detail Reports on the database making use of local authentication for most of its operations and Sql*Loader to load the data, so this “2minutes of waiting” when accumulated greatly consumes significant time on their operations and greatly impacts the business.

When I arrived on the client I first checked the alert logs of both ASM (they have a separate home for ASM) and RDBMS, there were no errors…

Then I checked on the server to see if there were any CPU, IO, memory, swap, and network bottlenecks going on

The CPU run queue was zero and most of the time 90% idle

The disks were also most of the time idle

The memory utilization was low with 430MB free