Thank you all those who attended my session Cache Fusion Demystified.
As I mentioned, it was impossible to fit in a demo within that compressed 45 minute timeline. I put the details of the demo in a separate file with all the scripts and instructions that allows you to execute the demos at your own site.Download the slide deck as well as the scripts. In addition I have also written a whitepaper to explain the concepts clearer.
Download the slides, the paper, the demo scripts and instructions here.
I sincerely hope it demystified one of the most complex topics in Oracle RAC. As always, I would love to hear from you about your experience.
This is in the Series "100 Things You Probably Didn't Know About Oracle". If you haven't already, I urge you to read the other parts -
During the recently concluded Collaborate 11 (http://collaborate11.ioug.org) I was asked a question which led me to the idea for this entry - the fourth in the series. If the database size is 100GB and you have a 100GB buffer cache, all the blocks will be in the memory and there will never be an I/O to the datafiles other than the regular DBWR lazy writes, right?
The other day I was putting together my presentation for Oracle Open World on Application Profiling in RAC. I was going to describe a methodology for putting a face to an app by measuring how it behaves in a database – a sort of a signature of that application. I was going to use the now-ubiquitous 10046 trace for wait events and other activities inside the database. For resource consumption such as redo generated, logical I/Os, etc., I used the v$sesstat; but then I was stuck. How would I collect the stats of a session when the session has not even started and I don’t know the SID. That problem led to the development of this tool where the stats of a future session can be recorded based on some identifying factors such as username, module, etc. Hope this helps in your performance management efforts.
Suppose you want to find out the resource consumed by a session. The resources could be redo generation, CPU used, logical I/O, undo records generated – the list is endless. This is required for a lot of things. Consider a case where you want to find out which apps are generating the most redo; you would issue a query like this:
select sid, value
from v$sesstat s, v$statname n
where n.statistic# = s.statistic#
and n.name = 'redo size'
The value column will show the redo generated. From the SID you can identify the session. Your next stop is v$session to get the other relevant information such as username, module, authentication scheme, etc. Problem solved, right?
Not so fast. Look at the above query; it selects from v$sesstat. When the session is disconnected, the stats disappear, making the entries for that session go from v$sesstat. If you run the query, you will not find these sessions. You have to constantly select from the v$sesstat view to capture the stats of the sessions hoping that you would capture the stats before the session disconnects. But it will be not be guaranteed. Some short sessions will be missed in between collection samples. Even if you are lucky to capture some stats of a short session, the other relevant information from v$session will be gone.
Oracle provides a package dbms_monitor, where a procedure named client_id_stat_enable allows you to enable stats collection on a future session where the client_id matches a specific value, e.g. CLIENT1. Here is an example:
However there are three issues:
(1) It collects only about 27 stats, out of 400+
(2) It offers only three choices for selecting sessions – client_id, module_name and service_name.
(3) It aggregate them, sums up all stats for a specific client_id. That is pretty much useless without a detailed session level.
So, in short, I didn’t have a readily available solution.
Well, necessity is the mother of invention. When you can’t find a decent tool; you build it; and so did I. I built this tool to capture the stats. This is version 1 of the tool. It has some limitations, as shown at the end. These limitations do not apply to all situations; so the tool may be useful in a majority of the cases. Later I will expand the tool to overcome these limitations.
The fundamental problem, as you recall, is not the dearth of data (v$sesstat has plenty); it’s the sessions in the future. To capture those sessions, the tool relies on a post-logon database trigger to capture the values.
The second problem was persistence. V$SESSTAT is a dynamic performance view, which means the records of the session will be gone when the session disappears. So, the tool relies on a table to store the data.
The third problem is the getting the values at the very end of the session. The difference between the values captured at the end and beginning of the session are the stats. To capture the values at the very end; not anytime before, the tool relies on a pre-logoff database trigger.
The fourth challenge is identification of sessions. SID of a session is not unique; it can be reused for a new session; it will definitely be reused when the database is recycled. So, the tool uses a column named CAPTURE_ID, a sequentially incremented number for each capture. Since we capture once at the beginning and then at the end, I must use the same capture_id. I use a package variable to store that capture_Id.
Finally, the tool allows you to enable stats collections based on some session attributes such as username, client_id, module, service_name, etc. For instance you may want to enable stats for any session where the username = ‘SCOTT’ or where the os_user is ‘ananda’, etc. These preferences are stored in a table reserved for that purpose.
Now that you understand how the tool is structured, let me show the actual code and scripts to create the tool.
(1) First, we should create the table that holds the preferences. Let’s call this table RECSTATS_ENABLED. This table holds all the different sessions attributes (ip address, username, module, etc.) that can enable stats collection in a session.
CREATE TABLE SYS.RECSTATS_ENABLED
SESSION_ATTRIBUTE VARCHAR2(200 BYTE),
ATTRIBUTE_VALUE VARCHAR2(2000 BYTE)
If you want to enable stats collection of a session based on a session attribute, insert a record into this table with the session attribute and the value. Here are some examples. I want to collect stats on all sessions where client_info matches ‘MY_CLIENT_INFO1’. You would insert a record like this:
insert into recstats_enabled values ('CLIENT_INFO','MY_CLIENT_INFO1');
Here are some more examples. All sessions where ACTION is ‘MY_ACTION1’:
insert into recstats_enabled values ('ACTION','MY_ACTION1');
Those of user SCOTT:
insert into recstats_enabled values ('SESSION_USER','SCOTT')
Those with service name APP:
insert into recstats_enabled values ('SERVICE_NAME','APP')
You can insert as many preferences as you want. You can even insert multiple values of a specific attribute. For instance, to enable stats on sessions with service names APP1 and APP2, insert two records.
Important: the session attribute names follow the naming convention of the USERENV context used in SYS_CONTEXT function.
(2) Next, we will create a table to hold the statistics
CREATE TABLE SYS.RECSTATS
CAPTURE_POINT VARCHAR2(10 BYTE),
ACTION VARCHAR2(2000 BYTE),
CLIENT_DENTIFIER VARCHAR2(2000 BYTE),
CLIENT_INFO VARCHAR2(2000 BYTE),
CURRENT_EDITION_NAME VARCHAR2(2000 BYTE),
CURRENT_SCHEMA VARCHAR2(2000 BYTE),
CURRENT_USER VARCHAR2(2000 BYTE),
DATABASE_ROLE VARCHAR2(2000 BYTE),
HOST VARCHAR2(2000 BYTE),
IDENTIFICATION_TYPE VARCHAR2(2000 BYTE),
IP_ADDRESS VARCHAR2(2000 BYTE),
ISDBA VARCHAR2(2000 BYTE),
MODULE VARCHAR2(2000 BYTE),
OS_USER VARCHAR2(2000 BYTE),
SERVICE_NAME VARCHAR2(2000 BYTE),
SESSION_USER VARCHAR2(2000 BYTE),
TERMINAL VARCHAR2(2000 BYTE),
STATISTIC_NAME VARCHAR2(2000 BYTE),
Note, I used the tablespace USERS; because I don’t want this table, which can potentially grow to huge size, to overwhelm the system tablespace. The STATISTIC_NAME and STATISTIC_VALUE columns record the stats collected. The other columns record the other relevant data from the sessions. All the attributes here have been shown with VARCHAR2(2000) for simplicity; of course they don’t need that much of space. In the future versions, I will put a more meaningful limit; but 2000 does not hurt as it is varchar2.
The capture point will show when the values were captured – START or END of the session.
(3) We will also need a sequence to identify the sessions. Each session will have 400+ stats; we will have a set at the end and once at the beginning. We could choose SID as an identifier; but SIDs could be reused. So, we need something that is truly unique – a sequence number. This will be recorded in the CAPTURE_ID column in the stats table.
SQL> create sequence seq_recstats;
(4) To store the capture ID when the post-logon trigger is fired, to be used inside the pre-logoff trigger, we must use a variable that would be visible to entire session. A package variable is the best for that.
create or replace package pkg_recstats
(5) Finally, we will go on to the meat of the tool – the triggers. First, the post-logon trigger to capture the stats in the beginning of the session:
CREATE OR REPLACE TRIGGER SYS.tr_post_logon_recstats
after logon on database
l_capture_point recstats.capture_point%type := 'START';
l_matched boolean := FALSE;
pkg_recstats.g_recstats_id := null;
for r in (
select session_attribute, attribute_value
order by session_attribute
exit when l_matched;
-- we select the userenv; but the null values may cause
-- problems in matching; so let’s use a value for NVL
-- that will never be used - !_!_!
l_stmt := 'select nvl(sys_context(''USERENV'','''||
r.session_attribute||'''),''!_!_!_!'') from dual';
execute immediate l_stmt into l_attr_val;
if l_attr_val = r.attribute_value then
-- match; we should record the stats
-- and exit the loop; since stats should
-- be recorded only for one match.
l_matched := TRUE;
insert into recstats
from v$mystat s, v$statname n
where s.statistic# = n.statistic#;
The code is self explanatory. I have provided more explanation as comments where needed.
(6) Next, the pre-logoff trigger to capture the stats at the end of the session:
CREATE OR REPLACE TRIGGER SYS.tr_pre_logoff_recstats
before logoff on database
l_capture_point recstats.capture_point%type := 'END';
if (pkg_recstats.g_recstats_id is not null) then
insert into recstats
from v$mystat s, v$statname n
where s.statistic# = n.statistic#;
Again the code is self explanatory. We capture the stats only of the global capture ID has been set by the post-logoff trigger. If we didn’t do that all the sessions would have started recording stats at their completion.
Now that the setup is complete, let’s perform a test by connecting as a user with the service name APP:
SQL> connect arup/arup@app
In this session, perform some actions that will generate a lot of activity. The following SQL will do nicely:
SQL> create table t as select * from all_objects;
Now check the RECSTATS table to see the stats on this catured_id, which happens to be 1330.
col name format a60
col value format 999,999,999
select a.statistic_name name, b.statistic_value - a.statistic_value value
from recstats a, recstats b
where a.capture_id = 1330
and a.capture_id = b.capture_id
and a.statistic_name = b.statistic_name
and a.capture_point = 'START'
and b.capture_point = 'END'
and (b.statistic_value - a.statistic_value) != 0
order by 2
Here is the output:
workarea memory allocated -2
change write time 1
parse time cpu 1
table scans (long tables) 1
cursor authentications 1
sorts (memory) 1
user commits 2
opened cursors current 2
IMU Flushes 2
index scans kdiixs1 2
parse count (hard) 2
workarea executions - optimal 2
redo synch writes 2
redo synch time 3
rows fetched via callback 5
table fetch by rowid 5
parse time elapsed 5
recursive cpu usage 8
switch current to new buffer 10
cluster key scan block gets 10
cluster key scans 10
deferred (CURRENT) block cleanout applications 10
Heap Segment Array Updates 10
table scans (short tables) 12
messages sent 13
index fetch by key 15
physical read total multi block requests 15
SQL*Net roundtrips to/from client 18
session cursor cache hits 19
session cursor cache count 19
user calls 25
CPU used by this session 28
CPU used when call started 29
buffer is not pinned count 33
execute count 34
parse count (total) 35
opened cursors cumulative 36
physical read total IO requests 39
physical read IO requests 39
calls to get snapshot scn: kcmgss 45
non-idle wait count 67
user I/O wait time 116
non-idle wait time 120
redo ordering marks 120
calls to kcmgas 143
enqueue releases 144
enqueue requests 144
DB time 149
hot buffers moved to head of LRU 270
recursive calls 349
active txn count during cleanout 842
cleanout - number of ktugct calls 842
consistent gets - examination 879
IMU undo allocation size 968
physical reads cache prefetch 997
physical reads 1,036
physical reads cache 1,036
table scan blocks gotten 1,048
commit cleanouts 1,048
commit cleanouts successfully completed 1,048
no work - consistent read gets 1,060
redo subscn max counts 1,124
Heap Segment Array Inserts 1,905
calls to kcmgcs 2,149
consistent gets from cache (fastpath) 2,153
free buffer requested 2,182
free buffer inspected 2,244
HSC Heap Segment Block Changes 2,519
db block gets from cache (fastpath) 2,522
consistent gets 3,067
consistent gets from cache 3,067
bytes received via SQL*Net from client 3,284
bytes sent via SQL*Net to client 5,589
redo entries 6,448
db block changes 9,150
db block gets 10,194
db block gets from cache 10,194
session logical reads 13,261
IMU Redo allocation size 16,076
table scan rows gotten 72,291
session uga memory 88,264
session pga memory 131,072
session uga memory max 168,956
undo change vector size 318,640
session pga memory max 589,824
physical read total bytes 8,486,912
cell physical IO interconnect bytes 8,486,912
physical read bytes 8,486,912
redo size 8,677,104
This clearly shows you all the stats of that session. Of course the table recorded all other details of the session as well – such as username, client_id, etc., which are useful later for more detailed analysis. You can perform aggregations as well now. Here is an example of the stats collected for redo size:
select session_user, sum(STATISTIC_VALUE) STVAL
where STATISTIC_NAME = 'redo size'
group by session_user
… and so on …
select session_user, host, sum(STATISTIC_VALUE) stval
where STATISTIC_NAME = 'redo size'
group by session_user, host
SESSION_USER HOST STVAL
------------ ----------- -------
ARUP oradba2 12356
ARUP oradba1 264567
APEX oradba2 34567
… and so on …
When is the last time you looked at the listener logs? Perhaps never. Not a very good idea. Listener logs contain a wealth of information on security events - it shows you the profile of hosts connecting to the database, the programs they are using and attempting to communicate but failed, among other things. But reading the listener logs is not exactly a pleasant experience. A while ago I wrote a series of articles on an online eZine called DBAZine.com on how to create an external table to read the listener logs using simple SQL; but unfortunately DBAZine.com has folded.
I have placed the articles on my website for your reference. As always, I would love to hear from you how you felt about these, stories of your own use and everything in between.
Today I delivered the second of my four sessions - "Upgrade Case Study: Database Replay, Snapshot Standby and Plan Baselines".
For those you attended, I thank you very much. Here is the presentation.
Why do you come to Open World? I'm sure we will get all kinds of reasons, as many as there are stars in the sky. Some predominant themes are - getting to know more about the Oracle (or related) technologies by attending sessions, reconnecting with old friends and building networking. Of course, getting freebies from the Exhibit Halls, I'm sure, can't be far behind as a motivator.
I come to OOW for all those reasons as well. But high up in my list is the visit to the Exhibit Halls. No; not for the tee-shirts that do not fit me and graphics I don't really dig. I visit the demogrounds and exhibit halls to know about the products and tools that I should be aware of. Where else would you find 1000+ companies advertising the products at one place? Sure, I can call them and find out; but ho do I find them? OOW exhibit halls are prime "hunting" grounds to look for new ideas and tools that I should be interested in; or at least be aware of. I can not only look at the tools; I can actually get some relevant technical facts in 5 minutes which might take weeks of scheduling and hours of marketing talk. And, if I decide the product is not relevant; I can always walk away. I have the privilege of walking away; they don't. If I call them to my office, "they" have that option; not me :) If I find something attractive, I can always follow up and get to know more.
Oracle demogrounds are even better. Not only I can meet Oracle PMs there; but the people who never come out to the public world - developers, development managers, architects and so on. These unsung heroes are mostly the reason why Oracle is what it is now. I meet the known faces, get to know new ones and establish new relationships. They hear from me what customers want and I learn the innards of some features I am curious about.
So, I spent almost the whole day yesterday navigating through demo grounds and exhibit halls. I could cover only a small fraction. In between I had to attend some meetings at work. Going to OOW is never "going away". I wish it was.
One of the most valuable benefits of being an Oracle ACE Director is the briefings by Oracle Product Managers at the Oracle HQ. This year the briefing was on Friday Oct 9th at Oracle conference center rather than the customary Hilton Hotel.
While I was a little disappointed at the coverage of the database topics, I quickly recovered from the alphabet soup that makes up the netherworld of middleware and tools. However, a surprise visit by Thomas Kurian to address questions from the audience about the various product roadmaps was testimonial that Oracle is dead serious about the ACE Program. That proves the commitment Oracle has made for the user community - very heartening.
As always, Vikky Lira and Lillian Buziak did a wonderful job of organizing the event. Considering about 100 ACE Directors from 20+ countries, that is no small task. Perhaps the highlight of the organization was the detailed briefing sheets Lillian prepared for each one individually, down to what car service one takes and when - simply superb! No amount of thanks will be enough. From the bottom of my heart, thank you, Vikky and Lillian. And, thank you Justin Kestelyn - for kicking off and running the event year after year.
Oracle Open World 2009 has officially started with the User Group sessions today. I am presenting a session today. I started off by registering and getting my cool Blogger badge holder, hanging off the even cooler ACE Director lanyard.
I went off to the first session of today on the IOUG bucket - Workload Management by Alex Gorbachev. Alex is one of those people who know their stuff; so there is always something to be learned from there. Alex successfully demonstrated the difference between Connection Load Balancing and Server Side Listener Load Balancing, with pmon trace to show how the sessions are balanced. It sheds light on the question - why Oracle is not balancing the workload.
If you didn't attend this, you should definitely download the presentation and check it out later.