Top 60 Oracle Blogs

Recent comments


Exadata Smart Flash Logging Explained

I’ve seen some posts on the blogosphere where people attempt to explain (or should I say guess) how Exadata Smart Flash Logging works and most of them are wrong. Hopefully this post will help clear up some the misconceptions out there.

The following is an excerpt from the paper entitled “Exadata Smart Flash Cache Features and the Oracle Exadata Database Machine” that goes into technical detail on the Exadata Smart Flash Logging feature.

HCC – 2

Just a little follow-up to my previous note on hybrid columnar compression. The following is the critical selection of code I extracted from the trace file after tracing a run of the advisor code against a table with 1,000,000 rows in it:


Hybrid Columnar Compression is one of the big features of Exadata that can make fairly dramatic differences to the amount of space it takes to store your data. But how do you find out what’s going on under the covers if you haven’t got an Exadata machine in your garage ?

Here’s a simple starting point that occurred to me a couple of days ago after the product manager (or some such) pointed out that there was no need to make an Exadata emulator available to anyone because all you needed was the compression advisor which you could trust because it actually compressed a sample of your data to see how well it could compress.

Oracle Big Data Appliance — Oracle’s Bold Move Into Big Data Space

Oracle Big Data Appliance (BDA) is being announced at the Oracle OpenWorld keynote as I’m posting this. It will take some time for it to be actually available for shipment and some details will likely change but here is what we have so far about Oracle Big Data Appliance. A rack with InfiniBand, full of [...]

Oracle Big Data Appliance — What’s Cooking?

Many analysts are suggesting that a big data appliance will be announced at this OOW. Based on published Oracle OpenWorld focus sessions on (PDF documents), the following technologies will most likely be the key — Hadoop, NoSQL, Hadoop data loader for Oracle, R Language. Want more details — you have to wait for them. [...]

I’m No Longer An Oracle ACE But Even I Know This: SPARC SuperCluster Will “Redefine Information Technology.” Forget Best Of Breed (Intel, EMC, VMware, Etc).

Before Oracle recruited me in 2007, to be the Performance Architect in the Exadata development organization, I was an Oracle ACE. As soon as I got my Oracle employee badge I was surprised to find out that I was removed from the roles of the Oracle ACE program. As it turned out Oracle Employees could not hold Oracle ACE status. Shortly thereafter, the ACE program folks created the Oracle Employee ACE designation and I was put into that status. In March 2011 I resigned from my role in Exadata development to take on the challenge of Performance Architect in the Data Computing Division of EMC focusing on the Data Computing Appliance and Greenplum Database.

Oracle Expertise Within EMC
Knowing a bit about Oracle means that I’m involved in Oracle-related matters in EMC. That should not come as a surprise since there are more blocks of Oracle data stored on EMC storage devices than any other enterprise-class storage. So, while I no longer focus on Exadata I remain very involved in Oracle Database matters in EMC—in at least an oblique fashion. So you say, “Remind me what this has to do with SPARC SuperCluster.” Please, read on.

So, my status in the Oracle ACE program has gone from non-ACE to ACE to non-ACE to ACE to non-ACE. It turns out that readers of this blog have noticed that fact. Not just two weeks ago I received email from a reader with the following quote:

Kevin, I read your blog for many years. I really like learning about system and storage topics and Oracle. You are not an Oracle ACE so I want you to remove the logo from you (sic) front page

I responded in agreement to the reader and am about to remove the Oracle ACE logo from the front page. She is right and I certainly don’t want to misrepresent myself.

Ace Director
Some of my fellow OakTable Network members started the paperwork to refer me into ACE Director status. They needed me to supply some information for the form but before I filled it out I read the ACE Director requirements. As ACE Director I would be required to speak at a certain number of conferences, or other public forums, covering material that helps Oracle customers be more successful with Oracle products. I gave that some thought. I certainly have no problems doing that—and indeed, I have done that and continue to do that. But, Oracle has acquired so many companies that no matter where I decided to go after leaving Oracle I couldn’t avoid working for a company that Oracle views as competition. To put it another way, Oracle views everyone in the enterprise technology sector as competition and everyone in return views Oracle as co-opetition or competition.

In my assessment, Oracle’s acquisitions have moved the company into a co-opetitive posture where companies like EMC are concerned. EMC and Oracle share customers. Conversely, EMC shares customers with all of Oracle’s software competitors as well. That’s the nature of industry consolidation. What does this have to do with the ACE program? Well, my current role in EMC will not be lending itself to many public speaking opportunities—at least not in the foreseeable future. For that, and a couple other reasons, I decided not to move forward with the ACE Director nomination put in motion by my fellow OakTable cadre. And, no, I haven’t forgot that this post is about SPARC SuperCluster goodness.

Oracle dominates the database market today. That is a fact. Oracle got to that position because choosing Oracle meant no risk of hardware lock-in. Remember “Open Systems?” Oracle was ported and optimized for a mind-boggling number of hardware/operating system platforms. I was a part of that for 10 years in my role within Sequent Computer System’s database engineering group.

This is all fresh in mind because I had dinner with one of the Vice Presidents in Oracle Server Technology just three nights ago. We’ve been friend for many years–about 15 or so if I recall. When we get together we enjoy discussing what’s going on in the IT industry today while wincing over the fact that the industry in general seems to enjoy “re-discovery” of how to solve problems that we already solved at least once over the period of our relationship. That’s just called getting old in a fast-paced industry.

So, while I’m no longer in the Oracle ACE program I can still enjoy putting aside my day job as co-opetitor-at-large (my role at EMC) and enjoy the company of friends—especially with those of us who, in one way or another, helped Oracle become the dominant force in open systems database technology.

Your Best Interest In Mind: SPARC?
With the topics from my dinner three nights ago in mind, and my clean-slate feeling regarding my status in the Oracle ACE program, I sit here scratching my head and pondering current IT industry events. Consider the meltdown of Hewlett-Packard (I could have wiped out 50% of HP’s market cap for less than 25 million dollars and I speak a good bit of Deutsch to boot), Larry-versus-Larry, Oracle’s confusion over the fact that Exadata is in fact commodity x86 servers) and how, on September 26 2011, we get the privilege of hearing how a has-been processor architecture (SPARC) in the latest SuperCluster offering is going to “redefine the IT industry.”

Redefine the IT industry? Really? Sounds more like open systems lock-in to me.

I personally think cloud computing is more likely to redefine the IT industry than some SPARC-flavored goodies. That point of view, as it turns out, is just another case where a non-Oracle ACE co-opetitor like me disagrees with Oracle executives. Indeed, could the pro-cloud viewpoint I share with EMC and VMware be any different from that of Oracle corporation’s leadership? Does anyone remember the following quote regarding Oracle Corporation’s view of the cloud?

What is it? It’s complete gibberish. It’s insane. When is this idiocy going to stop?

We’ll make cloud computing announcements. I’m not going to fight this thing. But I don’t understand what we would do differently in the light of cloud.

Don’t understand what to do in light of cloud computing? Is that a mystery? No, it’s called DBaaS and vFabric Data Director is most certainly not just one of those me-too “cloud computing announcements” alluded to in the quote above.

Life Is A Series Of Choices
You (IT shops) can choose to pursue cloud computing. You can choose x86 or SPARC stuff. You can choose to fulfill your x86 server sourcing requirements from a vendor committed to x86 or not.  You can fulfill your block and file storage requirements with products from a best of breed neutral storage vendor or not.  And, finally, you can choose to read this blog whether or not I hold Oracle ACE program status. I’d prefer you choose the former rather than the latter.

By the way, Oracle announced the SuperCluster about 9 months ago:

I lost my Oracle ACE designation because I became an Oracle employee, SPARC Supercluster isn’t going to redefine anything and I still remember the real definition of “Open Systems.” I also know, all to well, what the term co-opetition means.

Filed under: Exadata, Exadata Database Machine, oracle, Oracle Versus Intel, SPARC Supercluster, vFabric Data Director Tagged: OpenWorld 2011

Oracle Database Appliance — What Does It Mean for You and Your Business?

When I first heard about Oracle Database Appliance and what it does, I got really excited — I saw great potential in this product. When we got our hands dirty and started testing the appliance, I become confident that this product will be a hit. Now it’s finally the time when I can share my [...]

Collecting and analysing Exadata cell metrics

Recently I have been asked to write a paper about Exadata Flash Cache and its impact on performance. This was a challenge to my liking! I won’t reproduce the paper I wrote, but I’d like to demonstrate the methods I used to get more information about what’s happening in the software.


The Exadata Flash Cache is provided by four F20 PCIe cards in each cell. Currently the PCI Express bus is the most potent way to realise the potential of the flash disk in terms of latency and bandwidth. SSDs attached to a standard storage array will be slowed by fibre channel as the transport medium.

Each of the F20 cards holds 96G of raw space, totalling in 384GB of capacity per storage cell. The usable capacity is slightly less. The F20 card is subdivided into 4 so called FMODs, or solid state flash modules visible to the operating system using the standard SCSI SD driver.

Cellcli can also be used to view the FMODs using the “LIST PHYSICALDISK” command. The output is slightly different from the spinning disks as they are reported SCSI drivee’s [host:bus:target:lun] notation.

Now please don’t be tempted to take the FMODs and transform them into celldisks!

How it works

The flash cache is a write-through cache, and therefore won’t speed up write operations to disk. If an internal, undocumented algorithm decides that the data just written to disk is suitable for caching, it will put the data into the ESFC. It is quite clever in this respect and performs optimisations “ordinary” cache cannot perform.

Smart scans are going to ignore the flash cache by default and read directly from disk, unless the segment’s storage clause specifically instructs Oracle to read from flash cache as well. Meddling with the storage clause is also referred to as object pinning, a practice which is not recommended. Pinning objects to the flash cache immediately reminded me of the bad practice in the old 8i days of pinning objects in the shared pool. It didn’t necessarily made things go faster or help (except in some special cases).

The main statistics counter to query in relation to the smart cache is called “cell flash cache read hits”. The remainder of this paper has to do with the cell metrics which cannot be queried directly from the database. They are more granular though making them well suited for analysis.

The test case

The swingbench order entry benchmark has provided the data for the following text. The table ORDER_ITEMS contains most rows in the schema, and it is partitioned to reflect a realistic environment. Indexes have been made invisible as they may have impacted the performance statistics. Additionally cellcli’s WHERE clause isn’t very flexible and querying a number of objectIDs can be time consuming.

SOE.ORDER_ITEMS is divided into partitions with object IDs 74506 to 74570. These object IDs are required for mapping the output of cellcli to the data dictionary later.

A little bit of setup has been performed to capture the cell metrics. For pre and post comparison it was necessary to capture statistics in a statspack-like way and to analyse them. Rather than using Excel I decided to store the gathered information in the database in a dedicated schema.

I gathered two different sets of information from the cells: firstly the metrics for the objectType “FLASHCACHE”, and secondly the contents of the flash cache using the fabulous “dcli” command. The distributed command line interface can be used to execute arbitrary commands on a number of hosts simultaneously and reports the results back to the caller. I am redirecting the output from STDOUT to a CSV file in my home directory and use external tables to copy this information into the so-called “RESULT” table.

The table DDL is as follows:

create table cacheContents_csv (
cellname varchar2(50),
cachedKeepSize     number,
cachedSize         number,
hitCount           number,
missCount          number,
objectNumber             number
organization external (
type oracle_loader
default directory testdir
access parameters (
records delimited  by newline
fields  terminated by ';'
missing field values are null
location ('flashcontents.csv')
reject limit unlimited;

FROM cacheContents_csv A;

TYPE oracle_loader
"NAME" (14:36) CHAR,

"NAME"        VARCHAR2(50 BYTE),

I additionally added a table called “metricdefinition” which allows me to match the short metric name from metriccurrent with a more human readable description.

"NAME"        VARCHAR2(100 BYTE),
"UNIT"        VARCHAR2(20 BYTE)

The definitions, too, were loaded into the database using sqlldr.

First the metrics were captured:

# dcli -l root -g /tmp/martin_group 'cellcli -e "list METRICCURRENT where objecttype = \"FLASHCACHE\""' | sed -e 's/://' > /home/oracle/martin/metrics.csv

I admit it’s probably not the most elegant way to do so but it worked for me and I didn’t have weeks to prepare (as always…)

Now what does this command do? It will execute a call to cellcli on all nodes defined in /tmp/martin_group to list the current metrics for the Flash Cache. The result is then piped into sed which strips out any colons after the cell names and then saves the result in my TESTDIR directory on the first RDBMS node.

Similarly I am using the following command to get the contents of the flash cache to see if anything has changed after executing the query.

 # dcli -l root -g /tmp/martin_group 'cellcli -e "list flashcachecontent where dbUniqueName like 'TEST' and objectNumber > 74505 and objectNumber < 74571 attributes cachedKeepSize,cachedSize,hitCount,missCount,objectNumber"' | sed -e 's/://' | awk '{print $1";"$2";"$3";"$4";"$5";"$6}' > /home/oracle/martin/flashcontents.csv

This looks a little more complex, but it isn’t. The apparent complexity is required to limit the result set to the contents of the flash cache for database TEST and the object_ids of the table partitions. There is a little more transformation involved with AWK to convert the output in a “;” separated line.

With this done, it was possible to transfer the information in the staging tables into the RESULT tables. This is a simple “insert into table select testrun#, * from …CSV”.

The analysis I did was very simple once the data has been loaded into the RESULT tables. For the flashcache contents I used the following queries:

WITH prestats AS
( SELECT * FROM metriccurrent_result WHERE testrun# = &1
poststats AS
( SELECT * FROM metriccurrent_result WHERE testrun# = &2
SELECT pre.cellname,,
pre.metricvalue pre_metricvalue,
post.metricvalue post_metricvalue,
to_number(REGEXP_SUBSTR(post.metricvalue, '(\S*)'),'999G999G999D999999', 'NLS_NUMERIC_CHARACTERS = ''.,''') -
to_number(REGEXP_SUBSTR( pre.metricvalue, '(\S*)'),'999G999G999D999999', 'NLS_NUMERIC_CHARACTERS = ''.,''') delta
FROM prestats pre,
poststats post,
metricdefinition defn
WHERE pre.cellname = post.cellname
and =
and pre.objecttype = post.objecttype
and =
and = 'FC_IO_BYKEEP_R'
and (to_number(REGEXP_SUBSTR(post.metricvalue, '(\S*)'),'999G999G999D999999', 'NLS_NUMERIC_CHARACTERS = ''.,''') -
to_number(REGEXP_SUBSTR( pre.metricvalue, '(\S*)'),'999G999G999D999999', 'NLS_NUMERIC_CHARACTERS = ''.,''') ) <> 0
order by 1,2;

WITH prestats AS
( SELECT * FROM cachecontents_result WHERE testrun# = &1
poststats AS
( SELECT * FROM cachecontents_result WHERE testrun# = &2
SELECT pre.cellname,
pre.cachedkeepsize pre_cachedkeepsize,
pre.cachedsize pre_cachedsize,
pre.hitcount pre_hitcount,
pre.misscount pre_misscount,
post.cachedkeepsize post_cachedkeepsize,
post.cachedsize post_cachedsize,
post.hitcount post_hitcount,
post.misscount post_misscount,
(post.cachedkeepsize-pre.cachedkeepsize) delta_cachedkeepsize,
(post.cachedsize-pre.cachedsize) delta_cachedsize,
(post.hitcount-pre.hitcount) delta_hitcount,
(post.misscount-pre.misscount) delta_misscount
FROM prestats pre,
poststats post
WHERE pre.cellname   = post.cellname
AND pre.objectnumber = post.objectnumber

This revealed quite a lot about the underlying workings of the flash cache. The above queries are best executed with SQL Developer and the result is then copied and pasted into a spreadsheet application to be saved from accidental overwriting.


The above certainly isn’t bullet proof or very elegant, but a good starting point for your own experiments. Time permitting I would like to automate the generation of the CSV files via a scheduled job, and also the population of the result tables. There really should be some sort of unique key on the results tables to avoid duplicates. Maybe one day I’d like to write a statspack like version of the lot…

Oracle Exadata vs SAP HANA

Before I left on vacation (now almost a month ago – can’t remember when I had such a long vacation if I ever had), Mark Fontecchio organized a short video conference between myself and John Appleby. The idea was to compare Oracle Exadata with SAP HANA in a shot video discussion. Unfortunately, video part didn’t [...]

Application Developers Asking You For Urgent Response To A Database Provisioning Request? Tell Them: “Go Do It Yourself!”

…then calmly close the door and get back to work! They’ll be exceedingly happy!

The rate at which new applications pour forth from corporate IT is astounding. Nimble businesses, new and old, react to bright ideas quickly and doing so often requires a new application.  Sure, the backbone ERP system is critical to the business and without it there would be no need for any other application in the enterprise. This I know. However…

When an application developer is done white-boarding a high-level design to respond to a bright idea in the enterprise it’s off to the DBA Team to get the train rolling for a database to back-end the new application. I’d like to tell the DBA Team what to tell the application developer. Are you ready? The response should be:

Go do it yourself! Leave me alone. I’m busy with the ERP system

You see, the DBA Team can say that and still be a good corporate citizen because this hypothetical DBA Team works in a 21st century IT shop where Database As A Service is not just something they read about in the same blog I’ve been following for several years, namely Steve Bobrowski’s blog Database As A Service.

Steve’s blog contains a list of some of the pioneers in this technology space. I’m hoping that my trackback to his blog will entice him to include a joint VMware/EMC product on the list. I’d like to introduce readers of this blog to a very exciting technology that I think goes a long way towards realizing the best of what cloud database infrastructure can offer:

VMware vFabric(tm) Data Director

I encourage readers to view this demo of vFabric Data Director and  read the datasheet because this technology is not just chest-thumping IdeaWare™.  I am convinced this is the technology that will allow those in the DBA community to tell their application developers to “go do it yourself” and make their company benefit from IT even more by doing so.

What Can This Post Possibly Have To Do With Oracle Exadata?
Folks who read this blog know I can’t resist injecting trivial pursuit.

The architect and lead developer of vFabric Data Director technology is one of the three concept inventors of Oracle Exadata or, as it was soon to be called within Oracle, Storage Appliance for Grid Environments (SAGE). One of the others of that “team of three” was a crazy-bright engineer with whom I spent time scrutinizing the affect of NUMA on spinlocks (latches) in Oracle Database in the Oracle8i time frame.

It is a small world and, don’t forget, if a gifted application developer approaches your desk for a timely, urgent request for database provisioning just tell him/her to go do it yourself! They’ll be glad you did!

Filed under: Business Critical Systems, Cloud Infrastructure, Database As A Service, Exadata, Exadata Database Machine, oracle, Postgres, vFabric Data Director Tagged: DBaaS