Who's online

There are currently 0 users and 31 guests online.

Recent comments

Oakies Blog Aggregator

Direct path and buffered reads again: compressed tables.

In the previous post on the decision between buffered and direct path reads I showed the decision is depended on the version. Up to and including version the size of a segment needs to be five times small table threshold in order to be considered for direct path reads, and starting from the database starts considering direct path reads starting from small table threshold. The lower limit just discussed is small table threshold or five times small table threshold with lower versions, upper limit is called “very large object threshold” (VLOT) and is five times the size of the buffercache, which is the threshold after which a table scan always is going via direct path.

There seems to be an odd situation with and up that if a table has a size between small table threshold and five times small table threshold, it is always read via direct path, while if the table is bigger than five times small table threshold, it is depended on the amount of blocks in the cache, and switches to buffered reads if the amount of blocks in the cache is 99% or more. This seems to be the opposite of what is logical; a smaller sized table is likely a better candidate than a larger table for caching.

The last thing the previous blogpost showed was the impact of dirty blocks on the buffered/direct path decision. Up to a read that should go via direct path is done buffered when the amount of dirty blocks reaches around 75%. Starting from version, a read that should go via direct path is done buffered when the amount of dirty blocks for a table reaches approximately 50%. This is vital information for data warehouses!

A PL/SQL procedure was included so these tests could be to measure this for yourself. If you have any doubts, follow the link above and paste the procedure to test it for yourself.

The purpose of this blogpost is to further investigate the table direct path or buffered decision in various Oracle database versions when compression is used. I used compression using “COMPRESS FOR OLTP”, in order to see if compression changes the decisions.

The first measurement is reading, doing a full scan of a compressed table:

Version (_STT 3979; _STT*5:19895)

TS@v11201 > @test_direct_path
table: T2_OLTP size: 64512 blocks, 10000000 rows.
cached - physical reads cache/direct: (10)		6419/0
full scan - physical reads cache/direct:			  0/63911
cached - physical reads cache/direct: (20)		6487/0
full scan - physical reads cache/direct:			  0/63911
cached - physical reads cache/direct: (30)		6372/0
full scan - physical reads cache/direct:			  0/63911
cached - physical reads cache/direct: (40)		6376/0
full scan - physical reads cache/direct:			  0/63911
cached - physical reads cache/direct: (50)		6376/0
full scan - physical reads cache/direct:			  0/63911
cached - physical reads cache/direct: (60)		6376/0
full scan - physical reads cache/direct:			  0/63911
cached - physical reads cache/direct: (70)		6372/0
full scan - physical reads cache/direct:			  0/63911
cached - physical reads cache/direct: (80)		6376/0
full scan - physical reads cache/direct:			  0/63911
cached - physical reads cache/direct: (90)		6376/0
full scan - physical reads cache/direct:			  0/63911
cached - physical reads cache/direct: (100)		 6382/0
full scan - physical reads cache/direct:			  0/0

This pattern is repeated up to version if the amount of blocks is 99% or more, the database switches from direct path to buffered reads for table scans. So for simple reading, the decision does not seem different than for non-compressed tables.

When dirty blocks are included, it is different:


TS@v11201 > @test_direct_path.sql
table: T2_OLTP size: 64512 blocks, 10000000 rows.
full scan - physical reads cache/direct: (10)			 0/63911
full scan - physical reads cache/direct: (20)			 0/63911
full scan - physical reads cache/direct: (30)			 0/63911
full scan - physical reads cache/direct: (40)			 0/63911
full scan - physical reads cache/direct: (50)			 0/63911
full scan - physical reads cache/direct: (60)			 0/63911
full scan - physical reads cache/direct: (70)			 0/63911
full scan - physical reads cache/direct: (80)			 0/63911
full scan - physical reads cache/direct: (90)			 0/63911
full scan - physical reads cache/direct: (100)			  0/63911

If a large number of the blocks, or all of them, in the cache are dirty, a full scan does not flip to buffered reads at 99%.

However, it is interesting to see that this changes with version

TS@v11202 > @test_direct_path.sql
table: T2_OLTP size: 64512 blocks, 10000000 rows.
full scan - physical reads cache/direct: (10)			 0/63911
full scan - physical reads cache/direct: (20)			 0/63911
full scan - physical reads cache/direct: (30)			 0/63911
full scan - physical reads cache/direct: (40)			 0/63911
full scan - physical reads cache/direct: (50)			 0/63911
full scan - physical reads cache/direct: (60)			 0/63911
full scan - physical reads cache/direct: (70)			 0/63911
full scan - physical reads cache/direct: (80)			 0/63911
full scan - physical reads cache/direct: (90)			 0/63911
full scan - physical reads cache/direct: (100)			  461/0

Starting from version, the threshold of 99% for a full table scan to switch from direct path to buffered seems to be true for clean blocks and dirty blocks. The same is true for versions and

However, starting from, a full scan on a compressed table is done via direct path even if all the blocks are in the cache and dirty, which seems identical to the version behaviour:

TS@v12101 > @test_direct_path
table: T2_OLTP size: 65536 blocks, 10000000 rows.
full scan - physical reads cache/direct: (10)			 0/65174
full scan - physical reads cache/direct: (20)			 0/65174
full scan - physical reads cache/direct: (30)			 0/65174
full scan - physical reads cache/direct: (40)			 0/65174
full scan - physical reads cache/direct: (50)			 0/65174
full scan - physical reads cache/direct: (60)			 0/65174
full scan - physical reads cache/direct: (70)			 0/65174
full scan - physical reads cache/direct: (80)			 0/65174
full scan - physical reads cache/direct: (90)			 5/65174
full scan - physical reads cache/direct: (100)			  0/65174

So, the conclusion overall is there is a difference between regular heap tables and compressed heap tables. I tested OLTP compression, the [nsmtio] tracing indicates this is the same for HCC compression (hybrid columnar compression). In general, for simple reading there doesn’t seem to be a difference between normal heap tables and compressed heap tables. However, I tested on idle systems, on “real” systems with more “stress” on the buffercache management there might be more mechanisms in play.

When there are dirty blocks (blocks changed sitting in the buffercache waiting to be written by the database writer), there is different behaviour with respect to the buffered read or direct path read choice the instance makes. In general, with dirty blocks involved, compressed tables are less likely to be read into the buffercache.

It seems that with compressed tables and version and and up there is no switch to buffered reads even if all the blocks are in the cache, but dirty. The versions in between there, which are version, and do switch to buffered reads when 99% or more blocks are in the cache, regardless if they are dirty.

Also, please mind I explicitly mentioned tables. For a full scan on an index, indicated by the ‘FAST FULL INDEX SCAN’ line in an explain plan, entirely different rules are in play.

Tagged: buffercache, cache, IO, nsmtio, oracle, trace

NOUG Session: How Cache Fusion Works

For all those attended my session at Northeast Oracle User Group at Babson College in Wellesley, MA. Have you ever wondered how Cache Fusion knows where to get the block from? Or, how block locks vary from row locks? Or you are confused about the meaning and purpose of various Global Cache Service (GCS), Global Resource Directory (GRD) and Global Enqueue Service (GES). The session was meant to explain how all these actually work under the covers with live demos.

I hope you found the session useful and entertaining. As with all my sessions, I use slides as an aid to presentation; not to communicate the concepts. I have written an accompanying paper which explains these in detail. You can download the paper here. Or, the presentation here. All the scripts I used can be downloaded here.

As always, your feedback will be greatly appreciated.

Fedora 23 and Oracle 11gR2/12cR1

A few months ago I mentioned doing some Fedora 22 installations. At the time I did some pre-emptive installations on the Alpha release of Fedora 23 also.

Now the final release of Fedora 23 is out, I’ve run through the articles again to make sure things are all ship-shape.

It’s pretty much as it was before, with the nice bonus that the “MATE Desktop” package group has been fixed. Happy days! :)

As always, installations of Oracle server products on Fedora are not a great idea, as explained here.

If you do like playing with this stuff, knock yourself out… :)



Fedora 23 and Oracle 11gR2/12cR1 was first posted on November 9, 2015 at 7:32 pm.
©2012 "The ORACLE-BASE Blog". Use of this feed is for personal non-commercial use only. If you are not reading this article in your feed reader, then the site is guilty of copyright infringement.

The Eternal Newbie

It’s been over a decade since I first heard Tom Kyte talking about Project Marvel, which eventually became Application Express (APEX). Since then I’ve “used” just about every version of APEX. I use the term “used” very loosely, because I typically use APEX for a few days to get a job done, then never touch it again for months. By the time I come back, I pretty much have to start the learning process from the beginning again.

This is a perfect example of the “Eternal Newbie”. I could quite legitimately put 10+ years experience of APEX (including previous names) on my CV and it wouldn’t be a lie, but in reality I’m only about as good as any PL/SQL developer that’s been playing with APEX for a week.

It’s not that APEX is difficult, quite the contrary, but the process of getting good at anything takes time and repetition. You’ve probably heard the variations on “10,000 hours to become an expert” saying. It doesn’t matter that it’s not true or accurate. What matters is it highlights the need for time, repetition and constantly striving to improve. A little plodding once in a while does not count for experience in my book.

As an example of this, on Friday I was trying to get something to work with a Shuttle control in APEX 5.0.2. I found a couple of great examples on Dmitri‘s and Denes blogs, which (I thought) got me 90% of the way to what I wanted to achieve, but the last 10% took me half of Friday, then the whole of Saturday evening. If I was actually any good at this stuff it would have probably taken me 2 mins. What’s more, if I was good, I would have probably realised Dmitri and Denes’ examples actually got me to 99.9% of what I was trying to achieve, but my inexperience meant I kept shooting myself in the foot. After all that time playing and reading, I felt like a master of Shuttle controls in APEX, but if I don’t look at APEX for a week I’m going to be totally screwed. I have some knowledge now, but it will take repetition to make it stick, and based on past experience, that’s not going to happen. :)

I did a video a few months ago about the term “Senior” in relation to IT jobs. My recent fumblings with APEX made me think about this subject again.

With my 10+ years of APEX experience, I’m obviously a “Senior APEX Developer”. Just remind me, how do I alter a breadcrumb? :)



The Eternal Newbie was first posted on November 9, 2015 at 6:56 pm.
©2012 "The ORACLE-BASE Blog". Use of this feed is for personal non-commercial use only. If you are not reading this article in your feed reader, then the site is guilty of copyright infringement.

Second wave of cloud migration

#222222; text-align: right;">photo by Sunova Surfboards
#222222;"> A second wave of cloud migration is now happening after the initial hiccups of


Reaction to Shadow IT

  • #222222;">Misaligned Expectations
  • #222222;">Increased Data Exposure and Administrative Surface Area of Risk
  • #222222;">Choice of newer better cloud options now available after initial “wrong choice”
#222222;">Reaction to Shadow IT: A significant amount of public cloud use over the past few years was done without IT oversight and approval. As companies develop their formal cloud strategies, it often means moving away from the cloud provider that was initially chosen by a line of business or an individual group and moving to a different public/private cloud or even back onsite. 
#222222;">Misaligned Expectations: Many companies are realizing that they had not accurately estimated things like cost savings when they selected their public cloud environment. For example: while migrating data into a public cloud may be inexpensive, many companies have been surprised by the hidden cost of extracting data from these environments or moving data between regions.  Also, without constant monitoring and management of public cloud environments, many enterprise companies failed to efficiently decommission under-utilized workloads and therefore never realize any significant pay-as-you-go cost model savings.
#222222;">Another area of misaligned expectations is in technical support. Simply put, most large public cloud providers do not offer the level of technical support expected by the majority of enterprise companies. This issue becomes critical given that the typical enterprise company may only have a handful of internal employees that have a deep understanding of public cloud architectures. When you combine limited cloud expertise with the inability to get enterprise-level phone and tech support, we often see the “scaling or phase two” portion of cloud projects hit an impassable roadblock.
#222222;">Increased Data Exposure and Administrative Surface Area of Risk: While network-layer and physical security in public cloud data centers has proven to be adequate for most enterprise companies, many CISOs and security execs are very concerned about the risk of data exposure. This risk can be greatly increased in public clouds for two reasons:One, there are new administrative accounts that will have access to data and workloads. These accounts must now be managed and monitored by the IT security team and these accounts represent both a data exposure risk and a data protection risk (for example if an admin accidentally deletes a workload that hasn’t been backed up).

Two, the ability for multiple copies of sensitive data to be replicated in a cloud environment. This may be due to the public cloud provider’s underlying replication technology, or simply due to a company’s lack of data security controls in a cloud environment. For example, if a company had planned to implement business analytics in the cloud, it may require multiple copies of sensitive data to be sent to the cloud in order to complete the analysis reports. Even if this data is protected with encryption and identity management, a simple identity breach (the most common type of breach today) could expose that data in cleartext.


The result of this increased risk, especially given today’s threat climate, is causing many companies to abandon certain cloud projects unless more comprehensive data protection technologies (such as tokenization or data masking) can be implemented.


Choice: One of the simplest reasons for these statistics is the recent availability of choice. While it may be hard to believe, computing giants such as Google, Microsoft and VMware had no public cloud offering for enterprises just a couple years ago – and even companies like  IBM had only very limited private and hybrid cloud options. As enterprises develop a more mature cloud strategy, they are now able to select the best cloud environment for their individual cloud projects.



#222222;">Second wave of cloud migration



As these initial experiences and hiccups have been encountered, understood and dealt with, we are seeing a more serious and consolidated move to the cloud – a second wave of cloud migration.


Wrong Results

Here’s a little gem in 12c that arrived in my email a few days ago: a query where the result depends on the SQL*Plus arraysize!

The email had a short description, and a script to create a small data set that would demonstrate the problem. I’m not going to show you the query, or the result set, but here’s a sample of the output from an SQL*Plus session after creating the data. This is, by the way, on a “single-user” system – there is no way that some other session is changing the data – especially after the opening “set transaction”:

SQL> set transaction read only;

Transaction set.

SQL> set arraysize 1
SQL> select ...


541 rows selected.

SQL> set arraysize 4
SQL> select ...


599 rows selected.

SQL> set arraysize 10
SQL> select ...


620 rows selected.

SQL> set arraysize 32
SQL> select ...


616 rows selected.

The correct result set should have had the 616 rows reported when the arraysize was set to 32 (of, course, it’s possible with an arraysize of 32 the 616 rows returned weren’t the correct 616 rows – rows seemed to get multiplied or deleted fairly arbitrarily as the arraysize changed).

The execution plan was a little unusual in that it forced a nested loop join with a tablescan on the inner table; and when I ran the query with rowsource execution statistics enabled the number of starts of the inner tablescan was 1,597 but the number of rows actually returned varied. My first thought was that some new mechanical optimisation of the tablescan code was losing track of where it had got to in the repeated tablescans – but it turned out I was wrong.

Here’s the execution plan (with camouflage) – the key detail is in a section I didn’t look at intially, the column projection:

select * from table(dbms_xplan.display_cursor('0dh0kh9qa88mz',1,'-note +projection'));

| Id  | Operation           | Name | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT    |      |       |       | 14118 (100)|          |
|   1 |  NESTED LOOPS       |      |    29 |  2958 | 14118   (2)| 00:00:01 |
|*  2 |   HASH JOIN         |      |   892 | 57088 |    35   (3)| 00:00:01 |
|   3 |    TABLE ACCESS FULL| ABC  |   549 | 21411 |    17   (0)| 00:00:01 |
|   4 |    TABLE ACCESS FULL| DEF  |   892 | 22300 |    17   (0)| 00:00:01 |
|*  5 |   TABLE ACCESS FULL | XYZ  |     1 |    38 |    16   (0)| 00:00:01 |

Predicate Information (identified by operation id):
   2 - access("DEF"."ABC_FK"="ABC"."ABC_ID")
   5 - filter(("DEF"."COL0"="XYZ"."COL0" AND "XYZ"."COL1"="ABC"."COL1"
              AND "XYZ"."COL2"="ABC"."COL2"))

Column Projection Information (identified by operation id):
   1 - "ABC"."ABC_ID"[NUMBER,22], "DEF"."ABC_FK"[NUMBER,22],
       "ABC"."COL2"[NUMBER,22], "ABC"."COL1"[NUMBER,22],
       "DEF"."COL0"[VARCHAR2,20], "XYZ"."COL1"[NUMBER,22],
       "XYZ"."COL2"[NUMBER,22], "XYZ"."COL0"[VARCHAR2,20]
   2 - (#keys=1) "ABC"."ABC_ID"[NUMBER,22], "DEF"."ABC_FK"[NUMBER,22],
       "ABC"."COL2"[NUMBER,22], "ABC"."COL1"[NUMBER,22],
   3 - (rowset=200) "ABC"."ABC_ID"[NUMBER,22], "ABC"."COL1"[NUMBER,22],
   4 - (rowset=200) "DEF"."ABC_FK"[NUMBER,22], "DEF"."COL0"[VARCHAR2,20]
   5 - "XYZ"."COL1"[NUMBER,22], "XYZ"."COL2"[NUMBER,22],

The predicate section is irrelevant in this case, and I’ve camouflaged the names of the tables and columns – the only interesting bit is the appearance of the (rowset=200) in the projection information. This is reporting a feature new in 12c (and not to be confused with Oracle Java Rowsets) that should improve the performance of some queries.

I didn’t actually look at the projection information until after I’d asked the Oak Table members if they had ever seen this type of anomaly before – and Stefan Koehler emailed back to suggest that the problem might be related to rowsets (there are a couple of similar bugs on MoS, e.g: 17016479 and 20960570) – so I checked the projection, then repeated my tests after disabling the feature with a call to: ‘alter session set “_rowsets_enabled”=false;’

Problem solved – although I’ve told the person who emailed me to report this discovery and workaround to Oracle support and see what they supply as the approved solution.

It is possible to affect the feature through event 10055 – different levels disable it at different locations in the code; the list of options is given in the $ORACLE_HOME/rdbms/mesg/oraus.msg file (if you’re not running Windows):

//            Level:
//            0x00000001 - turn off for table scan
//            0x00000002 - turn off for hash join consume
//            0x00000004 - turn off for hash join produce
//            0x00000008 - turn off for group by
//            0x00000010 - turn off for sort
//            0x00000020 - turn off for table-queue out
//            0x00000040 - turn off for table-queue in
//            0x00000080 - turn off for identity
//            0x00000100 - turn off for granule iterator
//            0x00000200 - turn off for EVA functions
//            0x00000400 - turn off for PL/SQL
//            0x00000800 - turn off for upgrade
//            0x00001000 - turn off for database startup
//            0x00002000 - turn off for blobs and clobs
//            0x00004000 - turn off for tracing row source
//            0x00008000 - turn off rowset information in explain plan
//            0x00010000 - disable hash join rowsets fast path
//            0x00020000 - turn off for bloom create
//            0x00040000 - turn off for bloom use
//            0x00080000 - disable prefetch for hash join
//            0x00100000 - disable prefetch for bloom
//            0x00200000 - disable semi blocking hash join
//            0x00400000 - turn off rowset for fixed table

I tried the first few levels and found that both levels 1 and 2 eliminated the problem (and eliminated the appearance of the (rowset=200) entry in the projection information). Given the shape of the plan I had thought that just one of 1,2 or 4 might have been relevant so I was a little surprised to find that both 1 and 2 were effective – but that’s probably just a question of interpretation of the brief descriptions.


I’ve asked the owner of the problem if it’s okay to post the script to create the tables and data – and the answer was yes: the content was already heavily camouflaged anyway. So here’s a file you can download if you want to test other environments: insert_script_12c_bug

It’s declared as a “.doc” file to get past the upload process, but it’s really a flat text file.

Update 16th Nov

The official “minimum impact” workaround is to set event 10055 at level 2097152 (disable semi blocking hash join). Alternatively there is also a patch available. See Mike Dietriech’s blog for details:

Parsing URLs from plsql

Just the other day I needed to parse a URL, and could not find one that acted like I was used to in python. So tired of always missing out in plsql, I want to do something about it.
So yesterday I decided to create a package with different URL utilities that I need from time to time. The package is
called url_ninja. It can split an URL into the different components (scheme, authority, path, parameters, query, fragments)
and back again to a complete URL. It can parse an array into a query string and it can parse a query string into an array.

Oracle ACE program

What is the Oracle ACE program?

The Oracle ACE program is a community advocacy program for Oracle technology evangelists and enthusiasts, sponsored by and managed by Oracle Corporation.  As stated on the ACE overview page at “”, it is both a network as well as a resource for everyone in the Oracle community.  It is not a certification program, and there is no course to study and no test to pass.  Rather, the ACE program is a recognition program by one’s colleagues and peers, and joining is a process of nomination, review, and acceptance.

Who are Oracle ACEs?

They are your colleagues in the worldwide Oracle technology community, of which you and the RMOUG community here in Colorado are a significant part.  There are now more than 500 people in 55 countries around the world who have been recognized by their peers and by Oracle Corporation.  They are not employees of Oracle, but rather partners and customers of Oracle.

The ACE program is now 10 years old and comprises three levels…

ACE Associate logoACE Associate – this is the entry-point for the program and an ideal form of recognition for technology community advocates who are building their reputations

ACElogoACE– established advocates for Oracle technology who are well-known in their community

ACEDlogoACE Directors – top-tier members of the worldwide community who engage with many users groups and advocates and with Oracle Corporation

The ACE program is always growing, and the current members of the program are always happy to help you make the step up and jump to the next level in your career.  ACEs are expected to contribute to all or some of the following:

  • contribute actively to technical forums like OTN community forum or the ORACLE-L email list
  • publishing articles in newsletters, magazines, blogs
  • publishing books
  • tweet, post on social media
  • organize community activities such as Oracle users groups
  • present sessions at Oracle conferences and Oracle user group meetings

Few people do all these things, so don’t think that everyone does, but also please be aware that there are some who do, driving their careers above and beyond.

Joining the ACE program is not a one-time task, but an on-going commitment to contribution and sharing to build your community, whether that community is local to where you live, national, or world-wide.

To find out more about the program, go to the web page at “” or just contact one of the Oracle ACEs near you and ask questions.

That’s why we’re here.

Friday Philosophy – We Could Be Heroes! {just for one day}.

At Open World I overheard a snippet of conversation which went something like this:

Bob – “How’s it going? Did the last talk go down well?”
Bill – “Sure, it was on time, the audience seemed to like it.”
Bob – “Will you be here next year?”
Bill – “Errmm….” pause…*sigh*… “I don’t know…. I’ll see how I feel.”
Bob – “Oh? In what way?”
Bill – “It just that, at Open World… I have to fend off two dozen people just to go have a pee!”

Wild horses could not drag out of me the name of the person who said that (though several pints and the offer of a curry might do the trick – try me). It both made me smile and made me think. There are down-sides to becoming highly respected in your sphere.

There are definitely different levels of renown and respect in the relatively small world of the Oracle Database Technologist. I’m not doing bad in that respect; I’d put myself in the third of the seven circles, maybe tapping on gates of circle two. Occasionally I think it would be nice to be either technically or entertainingly good enough to join the Big Names in the innermost circle – but I really don’t think I can face the Hem-Touching!

What do I mean about “Hem-Touching”? It’s something a few friends and I came up with at the UKOUG Tech conference about 5 or 6 years ago to describe people who will approach one of the Oracle Names with a mixture of awe and fear in their eyes and just want them to acknowledge their presence,be allowed to speak, maybe to touch the hem of their cloak. If you go up to the balcony that is above the exhibition hall at the Birmingham ICC, you can sometimes watch an Oracle Name walk through the exhibition and see some people suddenly swerve and hurry towards them – especially if the Name currently has no one with them. I’ve even seen someone suddenly stop when another acolyte gets to their hero first. I don’t know why, these people will speak to more than one person at a time. And the thing is, people in the UK and Europe are generally more reserved than our cousins in other continents, so we are less forward in, well, being forward.

Am I being mean to these people? Well, a little I guess, but it’s mainly because of the little story I started with. I’m friends with some of the Names and I know a lot of them are uncomfortable with Hero Worship. Being respected and held in high regard is great, most of them are very happy about that, as they have worked damned hard and long to be knowledgeable enough to hold that position. But when people treat them like a living saint or the bestower of blessings, it’s just a bit weird. This is just an area of technology after all, not the eradication of Malaria. They are “just” people – OK, they are people who are usually very smart and very capable, but they are also people who are happy to share and teach – otherwise they would not be at the conferences sharing and teaching. Most of them are idiots in other areas of their lives too, we all are.

I’ve never felt the need to hero-worship myself. Not because I do not deeply respect people who achieve great things, it’s just not in my psychology I think. I did not put up any posters in my bedroom of the people I most respected when I was a teenager. I used to know a Nobel Prize Winner (though I doubt he’d recognise me in the street now) but when I met him the first time I had no idea who he was and just treated him like a person – and we got on fine. He treated me like a person too. I’ve been lucky enough to meet some very smart academics, many of the Oracle Names and even the odd traditionally famous person. It’s amazing how like people they are – if you treat them like people.

I’m certainly not above being pleased when someone I respect mentions me or refers to something I have done though. I’ll grin like an idiot on the rare occasions someone has name-checked me in a presentation or they tell me they liked something I said. I’m tickled pink when a Name follows me on twitter. But I feel hero worship is not what they want. Respect yes, being told you appreciate what they have taught you fine. Going shiny-eyed and asking to touch the hem of their coat, weird; don’t do it.

Oracle Names are people, treat them as such. They’ll probably appreciate you more if you do.

And if you ever find yourself in a group of several others, all trying to say “hello” to some gal or guy you just saw presenting, and they are looking a little uncomfortable and shifting from foot to foot and looking towards a door over there – let the poor sod go to the loo will you?

Say DevOps one more time …

Screen Shot 2015-11-04 at 4.02.56 PM

Credit: Matthias Weinberger (CC2)

What is Devops?

As has been noted many of times, DevOps is a term that is hard to define.

#666666;">DevOps is a term that is so easily misapplied in the enterprise that it’s popularity threatens adoption.

#666666;">If you can’t define it for the bean counters – it might as well not exist at all.

 #999999;">Tim O’Brien, Gleanster

Thus being able to clearly define DevOps is crucial to its success.

DevOps is a goal 

One of the problems in defining DevOps is that no two  DevOps teams are doing the same thing, so defining DevOps by what they do is currently impossible. On the other hand all DevOps teams have the same goal. That goal has been clearly articulated by Gene Kim as

#000000;">DevOps enable fast flow of features from development to IT operations to the production.  

One could expand a bit more and say

#000000;">DevOps enables Development, QA, IT Operations, Information Security and Product Management to work together to enable the fast flow of features from the business, to development, to QA, to IT to the customer, while preserving the stability, availability, security, and manageability of the production environment.

Screen Shot 2015-11-04 at 4.04.58 PM

How is the DevOps goal obtained ?

Everyone is in agreement as to the goal but the 6 million dollar question is how does one attain the goal? What tools and processes ?  There is a gnawing lack of discussion on what are the best tools and processes required to achieve the DevOps goal of fast flow of features. In order to increase flow in any system, the only productive action to take is to improve the constraint. The constraint is the slowest point in the flow. Any improvement not made at the constraint is an illusion as demonstrated by the works of Eli Goldratt. Thus the first step in DevOps is to identify the top constraint(s) and optimize them.

Gene Kim who has worked with 100s of CIOs and surveyed 14,000 companies listed the top 5 constraints in the fast flow of features as

    #555555; padding-left: 30px;">
  1. Provisioning environments for development
  2. Setting up test and QA environments
  3. Architecting code in development to facilitate easy changes in code
  4. Development speed of coding
  5. Product management  feature input

Screen Shot 2015-11-04 at 4.05.06 PM

The only way to optimizer the fast flow is to take the top constraint tune it, then move to the next constraint. The first 2 constraints that are most commonly seen are both centered around environment provisioning.  As Gene Kim puts it

One of the most powerful things that organizations can do is to enable development and testing to get environment they need when they need it. – Gene Kim

Thus the question for most companies is how to improve the provisioning of environments for development and QA.

Why is environment provisioning such a bottleneck?

Environment provisioning is such a bottleneck because historically environments take a long time and many resources to create and provision and developers depend on these environments to code the applications.  QA depend on these environments to test the code . IT  depend on these environments to verify deployment processes. When environment provisioning can be done quickly, repeatedly and efficiently then one can accomplish the core  of DevOps: Continuous Integration and Continuous Deployment.  Continuos Deployment is an automated process where developers code check-ins are integrated multiple times a day, run through QA and verified in IT for correct deployment. Some companies take it even farther with Continuos Delivery where the changes  that pass testing are rolled via automation into production.

Continuos Deployment enables the fast flow of features from development, to QA, to UAT, to IT  (and even to production in the case of Continuos Delivery).

What tools enable Continuos Deployment ?

Continuos deployment depends on automation and efficient environment provisioning to work, thus tools that automate code management, automate environment provisioning and improve efficiencies are key.

Minimum prerequisites for continuous integration

    • version control system like GIT
    • test automation like Jenkins
    • environment provisioning

Version Control

Version control can be implemented with any number of tools, of which one of the most popular  now is Git.

Test automation

Test automation can such as Jenkins and Team City can orchestrate running tests when changes are introduced in source control repositories.

But to move to continuous integration where code changes can be QA’ed  and verified for deployment multiple times day requires efficient provisioning of QA and verification environments.

Environment provisioning

To reach the goal CI/CD of multiple deployment tests at day requires not only integrating  code checkins in source control with automated test management tools but it also requires fast efficient access to environments to run those tests.

Machine virtualization

One of the first decisive technologies that came along to support efficient environment provisioning was machine virtualization such as VMware. With machine virtualization, virtual machines (VM), could be spun up or down quickly and easily thus providing the efficiency gains needed for DevOps and opening up machine provisioning to automation now that it was controlled by software.

Configuration tools automate VM provisioning

#555555;">Configuration tools like Chef, Puppet and Ansible are programmed to automate and manage the provisioning of VMs with the desired configurations, packages and software.

Continuous Integration (CI/CD)  Landscape

Thus to implement CI/CD requires

  1. version control system like GIT
  2. test automation like Jenkins and Team City
  3. efficient environment provisioning
    • Machine virtualization (VM) : VMware, KVM, Docker etc
    • VM provisioning tools : Chef, Puppet, Ansible

Screen Shot 2015-11-04 at 4.01.39 PM


Biggest Obstacle to  CI/CD 

Now that code changes are automatically QA’ed and VMs can be spun up in the right configuration, there is still the problem of “how do we get data to run QA tests on?” For example what if the application depends on data managed by a large Oracle database? How can one efficiently produce copies of this database for use in the flow from development to testing to deployment ?

#555555;">Fortunately there is a technology called data virtualization. As virtual machine technology opened the door to  continuos integration, data virtualization swings it wide open for enterprise level application development that depend on large databases.

#555555;">Data virtualization is an architecture (that can be encapsulated in software as Delphix has done) which connects to source data or database, takes an initial copy and then and forever collects only the changes from the source (like EMC SRDF, Netapp SMO, Oracle Standby database). The data is saved on storage that has either snapshot capabilities (as in Netapp & ZFS or software like Delphix that maps a snapshot filesystem onto any storage even JBODs). The data is managed as a timeline on the snapshot storage. For example Delphix saves by default 30 days of changes. Changes older than the 30 days are purged out, meaning that a copy can be made down to the second anywhere within this 30 day time window.

Thus to implement CI/CD requires efficient environment provisioning. Efficient environment provisioning depends upon tools and technology such as:

  1. Configuration Management: Chef, Puppet, Ansible
  2. Machine virtualization  : VMware, KVM, Docker etc
  3. Data virtualization :  Delphix

Once environment provisioning is streamlined then the next step is automating the flow of developer commits through QA and deployment testing with test automation tools like  like Jenkins and Team City.

There are examples coming out to document how to wire these tools togethers such as a recent one on Jenkins and Ansible.

Efficient environment provisioning and test automation are just the core.  From there we can build out more of the processes to attain the goals of DevOps.


The ambiguous descriptions of DevOps are undermining the movement, but DevOps is defined as the tools and culture that support a continuous delivery value chain. Now with a stake in the ground to rally around, the goal is no longer saying what is or what is not DevOps but instead becomes mapping out the best tools, processes, methods and cultural changes  to support the DevOps definition and mapping these to the various situations encountered in the industry. Everyone is doing DevOps, i.e. software development, integration, QA and release, the questions is how well are you doing it and can you improve it?

#555555;">Virtual data improves businesses’ bottom line by eliminating the enormous infrastructure, bureaucracy and time drag that it takes to provision databases and data for business intelligence groups and development environments. Development environments and business intelligence groups depend on having a copies of production data and databases and data virtualization allows provisioning in a few minutes with almost no storage overhead by sharing duplicate blocks among all the copies.


Screen Shot 2015-11-04 at 4.00.54 PM


Here is an episode from Arrested DevOps talking about the problem with data and databases in DevOps