Search

Top 60 Oracle Blogs

Recent comments

Oakies Blog Aggregator

“Flash” Storage Will Be Cheap – The End of the World is Nigh

A couple of weeks ago I tweeted a projection that the $/GB for flash drives will meet the $/GB for hard drives within 3-4 years. It was more of a feeling based upon current pricing with Moore’s Law applied than a well researched statement, but it felt about right. I’ve since been thinking some more about this within the context of current storage industry offerings from the likes of EMC, Netapp and Oracle, wondering what this might mean.

First of all I did a bit of research – if my 3-4 years guess-timate was out by an order of magnitude then there is not much point in writing this article (yet). I wanted to find out what the actual trends in flash memory pricing look like and how these might project over time, and I came across the following article: Enterprise Flash Drive Cost and Technology Projections. Though this article is now over a year old, it shows the following chart which illustrates the effect of the observed 60% year on year decline in flash memory pricing:

Flash Drive Pricing Projectionshttps://jamesmorle.files.wordpress.com/2010/05/flashdriveprojections2009... 600w, https://jamesmorle.files.wordpress.com/2010/05/flashdriveprojections2009... 150w" sizes="(max-width: 300px) 100vw, 300px" />

This 60% annual drop in costs is actually an accelerated version of Moore’s Law, and does not take into account any radical technology advances that may happen within the period.  This drop in costs is probably driven in the most part by the consumer thirst for flash technology in iPods and so forth, but naturally ripples back up into the enterprise space in the same way that Intel and AMD’s processor technologies do.

So let’s just assume that my guess-timate and the above chart are correct (they almost precisely agree) – what does that mean for storage products moving forward?

Looking at recent applications of flash technology, we see that EMC were the first off the blocks by offering the option of relatively large flash drives as drop-in replacements for their hard drives in their storage arrays. Netapp took a different approach of putting the flash memory in front of the drives as another caching layer in the stack. Oracle have various options in their (formerly Sun) product line and a formalised mechanism for using flash technology built into the 11g database software and into the Exadata v2 storage platform. Various vendors offer internal flash drives that look like hard drives to the operating system (whether connected by traditional storage interconnects such as SATA or by PCI Express). If  we assume that the cost of flash technology becomes equivalent to hard drive storage in the next three years, I believe all these technologies will quickly become the wrong way to deploy flash technology, and only one (Oracle) has an architecture which lends itself to the most appropriate future model (IMHO).

Let’s flip back to reality and look at how storage is used and where flash technology disrupts that when it is cheap enough to do so.

First, location of data: local or networked in some way? I don’t believe that flash technology disrupts this decision at all. Data will still need to be local in certain cases and networked via some high-speed technology in others, in much the same way as it is today. I believe that the networking technology will need to change for flash technology, but more on that later.

Next, the memory hierarchy: Where does current storage sit in the memory hierarchy? Well, of course, it is at the bottom of the pile, just above tape and other backup technologies if you include those. This is the crucial area where flash technology disrupts all current thinking – the final resting place for data is now close or equal to DRAM memory speeds. One disruptive implication of this is that storage interconnects (such as Fibre Channel, Ethernet, SAS and SATA) are now a latency and bandwidth bottleneck. The other, potentially huge, disruption is what happens to the software architecture when this bottleneck is removed.

Next, capacity: How does the flash capacity sit with hard drive capacity? Well that’s kind of the point of this posting… it’s currently quite a way behind, but my prediction is that they will be equal by 2013/2014. Importantly though, they will then start to accelerate away from hard drives. Given the exponential growth of data volumes, perhaps only semiconductor based storage can keep up with the demand?

Next, IOPs: This is the hugely disruptive part of flash technology, and is a direct result of a dramatically lowered latency (access time) when compared to hard disk technology. Not only is the latency lowered, but semiconductor-based storage is more or less contention-free given the absence of serialised moving parts such as a disk head. Think about it – the service time for a given hard drive I/O is directly by the preceding I/O and where the head was left on the platter. With solid-state storage this does not occur and service times are more uniform (though writes are consistently slower than reads).

These disruptions mean that the current architectures of storage systems are not making the most of semiconductor-based storage. Hey, why do I keep calling it “semiconductor-based storage” instead of SSD or flash? The reason is that the technologies used in this area are changing frequently, from DRAM-based systems to NOR-based flash to NAND based flash to DRAM-fronted flash; Single-level cells to Multi-level cells; battery-backed to “Super Cap” backed. Flash, as we know it today, could be outdated as a technology in the near future, but “semiconductor-based” storage is the future regardless.

I think that we now need technologies that look more like Oracle Exadata v2, with low-latency RDMA interfaces directly into the Operating System/Database. However, they need to easily and natively support other types of storage (unstructured data such as files, VMware datastores and so forth). The Exadata architecture lends itself well to changes in this area in both hardware trends and access protocols.

Perhaps more importantly, we are also only just beginning to understand the implications in software architecture for the disrupted memory hierarchy. We simply cannot continue to treat semiconductor-based storage as “fast disk” and need to start thinking, literally, outside the box.

Oracle OTN Developer Day VirtualBox Appliance

I have forgotten and searched for this cool VirtualBox Appliance, now so often that I found it time to make a small notepad/placeholder for this.

For those who haven't found it yet, there is a OTN Developer VirtualBox appliance out there that is a great start-up environment to use as a test, presentation, try-out or development environment. Probably officially used for Oracle demo's and or Hands on Exercises (HOL) but also very usable on a laptop.

Its a VirtualBox environment with Hands on Lab (HOL) exercises and the following software environment/software:

  • Oracle Enterprise Linux 5
  • Oracle Database 11g Release 2 Enterprise Edition
  • Oracle TimesTen In-Memory Database Cache
  • Oracle XML DB
  • Oracle SQL Developer
  • Oracle SQL Developer Data Modeler
  • Oracle Application Express
  • Oracle JDeveloper
  • Hands-On-Labs (accessed via the Toolbar Menu in Firefox)

Be aware that, as it is described as well on the download instruction page, that this appliance is for testing purposes only, as such it is unsupported and should not to be used in production environment. I really hope they make more of those cool VirtualBox example appliances.

Click here for the Pre-Workshop Setup Instructions - Hands-On Labs

 

Oracle OTN Developer Day VirtualBox Appliance

Oracle memory troubleshooting article

Randolf Geist has written a good article about systematic troubleshooting of a PL/SQL memory allocation & CPU utilization problem – and he has used some of my tools too!

http://oracle-randolf.blogspot.com/2010/05/advanced-oracle-troubleshooting-session.html

Share/Bookmark

Oracle memory troubleshooting article

Randolf Geist has written a good article about systematic troubleshooting of a PL/SQL memory allocation & CPU utilization problem – and he has used some of my tools too!
http://oracle-randolf.blogspot.com/2010/05/advanced-oracle-troubleshooti...

Oracle memory troubleshooting article

Randolf Geist has written a good article about systematic troubleshooting of a PL/SQL memory allocation & CPU utilization problem – and he has used some of my tools too!
http://oracle-randolf.blogspot.com/2010/05/advanced-oracle-troubleshooti...

SQL Server 3

I haven’t had much spare time for writing this week – although I’ve got about 45 drafts of notes to publish – but Simpletalk have just published the “learning SQL Server” paper I sent them a couple of weeks ago, and it’s worth a link. Filed under: SQL Server

Extended statistics and function-based indexes

About a year ago I’ve discovered nice feature of Oracle 10gR2 CBO: to overcome an issue with calculated selectivity for predicates on multiple columns, it can use DISTINCT_KEYS of the available index. Couple of weeks ago the bug fix mentioned in the OTN thread actually helped to solve a performance issue of a query. And [...]

ACT Oracle User Group Seminar Session: 2 June 2010

Just a short note for anyone near the Canberra region that the next ACT Oracle User Group Seminar Session will be held next Wednesday, 2 June 2010 at the Oracle Offices in Turner. I’ll be presenting “Indexing New Features and Improvements Introduced in Oracle 11g Release 1 & 2″. Follow the above link for the full agenda. [...]

Visual SQL Tuning Webinar with Jonathan Lewis June 10, 2010

Register for the live webinar on June 10th here:

Tune in for the Ultimate SQL Tune-off

According to Jonathan Lewis, there is a basic strategy to writing efficient SQL, but it requires knowledge of the data, the metadata, and the intent of the query.

Jonathan will start with the basic principles of efficient data access and then show different ways of analyzing the data. He'll then describe a graphical approach to finding the most efficient access path for a query and then apply the method to a problem he recently saw with one of his clients.

But wait... is there more than one way to skin a cat? Kyle Hailey thinks so, and he's not afraid to let Jonathan know it!

Kyle's fancy footwork will include his own approach to Visual SQL Tuning (VST) with some ring side support from DB Optimizer XE. Hitting below the belt? You be the judge.

Live! June 10, 2010 at 11AM Pacific

Visual Tuning

I’ll be doing a joint event with Kyke Hailey of Embarcadero in a few days time. It’s going to be a little unusual – a simultaneous webcast from opposite sides of the Atlantic – mixing my presentation about using a graphic method of tuning SQL with Kyle taking on the role of an active audience [...]