An interesting problem with ext4 on Oracle Linux 5.5

I have run into an interesting problem with my Red Hat 5.5 installation. Naively I assumed that ext4 has been around for a long time it would be stable. For a test I performed for a friend, I created my database files on a file system formatted with ext4 and mounted it the same way I would have mounted an ext3 file system:

$ mount | grep ext4
/dev/mapper/mpath43p1 on /u02/oradata type ext4 (rw)

Now when I tried to create a data file within a tablespace of a certain size, I got block corruption which I found very interesting. My first thought was: you must have a corruption of the file system. So I shut down all processes accessing /u02/oradata and gave the file system a thorough checking.

# umount /u02/oradata
#
# fsck.ext4 -cfv /dev/mapper/mpath43p1
e4fsck 1.41.9 (22-Aug-2009)
Checking for bad blocks (read-only test): done
/dev/mapper/mpath43p1: Updating bad block inode.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/mapper/mpath43p1: ***** FILE SYSTEM WAS MODIFIED *****

42 inodes used (0.00%)
14 non-contiguous files (33.3%)
0 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 18/14
3679655 blocks used (61.82%)
0 bad blocks
5 large files

26 regular files
7 directories
0 character device files
0 block device files
0 fifos
0 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
33 files
#

Have a look at the command line options and make sure you understand them before letting fsck loose on your file systems! By the way if Linux tells you the file system cannot be unmounted, use “fuser -m /u02/oradata” to list all PIDs accessing the mount point.

As you can see from the output the file system was fine-which struck me as odd. I mounted it again and started my database, surely that was only a glitch.

After repeating my test, I got the same block corruption. I now had a suspicion that ext4 might be the problem. Instead of creating a tablespace with 1 8GB data file, I used 4 2 GB data files and the problem went away.

After some experimentation I found out that the magical boundary is somewhere > 3G for single data files. Here is the proof:

SQL> create tablespace WillIBeUnusable datafile '/u02/oradata/orcl/corrupt.dbf' size 3G;

Tablespace created.

SQL> !dbv file=/u02/oradata/orcl/corrupt.dbf

DBVERIFY: Release 11.2.0.2.0 - Production on Thu Nov 3 10:32:02 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata/orcl/corrupt.dbf

DBVERIFY - Verification complete

Total Pages Examined         : 393216
Total Pages Processed (Data) : 0
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 127
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 393089
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3371559 (0.3371559)
$

However a 5G data file reports these:

$ dbv file=/oradata/orcl/corrupt.dbf

DBVERIFY: Release 11.2.0.2.0 - Production on Thu Nov 3 10:33:49 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata/orcl/corrupt.dbf

...

Page 655357 is marked corrupt
Corrupt block relative dba: 0x0289fffd (file 10, block 655357)
Completely zero block found during dbv:

Page 655358 is marked corrupt
Corrupt block relative dba: 0x0289fffe (file 10, block 655358)
Completely zero block found during dbv:

Page 655359 is marked corrupt
Corrupt block relative dba: 0x0289ffff (file 10, block 655359)
Completely zero block found during dbv:

Page 655360 is marked corrupt
Corrupt block relative dba: 0x028a0000 (file 10, block 655360)
Completely zero block found during dbv:

DBVERIFY - Verification complete

Total Pages Examined         : 655360
Total Pages Processed (Data) : 0
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 127
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 524160
Total Pages Marked Corrupt   : 131073
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3371948 (0.3371948)
$

That’s not good. A search on supporthtml.oracle.com revealed that ext4 is supported only in 5.6 and later. At that point I stopped bothering with it since even if I worked out where the problem was, I would still be running in an unsupported configuration. The main reason I prefer not to be in “unsupported” terrain is that I don’t like Oracle support to get away by simply dismissing the service request.

The Alternative: XFS

However, as Greg Rahn, Kevin Closson and others will tell you, you shouldn’t use ext4 for databases anyway: use XFS! The latter has been substantially improved in Oracle/Red Hat Linux 6 and should be your file system of choice.

Almost forgot the hard facts: