Subquery Factoring

It’s always worth browsing through the list of Oracle’s bug fixes each time a new release or patch comes out because it can give you clues about where to look for problems in your production release – and how to anticipate problems on the upgrade. This article is an example of a fix that I found while while looking at the note for 11.2.0.3 quite recently.

Over the last thre or four years I’ve made several commentsabout how subquery factoring could result in changes in execution plans even if the “factored subquery” (or “common table expression” to use the more appropriate technical term) was moved in line. In the 11.2.0.3 bug fixes list, though, you’ll find the following bug fixes:

Bug 11740670 Different execution plans with and without subquery factoring (WITH clause)
Bug 9002661 Transitive predicate were not generated in WITH clause.

So here’s an example that’s been sitting on my laptop for a few years, waiting for the right bug fix (it’s using my standard setup of 8KB block size, LMT with 1MB uniform extents, but not ASSM, and CPU costing disabled):

create table emp (
	dept_no		not null,
	sal,
	emp_no,
	padding,
	constraint e_pk primary key(emp_no)
)
as
select
	mod(rownum,6),
	rownum,
	rownum,
	rpad('x',60)
from
	all_objects
where
	rownum <= 20000
;

-- collect stats, compute, no histograms

select
	outer.*
from
	emp outer
where
	sal > 1000000
and	outer.sal > (
		select
			avg(inner.sal)
 		from	emp inner
		where	inner.dept_no = outer.dept_no
	)
;

with subq as (
	select
		outer.*
	from
		emp outer
	where
		sal > 1000000
	and	outer.sal > (
			select
				avg(inner.sal)
 			from	emp inner
			where	inner.dept_no = outer.dept_no
		)
	)
select	*
from	subq
;

In 11.1.0.7 (and earlier) the first version of this query uses a simple filter subquery, and the second version inlines the factored subquery (which should, therefore, produce the same plan) but then unnests the subquery and uses “group by placement” to resolve the plan as a hash join followed by hash aggregation. This gives us two completely different plans:


Plan for first form of the query
------------------------------------------------------------
| Id  | Operation           | Name | Rows  | Bytes | Cost  |
------------------------------------------------------------
|   0 | SELECT STATEMENT    |      |     1 |    72 |    70 |
|*  1 |  FILTER             |      |       |       |       |
|*  2 |   TABLE ACCESS FULL | EMP  |     1 |    72 |    35 |
|   3 |   SORT AGGREGATE    |      |     1 |     8 |       |
|*  4 |    TABLE ACCESS FULL| EMP  |  3333 | 26664 |    35 |
------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("OUTER"."SAL"> (SELECT AVG("INNER"."SAL") FROM "EMP"
              "INNER" WHERE "INNER"."DEPT_NO"=:B1))
   2 - filter("SAL">1000000)
   4 - filter("INNER"."DEPT_NO"=:B1)

Plan with factored subquery moved in line
-------------------------------------------------------------
| Id  | Operation            | Name | Rows  | Bytes | Cost  |
-------------------------------------------------------------
|   0 | SELECT STATEMENT     |      |     1 |    83 |    93 |
|*  1 |  FILTER              |      |       |       |       |
|   2 |   HASH GROUP BY      |      |     1 |    83 |    93 |
|*  3 |    HASH JOIN         |      |  1667 |   135K|    71 |
|*  4 |     TABLE ACCESS FULL| EMP  |     1 |    72 |    35 |
|   5 |     TABLE ACCESS FULL| EMP  | 20000 |   214K|    35 |
-------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("OUTER"."SAL">AVG("INNER"."SAL"))
   3 - access("INNER"."DEPT_NO"="OUTER"."DEPT_NO")
   4 - filter("SAL">1000000)

Under 11.2.0.3, however, the first plan appears for both forms of the query.

On the plus side – this is good, because Oracle is now behaving consistently. On the minus side – this is bad, because someone, somewhere, is going to be the unlucky person who managed to get a good execution plan by accident because of the bugs.

Maxim: when upgrading to 11.2.0.3, keep a close eye on any queries you’ve got that include subquery factoring – in fact, you might decide to capture SQL Baselines (without activating them) for every single one of them before doing the upgrade so that you can enforce the pre-upgrade plan if the post-upgrade plan is less efficient.

Footnote: I haven’t finished with this example – there’s an interesting follow-up that appeared in the remainder of my test code when I hinted the subquery into materialization. But that’s a story for another day.


Installing Oracle VM Manager 3.0.3 under Dom0 host, or How to save resources on your sandbox

If you are wondering why anyone would possibly need to Install Oracle VM Manager in such an unsupported way, please read my previous Oracle VM 3 related blog post. In this post, I will share with you my experience installing OVMM 3.0.3 under OVM 3.0.3 Dom0 host, including: A short and sweet action plan to [...]

right said Fred

I think it is entirely right and proper that Sir Fred Goodwin has been stripped of his knighthood.

After all, I am a shareholder in the Royal Bank of Scotland and his incompetence has cost me a significant amount of money and pushed back my retirement age to 87.

In fact, why stop there ? Personally, I find it objectionable that Freddie's surname includes the word 'Good'. This man is not good - in fact, he is a very bad man so I suggest he should henceforth be known as 'Fred Win'.

Oh no. Wait hang on - the word 'win' also has positive connotations associated with victory and happiness (see also Charlie Sheen and 'Winning'). This will never do so I insist the word 'Win' is also stripped from his legal name.

So now, the former head of RBS should simply be known as 'Fred'.

Investigations continue into whether Fred has a middle name.

Understanding Storage Masterclass – Dates Available

We’ve just booked the first European venue for the Understanding Storage Masterclass. I will be presenting the Masterclass on April 24/25 2012 at Prospero House in London, tickets are available HERE. I’m pretty excited to host this training session in my home country, and I hope to see you there!

Hotsos Symposium 2012 Speaker Spotlight – Doug Burns

Hotsos is very pleased to announce that Doug Burns is returning for Hotsos Symposium 2012! He has graciously accepted the spots formerly held by Randolf Geist, who had to cancel. While we will miss meeting Mr. Geist, we are very happy that Doug agreed to take his slots.

Doug Burns is an independent contractor who has 20 years of experience working with Oracle in a range of industries and applications. As well as presenting at a number of conferences, he has developed, taught, and edited performance-related courses for both Oracle UK and Learning Tree International. He is a member of The Oak Table Network and an Oracle Ace Director.

Doug's Topics: Statistics on Partitioned Objects 2012 and Falling in Love All Over Again ... OEM 12c Performance Page Enhancements

Descriptions of Sessions: Statistics on Partitioned Objects 2012 – Optimal SQL execution plans are rooted in the quality of the statistics describing the underlying objects that are used by the Cost Based Optimizer.

Statistics maintenance is a particular challenge on Data Warehouse systems holding large data volumes that support complex and flexible user reporting requirements. Such systems are likely to use partitioned tables and indexes to complicate the picture further.

Continuing the theme of last years presentation, the 2012 version will look at additional lessons learned during the past year and focuses on 11g features.

Review of Core Issues

  • The quality/performance trade off
    • Global and Approximate Global statistics METHOD_OPT and Column Statistics
  • Alternative Strategies
    • Copying statistics
    • Setting statistics
  • 11g improvements
    • AUTO_SAMPLE_SIZE
    • Incremental Statistics

The presentation will discuss a variety of problems and bugs with the various approaches that should be avoided as well as offering guidelines to help you design the best statistics collection strategy for your requirements.

Falling in Love All Over Again ... OEM 12c Performance Page Enhancements – Oracle 10g introduced significant instrumentation enhancements and the OEM 10g Performance Pages illustrated them to make performance analysis much easier. OEM 12c is the latest evolutionary stage of the Performance Pages and this presentation shows the new features that make them even more compelling and useful in more situations.


There are a few more seats available for Hotsos Symposium 2012, but you'll need to register soon! We don't want you to miss out on all these fabulous speakers. ;)

Bug fixes

From MOS (Metalink) a search for “Patch Set – List of Bug Fixes by Problem” is a useful search, andother is “Availability and Known Issues”. Whenever you find some behaviour that looks like a bug, it’s worth checking the patch sets for the patches or release that are newer than the version that you’re running – you may find that your problem is a known bug with a patch that might be back-ported.

For ease of reference, here are some of the results I got (sorted in reverse order of version) from the searches; you will need a MOS account to follow the links:


This system is for the use of authorized users only.

How to bypass the login banners?

There is actually more than one banner to bypass. One of the them is the message of the day banner, commonly located in /etc/motd. Typically friendly, example in AIX

****************************************************************************
*                                                                          *
*                                                                          *
*  Welcome to AIX Version 6.1!                                             *
*                                                                          *
*                                                                          *
*  Please see the README file in /usr/lpp/bos for information pertinent to *
*  this release of the AIX Operating System.                               *
*                                                                          *
*                                                                          *
***************************************************************************

This is easy to bypass, simply place .hushlogin file on your serverside homedirectory :


$ touch $HOME/.hushlogin
$

Yes, it is that easy.

A bit more cumbersome is the ssh banner. Which rather have an aggressive look with criminal punishment threats.


|-----------------------------------------------------------------|
| This system is for the use of authorized users only.            |
| Individuals using this computer system without authority, or in |
| excess of their authority, are subject to having all of their   |
| activities on this system monitored and recorded by system      |
| personnel.                                                      |
|                                                                 |
| In the course of monitoring individuals improperly using this   |
| system, or in the course of system maintenance, the activities  |
| of authorized users may also be monitored.                      |
|                                                                 |
| Anyone using this system expressly consents to such monitoring  |
| and is advised that if such monitoring reveals possible         |
| evidence of criminal activity, system personnel may provide the |
| evidence of such monitoring to law enforcement officials.       |
|-----------------------------------------------------------------|

Well, you could delete it from your ssh server but this would not please your sysadmins. The banner is actually a pre-authentication banner, so you must configure it on the client. With the unix ssh client, you lower the log level. Either with a command line option, -o LogLevel=quiet, or in a configfile

$ cat $HOME/.ssh/config
LogLevel=Quiet
StrictHostKeyChecking=no
NumberOfPasswordPrompts=1

This is so quiet that you will not get any feedback if you cannot connect, but I prefer quiet than noisy.

A very commonly used Windows ssh client is putty, and there, O miracle, there is a pre-authentication-banner option to uncheck in SSH-Auth.

This appeared in putty 0.62 and it made my day today :-)

Developer’s Toolbox

I had the privilege to be a guest blogger on the ODTUG blog. I wrote about Kscope and my role in putting together the Developer’s Toolbox track as well as why you should attend Kscope. Here is a short excerpt:

This year’s Kscope is a special one to me. I volunteered to lead the Developer’s Toolbox track team. My task was to build a team of volunteers from the Oracle community and then together determine appropriate sub-tracks, review and rate the submitted abstracts and create a draft schedule for the track.

Read the full blog post here.

Related articles:

VArrays 101

From An Expert’s Guide to Oracle Technology

 

Continuing on in my 101 series - I wrote about associative arrays, nested tables and An Expert's Guide to Oracle Technology § Comments Off °

Index Hash

You might think from the title that this little note is going to be about the index hash join – you would be half right, it’s also about how the optimizer seems to make a complete hash of dealing with index hash joins.

Let’s set up a simple data set and a couple of indexes so that we can take a closer look:

create table t1
as
with generator as (
	select	--+ materialize
		rownum id
	from dual
	connect by
		level <= 10000
)
select
	rownum			id,
	mod(rownum-1,20)	flag,
	lpad(rownum,10,'0')	v10,
	lpad(rownum,20,'0')	v20,
	lpad(rownum,30,'0')	v30,
	rpad('x',100)		padding
from
	generator	v1,
	generator	v2
where
	rownum <= 100000
;

-- gather stats, compute, no histograms

alter table t1 add constraint t1_pk primary key(id)
	using index (create index t1_pk on t1(id))
;

create index t1_flag on t1(flag);

create index t1_ia on t1(id, v20);
create index t1_ib on t1(id, v10);
create index t1_ic on t1(id, v30);

The indexing is a little unusual, but does represent the sort of thing I see from time to time. In particular, the “primary key plus one more column” can be helpful to make critical queries visit only the index and not visit the table; having three such indexes is a bit over the top. Look carefully and you’ll notice that the ia, ib, ic indexes look a little out of order compared to the v20, v10, v30 columns they are built on.

Now try running the following SQL with autotrace enabled:

select /*+ index_ffs(t1 t1_pk) */ count(*) from t1;
select /*+ index_ffs(t1 t1_ia) */ count(*) from t1;
select /*+ index_ffs(t1 t1_ib) */ count(*) from t1;
select /*+ index_ffs(t1 t1_ic) */ count(*) from t1;
select /*+ index_ffs(t1 t1_flag) */ count(*) from t1 where flag is not null;

These are the results I got from 11.2.0.3 (and this test reproduces under 10.2.0.3 and 11.1.0.7). I’ve shown the full plan for the first query, but only the critical operation with its cost for the rest of them:

-------------------------------------------------------
| Id  | Operation             | Name  | Rows  | Cost  |
-------------------------------------------------------
|   0 | SELECT STATEMENT      |       |     1 |    35 |
|   1 |  SORT AGGREGATE       |       |     1 |       |
|   2 |   INDEX FAST FULL SCAN| T1_PK |   100K|    35 |
-------------------------------------------------------

|   2 |   INDEX FAST FULL SCAN| T1_IA |   100K|    80 |
|   2 |   INDEX FAST FULL SCAN| T1_IB |   100K|    59 |
|   2 |   INDEX FAST FULL SCAN| T1_IC |   100K|   101 |

|*  2 |   INDEX FAST FULL SCAN| T1_FLAG |   100K|   292K|    31 |

As you might expect, the longer the “extra” column, the higher the cost. It’s an interesting little detail that the queries that didn’t need to look at a data column didn’t report a “bytes” column in the execution plan – just one clue that there’s a special optimisation for the generic “count everything” query.

Finally, we can get to the index hash join.

select	sum(id)
from
	t1
where
	flag = 0;
;

Clearly, this query could do a full tablescan, but it could do a hash join between the t1_flag index and one of the other indexes. Given that the primary key index has the lowest fast full scan cost at 35, and the index fast full scan on the t1_flag index is 31, we might hope to see an index hash join with a cost of about 66. Here’s the default plan:

-----------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost  |
-----------------------------------------------------------
|   0 | SELECT STATEMENT   |      |     1 |     8 |   380 |
|   1 |  SORT AGGREGATE    |      |     1 |     8 |       |
|*  2 |   TABLE ACCESS FULL| T1   |  5000 | 40000 |   380 |
-----------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter("FLAG"=0)

It’s a full tablescan, with a cost of 380. So let’s hint an index hash join with the two indexes we hoped to see, and find out what happens. I added the hint /*+ index_join(t1) */ and got the following execution plan:

----------------------------------------------------------------------------
| Id  | Operation               | Name             | Rows  | Bytes | Cost  |
----------------------------------------------------------------------------
|   0 | SELECT STATEMENT        |                  |     1 |     8 |   533 |
|   1 |  SORT AGGREGATE         |                  |     1 |     8 |       |
|*  2 |   VIEW                  | index$_join$_001 |  5000 | 40000 |   533 |
|*  3 |    HASH JOIN            |                  |       |       |       |
|*  4 |     INDEX RANGE SCAN    | T1_FLAG          |  5000 | 40000 |    10 |
|   5 |     INDEX FAST FULL SCAN| T1_IA            |  5000 | 40000 |   645 |
----------------------------------------------------------------------------

Note three anomalies:
a) Oracle has obeyed the index_join directive, but it’s used the “wrong” index
b) the cost of the query (533) is less than the cost of one of its operations (645 at line 5)
c) the cost of an index fast full scan on t1_ia has jumped from 80 (previous test) to 645

So let’s add a hint to get the right index used:

select
	/*+
		qb_name(main)
		index_join(@main t1 t1_flag t1_pk)
	*/
	sum(id)
from
	t1
where
	flag = 0
;

This makes no difference, Oracle still uses index t1_ia instead of t1_pk. It’s only when I include an undocumented hint that Oracle does what I want:

explain plan for
select
	/*+
		qb_name(main)
		outline_leaf(@main)
		index_join(@main t1 t1_flag t1_pk)
	*/
	sum(id)
from
	t1
where
	flag = 0
;

select * from table(dbms_xplan.display(null,null,'outline'));

----------------------------------------------------------------------------
| Id  | Operation               | Name             | Rows  | Bytes | Cost  |
----------------------------------------------------------------------------
|   0 | SELECT STATEMENT        |                  |     1 |     8 |   240 |
|   1 |  SORT AGGREGATE         |                  |     1 |     8 |       |
|*  2 |   VIEW                  | index$_join$_001 |  5000 | 40000 |   240 |
|*  3 |    HASH JOIN            |                  |       |       |       |
|*  4 |     INDEX RANGE SCAN    | T1_FLAG          |  5000 | 40000 |    10 |
|   5 |     INDEX FAST FULL SCAN| T1_PK            |  5000 | 40000 |   279 |
----------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      INDEX_JOIN(@"MAIN" "T1"@"MAIN" ("T1"."FLAG") ("T1"."ID"))
      OUTLINE(@"MAIN")
      OUTLINE_LEAF(@"MAIN")
      OUTLINE_LEAF(@"SEL$998059AF")
      ALL_ROWS
      OPT_PARAM('_optimizer_cost_model' 'io')
      DB_VERSION('11.2.0.3')
      OPTIMIZER_FEATURES_ENABLE('11.2.0.3')
      IGNORE_OPTIM_EMBEDDED_HINTS
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter("FLAG"=0)
   3 - access(ROWID=ROWID)
   4 - access("FLAG"=0)

You might note that with the right index in play
a) The cost of the plan is now lower than the cost of the tablescan
b) The total cost of the plan is still lower than the cost of the fast full scan
c) The cost of the fast full scan in the hash join is much higher than the cost of a standalone fast full scan

Analysis

I’ve said many times that I hardly ever look at a 10053 trace. The trouble is, when I do have to look at a 10053 I usually want to tell everyone how clever I’ve been; this means that I can give people the impression that the only way to solve optimizer problems is to look at 10053 traces – it’s not, but this time around it seemed like a good idea.

Problem 1: in the calculation for the cost of the fast full scan for an index hash join, the optimizer starts by calculating the cost of an index full scan, then adds the original cost of the fast full scan to that figure.

Problem 2: when considering the index hash join, the optimizer looks at the indexes in alphabetical order of name, and selects the legal candidate. This is why my tests had 3 extra candidates with three different sizes. Try dropping index t1_ia and Oracle will use t1_ib; alernatively change the name of t1_ia to t1_id and, again, Oracle will use t1_ib. The primary key didn’t have a chance, being way down the alphabet at t1_pk. (In passing, I also tried re-arranging column orders – with the same results, the anomaly is not dependent on the index starting with the important column(s)).

Problem 3: why is the whole smaller than the sum of its parts ? Why worry about the little detail when the big picture is smashed ?

Conclusion

As I pointed out, I’ve run these tests on 10.2.0.3, 11.1.0.7 and 11.2.0.3 – the index hash join arithmetic is wrong. This means the optimizer may be missing opportunities where the index hash join is a good execution path. Keep an eye open for this, you may want to hint some of your SQL (and then switch the hints into SQL Baselines, if you’re running 11g). The trouble is, if you’ve got multiple candidate indexes you may sometimes have to choose between renaming indexes (to get the right one chosen “by accident”) or using an undocumented hint (and in more complex queries you’ll have to look at the outline to find out which query blocks the hint should reference).