Here’s the list of things I’ve pencilled in on my timetable so far. (Some of the gaps are there because I’m doing three presentations myself, some are there because I haven’t decided what to see yet.)

If nothing else catches your eye, don’t miss out the opportunity to hear Maria Colgan talking about the optimizer. She’s doing three presenations (and only one of them coincides with one of mine) and they’re all worth hearing.

First the top five across the lifetime of the blog:

If we look at just the last year, the top five are:

The Cartesian Merge Join has finally been pushed off the top 5 in recent history, but perhaps that’s only a temporary effect while the Addenda to Oracle Core is still fresh in people’s minds. It’s interesting to see how many old articles are still popular reading, though.

The other statistic of note is the number of followers the blog has, currently: 828

While we’re looking at numbers and success stories for the 12 months – the UKOUG annual conference reviews have just come in, and one of mine managed to score 5.83 out of 6 for “Presentation Skills” and 5.65 for content – with one commentator saying:

“Jonathan Lewis is a legend in Oracle circles and now, after my first session with him as speaker, I know why! This was a really well-thought out talk with loads of great ideas and info. My only criticism is that I’d like to attend it again to pick up the bits I missed the first time!”

Don’t worry – it was the one about “Beating the Oracle Optimizer”, 60 minutes featuring seven different strategies for optimising a two-table join, and I’ll be presenting it at least a couple more times in the UK (probably 27th March and 13th June in Manchester and Linlithgow).

I also won two the of UKOUG’s “Inspiring Presenter” awards this year. Not sure I’ll be able to do that again next year since Connor McDonald was in town this year – if you ever get a chance to hear him, make sure you don’t miss it.

Over the last thre or four years I’ve made several commentsabout how subquery factoring could result in changes in execution plans even if the “factored subquery” (or “common table expression” to use the more appropriate technical term) was moved in line. In the 11.2.0.3 bug fixes list, though, you’ll find the following bug fixes:

Bug 11740670 Different execution plans with and without subquery factoring (WITH clause)
Bug 9002661 Transitive predicate were not generated in WITH clause.

So here’s an example that’s been sitting on my laptop for a few years, waiting for the right bug fix (it’s using my standard setup of 8KB block size, LMT with 1MB uniform extents, but not ASSM, and CPU costing disabled):

create table emp (
	dept_no		not null,
	sal,
	emp_no,
	padding,
	constraint e_pk primary key(emp_no)
)
as
select
	mod(rownum,6),
	rownum,
	rownum,
	rpad('x',60)
from
	all_objects
where
	rownum <= 20000
;

-- collect stats, compute, no histograms

select
	outer.*
from
	emp outer
where
	sal > 1000000
and	outer.sal > (
		select
			avg(inner.sal)
 		from	emp inner
		where	inner.dept_no = outer.dept_no
	)
;

with subq as (
	select
		outer.*
	from
		emp outer
	where
		sal > 1000000
	and	outer.sal > (
			select
				avg(inner.sal)
 			from	emp inner
			where	inner.dept_no = outer.dept_no
		)
	)
select	*
from	subq
;

In 11.1.0.7 (and earlier) the first version of this query uses a simple filter subquery, and the second version inlines the factored subquery (which should, therefore, produce the same plan) but then unnests the subquery and uses “group by placement” to resolve the plan as a hash join followed by hash aggregation. This gives us two completely different plans:

Plan for first form of the query
------------------------------------------------------------
| Id  | Operation           | Name | Rows  | Bytes | Cost  |
------------------------------------------------------------
|   0 | SELECT STATEMENT    |      |     1 |    72 |    70 |
|*  1 |  FILTER             |      |       |       |       |
|*  2 |   TABLE ACCESS FULL | EMP  |     1 |    72 |    35 |
|   3 |   SORT AGGREGATE    |      |     1 |     8 |       |
|*  4 |    TABLE ACCESS FULL| EMP  |  3333 | 26664 |    35 |
------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("OUTER"."SAL"> (SELECT AVG("INNER"."SAL") FROM "EMP"
              "INNER" WHERE "INNER"."DEPT_NO"=:B1))
   2 - filter("SAL">1000000)
   4 - filter("INNER"."DEPT_NO"=:B1)

Plan with factored subquery moved in line
-------------------------------------------------------------
| Id  | Operation            | Name | Rows  | Bytes | Cost  |
-------------------------------------------------------------
|   0 | SELECT STATEMENT     |      |     1 |    83 |    93 |
|*  1 |  FILTER              |      |       |       |       |
|   2 |   HASH GROUP BY      |      |     1 |    83 |    93 |
|*  3 |    HASH JOIN         |      |  1667 |   135K|    71 |
|*  4 |     TABLE ACCESS FULL| EMP  |     1 |    72 |    35 |
|   5 |     TABLE ACCESS FULL| EMP  | 20000 |   214K|    35 |
-------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("OUTER"."SAL">AVG("INNER"."SAL"))
   3 - access("INNER"."DEPT_NO"="OUTER"."DEPT_NO")
   4 - filter("SAL">1000000)

Under 11.2.0.3, however, the first plan appears for both forms of the query.

On the plus side – this is good, because Oracle is now behaving consistently. On the minus side – this is bad, because someone, somewhere, is going to be the unlucky person who managed to get a good execution plan by accident because of the bugs.

Maxim: when upgrading to 11.2.0.3, keep a close eye on any queries you’ve got that include subquery factoring – in fact, you might decide to capture SQL Baselines (without activating them) for every single one of them before doing the upgrade so that you can enforce the pre-upgrade plan if the post-upgrade plan is less efficient.

Footnote: I haven’t finished with this example – there’s an interesting follow-up that appeared in the remainder of my test code when I hinted the subquery into materialization. But that’s a story for another day.

For ease of reference, here are some of the results I got (sorted in reverse order of version) from the searches; you will need a MOS account to follow the links:

• 11.2.0.3 patch set bug list
• 11.2.0.3 known issues
• 11.1.0.7 patch set bug list
• 11.1.0.7 known issues
• 10.2.0.5 patch set bug list
• 10.2.0.5 known issues
• 10.1.0.5 patch set bug list
• 10.1.0.5 known issues
• 9.2.0.7 patch set bug list (no bug list published for 9.2.0.8)
• 9.2.0.8 list of actual patches (see readme for bugs fixed)
• 9.2.0.8 known issues

Let’s set up a simple data set and a couple of indexes so that we can take a closer look:

create table t1
as
with generator as (
	select	--+ materialize
		rownum id
	from dual
	connect by
		level <= 10000
)
select
	rownum			id,
	mod(rownum-1,20)	flag,
	lpad(rownum,10,'0')	v10,
	lpad(rownum,20,'0')	v20,
	lpad(rownum,30,'0')	v30,
	rpad('x',100)		padding
from
	generator	v1,
	generator	v2
where
	rownum <= 100000
;

-- gather stats, compute, no histograms

alter table t1 add constraint t1_pk primary key(id)
	using index (create index t1_pk on t1(id))
;

create index t1_flag on t1(flag);

create index t1_ia on t1(id, v20);
create index t1_ib on t1(id, v10);
create index t1_ic on t1(id, v30);

The indexing is a little unusual, but does represent the sort of thing I see from time to time. In particular, the “primary key plus one more column” can be helpful to make critical queries visit only the index and not visit the table; having three such indexes is a bit over the top. Look carefully and you’ll notice that the ia, ib, ic indexes look a little out of order compared to the v20, v10, v30 columns they are built on.

Now try running the following SQL with autotrace enabled:

select /*+ index_ffs(t1 t1_pk) */ count(*) from t1;
select /*+ index_ffs(t1 t1_ia) */ count(*) from t1;
select /*+ index_ffs(t1 t1_ib) */ count(*) from t1;
select /*+ index_ffs(t1 t1_ic) */ count(*) from t1;
select /*+ index_ffs(t1 t1_flag) */ count(*) from t1 where flag is not null;

These are the results I got from 11.2.0.3 (and this test reproduces under 10.2.0.3 and 11.1.0.7). I’ve shown the full plan for the first query, but only the critical operation with its cost for the rest of them:

-------------------------------------------------------
| Id  | Operation             | Name  | Rows  | Cost  |
-------------------------------------------------------
|   0 | SELECT STATEMENT      |       |     1 |    35 |
|   1 |  SORT AGGREGATE       |       |     1 |       |
|   2 |   INDEX FAST FULL SCAN| T1_PK |   100K|    35 |
-------------------------------------------------------

|   2 |   INDEX FAST FULL SCAN| T1_IA |   100K|    80 |
|   2 |   INDEX FAST FULL SCAN| T1_IB |   100K|    59 |
|   2 |   INDEX FAST FULL SCAN| T1_IC |   100K|   101 |

|*  2 |   INDEX FAST FULL SCAN| T1_FLAG |   100K|   292K|    31 |

As you might expect, the longer the “extra” column, the higher the cost. It’s an interesting little detail that the queries that didn’t need to look at a data column didn’t report a “bytes” column in the execution plan – just one clue that there’s a special optimisation for the generic “count everything” query.

Finally, we can get to the index hash join.

select	sum(id)
from
	t1
where
	flag = 0;
;

Clearly, this query could do a full tablescan, but it could do a hash join between the t1_flag index and one of the other indexes. Given that the primary key index has the lowest fast full scan cost at 35, and the index fast full scan on the t1_flag index is 31, we might hope to see an index hash join with a cost of about 66. Here’s the default plan:

-----------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost  |
-----------------------------------------------------------
|   0 | SELECT STATEMENT   |      |     1 |     8 |   380 |
|   1 |  SORT AGGREGATE    |      |     1 |     8 |       |
|*  2 |   TABLE ACCESS FULL| T1   |  5000 | 40000 |   380 |
-----------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter("FLAG"=0)

It’s a full tablescan, with a cost of 380. So let’s hint an index hash join with the two indexes we hoped to see, and find out what happens. I added the hint /*+ index_join(t1) */ and got the following execution plan:

----------------------------------------------------------------------------
| Id  | Operation               | Name             | Rows  | Bytes | Cost  |
----------------------------------------------------------------------------
|   0 | SELECT STATEMENT        |                  |     1 |     8 |   533 |
|   1 |  SORT AGGREGATE         |                  |     1 |     8 |       |
|*  2 |   VIEW                  | index$_join$_001 |  5000 | 40000 |   533 |
|*  3 |    HASH JOIN            |                  |       |       |       |
|*  4 |     INDEX RANGE SCAN    | T1_FLAG          |  5000 | 40000 |    10 |
|   5 |     INDEX FAST FULL SCAN| T1_IA            |  5000 | 40000 |   645 |
----------------------------------------------------------------------------

Note three anomalies:
a) Oracle has obeyed the index_join directive, but it’s used the “wrong” index
b) the cost of the query (533) is less than the cost of one of its operations (645 at line 5)
c) the cost of an index fast full scan on t1_ia has jumped from 80 (previous test) to 645

So let’s add a hint to get the right index used:

select
	/*+
		qb_name(main)
		index_join(@main t1 t1_flag t1_pk)
	*/
	sum(id)
from
	t1
where
	flag = 0
;

This makes no difference, Oracle still uses index t1_ia instead of t1_pk. It’s only when I include an undocumented hint that Oracle does what I want:

explain plan for
select
	/*+
		qb_name(main)
		outline_leaf(@main)
		index_join(@main t1 t1_flag t1_pk)
	*/
	sum(id)
from
	t1
where
	flag = 0
;

select * from table(dbms_xplan.display(null,null,'outline'));

----------------------------------------------------------------------------
| Id  | Operation               | Name             | Rows  | Bytes | Cost  |
----------------------------------------------------------------------------
|   0 | SELECT STATEMENT        |                  |     1 |     8 |   240 |
|   1 |  SORT AGGREGATE         |                  |     1 |     8 |       |
|*  2 |   VIEW                  | index$_join$_001 |  5000 | 40000 |   240 |
|*  3 |    HASH JOIN            |                  |       |       |       |
|*  4 |     INDEX RANGE SCAN    | T1_FLAG          |  5000 | 40000 |    10 |
|   5 |     INDEX FAST FULL SCAN| T1_PK            |  5000 | 40000 |   279 |
----------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      INDEX_JOIN(@"MAIN" "T1"@"MAIN" ("T1"."FLAG") ("T1"."ID"))
      OUTLINE(@"MAIN")
      OUTLINE_LEAF(@"MAIN")
      OUTLINE_LEAF(@"SEL$998059AF")
      ALL_ROWS
      OPT_PARAM('_optimizer_cost_model' 'io')
      DB_VERSION('11.2.0.3')
      OPTIMIZER_FEATURES_ENABLE('11.2.0.3')
      IGNORE_OPTIM_EMBEDDED_HINTS
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter("FLAG"=0)
   3 - access(ROWID=ROWID)
   4 - access("FLAG"=0)

You might note that with the right index in play
a) The cost of the plan is now lower than the cost of the tablescan
b) The total cost of the plan is still lower than the cost of the fast full scan
c) The cost of the fast full scan in the hash join is much higher than the cost of a standalone fast full scan

Analysis

I’ve said many times that I hardly ever look at a 10053 trace. The trouble is, when I do have to look at a 10053 I usually want to tell everyone how clever I’ve been; this means that I can give people the impression that the only way to solve optimizer problems is to look at 10053 traces – it’s not, but this time around it seemed like a good idea.

Problem 1: in the calculation for the cost of the fast full scan for an index hash join, the optimizer starts by calculating the cost of an index full scan, then adds the original cost of the fast full scan to that figure.

Problem 2: when considering the index hash join, the optimizer looks at the indexes in alphabetical order of name, and selects the legal candidate. This is why my tests had 3 extra candidates with three different sizes. Try dropping index t1_ia and Oracle will use t1_ib; alernatively change the name of t1_ia to t1_id and, again, Oracle will use t1_ib. The primary key didn’t have a chance, being way down the alphabet at t1_pk. (In passing, I also tried re-arranging column orders – with the same results, the anomaly is not dependent on the index starting with the important column(s)).

Problem 3: why is the whole smaller than the sum of its parts ? Why worry about the little detail when the big picture is smashed ?

Conclusion

As I pointed out, I’ve run these tests on 10.2.0.3, 11.1.0.7 and 11.2.0.3 – the index hash join arithmetic is wrong. This means the optimizer may be missing opportunities where the index hash join is a good execution path. Keep an eye open for this, you may want to hint some of your SQL (and then switch the hints into SQL Baselines, if you’re running 11g). The trouble is, if you’ve got multiple candidate indexes you may sometimes have to choose between renaming indexes (to get the right one chosen “by accident”) or using an undocumented hint (and in more complex queries you’ll have to look at the outline to find out which query blocks the hint should reference).

(The obvious flaw in the concept appears  in comment 22.)

select
        name, value
from    v$sysstat
where
        name in (
                'consistent gets - examination',
                'consistent gets',
                'session logical reads'
        )
or      name like 'transaction tables%'
;

NAME                                                                            VALUE
---------------------------------------------------------------- --------------------
session logical reads                                                 102,731,023,313
consistent gets                                                       102,716,499,376
consistent gets - examination                                          98,170,595,252
transaction tables consistent reads - undo records applied             96,590,314,116
transaction tables consistent read rollbacks                                2,621,019

5 rows selected.

The instance has been up for about 60 hours – and 95% of the work it has done has been trying to find the commit times for transactions affecting blocks that are in need of cleanout. If you look at the two statistics about the transaction tables (those are the things in the undo segment header blocks) you can see that the average work done to find a commit time was a massive 48,000 visits to undo blocks.

The solution was fairly simple – kill all the reports which had been running for the last six hours, because they were the ones that were causing a problem, while simultaneously suffering most from the problem – at the time I killed the worst offender it was managing to read about 50 blocks per minute from the database, and doing about 100,000 buffer visits to undo blocks per second.

You probably won’t see this every again, but if you do, a quick check is:

select * from v$sess_io order by consistent_changes; 

In general, if you have a three-column index that starts with the same columns in the same order as the two-column index then the three-column index will be bigger and have a higher clustering_factor.

So what scenarios can you come up with that fall outside the general case ?
Alternatively, what argument could you put forward that justifies the general claim ?

I’ll try to respond to comments on this post a little more quickly than the last one, but I still have quite a lot of other comments to catch up on.

… the index has to start with the columns (product_group, id) in that order – with preference given to an exact match, otherwise using the lowest cost index that starts the right way.

On reading this statement I suddenly realised that I hadn’t actually proved (to myself, even) that if I had the indexes (product_group, id) and (product_group, id, other_col) then a two-column hint forced Oracle to use the two column index in all (legal) circumstances.

So, tonight’s quiz – are there any edge cases, and what easy ways can you think of to prove (or disprove) the claim for the general case.

Footnote: you don’t have to demonstrate the method, just a brief outline of the idea will be sufficient.

Update Jan 19th

I’m sorry it’s taken so long to respond to this. It was a post that I pre-dated at the end of December, and I forgot that it would launch itself.

The most interesting comment, from my perspective, came from Valentin Nikotin – “what if the index you are hinting is in another schema?” Virtually every test I do is based on a single schema, I rarely use two, or more, schemas at the same time. So what if there’s a special case that somehow the hint has a precedence that assumes the indexes will be found by (in effect) querying user_ind_columns rather than all_ind_columns ? I wouldn’t expect this to be the case – but when you’re trying to do pre-emptive trouble-shooting it’s this type of case (i.e. “nobody does that sort of thing”) that you can easily overlook.

The other responents came up with the type of thing that I would call the correct strategy. We need to start with an SQL statement that uses the three-column index by default and uses the two-column index only when it’s hinted. Of course, if we can examine the 10053 trace and see the change, especially the restriction to the two-column index explicitly being labelled with something like “index demanded by user hint”, then we can be pretty confident that the interpretation is correct.

The question then is, how can we construct a suitable demonstration.

In general, if you have a three-column index that starts with the same columns in the same order as the two-column index then the three-column index will be bigger and have a higher clustering_factor so, from a purely arithmetic perspective, it will be less desirable than the two column index except for one special class of queries. Consider a query of the form:

select  col3
from    t1
where   col1 = {constant}
and     col2 = {constant}

With a suitable three-column index this query can be answered from completely within the index, with only the first two columns Oracle has to visit the table. So if we ensure that there are several rows scattered around the table we can be confident (in the general case) that the optimizer will see that the three-column query has a lower cost than the two-column query.

I don’t need to set up the model for you – Charles Hooper has already done the work.

Footnote 2:
Following Valentin Nikotin’s comment, you might like to consider that any reports you currently run against the user_indexes and user_ind_columns views (and their partition-related equivalents) should be run against the all_indexes and all_ind_columns views with a restriction of the form: “table_owner = user”.

Key feature: it supplies a query that could be very useful for capturing short, but nasty, events;  and has links to a couple of documents explaining what it’s trying to do and why. It’s a query that could do with more exercise on production systems so that Doug can get some feedback to JB about how effect it is, and how it could be improved.

Start here.