ORACLE-BASE Blog Aggregator

[One thing that's great about jet-lag is that it allows you to catch up on blogging and all the email that's built up while you've been away at the conference. Not much else you can do at 2:30 in the morning.]

I’m glad I went to the Hotsos Symposium again this year. Attending conferences is an expensive business when you’re a contractor as it means more lost income, so I can’t go to every conference I’d like to attend but this one should probably remain near the top of my list.

Lows

- My presentation, obviously. There was a funny moment in the office today when I was looking at the OEM Top Activity page – because, yet again, it had highlighted a performance problem we were previously unaware of – and our architect said perhaps I shouldn’t be using it because I couldn’t be trusted not to break OEM!

- Somehow contriving to miss both of Richard Foote’s presentations but at least I was able to enjoy his entertaining company at dinner one night.

- The weather – not what I had in mind at all!

Highs

- The people, as always. Not only are there lots of smart speakers and attendees at Hotsos, but the informal setting makes it easier to catch up with them. It was particularly good to get time to speak properly to a few people I’ve only met briefly before, buy Kerry Osborne that beer and to see Polish Paul Matuszyk there after I recommended it to him 4 years ago! As for Monique, she’s one of a kind

- Disco Night. Can I request that we have more party nights themed on women wearing glittery mini-dresses and Oracle geeks dancing in duck costumes?

- Alex G playing chopsticks on the piano in a large suite. I was expecting something a little more elegant!

- Quite a few presentations, not least, Bob Sneed’s CPU QoS, Kevin Closson ranting intelligently about where we are today and where we’re heading, Alex G reducing the room to fits of laughter whilst reinforcing the Battle Against Any Guess message and Wolfgang’s Anatomy of a SQL Tuning Session.

- Feeling inspired to think about performance even more now that I’m back in the office.

So, expensive or not, I’m glad I went but next year I might stick to text-mode demos that don’t need a network connection!

There she goes, squirreling into the corner of your carry-on.  Ready for another adventure.  Tucked between a fun floppy hat and the flourescent sun-block, of which she uses plenty, pale creature that she is, Monique is situated to prove to you that a seasoned traveller knows best.  Dressed in a curly mohair (so as not to show the wrinkles that betray one new to the jet set) and sporting a pale chiffon scarf – to double as a disquise in those cities where she is already known for her somewhat silly escapades – she seems to wink at you and beckon you along.  Forget your cares, but by no means your playing cards, and step lightly into whatever dreams your next voyage should hold for you:  let Monique be your guide.  

Her very presence alleviates all the maladies brought on by the jostling of trains, the dipping of ships, the swerving of buses and even by the simple fact that you are far from home.  For she has discovered the secret to successful travel; with the exception of the luscious few minutes in the X-ray machine, where she delights in making funny faces at the operators searching for more dangerous contents.  Monique never looks back.

Charleen Kinser Designs

(With thanks to Carol Dacko for bringing along an interesting dinner companion for my own cuddly friends. Somehow I doubt they’ll ever be the same – she’s terribly sophisticated for my lot!)

First up was Cary Millsap’s – Lessons Learned, Version 2010.03 As Cary pointed out, they always try to put the best speakers in the toughest slots – 8:30 in the morning post-party. I think local guys are slightly more reliable too because they might have actually gone home the night before! He started with a quick Hangover Survey (me – check!) and then pressed on talking about how we test system performance.

He showed a video of Boeing stress-testing the wing of the 787 and, as he pointed out, aircraft manufacturers really know how to stress-test! (Of course whether that reassures you as it does me, or makes you wish no-one would talk about wings disintegrating, as it probably would Mads, is personal.) They showed Boeing test equpiment which is complicated, expensive and non-revenue generating. Those tests are expensive but when people’s lives are on the line, what choice is there? Boeing knows that it has to test the analytic models used in the design. He spent a lot of time talking about good test design. A few thoughts that stood out to me …

- Some stress tests are a waste of time. Will the Boeing 787 land on the moon? If this test fails, what has it proven? If it passes, then it’s awesome but it would be a very expensive way to prove it can cope with commercial flights in Earth’s atmosphere.

- Why test for more than you will see in Production? Because you don’t really know for sure what you’ll see in Production.

- At some point, but I can’t remember the context, he used a Scottish phrase that he’d heard Billy Connolly shout (although the Big Yin was only fully credited later in the day) …

    "There’s no such thing as bad weather, just the wrong clothes"

… looked over at me and said – "I’d love to hear you say that, with the proper accent". I declined politely.

- Most people try to prove only that their systems will work.

- Most tests of systems that are destined to fail never proved it in advance.

- Test to destruction

    a) Test
    b) Until the system melts
    c) Decide whether your real requirements are likely to be lower or higher than melting point.

There was a small amount of time for questions and once it looked like they were done, I granted Cary’s wish (never thought I’d say that), stuck my hand up and repeated The Big Yin’s words. It was only after the laughter had stopped that I realised I might have ruined his big closing, but I think he was ok about it

Next was Tanel Poder talking about LGWR, log file sync waits and COMMIT performance and shock, horror, I was actually going to say that this was one of the least rewarding sessions of the week for me. What?!? Tanel? But he’s, like, an Oracle God! LOL But there were reasons

- I realise that I know a *lot* about how log file sync and log file parallel write work, how they relate to each other and some of the problems they might help you identify. Because it’s a subject I’m *so* familiar with, I didn’t learn much.

- His main demo didn’t quite show what he wanted it to because it didn’t run multiple sessions but, frankly, I’m in no position to talk about demos this week!

By the end, the presentation turned out ok, not least because there was another unexpected appearance from Bob Sneed to talk about the I/O components involved in  redo log management including a suggestion that LGWR be put into a higher scheduling class (but not Real Time!) Updated later – make sure you read Bob and Kevin’s comments below. I’ll try to find a link to his slides and let you take a look yourself.

I loved Tanel’s Big Log File Sync Tuning Secret, though …

    COMMIT LESS!

It was particularly relevant to me because I had a Big Log File Sync Tuning Secret as the closing moment of my own presentation. The problem was I couldn’t use it after the demos went wrong!

    USE ASYNCHRONOUS COMMITS

But, in my case, that was supposed to be funny, too.

I ran off to try and use the free breakfast voucher that Marco had given me but I was just too late. No food again, then Well, I had a couple of slices of cold meat at lunchtime, but mainly to catch up with Alex G before he had to present and then head back to Ottawa. I managed to skip one session at this stage but, after a quick call home, I decided to go along to Alex’s RAC Connection Management presentation after all (a little late). Although I have seen some of this stuff before, I always enjoy watching Alex’s demos and was particularly impressed by the fact that he’d managed to write his own RAC connection load balancer! I was waiting for the applause in the room but either people didn’t quite get it or there was just a lack of energy post-lunch on the last day. I suspect the latter.

Of course, once I’d said goodbye to Alex properly (don’t see him nearly enough), I was a little late for whichever session was going to be my final one of the conference and I was hopelessly torn between Kyle Hailey’s modern SQL performance tools presentation (Kyle’s done a lot of cool work in the area of Oracle Performance Visualisation) and Chris Antognini’s Diagnosing Parallel Executions Performance. In the end I plumped for the latter because I thought it was going to be like something I’d unsuccessfully attempted a couple of years ago and I wanted to see if Chris had a different angle on it and had been more successful. In the end, I probably made the wrong choice because although Chris’ presentation was great, it was really all stuff I already knew. Definitely my bad call, though. Hopefully I’ll get a chance to catch up with Kyle’s presentation at some point in the future too!

After that there was just the usual short farewell and thanks from Gary Goodman of Hotsos. Although the thanks were appreciated, I’m glad they were spread around everybody because the attendees are one of the things that make this conference great and Becky and Rhonda did their usual sterling job of organising everything.

Then it was time for some Fajitas with a few friends (actually, a whopping great number of friends who practically filled the Mexican restaurant!) and a few very sedate beers. (We are old men (and women) now and the night before was a big one!) While we were waiting to go to the Mexican, I had one great surprise left – Alex’s flights weren’t going to get him home, so he came back from the airport and had to check in overnight! At least I got a chance to talk to him properly when I wasn’t hopelessly drunk and didn’t try to seduce him this time.

Now I need to stop blogging and get back to listening to Tanel’s Training Day (good stuff, too, but more about that later)

Well, that was a nice start to the day! Someone came up to me just before Wolfgang Breitling’s presentation to point out that something from my presentation had helped him fix a problem at work last night. It was the OEM Raw Data drill-down that shows you the underlying ASH data for sessions, including backgrounds. He’d used it to identify the timed event leading up to a problem with a crashing smon process. Actually, the more people I speak to, the more I get over the presentation. Most people recognised I made the best of a bad situation but I’m glad that someone actually learned something, too!

Wolfgang’s "Anatomy of a SQL Tuning Session" was one that I managed to miss at UKOUG and I’m glad I made it this time. It revolved around taking a single SQL statement that took 9 minutes to execute and walking through various tuning iterations, using modern tools and optimiser possibilities, leading to a sub-second execution time. It was a natural extension of Wolfgang’s Tuning by Cardinality Feedback that I’ve often recommended to colleagues. He covered a variety of techniques including :-

- Converting parts of the statement to scalar sub-queries
- Subquery factoring
- Transitive closure, with a useful tip that it may be worth experimenting with specifying apparently redundant join predicates to give the optimiser more information to work with.
- Using the Outline part of 10g DBMS_XPLAN to identify the set of hints that would create a specific plan and then using some of them yourself (but this is far easier if you name your query blocks).

What I particularly liked about this presentation was the way that Wolfgang illustrated execution plan steps and changes with some nice slides, highlighting a few steps at a time. But it made me feel a lot better about my mobile phone going off the previous day when Wolfgang’s *own* mobile went off during this one

Next up was Neil Gunther with "How to Quantify Oracle Scalability – Part 1", a presentation about applying his Universal Scalability Law (USL) to Oracle systems. There were quite a few high-level points I picked up from this.

People often assume that the purpose of a model is purely to predict the future but it’s probably just as (or more) useful as a method of validating test results because in his view ‘Data comes from the devil and models come from god’. I’m not sure I agree with that. Test results, even incorrect results, represent reality to me that can’t just be explained away by Maths which doesn’t agree with them but I suspect that’s my peculiar perspective. I much preferred the suggestion that we use models and data together because, whilst he might not trust my data, maybe I don’t trust his model yet?

He talked about how the USL allows for the phenomenon of reduced throughput as workload increases which is something I think I’ve seen before by adding Coherency to Amdahl’s Law. The USL is definitely worth more investigation. As a non-mathematician, though, I suspect I always struggle with this stuff.

Which was why I was *so* relieved that I made the tough decision to skip Riyaj’s presentation and stayed for the second part of this two-hander by Peter Stadler – "How to Quantify Oracle Scalability – Part 2". This was a more practical examination of the USL in relation to Oracle systems and as someone who is very interested in performance in general and the relationship between Response Time and Throughput in particular, this hit the spot. What was slightly bizarre, though was when I recognised the URL for this blog post and the test results come up on screen. I think I’m right to say that this is the second consecutive Hotsos Symposium where this one post has been discussed (by Cary Millsap last year) so I must be doing something right

Peter spent the next 20 minutes or so talking about some of the comments on the post and plugging the results into the USL. I must admit to being slightly surprised by the fact that Peter didn’t think to drop me a mail to let me know he was going to talk about it so much because I might have missed it and it was fascinating! He talked about the lack of detail in the results, but that was because the blog post had an extremely simple message – are you looking for High Throughput, Low Response Times or both? Regardless, if he’d asked me, I could have given him some more information to work with. For example, there was some discussion about measurement errors in data and performing multiple runs to address that which is something I did, but only published one set of fairly representative results.

So it was all a bit strange and unexpected, but utterly fascinating to see someone apply a mathematical approach to my empirical results. I hope that Peter might post the slides and add a URL to the blog post so that everyone can share what he found. I think that’s the point of the comments thread and of blogging in general – sharing information and knowledge and building a discussion.

Next I managed to eat a little bit of much-needed lunch with Paul Matuszyk and then had to get ready for my important work call.

I really don’t know how to blog about this because every time I feel I’m honestly self-critical, everyone thinks I’m close to suicide or something. I like to think I notice both the good and the bad but am probably more likely to speak openly about the bad. If you ask me it’s a Scottish thing about not getting above yourself or blowing your own trumpet too loudly. Or maybe it’s just that the only way you can improve is by noticing the bad stuff and fixing it? Someone once said to me "You can’t be an insecure overachiever without first being insecure" and they aren’t Scottish, so I shouldn’t generalise. Updated later – it was Cary Millsap who came up with this line. I loved it when he mentioned it a few months ago and still do.

In this case I’ll try to be even-handed and finish with the positives!

Negatives

The whole point of this presentation is that it’s about 50 minutes of demonstrations and about 10 of slides. For the first 30 minutes, the demonstrations would not work. If you think that’s a success, there’s something the matter with you and you should probably never give any presentations!

This is the killer. In retrospect I know what the problem was and have fixed it previously with the help of others but did not do the same on the new laptop! As I moved in and out of wireless range, the lack of a network connection at the Windows end completely threw VMWare and my demos. But, believe me, when your demos have been working for a couple of weeks, you don’t have long until your presentation and they stop working intermittently, it’s difficult to be cool and analytical. I was so distracted by other things that might go wrong, I missed one and, having missed it, I wasn’t cool enough to recognise the symptoms. Fortunately, an old Hotsos friend in the room came up with the goods in the form of a little Sprint wireless box that got me hooked up. I kissed him, but there were no tongues involved.

With only 30 minutes left, there was very little I could show and it completely ruined the whole flow of the presentation, which I’d worked so hard to get right and which I know can be terrific, because I’ve done similar presentations before and had been looking forward to doing the best version yet. People took the time to come and see it, I let them down and I’m sorry about that. Nobody is likely to change my view on that.

This presentation is a big deal to me, never mind anyone else. I work hard on these things, try to cover all the angles, take time off work and all because I like teaching people new stuff. When I take a week off to attend a conference, but also to present, it’s a bit of a blow when your one shot fails. If you’re not passionate about your presentations (and I somehow doubt anyone isn’t) again, you shouldn’t be presenting.

Positives

Sh*t happens when you get involved with computers. Yes, folks, I know that, but that’s also why you play around with demos for a long time to minimise the possibility that it will!

This might not sound like a positive and it’s dangerously close to sounding like an excuse, but the fact is that I’ve had 2 out of 3 presentations go very wrong recently. One of the consistent factors in this is that I switched to VMWare because I had to to run 11gR2 on Windows. It’s not the same as blaming VMWare to say that I’ve been having to deal with stuff I haven’t for a long time. When everyone was talking about VMWare as the way to go for presentations, I remember thinking ‘I don’t know, just seems like more moving parts that might break to me.’ and I kept reading blog posts about demos being broken and then fixed just in time, all of which were on some virtualisation platform or other, but didn’t have the confidence to say something. All I know is, say what you like about Windows, but I’ve hardly ever had a problem in multiple 2-day course teaches of performance and OEM stuff! Still, it’s down to me to get on top of what are some simple issues.

Listen, I know I can present. No false modesty round here. So, to wrap this up on a positive note, I’m well aware that there are few of the people who I see present who could have managed to get through that first half hour, make people laugh, keep thinking about the problem and manage to get a short demo of Swingbench into the bargain. I think I also managed to salvage something out of the last half of the presentation without completely collapsing into a heap. Sure, I was a bit brain-addled by then, but I would have liked to have seen how others might have coped Frankly, I kept waiting for the room to empty (I’ve seen it happen) but the vast majority stuck around to the end. Maybe they were sadists! LOL

I am not and will never manage to be happy with that presentation but lessons have been learned, it’s just a presentation and there’ll be lots of others. I know that.

Postive Solution 1 – If anyone wants to try to grab me while I’m at the conference, I’ll show you the screens and demos. They’re very cool

Positive Solution 2 – I think Alex Gorbachev might try to arrange for me to repeat the presentation properly as a webinar. I’ve already done this at my current customer site once (on the 10g stuff) and it went reasonably well. I’d also … get this … asked Marco to video it for me yesterday with his snazzy mini-setup because I thought I might post a few bits online if they were particularly good so those that can’t make it to conferences could get a taste of it. Actually, it appeals to my cold, self-deprecating sense of humour to post some of it online soon. I promise I won’t make it too self-flaggelating though!

P.S. For Paul Vallee. Paul it is not all good