I was analyzing some unexpected RMAN termination, a RMAN-10038/RMAN-03009 combo:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: Fehler bei REFAF Befehl in c13 Kanal auf 10/31/2011 08:21:50
RMAN-10038: Datenbank-Session für Kanal c13 unerwartet beendet

Nothing, not even an RMAN tracing was able to reveal more hints. The trace just told it in other words. No underlying ORA/TNS error or similar.

It finally turned out it were some parameters recently added to sqlnet.ora, I set SQLNET.RECV_TIMEOUT and SQLNET.SEND_TIMEOUT and the beast silently dropped RMAN channels that were idle for a while. The next command in this channel blew up the whole RUN block of the backup script.

I removed the parameters and it works again.

Be careful with your sqlnet.ora

• / (slash) to separate username and password
• @ (at) to separate the TNS descriptor string

If you need to use those characters literally in the logon string, you need to tag them with literal double quotes (literal means: the quotes need to be passed to SQL*Plus, I’m not talking about the UNIX shell):

• Less readable:

  $ sqlplus USER/\"PASS/WORD\"

• More readable:

  $ sqlplus USER/'"PASS/WORD"'

In the past, all storage migration operations there had two problems that lead to downtime:

• CRS votedisk migration at best only with offline CRS
• kick old devices from the device management in the operating system

In December I migrated the system to something more modern

• SLES11
• Oracle Grid Infrastructure (CRS/ASM) 11.2
• Oracle Database 10.2.0.5 (yes, still quite old, but it has to be a 10.2 for now)

The use of a modern Linux kernel (especially a modern SCSI stack) and 11gR2 infrastructure fixed all my trouble from the past.

Votedisk

With 11gR2 Grid Infrastructure, the clusterware manages its vital files (VD, OCR) using an ASM instance. A migration of the ASM diskgroups holding these files is now as easy as a migration of a normal diskgroup. ASM takes care of moving the votedisks and the OCR and collaborates with CRS here.

Linux devices

The new Linux kernels with a finally sane SCSI stack and native multipathing helps with the second problem. It’s not a problem anymore to remove old device references from the stack:

• deconfigure the devices from multipathd (not needed technically, since multipathd itself holds no device references)
• remove the device references from the Linux device mapper

  # see /dev/mapper/* for the name
  dmsetup remove mpathX

• remove the LUNs from the SCSI stack

  # X:X:X:X LUN number
  echo 1 > /sys/bus/scsi/X:X:X:X/delete

After that, you can safely edit the FC zone – no errors should occur in any logs.

ALTER SYSTEM SWITCH LOGFILE;

In a RAC with some more nodes, it can be quite disturbing to do that on all instances.

When you inspect the protocol files of the right services, you will find another, undocumented method:

ALTER SYSTEM SWITCH ALL LOGFILE;

This works at least on 10.2.0.5. Since it’s undocumented, I can’t tell you under which circumstances this command works or not, but it looks like always possible. Please find out yourself and let me know.

The export routines were done with datapump export. I watched a manual run and…

• datapump started and connected to a database server process: 3 seconds
• datapump estimated and analyzed the data to export: 36 minutes
• datapump exported the relevant tables: 4 seconds

Do you see what I see? Yes? Right, datapump played with itself. Over 36 minutes of runtime, where 99% of the time not one bit of data was moved. Reproducable 100%. A datapump TRACE=480300 generated trace set revealed a huge delay in the worker process when calling DBMS_METADATA.FETCH_XML_CLOB (exactly my 36 minutes).

The solution was a patch for bug

• 7710931: DATAPUMP EXPORT IS EXTREMELY SLOW WHEN EXTRACTING SCHEMA

Now datapump does what it’s designed for: Pumping data without playing with itself!

Dienstleister bittet die Datenbankadministration um die Beseitigung des folgenden (Zitat) Datenbankfehlers: ORA-00054: resource busy and acquire with NOWAIT specified

…der manchmal bei einem SELECT FOR UPDATE NOWAIT kommt.

Ist meine Datenbank kaputt?

Datenbank statt Gartenbank!

(vielen Dank an die zynischen Kollegen der Netzwerkadministration…)

Well, test server is test server: We (me as DBA, and a collegue as the responsible application developer) finally decided to step into the Oracle 11 world: The new system will be 11gR2

After some reading in I had no problems to understand the new separated architecture with its HA daemon (keyword: Grid Infrastructure) and install the whole software and create a database. And, very important, there was no, not one damn small, problem when importing 10gR2 datapump files (from the productive system). I heard some weird stories before, and I was a bit afraid about that. But nothing, all data was pumped in like a charm.

The first thing I changed, for me as DBA one of the most interesting features on those large databases, I really parallelized RMAN for backup. I hope I can reduce the backup time to a useful amount with that. This was one of the biggest discrepance in 10g: You can create very large datafiles, but you can’t backup them in a true parallel manner. I’ll check my logfiles today and hope…

Well, so far so good, I will see what 11g (especially R2 here) brings to me and, more important, the applications. It will take some time to really explore the pros.

For all of you still at 10g and loose a test system by accident (and have a developer that really wants it…): Try it

After configuring the ASM instance with the creation assistant and enabling the instance in /etc/oratab, I did a reboot. Finally I wrote a small shell function db_change() to change ORACLE_HOME, ORACLE_SID and PATH on the fly to operate on the different instances.

Nothing. I wasn’t able to connect with sqlplus. I verified the instance “+ASM” still was running, but I couldn’t connect. I checked my ORACLE_SID and ORACLE_HOME, but I couldn’t connect. I even called a friend of mine (thanks Martin!) to make sure I have no conceptual mistake. Nothing. Sqlplus always said “connected to an idle instance” – as if no ASM instance was running at all (or as if I had a typo in the SID).

Since the +ASM instance started fine during system startup, and started fine using the orarun-initscript from SuSE, something had to smell in my setup. After 30 minutes of checking all corners, I decided to compare my current process environment to the process environment of the started ASM PMON (thanks to Linux’ /proc/ filesystem and some shell vodoo, it’s not a big deal to get a good diff).

After eliminating all the unimportant differences, one was left:

ORACLE_HOME=/opt/oracle/product/10.2/asm_1

ORACLE_HOME=/opt/oracle/product/10.2/asm_1/

See it? Believe it or not, that was it! My shell function set an ORACLE_HOME with a trailing slash, and that drove sqlplus crazy! This may be documented somewhere, I didn’t look, but it’s definitely not something intuitive. Finally, that automatically means that things may fail if you put pathes with trailing slash in your /etc/oratab, too.

O.R.A.C.L.E!!!

Applies to the setup:

• SLES10SP2
• Oracle 10.2.0.4 w/ recommendec patchset 4
  

Have a nice day, and better don’t trust your common sense! Comments welcome, ’cause I’m sure this is a well known behaviour and I’m just too stupid

(and yes, I’ll rewrite my small shell function to actually parse /etc/oratab!)

super noob question , how do i query my oracle database online, do i need to setup bind9 and allow incoming connections so it can be accessed

I just wanted to share that

Have a good day!