$ echo $((2#$(date -d '10 oct 2010' +"%d%m%y")))
42

Be prepared!

On IRC or mailinglists I frequently see people not understanding the idea and concepts behind glue records. Many have no clue about it. It’s always the same, some people don’t get it and thus make their sites unavailable – when you explain it, they don’t understand.

Hence, I suggest a name change, just name them

The clue records

I have a small LDAP setup running for centralized Linux authentication since years, using the NSS and PAM modules from PADL Software Pty Ltd.

A few weeks ago I found the nss-pam-ldapd from Arthur de Jong and found the architecture is promising and fixes some problems with the design of the PADL modules.

This project needs contributors, so if you

• have time
• know some C
• like LDAP
• have ideas
• can test

it would be nice to see you there.

I also contributed a line or two.

Today, somebody (thanks exaltis on Freenode) provided me a link to an OSNews article during a discussion about X. He told me it’s fun to read – I clicked.

In this article (also read the comments), Thom Holwerda talks about why the X Window System is bad and Linux needs a different graphics architecture. This article is nice to read and discusses existing problems in some applications and X server software.

All problems he talks about are real, yes. I can understand that he’s pissed off. And he, as end-user, wants a solution and a stable system.

But his conclusions that these problems come from the design of the X Window System itself are just wrong and show that he doesn’t know what the design of X is. X has enough problems in its design, but none of them are related to the problems in his article. He talks about bad design, but he actually means bad implementation. Even his example with driver crashes or driver updates are completely irrelevant for the X design, the X Window architecture allows ways to implement this. That nobody does this doesn’t mean it’s a design problem.

From what I can see in the text he just would say to me that he’s a user and not a tech and he doesn’t care and he just wants a working system.

Yes, right, he’s a user. That’s why he shouldn’t conclude it’s a design problem, he should just talk about what he knows and not about what he thinks he knows.

After configuring the ASM instance with the creation assistant and enabling the instance in /etc/oratab, I did a reboot. Finally I wrote a small shell function db_change() to change ORACLE_HOME, ORACLE_SID and PATH on the fly to operate on the different instances.

Nothing. I wasn’t able to connect with sqlplus. I verified the instance “+ASM” still was running, but I couldn’t connect. I checked my ORACLE_SID and ORACLE_HOME, but I couldn’t connect. I even called a friend of mine (thanks Martin!) to make sure I have no conceptual mistake. Nothing. Sqlplus always said “connected to an idle instance” – as if no ASM instance was running at all (or as if I had a typo in the SID).

Since the +ASM instance started fine during system startup, and started fine using the orarun-initscript from SuSE, something had to smell in my setup. After 30 minutes of checking all corners, I decided to compare my current process environment to the process environment of the started ASM PMON (thanks to Linux’ /proc/ filesystem and some shell vodoo, it’s not a big deal to get a good diff).

After eliminating all the unimportant differences, one was left:

ORACLE_HOME=/opt/oracle/product/10.2/asm_1

ORACLE_HOME=/opt/oracle/product/10.2/asm_1/

See it? Believe it or not, that was it! My shell function set an ORACLE_HOME with a trailing slash, and that drove sqlplus crazy! This may be documented somewhere, I didn’t look, but it’s definitely not something intuitive. Finally, that automatically means that things may fail if you put pathes with trailing slash in your /etc/oratab, too.

O.R.A.C.L.E!!!

Applies to the setup:

• SLES10SP2
• Oracle 10.2.0.4 w/ recommendec patchset 4
  

Have a nice day, and better don’t trust your common sense! Comments welcome, ’cause I’m sure this is a well known behaviour and I’m just too stupid

(and yes, I’ll rewrite my small shell function to actually parse /etc/oratab!)

• Backup/exports
• Shutdown DB, remove userspace files
• Cleanup files inside ASM
• DBCA to make new one (no scripts available)

You see, nothing unusual. So far so good, everything fine except the last step. The scripts I had from the first installation didn’t run through, so I just wanted to click something together. But DBCA spit out:

DBCA could not startup the ASM instance configured on this node. To proceed with database creation using ASM you need the aSM instance to be up and running. Do you want to recreate the ASM instance on this node?

Er.. what? ASM instance is up and running and some minutes before the old database ran fine with it! So what? Metalink, Google: Environment variables, Listener configuration etc etc etc. Nothing applied. The only thing I really knew 100% was that the ASM setup was rocksolid.

DBCA failed in the following cases in my experiments (I removed and re-added ASM instance):

• it detected offline ASM and failed trying to start it
• it detected online ASM and failed
• it detected no ASM, successfully created one, and failed to add disks etc..

Long story short: Every combination of DBCA and ASM actions failed.

After several hours we got the solution. I was about to give up and manually rewrote another DB installation script to use it for this DB environment. The last hours we turned around nearly every bit of that installation, we nearly did everything except reinstall the software. We even rebooted the server . My collegue was hopeless, too. But we both were fanatic to find the “why” – it was one of these things where workarounds may work, but don’t satisfy the admins.

I hacked a glogin.sql startup file several weeks ago, that displays a nice prompt with instance and username and some timing statistics when you use SQL*Plus. This is something fine and useful for me (and in general).

But we (infact my collegue) found out:

DBCA seems to communicate with the ASM instance using SQL*Plus. My changed output (infact only the prompt and the time statistics) made the SQL*Plus text parsers of DBCA going crazy. Without one word in the right direction in any log of DBCA. DBCA log always said it found the ASM disks and listed them (in the log). So, it was a simple text parsing issue! The developers of DBCA or the JAVA-classes that manage the communication seem to assume that nobody personaziles his SQL*Plus. Only bad software makes such assumptions

Now I know why ORACLE recommends to use different ORACLE_HOMEs. I didn’t try it, but I can imagine that the SQL*Plus of the ORACLE_HOME from the ASM instance is used when different.

The evening in numbers:

• applies to ORACLE software 10.2.0.4, most likely others, too
• happened on SLES10 SP2
• may be documented somewhere, but I didn’t find it on min. 200 pages I read on the Internet
• gave us min. 500 million new grey hairs
• took us 5 hours

I really begin to love ORACLE At least life isn’t boring…

• Sat Feb 14 00:31:30 CET 2009 -or-
• Fri Feb 13 23:31:30 UTC 2009 -or-
• Epoch 1234567890

please don’t forget to celebrate UNIX!

Additionally, I hope this event helps me to remember the valentine day at least this year

I’ll post an update if Feb 13th has come, to wake you up…

If you send postcards (address in Imprint, also by email), I’ll publish them here!

UPDATE-01:

Found a link. You can even buy T-shirts and stuff. eek, 1234567890 Merchandising…

UPDATE-02:

Do not forget 1234567890!

UPDATE-03:

We survived that day. Good. See you @2000000000!

http://bash-hackers.org/wiki/doku.php/bash4 http://wiki.bash-hackers.org/bash4

Enjoy, comments and edits welcome.

<damien> I should mail Daniel to ask him what he thinks about that
<TheBonsai> daniel?
<TheBonsai> *scratches beard*
<Snark> *scratches head*
<damien> constantin mierla
<damien> from sip-router
<TheBonsai> aah
<Snark> ahh

And yes, we do that every day.