How to configure an Infiniband interconnect for Oracle RAC on SUSE Linux Enterprise Server 11

Wikipedia on Infiniband: “a switched fabric communications link used in high-performance computing and enterprise data centers”. The most useful feature of this type of link, in the context of a cluster interconnect, is very low latency.

This post describes the SUSE settings needed for a very simple setup for Oracle Real Application Cluster with two database nodes. The hardware:

  • QLogic QLE7340 Host Channel Adapters (HCA). Good to know: the infiniband business was acquired by Intel from QLogic, so Intel provides support for these cards now
  • Mellanox IS5022 switch
  • Cables

The switch does not need any configuration. Getting the cards to work on the servers needs a bit of work. It’s because these particular types of cards are not officially supported by SUSE, even though SUSE offers the compiled kernel driver module. Supposedly (according to Intel), SP3 for SLES11 will provide support. Until it is released, these are the steps to be taken.

Install the Infiniband (OFED) pattern, in YaST Software Management

Open YaST2/Software Management, select View/Patterns and install the Infiniband (OFED) pattern.

Install the drivers and packages for QLogic QLE7340 HCAs

  • in YaST2/Software repositories, enable the SLES11-Extras repository. Then install the kernel-default-extra package that’s available in it. This package contains the kernel driver module for the QLogic QLE7340 HCAs. It’s called “ib_qib”.
  • configure openibd to load the drivers by editing /etc/infiniband/openibd.conf and changing this line from no to yes:
    QIB_LOAD=yes
  • also, you have to install these two packages with userland “verbs” that are specific to these HCAs: libipathverbs, libipathverbs-32bit

If you don’t do these steps and just try to set an IP in YaST2 Networking Devices, you will get this error:

Unable to configure the network card because the kernel device (eth0, wlan0) is not present.
This is mostly caused by missing firmware (for wlan devices). See dmesg output for details.

Start openibd and enable it for boot

/etc/init.d/openibd start
chkconfig --add openibd

Start Open Subnet Manager and enable it for boot

This needs to be running on only one server that is connected to the switch. Switches with management have the subnet manager embedded in their firmware, but this particular entry-level switch does not, so it needs to run on a server. Without it, there will be no ping response. Edit /etc/sysconfig/opensm and change the last line to ONBOOT=yes.

/etc/init.d/opensmd start
chkconfig --add opensmd

Verify that HCAs work

You can use this command: ibv_devinfo. It should display something like this:

hca_id: qib0
        transport:                      InfiniBand (0)
        fw_ver:                         0.0.0
        node_guid:                      0011:7500:****:****
        sys_image_guid:                 0011:7500:****:****
        vendor_id:                      0x1175
        vendor_part_id:                 29474
        hw_ver:                         0x2
        board_id:                       InfiniPath_QLE7340
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_INIT (2)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 65535
                        port_lid:               65535
                        port_lmc:               0x00
                        link_layer:             IB

Assign an IP to ib0

Now you can assign an IP as you usually do for any network interface, in YaST Network Devices. Do it on both servers.

Verify that the link works

You can use ping between the IPs you have assigned. Also, ibnetdiscover should show something like this:

#
# Topology file: generated on 
#
# Initiated from node 00117500******** port 001175000******

vendid=0x2c9
devid=0xbd36
sysimgguid=0x2c902********
switchguid=0x2c902********(2c902********)
Switch  8 "S-0002c902********"          # "Infiniscale-IV Mellanox Technologies" base port 0 lid 2 lmc 0
[1]     "H-001175000*******"[1](1175000*******)                 # "server1 HCA-1" lid 1 4xDDR
[2]     "H-001175000*******"[1](1175000*******)                 # "server2 HCA-1" lid 3 4xQDR

vendid=0x1175
devid=0x7322
sysimgguid=0x117500********
caguid=0x1175000*******
Ca      1 "H-00117500********"          # "server2 HCA-1"
[1](117500*******)     "S-0002c9020*******"[2]         # lid 3 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 2 4xQDR

vendid=0x1175
devid=0x7322
sysimgguid=0x117500*********
caguid=0x11750********
Ca      1 "H-001175000******"          # "server1 HCA-1"
[1](1175000*******)     "S-0002c9020********"[1]         # lid 1 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 2 4xDDR
Advertisements

Installing Valve Steam with Adobe Flash Player on openSUSE

Unbelievably, we actually have Steam for Linux!

The first version is made for Ubuntu, but fear not, it can be installed easily on openSUSE, with 1-click. Just go here and choose your openSUSE version. Click on “Show unstable packages” to see the “1 Click Install” link. Fear not, it’s stable enough to use. In the window that opens you have an check-box that is ticket by default “Remain subscribed to these repositories after installation”. I would not recommend that, because it’s not a good idea to subscribe to a lot of repositories, it can create a mess.

After you installed Steam, you will want to be able to see video previews of games. Because Steam is 32 bit, it will need a 32 bit Flash Player. Most likely your computer is 64 bit and the Flash player is 64 bit too. What you need to do is download the 32 bit .tar.gz from Adobe. Then copy the file libflashplayer.so from inside that archive into /usr/lib/browser-plugins. You have to be root to do that copy operation. Only that library is enough, ignore the rest of the files.

Now run Steam either from the Desktop Environment menu, for example in KDE Menu: Games/More Programs, or from command line: /usr/bin/steam

Options for “less” without word wrapping and one-character horizontal scroll

Today, all of a sudden, I thought that less doesn’t function, by default, the way I would like. So I read the manual page. These options will make “less” not wrap long lines and also will make it scroll horizontally one character at a time, instead of half the screen width. This way it’s easier to follow which long line is which.

As root, add this line to /etc/bash.bashrc.local

alias less="less --chop-long-lines --shift 1"

Automatically starting graphics mode (GUI) KDE in BackTrack Linux 5 R3

A short post that is not about SUSE; it’s just something I had to get done. BackTrack starts by default in text mode, it the intended behavior. If you’d like to have a GUI that starts automatically, read on.

1. make sure you have KDM installed

dpkg --list kdm

It should be there, it’s installed by default. If you use Gnome, you need GDM, that is not installed by default, so install it this way:

apt-get install gdm

2. remove the “text” option in /etc/default/grub

Edit this line and delete the “text” option

GRUB_CMDLINE_LINUX_DEFAULT="text splash vga=791"

3. update grub.cfg

Run this:

update-grub

By now, you should have KDM on your screen. It’s going to appear when you reboot, too.

And since you’re here, maybe you also want to configure this:

Remote connection through VNC

1. install tightvncserver

apt-get install tightvncserver

2. edit /etc/services

Add the line below. You might want to do it at the proper sorted position, by port number.

vnc             5900/tcp                        # TightVNC server

3. add this line to /etc/inetd.conf

vnc stream tcp nowait nobody /usr/bin/Xvnc Xvnc -inetd -query localhost -once -geometry 1024x768 -depth 16

4. enable the service to start at boot

It’s not converted to upstart, it’s still sysinit

update-rc.d openbsd-inetd defaults

5. enable Xdmcp in /etc/kde4/kdm/kdmrc

Go into section [Xdmcp] and enable it

Enable=true

6. allow login

Edit /etc/kde4/kdm/Xaccess. Remove the comment sign (#) from the beginning of this line:

* #any host can get a login window

7. start the service

/etc/init.d/openbsd-inetd start

8. verify that you can connect

Use a VNC client, like Krdc, vncviewer from the command line, or the Windows TightVNC client. Connect to port 5900.

sftp only & chroot user accounts in openSUSE 12.2

This post walks you through the steps to make some user accounts that can connect only by sftp (Secure FTP, a subsystem of openSSH) and are chrooted into their home directories. That is, they can’t see anything else outside their homes. You will be able to have other regular users on the system, the sftp-only users will be a special group.

Create a volume for chroot homes and mount it with options nodev, nosuid

This volume is used by the chrooting part of the mechanism. It will contain the home directories of the sftp-only users. The easiest way to make this on an already-installed OS is to create a loop mounted image. You must be root for these commands to work.

su - # become root
dd if=/dev/zero of=/home/sftponly_image.ext3 bs=1M count=512 # this makes a 512 MB volume
mkfs.ext3 /home/sftponly_image.ext3 # create the filesystem. Answer y to "Proceed anyway?"
mkdir /home/sftponly # make a mountpoint
mount -o loop,nodev,nosuid /home/sftponly_image.ext3 /home/sftponly # mount the image

Also you’ll want the image to mount automatically at boot. Add this to /etc/fstab:

/home/sftponly_image.ext3 /home/sftponly ext3 loop,nodev,nosuid 0 0

Everything up until now was laying the foundation for the actual sftp-only configuration that follows. Without all that, you would see this error in /var/log/messages when you try to login with sftp-only accounts:

sshd[...]: fatal: chroot into directory without nodev or nosuid

Create a sftponly group

Members of this group will be able to connect only by SFTP and will be chrooted to their home directories.

groupadd --system sftponly

Create a sftp user

The following command creates a sftp-only user named “sftpexample” with the home directory inside the mounted image. The user is a member of sftponly group and does not have permission to login with ssh, only sftp.

useradd --comment "SFTP example" --create-home --home /home/sftponly/sftpexample --groups sftponly --shell /bin/false sftpexample
passwd sftpexample # set a password

Add options to SSH daemon configuration file

Add this to /etc/ssh/sshd_config

Match Group sftponly
        ChrootDirectory /home/sftponly/%u
        ForceCommand internal-sftp

Reload the ssh daemon:

systemctl reload sshd.service

Verify that it works

Login through sftp and verify the chroot environment:

sftp sftpexample@localhost
Password: 
Connected to localhost.
sftp> ls
bin          public_html  
sftp> cd /
sftp> ls
bin          public_html  
sftp> quit

Must not be able to login with ssh:

ssh sftpexample@localhost
Password: 
Connection to localhost closed by remote host.
Connection to localhost closed.

Setting up sound through HDMI in KDE on openSUSE 12.2

I use an older PC as media player, it’s connected through a HDMI cable to a LCD TV. Most of the people might use, in this case, a media center application like XBMC (packages for openSUSE are available in the packman repository). I have my own reasons for using just plain openSUSE:

  • I like KDE in openSUSE and I’m proficient in setting it up the way I want
  • my older PC has little memory, 1 GB with 256 MB shared as video memory for the on-board GeForce 8200 video
  • need to run a web-browser (Firefox) that I can set up with large fonts to be read at 3 m
  • I actually like more to run mplayer on the command line to play stuff and pass options to it this way, rather thank click through menus

By default, sound goes to the sound board jack plugs, which means you should connect speakers to hear it. However, the TV has it’s own speakers and sound from the PC can reach them through the HDMI cable.

Go to KDE Configure Desktop, in the Hardware section, choose Multimedia and then Phonon.

The Audio Hardware Setup should look like this:

The Device Preference should look like this:

Click on “Internal Audio Digital Stereo (HDMI)”, then on “Test”. You’d expect that to be enough, but if you’ve come to read this post it might be because it’s not working for you yet. The most probable reason is that the particular output is muted by default. Here’s the magic tool to un-mute it: alsamixer. It can be run in Konsole:

Select the sound card with [F6]

In the next screen, use right arrow [->] to scroll through available channels until you reach S/PDIF 1 and un-mute it with [M]

In your case it might be something else than S/PDIF 1. Just look for channels that are muted, they have a “MM” at the bottom, like “Line” has in the picture above. It helps if you play a song in a different window and just mute/un-mute channels, one by one, in alsamixer.