Tag: Linux

Running Sendmail In A CHROOT Jail

My employer’s OS-support model restricts root access to members of the Unix support team. Applications are normally installed into a package directory and run under a service ID. While this model works well for most applications, sendmail is tightly integrated into the OS and is not readily built into an application directory. We attempted to run sendmail as a non-root user with modified permissions on application directories such as /var/spool/mqueue – this worked, until OS patches were applied and permissions reset. We needed a way to run sendmail as a non-root user and allow the OS support team to patch servers without impacting the sendmail application.

Chroot is a mechanism that uses a supplied directory path as the environment’s root directory. The jailed process, and its children, should not be able to access any part of the file hierarchy outside of the new root. As a security mechanism, the approach has several flaws – abridged version of the story is that it’s not terribly difficult to break out of jail here; and there are far more effective security approaches (e.g. SELinux). However, chroot jails have their own copies of system owned directories (such as /var/spool/mqueue), binaries, and libraries. Using a chroot jail will allow us to maintain a sendmail application in the package directory that is not impacted by OS updates.

This approach works on relaying mail servers (i.e. those that queue mail to /var/spool/mqueue and send it on its merry way). If sendmail is hosting mailboxes, there are additional challenges to designing a chroot configuration that actually drops messages into mailbox files that users can access.

Preliminaries: To copy/paste, view the single article. Create a service account under which sendmail will run. The installation directory should be owned by the service account user.

Set up the chroot jail location in the installation directory. In this example, that directory is /smt00p20.

mkdir /smt00p20/sendmail
mkdir /smt00p20/sendmail/dev
mkdir /smt00p20/opendkim

We need a null and random in the sendmail jail. On a command line, run:

# Create sendmail jail /dev/null
mknod /smt00p20/sendmail/dev/null c 1 3
# Create sendmail jail /dev/random
mknod /smt00p20/sendmail/dev/random c 1 8

We need an rsyslog socket added under each jail. In /etc/rsyslog.conf, add the following:

# additional log sockets for chroot'ed jail
# Idea from http://www.ispcolohost.com/2014/03/14/how-to-get-syslog-records-of-chrooted-ssh-sftp-server-activity/
$AddUnixListenSocket /smt00p20/sendmail/dev/log
$AddUnixListenSocket /smt00p20/opendkim/dev/log

 

Additionally, these instructions assume both sendmail and sendmail-cf have been installed on the server. If they have not, you can download the RPMs, unpack them, and copy the files to the appropriate relative jail locations.

Chrooting Sendmail

Logged in with the sendmail ID, ensure you have a .bash_profile that loads .bashrc

-bash-4.2$ cat ~/.bash_profile
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

Edit ~/.bashrc and add the following, where smt00p20 is the appropriate installation directory, to allow copy/paste

export SENDMAILJAIL=/smt00p20/sendmail
export OPENDKIMJAIL=/smt00p20/opendkim

Log out of the service account and back in (or just source in the .bashrc file). Verify SENDMAILJAIL and OPENDKIMJAIL are set.

Copy a whole heap of ‘stuff’ into the jail – this includes some utilities used to troubleshoot issues within the jail which aren’t strictly needed. I’ve also unpacked the strace RPM to the respective directories within the jail.

mkdir $SENDMAILJAIL/bin
mkdir $SENDMAILJAIL/etc
mkdir $SENDMAILJAIL/etc/alternatives
mkdir $SENDMAILJAIL/etc/mail
mkdir $SENDMAILJAIL/etc/smrsh
mkdir $SENDMAILJAIL/lib64
mkdir $SENDMAILJAIL/lib
mkdir $SENDMAILJAIL/lib/tls
mkdir $SENDMAILJAIL/tmp
mkdir $SENDMAILJAIL/usr
mkdir $SENDMAILJAIL/usr/bin
mkdir $SENDMAILJAIL/usr/sbin
mkdir $SENDMAILJAIL/usr/lib
mkdir $SENDMAILJAIL/usr/lib/sasl2
mkdir $SENDMAILJAIL/var
mkdir $SENDMAILJAIL/var/log
mkdir $SENDMAILJAIL/var/log/mail
mkdir $SENDMAILJAIL/var/run
mkdir $SENDMAILJAIL/var/spool
mkdir $SENDMAILJAIL/var/spool/mqueue
mkdir $SENDMAILJAIL/var/spool/clientmqueue
 
cp /etc/aliases $SENDMAILJAIL/etc/
cp /etc/aliases.db $SENDMAILJAIL/etc/
cp /etc/passwd $SENDMAILJAIL/etc/
cp /etc/group $SENDMAILJAIL/etc/
cp /etc/resolv.conf $SENDMAILJAIL/etc/
cp /etc/host.conf $SENDMAILJAIL/etc/
cp /etc/nsswitch.conf $SENDMAILJAIL/etc/
cp /etc/services $SENDMAILJAIL/etc/
cp /etc/hosts $SENDMAILJAIL/etc/
cp /etc/localtime $SENDMAILJAIL/etc/
 

# If cloning an existing server, scp /etc/mail/* from source to /smt00p20/sendmail/etc/mail

# Verify the sendmail.mc has a RUNAS_USER set to the same service account you are using - the account on our servers is named 'sendmail'. Our old servers are not all set up with a runas user, and failing to have one will cause write failures to the jail /var/spool/mqueue

cp -r /etc/mail/ $SENDMAILJAIL/etc/etc/mail/
cp /usr/sbin/sendmail.sendmail $SENDMAILJAIL/usr/sbin/sendmail.sendmail

cd /smt00p20/sendmail/etc/alternatives
ln -s ../../usr/sbin/sendmail.sendmail ./mta

cd /smt00p20/sendmail/usr/sbin
ln -s ../../etc/alternatives/mta ./sendmail
ln -s ./sendmail ./newaliases
ln -s ./sendmail ./newaliases.sendmail

cd /smt00p20/sendmail/usr/bin
ln -s ../sbin/sendmail ./mailq
ln -s ../sbin/sendmail ./mailq.sendmail
ln -s ../sbin/sendmail.sendmail ./hoststat
ln -s ../sbin/sendmail.sendmail ./purgestat
ln -s ../sbin/makemap ./makemap
ln -s ./rmail.sendmail ./rmail
cp /usr/lib64/libssl.so.10 $SENDMAILJAIL/usr/lib64/libssl.so.10
cp /usr/lib64/libcrypto.so.10 $SENDMAILJAIL/usr/lib64/libcrypto.so.10
cp /usr/lib64/libnsl.so.1 $SENDMAILJAIL/usr/lib64/libnsl.so.1
cp /usr/lib64/libwrap.so.0 $SENDMAILJAIL/usr/lib64/libwrap.so.0
cp /usr/lib64/libhesiod.so.0 $SENDMAILJAIL/usr/lib64/libhesiod.so.0
cp /usr/lib64/libcrypt.so.1 $SENDMAILJAIL/usr/lib64/libcrypt.so.1
cp /usr/lib64/libdb-5.3.so $SENDMAILJAIL/usr/lib64/libdb-5.3.so
cp /usr/lib64/libresolv.so.2 $SENDMAILJAIL/usr/lib64/libresolv.so.2
cp /usr/lib64/libsasl2.so.3 $SENDMAILJAIL/usr/lib64/libsasl2.so.3
cp /usr/lib64/libldap-2.4.so.2 $SENDMAILJAIL/usr/lib64/libldap-2.4.so.2
cp /usr/lib64/liblber-2.4.so.2 $SENDMAILJAIL/usr/lib64/liblber-2.4.so.2
cp /usr/lib64/libc.so.6 $SENDMAILJAIL/usr/lib64/libc.so.6
cp /usr/lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/usr/lib64/libgssapi_krb5.so.2
cp /usr/lib64/libkrb5.so.3 $SENDMAILJAIL/usr/lib64/libkrb5.so.3
cp /usr/lib64/libcom_err.so.2 $SENDMAILJAIL/usr/lib64/libcom_err.so.2
cp /usr/lib64/libk5crypto.so.3 $SENDMAILJAIL/usr/lib64/libk5crypto.so.3
cp /usr/lib64/libdl.so.2 $SENDMAILJAIL/usr/lib64/libdl.so.2
cp /usr/lib64/libz.so.1 $SENDMAILJAIL/usr/lib64/libz.so.1
cp /usr/lib64/libidn.so.11 $SENDMAILJAIL/usr/lib64/libidn.so.11
cp /usr/lib64/libfreebl3.so $SENDMAILJAIL/usr/lib64/libfreebl3.so
cp /usr/lib64/libpthread.so.0 $SENDMAILJAIL/usr/lib64/libpthread.so.0
cp /usr/lib64/libssl3.so $SENDMAILJAIL/usr/lib64/libssl3.so
cp /usr/lib64/libsmime3.so $SENDMAILJAIL/usr/lib64/libsmime3.so
cp /usr/lib64/libnss3.so $SENDMAILJAIL/usr/lib64/libnss3.so
cp /usr/lib64/libnssutil3.so $SENDMAILJAIL/usr/lib64/libnssutil3.so
cp /usr/lib64/libplds4.so $SENDMAILJAIL/usr/lib64/libplds4.so
cp /usr/lib64/libplc4.so $SENDMAILJAIL/usr/lib64/libplc4.so
cp /usr/lib64/libnspr4.so $SENDMAILJAIL/usr/lib64/libnspr4.so
cp /usr/lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/usr/lib64/ld-linux-x86-64.so.2
cp /usr/lib64/libkrb5support.so.0 $SENDMAILJAIL/usr/lib64/libkrb5support.so.0
cp /usr/lib64/libkeyutils.so.1 $SENDMAILJAIL/usr/lib64/libkeyutils.so.1
cp /usr/lib64/librt.so.1 $SENDMAILJAIL/usr/lib64/librt.so.1
cp /usr/lib64/libselinux.so.1 $SENDMAILJAIL/usr/lib64/libselinux.so.1
cp /usr/lib64/libpcre.so.1 $SENDMAILJAIL/usr/lib64/libpcre.so.1
cp /usr/lib64/libnss_dns.so.2 $SENDMAILJAIL/usr/lib64/libnss_dns.so.2
cp /usr/lib64/libnss_files.so.2 $SENDMAILJAIL/usr/lib64/libnss_files.so.2

cd $SENDMAILJAIL/lib64
cp /lib64/libnss_dns-2.17.so $SENDMAILJAIL/lib64/libnss_dns-2.17.so
ln -s ./libnss_dns-2.17.so ./libnss_dns.so.2

cp /lib64/libresolv-2.17.so $SENDMAILJAIL/lib64/libresolv-2.17.so
ln -s ./lib64/libresolv-2.17.so ./libresolv.so.2

cp /lib64/libnss_files-2.17.so $SENDMAILJAIL/lib64/libnss_files-2.17.so
ln -s ./lib64/libnss_files-2.17.so ./libnss_files.so.2

cd $SENDMAILJAIL/lib 
cp /lib64/libnss_dns-2.17.so $SENDMAILJAIL/lib/libnss_dns-2.17.so
ln -s ./lib/libnss_dns-2.17.so ./libnss_dns.so.2

cp /lib64/libresolv-2.17.so $SENDMAILJAIL/lib/libresolv-2.17.so
ln -s ./lib/libresolv-2.17.so ./libresolv.so.2

cp /lib64/libnss_files-2.17.so $SENDMAILJAIL/lib/libnss_files-2.17.so
ln -s ./lib/libnss_files-2.17.so ./libnss_files.so.2

mkdir $SENDMAILJAIL/usr/lib64/sasl2
cp /usr/lib64/sasl2/* $SENDMAILJAIL/usr/lib64/sasl2/

mkdir $SENDMAILJAIL/lib64/sasl2/
cp /lib64/sasl2/* $SENDMAILJAIL/lib64/sasl2/
cp /etc/sasl2/Sendmail.conf $SENDMAILJAIL/usr/lib64/sasl2/

mkdir $SENDMAILJAIL/etc/sasl2
cp /etc/sasl2/Sendmail.conf $SENDMAILJAIL/etc/sasl2/


cp /usr/sbin/makemap $SENDMAILJAIL/usr/sbin/makemap
ln -s ../sbin/makemap ./makemap
cp /usr/bin/rmail.sendmail $SENDMAILJAIL/usr/bin/rmail.sendmail
ln -s ./rmail.sendmail ./rmail

cp /usr/sbin/mailstats $SENDMAILJAIL/usr/sbin/mailstats
cp /usr/sbin/makemap $SENDMAILJAIL/usr/sbin/makemap
cp /usr/sbin/praliases $SENDMAILJAIL/usr/sbin/praliases
cp /usr/sbin/smrsh $SENDMAILJAIL/usr/sbin/smrsh

cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libcom_err.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libcrypt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libcrypto.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libdb-5.3.so $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libfreebl3.so $SENDMAILJAIL/lib64/
cp /lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libhesiod.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libidn.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3: $SENDMAILJAIL/lib64/
cp /lib64/libkeyutils.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5support.so.0 $SENDMAILJAIL/lib64/
cp /lib64/liblber-2.4.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libldap-2.4.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libnsl.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libnspr4.so $SENDMAILJAIL/lib64/
cp /lib64/libnss3.so $SENDMAILJAIL/lib64/
cp /lib64/libnssutil3.so $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libplc4.so $SENDMAILJAIL/lib64/
cp /lib64/libplds4.so $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/librt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libsasl2.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libsmime3.so $SENDMAILJAIL/lib64/
cp /lib64/libssl.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libssl3.so $SENDMAILJAIL/lib64/
cp /lib64/libwrap.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /usr/lib64/libk5crypto.so.3 $SENDMAILJAIL/usr/lib64/

cp /lib64/libdns.so.100 $SENDMAILJAIL/lib64/
cp /lib64/liblwres.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libbind9.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisccfg.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisccc.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisc.so.95 $SENDMAILJAIL/lib64/
cp /lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libcom_err.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libcrypto.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libGeoIP.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libxml2.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libm.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libidn.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5support.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libkeyutils.so.1 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/liblzma.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /bin/dig $SENDMAILJAIL/bin/

cp /lib64/libtinfo.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /bin/bash $SENDMAILJAIL/bin/

cp /bin/ls $SENDMAILJAIL/bin/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libacl.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/

cp /bin/vi $SENDMAILJAIL/bin/
cp /usr/sbin/pidof $SENDMAILJAIL/usr/sbin/pidof
cp /lib64/libprocps.so.4 $SENDMAILJAIL/lib64/
cp /lib64/libsystemd.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libm.so.6 $SENDMAILJAIL/lib64/
cp /lib64/librt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/liblzma.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libgcrypt.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libgpg-error.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libdw.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libgcc_s.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libelf.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libbz2.so.1 $SENDMAILJAIL/lib64/
cp /bin/rm $SENDMAILJAIL/bin/

Under your ID, ensure the proper permissions are set on the chroot jail

sudo chown -R sendmail:mail /smt00p20/sendmail/
sudo chown sendmail /smt00p20/sendmail/var/spool/mqueue
sudo chmod 0700 /smt00p20/sendmail/var/spool/mqueue
sudo chmod -R go-w /smt00p20/sendmail
sudo chmod 0400 /smt00p20/sendmail/etc/mail/*.cf

Now verify it works – still under your ID as you have sudo permission to run chroot.

sudo /sbin/chroot /smt00p20/sendmail /bin/ls
# You should see a directory listing like this, not an error
bin  dev  etc  lib  lib64  tmp  usr  var

Assuming there are no problems, run sendmail:

sudo /sbin/chroot /smt00p20/sendmail /usr/sbin/sendmail -bd -q5m

Test sending mail through the server to verify proper functionality.

Unit Config: Edit the systemd unit file and add the “RootDirectory” directive

sudo vi /etc/systemd/system/multi-user.target.wants/sendmail.service

[Unit]
Description=Sendmail Mail Transport Agent
After=syslog.target network.target
Conflicts=postfix.service exim.service
Wants=sm-client.service

[Service]
RootDirectory=/smt00p20/sendmail
Type=forking
StartLimitInterval=0
# Known issue – pid causes service hang/timeout that bothers Unix guys
# https://bugzilla.redhat.com/show_bug.cgi?id=1253840
#PIDFile=/run/sendmail.pid
Environment=SENDMAIL_OPTS=-q15m
EnvironmentFile=-/smt00p20/sendmail/etc/sysconfig/sendmail
ExecStart=/usr/sbin/sendmail -bd $SENDMAIL_OPTS $SENDMAIL_OPTARG

[Install]
WantedBy=multi-user.target
Also=sm-client.service

Then run “systemctl daemon-reload” to ingest the changes.

You can now use systemctl to start and stop the sendmail service.

Chrooting opendkim

Create the chroot jail and lib64 directory, then populate with required files:

mkdir $OPENDKIMJAIL
mkdir $OPENDKIMJAIL/lib64
cp /lib64/libtinfo.so.5 $OPENDKIMJAIL/lib64/
cp /lib64/libdl.so.2 $OPENDKIMJAIL/lib64/
cp /lib64/libc.so.6 $OPENDKIMJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $OPENDKIMJAIL/lib64/

cp /bin/bash $OPENDKIMJAIL/bin/
cp /lib64/libstdc++.so.6* $OPENDKIMJAIL/lib64
cp /lib64/libm.so.6 $OPENDKIMJAIL/lib64
cp /lib64/libgcc_s.so.1 $OPENDKIMJAIL/lib64
cp /lib64/libnss_files* $OPENDKIMJAIL/lib64/

Unpack the following RPMs:

rpm2cpio opendkim-2.11.0-0.1.el7.x86_64.rpm | cpio -idmv
rpm2cpio libopendkim-2.11.0-0.1.el7.x86_64.rpm | cpio -idmv
rpm2cpio sendmail-milter-8.14.7-5.el7.x86_64.rpm | cpio -idmv
rpm2cpio opendbx-1.4.6-6.el7.x86_64.rpm | cpio -idmv
rpm2cpio libmemcached-1.0.16-5.el7.x86_64.rpm | cpio -idvm
rpm2cpio libbsd-0.6.0-3.el7.elrepo.x86_64.rpm | cpio -idvm

Then move the unpacked files into the corresponding location in the $OPENDKIMJAIL directory.

Copy host configuration ‘stuff’ from /etc

cp /etc/aliases $OPENDKIMJAIL/etc/
cp /etc/aliases.db $OPENDKIMJAIL/etc/
cp /etc/passwd $OPENDKIMJAIL/etc/
cp /etc/group $OPENDKIMJAIL/etc/
cp /etc/resolv.conf $OPENDKIMJAIL/etc/
cp /etc/host.conf $OPENDKIMJAIL/etc/
cp /etc/nsswitch.conf $OPENDKIMJAIL/etc/
cp /etc/services $OPENDKIMJAIL/etc/
cp /etc/hosts $OPENDKIMJAIL/etc/
cp /etc/localtime $OPENDKIMJAIL/etc/

Configure OpenDKIM ($DKIMJAIL/etc/opendkim.conf) and populate keys (copy from server being replaced or generate new keys). Then, under your ID, run:

sudo /sbin/chroot /smt00p20/opendkim /usr/sbin/opendkim -u sendmail -v

The systemd unit file, /usr/lib/systemd/system/opendkim.service, needs to contain:

# If you are using OpenDKIM with SQL datasets it might be necessary to start OpenDKIM after the database servers.
# For example, if using both MariaDB and PostgreSQL, change "After=" in the "[Unit]" section to:
# After=network.target nss-lookup.target syslog.target mariadb.service postgresql.service

[Unit]
Description=DomainKeys Identified Mail (DKIM) Milter
Documentation=man:opendkim(8) man:opendkim.conf(5) man:opendkim-genkey(8) man:opendkim-genzone(8) man:opendkim-testadsp(8) man:opendkim-testkey http://www.opendkim.org/docs.html
After=network.target nss-lookup.target syslog.target

[Service]
RootDirectory=/smt00p20/opendkim
Type=forking
PIDFile=/smt00p20/opendkim/var/run/opendkim/opendkim.pid
EnvironmentFile=-/etc/sysconfig/opendkim
ExecStart=/usr/sbin/opendkim -u sendmail -v $OPTIONS
ExecReload=/bin/kill -USR1 $MAINPID
User=sendmail
Group=mail

[Install]
WantedBy=multi-user.target

 

Upgrading Sendmail – After Unix Applies Patches

This process grabs a new copy of sendmail, associated diagnostic utilities, and their dependencies from the OS installation. If you want to apply patches prior to Unix support doing so, you can stage a sendmail build (everything up to ‘make install’) and copy the files out or, if an updated RPM is in the repo but just not installed, download the RPMs, unpack them, and copy the files in. I would do that in addition to (and after) this process to ensure library updates are reflected in our jailed sendmail installation (i.e. if there’s an update to the crypto libraries, we get those updates).

cp /usr/sbin/sendmail.sendmail $SENDMAILJAIL/usr/sbin/sendmail.sendmail
cp /usr/lib64/libssl.so.10 $SENDMAILJAIL/usr/lib64/libssl.so.10
cp /usr/lib64/libcrypto.so.10 $SENDMAILJAIL/usr/lib64/libcrypto.so.10
cp /usr/lib64/libnsl.so.1 $SENDMAILJAIL/usr/lib64/libnsl.so.1
cp /usr/lib64/libwrap.so.0 $SENDMAILJAIL/usr/lib64/libwrap.so.0
cp /usr/lib64/libhesiod.so.0 $SENDMAILJAIL/usr/lib64/libhesiod.so.0
cp /usr/lib64/libcrypt.so.1 $SENDMAILJAIL/usr/lib64/libcrypt.so.1
cp /usr/lib64/libdb-5.3.so $SENDMAILJAIL/usr/lib64/libdb-5.3.so
cp /usr/lib64/libresolv.so.2 $SENDMAILJAIL/usr/lib64/libresolv.so.2
cp /usr/lib64/libsasl2.so.3 $SENDMAILJAIL/usr/lib64/libsasl2.so.3
cp /usr/lib64/libldap-2.4.so.2 $SENDMAILJAIL/usr/lib64/libldap-2.4.so.2
cp /usr/lib64/liblber-2.4.so.2 $SENDMAILJAIL/usr/lib64/liblber-2.4.so.2
cp /usr/lib64/libc.so.6 $SENDMAILJAIL/usr/lib64/libc.so.6
cp /usr/lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/usr/lib64/libgssapi_krb5.so.2
cp /usr/lib64/libkrb5.so.3 $SENDMAILJAIL/usr/lib64/libkrb5.so.3
cp /usr/lib64/libcom_err.so.2 $SENDMAILJAIL/usr/lib64/libcom_err.so.2
cp /usr/lib64/libk5crypto.so.3 $SENDMAILJAIL/usr/lib64/libk5crypto.so.3
cp /usr/lib64/libdl.so.2 $SENDMAILJAIL/usr/lib64/libdl.so.2
cp /usr/lib64/libz.so.1 $SENDMAILJAIL/usr/lib64/libz.so.1
cp /usr/lib64/libidn.so.11 $SENDMAILJAIL/usr/lib64/libidn.so.11
cp /usr/lib64/libfreebl3.so $SENDMAILJAIL/usr/lib64/libfreebl3.so
cp /usr/lib64/libpthread.so.0 $SENDMAILJAIL/usr/lib64/libpthread.so.0
cp /usr/lib64/libssl3.so $SENDMAILJAIL/usr/lib64/libssl3.so
cp /usr/lib64/libsmime3.so $SENDMAILJAIL/usr/lib64/libsmime3.so
cp /usr/lib64/libnss3.so $SENDMAILJAIL/usr/lib64/libnss3.so
cp /usr/lib64/libnssutil3.so $SENDMAILJAIL/usr/lib64/libnssutil3.so
cp /usr/lib64/libplds4.so $SENDMAILJAIL/usr/lib64/libplds4.so
cp /usr/lib64/libplc4.so $SENDMAILJAIL/usr/lib64/libplc4.so
cp /usr/lib64/libnspr4.so $SENDMAILJAIL/usr/lib64/libnspr4.so
cp /usr/lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/usr/lib64/ld-linux-x86-64.so.2
cp /usr/lib64/libkrb5support.so.0 $SENDMAILJAIL/usr/lib64/libkrb5support.so.0
cp /usr/lib64/libkeyutils.so.1 $SENDMAILJAIL/usr/lib64/libkeyutils.so.1
cp /usr/lib64/librt.so.1 $SENDMAILJAIL/usr/lib64/librt.so.1
cp /usr/lib64/libselinux.so.1 $SENDMAILJAIL/usr/lib64/libselinux.so.1
cp /usr/lib64/libpcre.so.1 $SENDMAILJAIL/usr/lib64/libpcre.so.1
cp /usr/lib64/libnss_dns.so.2 $SENDMAILJAIL/usr/lib64/libnss_dns.so.2
cp /usr/lib64/libnss_files.so.2 $SENDMAILJAIL/usr/lib64/libnss_files.so.2
cp /lib64/libnss_dns-2.17.so $SENDMAILJAIL/lib64/libnss_dns-2.17.so
cp /lib64/libresolv-2.17.so $SENDMAILJAIL/lib64/libresolv-2.17.so
cp /lib64/libnss_files-2.17.so $SENDMAILJAIL/lib64/libnss_files-2.17.so
cp /lib64/libnss_dns-2.17.so $SENDMAILJAIL/lib/libnss_dns-2.17.so
cp /lib64/libresolv-2.17.so $SENDMAILJAIL/lib/libresolv-2.17.so
cp /lib64/libnss_files-2.17.so $SENDMAILJAIL/lib/libnss_files-2.17.so
cp /usr/lib64/sasl2/* $SENDMAILJAIL/usr/lib64/sasl2/
cp /lib64/sasl2/* $SENDMAILJAIL/lib64/sasl2/
cp /etc/sasl2/Sendmail.conf $SENDMAILJAIL/usr/lib64/sasl2/
cp /etc/sasl2/Sendmail.conf $SENDMAILJAIL/etc/sasl2/
cp /usr/sbin/makemap $SENDMAILJAIL/usr/sbin/makemap
cp /usr/bin/rmail.sendmail $SENDMAILJAIL/usr/bin/rmail.sendmail
cp /usr/sbin/mailstats $SENDMAILJAIL/usr/sbin/mailstats
cp /usr/sbin/makemap $SENDMAILJAIL/usr/sbin/makemap
cp /usr/sbin/praliases $SENDMAILJAIL/usr/sbin/praliases
cp /usr/sbin/smrsh $SENDMAILJAIL/usr/sbin/smrsh

cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libcom_err.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libcrypt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libcrypto.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libdb-5.3.so $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libfreebl3.so $SENDMAILJAIL/lib64/
cp /lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libhesiod.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libidn.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3: $SENDMAILJAIL/lib64/
cp /lib64/libkeyutils.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5support.so.0 $SENDMAILJAIL/lib64/
cp /lib64/liblber-2.4.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libldap-2.4.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libnsl.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libnspr4.so $SENDMAILJAIL/lib64/
cp /lib64/libnss3.so $SENDMAILJAIL/lib64/
cp /lib64/libnssutil3.so $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libplc4.so $SENDMAILJAIL/lib64/
cp /lib64/libplds4.so $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/librt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libsasl2.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libsmime3.so $SENDMAILJAIL/lib64/
cp /lib64/libssl.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libssl3.so $SENDMAILJAIL/lib64/
cp /lib64/libwrap.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /usr/lib64/libk5crypto.so.3 $SENDMAILJAIL/usr/lib64/

cp /lib64/libdns.so.100 $SENDMAILJAIL/lib64/
cp /lib64/liblwres.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libbind9.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisccfg.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisccc.so.90 $SENDMAILJAIL/lib64/
cp /lib64/libisc.so.95 $SENDMAILJAIL/lib64/
cp /lib64/libgssapi_krb5.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libk5crypto.so.3 $SENDMAILJAIL/lib64/
cp /lib64/libcom_err.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libcrypto.so.10 $SENDMAILJAIL/lib64/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libGeoIP.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libxml2.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libm.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libidn.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libkrb5support.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libkeyutils.so.1 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/liblzma.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /bin/dig $SENDMAILJAIL/bin/

cp /lib64/libtinfo.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /bin/bash $SENDMAILJAIL/bin/

cp /bin/ls $SENDMAILJAIL/bin/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libacl.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/

cp /bin/vi $SENDMAILJAIL/bin/
cp /usr/sbin/pidof $SENDMAILJAIL/usr/sbin/pidof
cp /lib64/libprocps.so.4 $SENDMAILJAIL/lib64/
cp /lib64/libsystemd.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libdl.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libc.so.6 $SENDMAILJAIL/lib64/
cp /lib64/libcap.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libm.so.6 $SENDMAILJAIL/lib64/
cp /lib64/librt.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libselinux.so.1 $SENDMAILJAIL/lib64/
cp /lib64/liblzma.so.5 $SENDMAILJAIL/lib64/
cp /lib64/libgcrypt.so.11 $SENDMAILJAIL/lib64/
cp /lib64/libgpg-error.so.0 $SENDMAILJAIL/lib64/
cp /lib64/libdw.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libgcc_s.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpthread.so.0 $SENDMAILJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $SENDMAILJAIL/lib64/
cp /lib64/libattr.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libpcre.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libelf.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libz.so.1 $SENDMAILJAIL/lib64/
cp /lib64/libbz2.so.1 $SENDMAILJAIL/lib64/

cp /bin/rm $SENDMAILJAIL/bin/

 

Under your ID, ensure the proper permissions are set on the chroot jail

sudo chown -R sendmail:mail /smt00p20/sendmail/
sudo chown sendmail /smt00p20/sendmail/var/spool/mqueue
sudo chmod 0700 /smt00p20/sendmail/var/spool/mqueue
sudo chmod -R go-w /smt00p20/sendmail
sudo chmod 0400 /smt00p20/sendmail/etc/mail/*.cf

Then start sendmail and verify functionality.

Updating OpenDKIM

cp /lib64/libtinfo.so.5 $OPENDKIMJAIL/lib64/
cp /lib64/libdl.so.2 $OPENDKIMJAIL/lib64/
cp /lib64/libc.so.6 $OPENDKIMJAIL/lib64/
cp /lib64/ld-linux-x86-64.so.2 $OPENDKIMJAIL/lib64/
cp /bin/bash $OPENDKIMJAIL/bin/
cp /lib64/libstdc++.so.6* $OPENDKIMJAIL/lib64
cp /lib64/libm.so.6 $OPENDKIMJAIL/lib64
cp /lib64/libgcc_s.so.1 $OPENDKIMJAIL/lib64
cp /lib64/libnss_files* $OPENDKIMJAIL/lib64/

 

If there is an update to the opendkim packages, unpack the updated RPM files and move the new files into the corresponding jail locations.

rpm2cpio opendkim-2.11.0-0.1.el7.x86_64.rpm | cpio -idmv
rpm2cpio libopendkim-2.11.0-0.1.el7.x86_64.rpm | cpio -idmv
rpm2cpio sendmail-milter-8.14.7-5.el7.x86_64.rpm | cpio -idmv
rpm2cpio opendbx-1.4.6-6.el7.x86_64.rpm | cpio -idmv
rpm2cpio libmemcached-1.0.16-5.el7.x86_64.rpm | cpio -idvm
rpm2cpio libbsd-0.6.0-3.el7.elrepo.x86_64.rpm | cpio -idvm

 

Extracting RPM Packages

I’ve encountered a few scenarios of late where I couldn’t install an RPM package but needed its content. One is the security config at work where I have sudo access for cp but not install rights. Sigh! But more recently, I needed to compare a library from an updated package to the currently installed one. Listing package content confirms it is the same file name and path.

[root@fedora02 tmp]# rpm -q --filesbypkg -p ./mariadb-libs-10.2.13-2.fc27.i686.rpm
mariadb-libs              /etc/my.cnf.d/client.cnf
mariadb-libs              /usr/lib/.build-id
mariadb-libs              /usr/lib/.build-id/7c
mariadb-libs              /usr/lib/.build-id/7c/c8e65deafbdcc28b3089da60f295a6f757cf4f
mariadb-libs              /usr/lib/libmariadb.so.3

 

Extracting the rpm allowed me to actually compare the files, swap back and forth to see which worked, etc.

[lisa@fedora tmp]# rpm2cpio mariadb-libs-10.2.13-2.fc27.x86_64.rpm | cpio -idmv

Linux Authentication Over Key Exchange

On Linux, you can log in without logging in (essential for non-interactive processes that run commands on remote hosts, but also nice accessing hosts when you get paged at 2AM to look into an issue). The first thing you need is a key. You can use the openssh installation on a server to generate the key:

ssh-keygen -t rsa -b 2048

You’ll get an id_rsa and id_rsa.pub. Your private key (keep it somewhere safe) is in id_rsa; your public key is in id_rsa.pub.

Alternately you can run puttygen.exe (www.chiark.greenend.org.uk/~sgtatham/putty/download.html) for a GUI key generator. Click the “Generate” button & then move the mouse around over the blank area of the PuttyGen window – your coordinates are used as random data for the key seed.

Once the key is generated, click “save public key” and store it somewhere safe. Click “save private key” and store it somewhere safe. Copy the public key at the top of the window. You don’t have to – you can drop the newline characters from the saved public key file, but this saves time.

Either way, you’ve got a base 64 encoded public and private key.

** Key recovery isn’t a big deal – you can always generate a new public/private key pair and set it up. Time consuming if your public key is all over the place, but it isn’t a data loss kind if thing.

*** Anyone who gets your private key can log in as you anywhere you set up this key exchange. You can add a passphrase to your key for additional security.

 

Go to whatever box you want to log into using the key exchange. ** I have a key exchange set up from my Windows boxes (laptop, terminal server) to myid@jumphost. I then have a different key used from myid@jumphost to all of our other boxes. This allows me to change my on laptop key (i.e. the one more likely to get lost) out more frequently without having to get a new public key on dozens of hosts.

Once you are on the box you want as the ID you want (you can do a key exchange to any id for which you know the password – so you can log into serviceaccount@hostname or otherserviceaccount@otherhostname and do this, or you can be logged in as yourid@hostname). Run “cd ~/.ssh” – if it says no such file, run “ssh localhost” – it will ask you if you want to store the server public key – say yes, that creates the .ssh folder with proper permissions. Ctrl-c and cd ~/.ssh again. Now determine if there is an authorized_keys, authorized_keys2, or both. Vim the one you find – if there aren’t any, try “vi authorized_keys” first (authorized_keys2 on RedHat/Fedora, long story) – go into edit mode and paste in the public key line we copied earlier. Save the file. If you get an error like “The server refused our key”, you can “mv authorized_keys authorized_keys2” (or “mv authorized_keys2 authorized_keys” if you started with keys2).

In putty, load in your configuration for whatever host we just pasted the public key into. Under Connection -> Data, find the “Auto-login username” section. Put in whatever ID you used when you added the public key (my use case is me e0082643 … but if you were using ldapAdmin@hostname, you would put ldapAdmin in here)

Then under Connection ->SSH->Auth, find the “private key file for authentication” section and put in your private key location. Go back to the Session section and save the configuration changes.

Now connect & you shouldn’t need to supply a password (or you only need to supply your key passphrase).

** OpenSSH automatically uses the id_dsa or id_rsa (private keys) from ~/.ssh/ when you attempt to authenticate to other hosts. If the destination id@host has your public key in its ~/.ssh/authorized_keys (or ~/.ssh/authorized_keys2), then you’ll get magic key based authentication too. Caveat: on the source Linux host, your private key cannot be group or other readable. Run “chmod go-rw ~/.ssh/id_rsa” to ensure it is sufficiently private, otherwise auth will fail due to permissive access.

** Once you have a key exchange in place, it is fairly easy to update your key. Create a new one but do not yet replace your old one. You can make a shell script that updates all remote hosts with your new public key – per host, run:

ssh user@remoteHost “echo \”`cat ~/.ssh/new_id_rsa.pub`\” >> ~/.ssh/authorized_keys”

Once the new public key info has been pushed out, test it using “ssh -i new_id_rsa user@remoteHost” and verify the key authentication works. Once confirmed, rename your old id_rsa and id_rsa.pub files to something else. Then rename your new_id_rsa to id_rsa and new_id_rsa.pub to id_rsa.pub

Persisting Port Names For OpenHAB

If you only have one device connected to your Linux box, your controller may well always be assigned the same port when the system is rebooted. In the real world, multiple connected devices make this unlikely. We use udev rules to create symlinks used within the OpenHAB configuration. The udev rule essentially uses attributes of the device to identify the real port and creates a statically named symlink to that port on boot. So the first thing you need to identify is something unique about the device. We have several video capture cards and a Z-Wave/ZigBee combo controller attached to our server. If you do not have udevinfo, you can use lsusb to find details about the device. First list the devices, then identify the proper one and use the -d switch with the ????:???? formatted ID number. The -v switch outputs verbose information. Find a unique attribute or set of attributes that create a unique identifier. In this case, the interface name is unique and we can stop there.

[lisa@server ~]#  lsusb
Bus 001 Device 004: ID 0bda:0111 Realtek Semiconductor Corp. RTS5111 Card Reader Controller
Bus 001 Device 003: ID 1058:1230 Western Digital Technologies, Inc. My Book (WDBFJK0030HBK)
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 10c4:8a2a Cygnal Integrated Products, Inc.
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
[lisa@server ~]# lsusb -d 10c4:8a2a -v
Bus 002 Device 002: ID 10c4:8a2a Cygnal Integrated Products, Inc.
Device Descriptor:
 bLength 18
 bDescriptorType 1
 bcdUSB 2.00
 bDeviceClass 0
 bDeviceSubClass 0
 bDeviceProtocol 0
 bMaxPacketSize0 64
 idVendor 0x10c4 Cygnal Integrated Products, Inc.
 idProduct 0x8a2a
 bcdDevice 1.00
 iManufacturer 1 Silicon Labs
 iProduct 2 HubZ Smart Home Controller
 iSerial 5 90F0016B
 bNumConfigurations 1
 Configuration Descriptor:
 bLength 9
 bDescriptorType 2
 wTotalLength 55
 bNumInterfaces 2
 bConfigurationValue 1
 iConfiguration 0
 bmAttributes 0x80
 (Bus Powered)
 MaxPower 100mA
 Interface Descriptor:
 bLength 9
 bDescriptorType 4
 bInterfaceNumber 0
 bAlternateSetting 0
 bNumEndpoints 2
 bInterfaceClass 255 Vendor Specific Class
 bInterfaceSubClass 0
 bInterfaceProtocol 0
 iInterface 3 HubZ Z-Wave Com Port
 Endpoint Descriptor:
 bLength 7
 bDescriptorType 5
 bEndpointAddress 0x81 EP 1 IN
 bmAttributes 2
 Transfer Type Bulk
 Synch Type None
 Usage Type Data
 wMaxPacketSize 0x0040 1x 64 bytes
 bInterval 0
 Endpoint Descriptor:
 bLength 7
 bDescriptorType 5
 bEndpointAddress 0x01 EP 1 OUT
 bmAttributes 2
 Transfer Type Bulk
 Synch Type None
 Usage Type Data
 wMaxPacketSize 0x0040 1x 64 bytes
 bInterval 0
 Interface Descriptor:
 bLength 9
 bDescriptorType 4
 bInterfaceNumber 1
 bAlternateSetting 0
 bNumEndpoints 2
 bInterfaceClass 255 Vendor Specific Class
 bInterfaceSubClass 0
 bInterfaceProtocol 0
 iInterface 4 HubZ ZigBee Com Port
 Endpoint Descriptor:
 bLength 7
 bDescriptorType 5
 bEndpointAddress 0x82 EP 2 IN
 bmAttributes 2
 Transfer Type Bulk
 Synch Type None
 Usage Type Data
 wMaxPacketSize 0x0020 1x 32 bytes
 bInterval 0
 Endpoint Descriptor:
 bLength 7
 bDescriptorType 5
 bEndpointAddress 0x02 EP 2 OUT
 bmAttributes 2
 Transfer Type Bulk
 Synch Type None
 Usage Type Data
 wMaxPacketSize 0x0020 1x 32 bytes
 bInterval 0

We then need to create a file under /etc/udev/rules.d. The file name begins with a number that is used for a load order – I generally number my custom files 99 to avoid interfering with system operations. The bit in “ATTRS” is the attribute name and value to match. The KERNEL section contains the search domain (i.e. look at all of the ttyUSB### devices and find ones where the interface is this). The symlink bit is the name you want to use (more on this later). Set the group and mode to ensure OpenHAB is able to use the symlink.

[root@fedora01 rules.d]# cat 99-server.rules
KERNEL=="ttyUSB[0-9]*", ATTRS{interface}=="HubZ Z-Wave Com Port", SYMLINK+="ttyUSB-5", GROUP="dialout", MODE="0666"
KERNEL=="ttyUSB[0-9]*", ATTRS{interface}=="HubZ ZigBee Com Port", SYMLINK+="ttyUSB-55", GROUP="dialout", MODE="0666"

The symlink name can be anything – when I created udev rules for our video capture cards, I named them something immediately obvious: video-hauppauge250 is the Hauppauge 250 card. Tried to do the same thing here, naming the ports controller-zigbee; while the symlink appeared and had the expected ownership and permissions … OpenHAB couldn’t use it.

Turns out there’s a nuance to Java RxTx where non-standard port names need to be accommodated in the java options. So I could have added -Dgnu.io.rxtxSerialPorts=/dev/controller-zigbee and -Dgnu.io.rxtxSerialPorts=/dev/controller-zwave to the OpenHAB startup and been OK (in theory), it was far easier to name the symlinks using the Linux standard conventions. Hence I have symlinks named ttyUSBsomethingsomethingsomething. 

Linux Primer

We’ve got a few new people at work who don’t have any Linux experience, and I was asked to do a quick crash course on some super fundamental logging in / navigating / restarting service stuff so their first on call rotation wouldn’t be quite so stressful. Publishing the overview here in case it is useful for anyone else.

Linux Primer:

Connecting – We use both putty and Cygwin to connect to our Linux hosts via SSH (secure socket shell). Each has its own advantages and disadvantages – try them both and see which you prefer. If you need X redirection (you need the GUI ‘stuff’ to magic itself onto your computer), use Cygwin-X.

Logging In – Our Linux hosts authenticate users via cusoldap.windstream.com, so (assuming you are set up for access to the specific host) you will use your CSO userID and password to log in.

  • We often use a jump box – log into the jump box with your ID or using a key exchange. From there, we have key exchanges with our other boxes that allow us to connect without entering credentials again.
  • You can set up key exchanges on your own ID too – even from your Windows desktop – and avoid typing passwords.

Once you are logged in, you can start a screen session. Normally, anything you are running is terminated if your SSH session terminates (e.g. if you use Cygwin or Putty to connect to a box from your laptop that is VPN’d into the network & your VPN drops … everything you were doing in the SSH session is terminated.). You can use screen to set up a persistent session – you can reconnect to that session should your SSH connection get interrupted, other people can connect to the session to monitor a long running script, or multiple people can connect to the session and all see the same thing (screen sharing).

To start a new screen session, screen -S SessionName where SessionName is something that identifies the screen session as yours (e.g. LJRPasswordResync was the session I used when resyncing all employee and contractor passwords for OIDM – this includes both my initials and the function I’m running in the session). To see the currently running sessions, use screen –ls

[lisa@server810 ~]# screen -ls

There is a screen on:

8210.LJR        (Detached)

1 Socket in /tmp/screens/S-lisa.

The output contains both a session ID number (green) and a session name (blue) separated by a full stop. You can use either to connect to a screen session (the name is case sensitive!). To reconnect, use screen –x SessionName or screen –x SessionID

To determine if you are currently in a screen session, look at the upper left hand corner of your Putty window. The title will change to include screen when you are in a screen session. Alternately echo the STY environment variable. If you get nothing, it is not a screen session. If you get output, it is the PID and name of your current screen session.

[lisa@server810 ~]# echo $STY
43116.LJR

SUDO – The sudo command lets you execute commands that your ID is not normally privileged to run. There is configuration to sudo (maintained by ITSecurity) that defines what you can run through sudo. If, for example, you are unable to edit a file but are permitted to sudo vim … editing a file using “vi /path/to/file.xtn” will throw an error if you attempt to save changes, but running “sudo vi /path/to/file.xtn” would allow you to save changes to the file.

Substitute user – The command su lets you substitute a uidnumber for yours – this means you become that user.

Combining SUDO and SU – Once we are logged into LX810 with our user ID, we can use sudo su – root to become root without actually knowing the root password. The “space dash space” in the su command means the user’s environment is loaded. If you omit the space dash space, you’ll still be logged in as the root user, but your environment will be left in place.

Generally speaking, allowing sudo to root is a bad idea (i.e. don’t do this even though you’ll see it on a lot of our old servers). This is because root has full access to everything and running the shell as root is insecure and typos can be disastrous.

Navigating – You are in a DOS-like command line interface. The interface is known as a shell – root on LX810 is a bash shell. The default for a CUSO ID is the korn shell (/bin/ksh) – you can change your shell in your LDAP account to /bin/bash (or /bin/csh for the C shell) and subsequent logons will use the new shell. You can try each one and see which you prefer, you can use korn because it is the default from CUSO, or you can use bash because it matches the instructions I write.

From a file system navigation perspective, you will be in the logon user’s home directory. If you aren’t sure where you are in the file system, type pwd and the present working directory will be output.

To see what is in a directory, use ls … there are additional parameters you can pass (in Linux parameters are passed with a dash or two dashes). Adding -a lists *all* files (including the hidden ones, any file where the name starts with a full stop is a hidden file). Adding -l provides a long listing (file owners, sizes, modified dates). Adding -h lists file sizes in human readable format. You can pass each parameter separately (ls –a –l –h) or by concatenating them together (ls –alh)

You can use wc to count the number of lines either in a file (wc –l /path/to/file.xtn) or the output of ls (ls –al | wc –l) – this is useful on our sendmail servers when you have received a queue length alert and done something to clear out some of the queue. In sendmail particularly, there are two files for each message so you need to divide the line count by 2.

To change to a different directory, use cd – e.g. cd /etc/mail will change the working directory to /etc/mail.

To delete a file, use rm /path/to/file.xtn – this is the safe way to run it, it will prompt for confirmation for each file being deleted. You can use wildcards (rm /path/to/files*) to delete multiple files. You can add a -f parameter to not be prompted – which is more dangerous as you may have typed the wrong thing and it’ll be deleted without prompting. You can add a –r parameter for recursive (get rid of everything under a path). Not too dangerous as long as you have the prompt coming up – but if you use –r in conjunction with –f (rm –rf) … you can do a lot of damage. Absolute worst case would be recursive force delete from / … which would mean every file on disk goes away. Don’t do that J

If you are not sure where a file you need is located, you can use either find or locate. The locate command is not always installed, so you would need to use the find command. Locate uses an index database – so it’s quicker, but it doesn’t know about files created/deleted since the index was updated.

To use locate, use locate -i filename where filename is some part of the filename. The -i performs a case insensitive search – if you know the proper casing, you do not need to include this parameter.

To use find, you need to indicate the root of the search (if you have no clue, use ‘/’ which is the top level directory) as well as the exact file name that you want (not a substring of the file name like locate will let you do). Finding a file named audit.log that is somewhere on the disk would be find / -name audit.log

Customizing shell environment – You can customize your shell environment. The system-wide shell environment settings are in /etc and are specific to the shell. For a bash shell, it is /etc/bashrc

Individual user settings are in a hidden file within their home directory. For the bash shell, the user specific settings are in $HOME/.bashrc ($HOME is a variable for the current logon user’s home directory).

For a shared account, adding things to $HOME/.bashrc isn’t the best idea – your preferred settings can differ from someone else’s preferences. We make our own rc file in $HOME for the shared account (I actually set my .bashrc as world-readable and linked the shared ID $HOME/.ljlrc to my personal .bashrc file so I only have to remember to edit one file). You can load your personal preferences using source $HOME/.yourrc or you can load someone else’s preferences by sourcing their file in the shared account’s home directory (source $HOME/.ljlrc will load in mine).

Service Control – Most of our Linux systems still use systemd (init.d scripts) to start and stop services. You can find the scripts in /etc/init.d – these are readable text scripts. All scripts will have a start and stop command, and many have restart and status as additional commands. To control a service, you can use service servicename command, /sbin/service servicename command or /etc/init.d/servicename command – same thing either way. If you are controlling the service through sudo, though, you need to use the technique that is permitted to your UID in the sudo configuration.

If you use a command that isn’t implemented in the script, you will get usage information. You can use a semicolon to chain commands (like the & operator in DOS) – so /etc/init.d/sendmail restart is the same thing as running /etc/init.d/sendmail stop;/etc/init.d/sendmail start

Process utilization – To see what the processor and memory utilization is like on a box (as well as which processes are causing this utilization), use top. When top has launched, the first few lines give you the overall usage. The load average (blue below) tells you the load during the last one, five, and fifteen minutes – 1.00 is 100% on a single core system, 2.00 is 100% on a two core system, etc. Over the 100% number for a system means stuff got queued waiting for CPU cycles to become available.

The current CPU utilization (green below) breaks out usage by user tasks, system tasks, nice’d processes (generally nothing here), idle, io wait, hardware irq, software irq.

The memory usage (red below) shows used and free memory.

top – 13:58:30 up 486 days,  2:16,  9 users,  load average: 0.34, 0.24, 0.25

Tasks: 162 total,   1 running, 161 sleeping,   0 stopped,   0 zombie

Cpu(s):  0.4% us,  0.1% sy,  0.0% ni, 99.5% id,  0.0% wa,  0.0% hi,  0.0% si

Mem:   4147208k total,  2107876k used,  2039332k free,    62372k buffers

Swap:  2064376k total,     1352k used,  2063024k free,  1167652k cached

 

The process list can be sorted by whatever you need – if the box is CPU-bound, type an upper case C to sort by CPU usage. If it is memory bound, type an upper case M to sort by memory usage.

PID USER      PR  NI %CPU    TIME+  %MEM  VIRT  RES  SHR S COMMAND

23190 root      15   0    1   5:43.81 14.9  608m 605m 2872 S perl

14225 root      16   0    0   7:14.20  1.7  170m  69m  60m S cvd

14226 root      16   0    0   1:30.32  1.4  147m  57m  50m S EvMgrC

4585 root      16   0    0 212:01.99  1.1  230m  43m 6368 S dsm_om_connsvc3

4003 root      16   0    0   2729:44  0.6  171m  24m 3364 S dsm_sa_datamgr3

24552 root      16   0   13   0:36.16  0.3 17804  12m 2900 S perl

 

The first column shows the PID (process ID). Some commands as listed in top are obvious what they actually are (httpd is the apache web server, for instance) and others aren’t (perl, above, doesn’t really tell us *what* is using the CPU). To determine what the PID actually is, use ps –efww | grep PID#

[lisa@server810 Sendmail-CheckQSize]# ps -efww | grep 23190

root     23190 23187  0 01:23 ?        00:05:44 /usr/bin/perl /home/NDSSupport/Scripts/osrOCSProvisioning/_syncIMEnabledFromCSO.pl

root     24645 16640  0 14:10 pts/10   00:00:00 grep 23190

 

You will see the full command that is running – in this case a particular perl script. Note that you may also find your grep command in the list … depends a bit on timing if it shows up or not.

You may need to restart a service to clear something that has a memory leak. You may need to stop the process outside of the service control (e.g. stopping the sendmail service doesn’t shut down all current threads). To stop a process, use kill PID# … this is basically asking a process nicely to stop. It will clean up its resources and shut down cleanly. use ps –efww to see if the process is still running. If it still is, use kill -9 PID# which is not asking nicely. Most things to which a process is connected will clean up their own resources after some period of client inactivity (i.e. you aren’t causing a huge number of problems for someone else by doing this) but it is cleaner to use kill without the “do it NOW!!!” option first.

Tail and Grep – Tail is a command that outputs the last n lines of a file. It has a parameter that outputs new lines as they get appended to the file. On *n?x systems, you can use tail –F /path/to/file.xtn and lines will be output as they show up. This is particularly useful on log files where the system is continually adding new info at the bottom of the file. We put Windows ports of these utilities on our Windows servers – but the Windows port of tail does not support –F (there’s a good reason that has to do with the difference between Unix-like and Windows file systems). You can use tail –f instead – if the log file rolls (gets moved to another file and a new file is started) you won’t continue to get output like you will with –F … but you can ctrl-c to terminate the tail & start it again to start seeing the new lines again.

Grep is a command line search program. You can use grep to find lines in a file containing a string (or regex pattern, but learning regex is a question for LMGTFY.com) – to find all of the mail addressed to or from me in a sendmail log, grep –i rushworth /var/log/maillog – the dash i means case insensitive search.

Grep will also search piped input instead of a file – this means you can send the output of tail to grep and display only the lines matching the pattern for which you search.

tail -f /var/log/maillog | grep –i rushworth will output new lines of the maillog as they come in, but only display the ones with my name.

VIM – The non-visual text editor is vim – which is usually invoked using ‘vi’, but vi is an actual program that is like but not exactly the same as vim (vim is improved vi). The vim installation contains a very nice tutorial – invoked by running vimtutor

VIM has both a command mode and an editing mode. When in command mode, different keys on the keyboard have different functions. There are “quick reference” guides and “cheat sheets” online for vim – most people I know have a quick ref guide or cheat sheet taped next to their computer for quite some time before vim commands become well known.

History – Linux maintains a history of commands run in a session. This spans logons (you’ll see commands run last week even through you’ve logged on and off six times between then) but when there are multiple sessions for the same user, there can be multiple history files. Which is all a way of saying you may not see something you expect to see, or you may see things you don’t expect. The output of history shows the command history for the current logon session. You can pipe the output to grep and find commands in the history – for example, if you don’t remember how to start a service, you can use history | grep start and get all commands that contain the string start

[lisa@server855 ~]# history | grep start

7  service ibmslapd start

15  service ibmslapd restart

42  service ibmslapd start

56  service ibmslapd restart

71  service ibmslapd start

95  service ibmslapd start

107  service ibmslapd start

115  service ibmslapd restart

289  service ibmslapd start

303  service ibmslapd start

408  service ibmslapd start

419  service ibmslapd start

430  service ibmslapd start

443  service ibmslapd start

If a command fails, it will still be in the history (all of my typo’s are in there!), but if you see the same command a number of times … it’s probably correct. You can copy/paste the command if you need to edit it to run (or even to run it as-is). You can run the exact command again by typing bang followed by the line number of the history output (!115 with the history above would re-run “service ibmslapd restart”).

Symbolic Links

Linux symbolic links are nothing like Windows shortcuts, although I see people saying that. Shortcuts are independent files that contain a path to the referenced file. Linux sym links are just pointers to the inode for the file. They are the file, just allowing it to be used in a different location. This is a bit like memory addressing in programming — anything that reads from the memory address will get the same data, and anything that writes to the memory address. When you do a long list (ls -al or just ll), you will see both the file name and the file to which it points:

lrwxrwxrwx 1 root root 19 Aug 17 13:54 ljrtest -> /tmp/dnsexit-ip.txt

The “l” at the start of the line indicates that it is a link.