Author: Lisa

DevOps Alternatives

While many people involved in the tech industry have a wide range of experience in technologies and are interested in expanding the breadth of that knowledge, they do not have the depth of knowledge that a dedicated Unix support person, a dedicated Oracle DBA, a dedicated SAN engineer person has. How much time can a development team reasonably dedicate to expanding the depth of their developer’s knowledge? Is a developer’s time well spent troubleshooting user issues? That’s something that makes the DevOps methodology a bit confusing to me. Most developers I know … while they may complain (loudly) about unresponsive operational support teams, about poor user support troubleshooting skills … they don’t want to spend half of their day diagnosing server issues and walking users through basic how-to’s.

The DevOps methodology reminds me a lot of GTE Wireline’s desktop and server support structure. Individual verticals had their own desktop support workforce. Groups with their own desktop support engineer didn’t share a desktop support person with 1,500 other employees in the region. Their tickets didn’t sit in a queue whilst the desktop tech sorted issues for three other groups. Their desktop support tech fixed problems for their group of 100 people. This meant problems were generally resolved quickly, and some money was saved in reduced downtime. *But* it wasn’t like downtime avoidance funded the tech’s salary. The business, and the department, decided to spend money for rapid problem resolution. Some groups didn’t want to spend money on dedicated desktop support, and they relied on corporate IT. Hell, the techs employed by individual business units relied on corporate IT for escalation support. I’ve seen server support managed the same way — the call center employed techs to manage the IVR and telephony system. The IVR is malfunctioning, you don’t put a ticket in a queue with the Unix support group and wait. You get the call center technologies support person to look at it NOW. The added advantage of working closely with a specific group is that you got to know how they worked, and could recommend and customize technologies based on their specific needs. An IM platform that allowed supervisors and resource management teams to initiate messages and call center reps to respond to messages. System usage reporting to ensure individuals were online and working during their prescribed times.

Thing is, the “proof” I see offered is how quickly new code can be deployed using DevOps methodologies. Comparing time-to-market for automated builds, testing, and deployment to manual processes in no way substantiates that DevOps is a highly efficient way to go about development. It shows me that automated processes that don’t involve waiting for someone to get around to doing it are quick, efficient, and generally reduce errors. Could similar efficiency be gained by having operation teams adopt automated processes?

Thing is, there was a down-side to having the major accounts technical support team in PA employ a desktop support technician. The major accounts technical support did not have broken computers forty hours a week. But they wanted someone available from … well, I think it was like 6AM to 10PM and they employed a handful of part time techs, but point remains they paid someone to sit around and wait for a computer to break. Their techs tended to be younger people going to school for IT. One sales pitch of the position, beyond on-the-job experience was that you could use your free time to study. Company saw it as an investment – we get a loyal employee with a B.S. in IT who moved into other orgs, college kid gets some resume-building experience, a chance to network with other support teams, and a LOT of study time that the local fast food joint didn’t offer. The access design engineering department hired a desktop tech who knew the Unix-based proprietary graphic workstations they used within the group. She also maintained their access design engineering servers. She was busier than the major accounts support techs, but even with server and desktop support she had technical development time.

Within the IT org, we had desktop support people who were nearly maxed out. By design — otherwise we were paying someone to sit around and do nothing. Pretty much the same methodology that went into staffing the call center — we might only expect two calls overnight, but we’d still employ two people to staff the phones between 10P and 6A *just* so we could say we had a 24×7 tech support line. During the day? We certainly wouldn’t hire two hundred people to handle one hundred’s worth of calls. Wouldn’t operations teams be quicker to turn around requests if they were massively overstaffed?

As a pure reporting change, where you’ve got developers and operations people who just report through the same structure to ensure priorities and goals align … reasonable. Not cost effective, but it’s a valid business decision. In a way, though, DevOps as a vogue ideology is the impetus behind financial decisions (just hire more people) and methodology changes (automate it!) that would likely have similar efficacy if implemented in silo’d verticals.

 

The Spoils System

How a company, political organization, or non-profit spends their money doesn’t bother me that much — it’s a factor I consider when donating to “the cause” or purchasing from a company. I can appreciate that one feels betrayed when, say, some of your University tuition is used to pay off women and pizza delivery people who have been attacked by the school’s hockey team, thus suppressing reporting and criminal charges (then bragging to prospective students about the ZERO on-campus crime rate). Or the quasi-celebrity fronted charity uses donations to settle legal disputes involving the quasi-celebrity. Or your political party offers a decent salary to someone they simultaneously claim is a terrible employee who needed to be fired. *But* those were all choices you made. And you’re free to make new choices next time around based on new information. I can even see how it would really suck to believe the Republican party has the right way of things, and you *want* to provide financial support, but you also have to accept that some percent of your contributions are going to salary for unqualified individuals who they want to keep quiet.

What bothers *me* is that Trump has essentially admitted he handed a 179,700$ per year job to an utterly unqualified individual because of their personal relationship.

*I* didn’t chose the grifter-in-chief, but it’s still my tax money going to his friends (at that, to his “racist dude’s obligatory non-white, I cannot be a racist because Bob’s my friend” friend … although there are *whole binders full* of unqualified old white dudes grifting in Trump’s administration too.)

Our tax money going to political supporters who are unqualified for the job? That’s why we’ve got the Pendleton Act and the subsequent civil service system — every politician could find somewhere for their friends and supporters to suck up taxpayer money for a few years. And the inverse: a politician could garner key supporters by offering them cushy government positions. And an unlucky politician could garner a supporter who considered himself key, anger the chap by not giving him one of those cushy jobs, and get shot. Maybe it’s time to eliminate the remaining “at the pleasure of the president” job appointments.

openHAB With Custom Built Serial Binding – fix to locking permission issue

When we updated our openHAB server to Fedora 28 and changed to a non-root user, the openhab user was unable to create lock files in /run/lock. As an interim fix, we just changed the permission on the lock folder to allow the openhab account to create files. As a more elegant solution, I’ve built the nrjavaserial JAR file from the source in NeuronRobotics’ repository.

The process to build and use a JAR built from this source follows. Before attempting to build the nrjavaserial jar from source, ensure you have gradle (which will install a LOT of additional packages), lockdev, lockdev-devel, some jdk, and some jdk-devel (I used java-1.8.0-openjdk-1.8.0.181-7.b13.fc28.x86_64 and java-1.8.0-openjdk-devel-1.8.0.181-7.b13.fc28.x86_64 because they were already installed for other projects).

# Set ossrhUsername and ossrhPassword values for the account used to build the project – username and password can be null
[lisa@server ~]# cat ~/.gradle/gradle.properties
ossrhUsername=
ossrhPassword=

# Grab the source
[lisa@server ~]# git clone https://github.com/NeuronRobotics/nrjavaserial.git

# Build the project
[lisa@server ~]# cd nrjavaserial
[lisa@server nrjavaserial]# make linux64 # assuming you’ve got 64-bit linux

# Voila, a jar file
[lisa@server nrjavaserial]# cd build/libs
[lisa@server libs]# ll
total 852
-rw-r–r– 1 root root 611694 Aug 16 10:08 nrjavaserial-3.14.0.jar
-rw-r–r– 1 root root 170546 Aug 16 10:08 nrjavaserial-3.14.0-javadoc.jar
-rw-r–r– 1 root root 85833 Aug 16 10:08 nrjavaserial-3.14.0-sources.jar

Before installing the newly built nrjavaserial-3.14.0.jar into openHAB, ensure you have lockdev installed on your Fedora machine and add your openhab user account to the lock group.

# Verify the lockdev folder was created
[lisa@server ~]# ll /run/lock/
total 4
-rw-r–r– 1 root root 22 Aug 10 15:35 asound.state.lock
drwx—— 2 root root 60 Aug 10 15:30 iscsi
drwxrwxr-x 2 root lock 140 Aug 16 12:19 lockdev
drwx—— 2 root root 40 Aug 10 15:30 lvm
drwxr-xr-x 2 root root 40 Aug 10 15:30 ppp
drwxr-xr-x 2 root root 40 Aug 10 15:30 subsys
# Add the openhab user to the lock group
[lisa@server ~]# usermod -a -G lock openhab

The openhab user account can now write to the /run/lock/lockdev folder. Install the new jar file into openHAB. When you restart openHAB, verify lock files are created as expected.
[lisa@server ~]# ll /run/lock/lockdev/
total 20
-rw-rw-r– 5 openhab openhab 11 Aug 16 12:19 LCK…31525
-rw-rw-r– 5 openhab openhab 11 Aug 16 12:19 LCK..ttyUSB-5
-rw-rw-r– 5 openhab openhab 11 Aug 16 12:19 LCK..ttyUSB-55
-rw-rw-r– 5 openhab openhab 11 Aug 16 12:19 LK.000.188.000
-rw-rw-r– 5 openhab openhab 11 Aug 16 12:19 LK.000.188.001

 

Zoneminder Snapshot With openHAB Binding

When we upgraded to Fedora 28 on our server, ZoneMinder ceased working because some CakePHP function names could no longer be used. To resolve the issue, I ended up running a snapshot build of ZoneMinder that included a newer build of CakePHP. Version 1.31.45 instead of 1.30.4-7 on the repository.

All of our cameras showed up, and although the ZoneMinder folks seem to have a bug in their SQL query when building out the table of event counts on the main page (that is, all of my monitors have blank instead of event counts and my apache log is filled with

[Wed Aug 15 12:08:37.152933 2018] [php7:notice] [pid 32496] [client 10.5.5.234:14705] ERR [SQL-ERR 'SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'and E.MonitorId = '13' ),1,NULL)) as EventCount1, count(if(1 and (  and E.Monito' at line 1', statement was 'select count(if(1 and ( E.MonitorId = '13' ),1,NULL)) as EventCount0, count(if(1 and (  and E.MonitorId = '13' ),1,NULL)) as EventCount1, count(if(1 and (  and E.MonitorId = '13' ),1,NULL)) as EventCount2, count(if(1 and (  and E.MonitorId = '13' ),1,NULL)) as EventCount3, count(if(1 and (  and E.MonitorId = '13' ),1,NULL)) as EventCount4, count(if(1 and (  and E.MonitorId = '13' ),1,NULL)) as EventCount5 from Events as E where MonitorId = ?' params:13]

… it works.

Until Scott checked openHAB, where all of the items are offline. Apparently the openHAB ZoneMinder binding is using the cgi-bin stuff to get the value of ZM_PATH_ZMS. A config option which was removed from the database as part of the upgrade process.

Upgrading database to version 1.31.1
Loading config from DBNo option 'ZM_DIR_EVENTS' found, removing.
No option 'ZM_DIR_IMAGES' found, removing.
No option 'ZM_DIR_SOUNDS' found, removing.
No option 'ZM_FRAME_SOCKET_SIZE' found, removing.
No option 'ZM_OPT_FRAME_SERVER' found, removing.
No option 'ZM_PATH_ARP' found, removing.
No option 'ZM_PATH_LOGS' found, removing.
No option 'ZM_PATH_MAP' found, removing.
No option 'ZM_PATH_SOCKS' found, removing.
No option 'ZM_PATH_SWAP' found, removing.
No option 'ZM_PATH_ZMS' found, removing.
 207 entries
Saving config to DB 207 entries
Upgrading DB to 1.30.4 from 1.30.3

The calls from openHAB yield 404 errors in the access_log

10.0.0.5 - - [15/Aug/2018:09:38:04 -0400] "GET /zm/api/configs/view/ZM_PATH_ZMS.json HTTP/1.1" 404 1751 "-" "Jetty/9.3.21.v20170918"

 

Unfortunately they’ve changed the URL to get these values — it’s “munged” from the config file as the parameters are no longer stored to the Config table.
http://zoneminder.domain.ccTLD/zm/api/configs/view/ZM_PATH_ZMS.json
is now
http://zoneminder.domain.ccTLD/zm/api/configs/viewByName/ZM_PATH_ZMS.json

So … that’s a problem!

Running OpenHAB2 As Non-Root User — With USB

I’ll prefix this saga with the fact my sad story is implementation specific (i.e. relevant to those using Fedora, RHEL, or CentOS). I know Ubuntu has its own history with handling locks, and I’m sure other distros do as well. But I don’t know the history there, nor do I know how they currently manage locking.

We switched our openHAB installation to use a systemd unit file to run as a service and changed the execution to a non-root user. Since we knew the openhab service account needed to be a member of dialout and tty, and we’d set the account up properly, we expected everything would work beautifully.

Aaaand … neither ZWave for ZigBee came online. Not because it couldn’t access the USB devices, but because the non-root user could not lock the USB devices. From journalctl, we see LOTS of error messages that are not reflected in openHAB:

-- Logs begin at Sun 2017-04-30 14:28:12 EDT, end at Sun 2018-08-12 19:10:32 EDT. --
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyUSB-55: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: [34B blob data]
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyUSB-5: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: [34B blob data]
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyUSB1: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: [34B blob data]
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyUSB0: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: [34B blob data]
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyS31: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: [34B blob data]
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyS30: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: testRead() Lock file failed
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyS29: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: testRead() Lock file failed
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyS28: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: testRead() Lock file failed
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyS27: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: testRead() Lock file failed
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyS26: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: testRead() Lock file failed
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyS25: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: testRead() Lock file failed
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: check_group_uucp(): error testing lock file creation Error details:Permission deniedcheck_lock_status: No permission to create lock fi>
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: RXTX fhs_lock() Error: opening lock file: /var/lock/LCK..ttyS24: Permission denied. FAILED TO OPEN: No such file or directory
Aug 12 18:36:19 server.domain.ccTLD start.sh[7448]: testRead() Lock file failed

And now my old-school Linux/Unix knowledge totally screws me over — I expected a uucp group with write access to /run/lock. Except … there’s no such group. Evidently in RHEL 7.2, they started using a group named lock with permission to /var/lock to differentiate between serial devices (owned by uucp) and lock files. Nice bit of history, that, but Fedora and RedHat don’t do that anymore either.

Having a group with write permission was deemed a latent privilege escalation vulnerability, and they played around with having a lockdev binary writing files to /run/lock/lockdev, the creation and configuration of lockdev was moved into systemd, and then removed from systemd in favor of approaches [flock(), for instance].

RXTX has a hard-coded path based on OS version — that is what is used to create the lock file. And as the /run/lock folder is writable only by the owner, root … that is what is failing.

#if defined(__linux__)
/*
	This is a small hack to get mark and space parity working on older systems
	https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=147533
*/
#	if !defined(CMSPAR)
#		define CMSPAR 010000000000
#	endif /* CMSPAR */
#	
#	define DEVICEDIR "/dev/"
#	define LOCKDIR "/var/lock"
#	define LOCKFILEPREFIX "LCK.."
#	define FHS
#endif /* __linux__ */

Which is odd because I see a few threads about how nrjavaserial has been updated and as soon as the newer nrjavaserial gets bundled into the application, locking will all be sorted. And there’s an open issue for exactly the problem we are having … which explains why I’m not seeing something different in their source code. Digging around more, it looks like they didn’t actually change the hardcoded paths but rather added support for liblockdev. Which prompted my hypothesis that simply installing the lockdev package would magically sort the issue. It did not.

In the interim, though, we can just add write permission for /run/lock thorough the config file /usr/lib/tmpfiles.d/legacy.conf — the distro creates the lock directory owned by root:root. Original config lines:

d /run/lock 0755 root root -
L /var/lock - - - - ../run/lock

We can create the folder as owned by the lock group group and add group write permissions (realizing that creates the potential for privilege escalation attacks). Updated config lines:

#d /run/lock 0755 root root -
d /run/lock 0775 root lock -
L /var/lock - - - - ../run/lock

Adding the openhab account to the lock group allows the LCK.. files to be created.

[lisa@server run]# usermod -a -G lock openhab
[lisa@server run]# id openhab
uid=964(openhab) gid=963(openhab) groups=963(openhab),5(tty),18(dialout),54(lock)

Either reboot to reprocess legacy.conf or manually change the ownership & permissions on /run/lock. Either way, confirm that the changes are successful.

[lisa@server run]# chown root:lock /run/lock
[lisa@server run]# chmod g+w lock
[lisa@server lock]# ll /run | grep lock
drwxrwxr-x  7 root           lock             200 Aug 13 14:03 lock

If you manually set the permissions, restart openHAB. Our devices are online, and we have lock files:

[lisa@seerver lock]# ll
total 12
-rw-r--r-- 1 root root 22 Aug 10 15:35 asound.state.lock
drwx------ 2 root root 60 Aug 10 15:30 iscsi
-rw-r--r-- 1 openhab openhab 11 Aug 13 14:03 LCK..ttyUSB-5
-rw-r--r-- 1 openhab openhab 11 Aug 13 14:03 LCK..ttyUSB-55
drwxrwxr-x 2 root lock 40 Aug 10 15:30 lockdev
drwx------ 2 root root 40 Aug 10 15:30 lvm
drwxr-xr-x 2 root root 40 Aug 10 15:30 ppp
drwxr-xr-x 2 root root 40 Aug 10 15:30 subsys

 

Testing udev Rules

The first step of testing a udev rule is to determine the actual device you want to test. Get the info for the /dev/thing and find the real /device/path/…. (note, make sure you’re not in a “looking at parent” section — you want the one all the way at the top)

[lisa@linuxhost dev]# udevadm info -a /dev/ttyUSB0 | more
...
looking at device '/devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/ttyUSB0/tty/ttyUSB0':
...

Once you have the device, use udevadm in test mode and you will see the results from all of your udev rules. Including group & permission mask applied to the device.

[lisa@linuxhost dev]# udevadm test /devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/ttyUSB0/tty/ttyUSB0 | more
calling: test
version 238
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
...
47437 strings (385590 bytes), 43385 de-duplicated (347383 bytes), 4053 trie nodes used
PROGRAM 'usb_modeswitch --symlink-name /devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/ttyUSB0/tty/ttyUSB0 10c4 8a2a ' /usr/lib/udev/rules.d/40-usb_modeswitch.rules:10
starting 'usb_modeswitch --symlink-name /devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/ttyUSB0/tty/ttyUSB0 10c4 8a2a '
Process 'usb_modeswitch --symlink-name /devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/ttyUSB0/tty/ttyUSB0 10c4 8a2a ' succeeded.
GROUP 18 /usr/lib/udev/rules.d/50-udev-default.rules:25
IMPORT builtin 'hwdb' /usr/lib/udev/rules.d/60-serial.rules:7
IMPORT builtin 'usb_id' /usr/lib/udev/rules.d/60-serial.rules:8
/sys/devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0: if_class 255 protocol 0
IMPORT builtin 'hwdb' /usr/lib/udev/rules.d/60-serial.rules:8
IMPORT builtin 'path_id' /usr/lib/udev/rules.d/60-serial.rules:15
LINK 'serial/by-path/pci-0000:00:02.0-usb-0:1:1.0-port0' /usr/lib/udev/rules.d/60-serial.rules:17
IMPORT builtin skip 'usb_id' /usr/lib/udev/rules.d/60-serial.rules:19
LINK 'serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_90F0016B-if00-port0' /usr/lib/udev/rules.d/60-serial.rules:24
GROUP 18 /etc/udev/rules.d/99-server.rules:5
MODE 0666 /etc/udev/rules.d/99-server.rules:5
LINK 'ttyUSB-5' /etc/udev/rules.d/99-server.rules:5
handling device node '/dev/ttyUSB0', devnum=c188:0, mode=0666, uid=0, gid=18
preserve permissions /dev/ttyUSB0, 020666, uid=0, gid=18
preserve already existing symlink '/dev/char/188:0' to '../ttyUSB0'
found 'c188:0' claiming '/run/udev/links/\x2fserial\x2fby-id\x2fusb-Silicon_Labs_HubZ_Smart_Home_Controller_90F0016B-if00-port0'
creating link '/dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_90F0016B-if00-port0' to '/dev/ttyUSB0'
preserve already existing symlink '/dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_90F0016B-if00-port0' to '../../ttyUSB0'
found 'c188:0' claiming '/run/udev/links/\x2fserial\x2fby-path\x2fpci-0000:00:02.0-usb-0:1:1.0-port0'
creating link '/dev/serial/by-path/pci-0000:00:02.0-usb-0:1:1.0-port0' to '/dev/ttyUSB0'
preserve already existing symlink '/dev/serial/by-path/pci-0000:00:02.0-usb-0:1:1.0-port0' to '../../ttyUSB0'
found 'c188:0' claiming '/run/udev/links/\x2fttyUSB-5'
creating link '/dev/ttyUSB-5' to '/dev/ttyUSB0'
preserve already existing symlink '/dev/ttyUSB-5' to 'ttyUSB0'
...

Or as a one-liner:

udevadm test `udevadm info -a /dev/ttyUSB0 | grep "looking at device" | sed "s/looking at device '//" | sed "s/'://"`

Just say the word

It worries me — the did he / didn’t he “say the n word” question once again surrounding Trump. Not because I think he did or did not use the term, but because the discussion is meaningless. Trump will deny saying it — hell, he denies writing things that are archived in his Twitter account. He denies saying things that even when told there is a publicly available recording of him saying it. The racists among his supporters will see a wink with that denial. Some willfully blind supports will believe the denial. Opponents will assume the tape exists. Whatever.

Words are powerful, but not in the way this debate seems to imply. Not saying a specific word does not magically cure the social, political, and economic problems in this country any more than having a president say radical Islamic terrorism magically solved the real social, political, and economic problems that lead to terrorist attacks.

But what makes a single word the arbiter of racism? I had a physical education instructor who was sexist. The fact he called every girl in his class “chick” didn’t make him sexist. His belief that we were less capable because of our gender, that our time in physical education would best be spent sitting on the bleachers fixing up our nails, that we did not have the mental capacity to be taught. That made him racist. Had the man respectfully called me by my proper name every single time … he would still have been a sexist asshole who had no business teaching school children.

Someone who made his political name demanding a president show him some ID, who knowingly called Mexican emigrants a bunch of rapists, who thinks shutting down Mosques bears consideration, who refers to hut-dwellers from shit-hole countries … words are powerful, but refraining from uttering a specific word does not negate his racism. Maybe the dude was not a racist (discrimination against tenants and such says otherwise, but just pretend). Maybe Trump’s political persona is a role he’s playing. Willing injuring others by inciting racist violence. Scapegoating others for serious economic problems. Fomenting a social environment where racist actions are acceptable. Maybe that’s not racist. It’s still horrendous behavior.

Zoneminder and PHP 7.2

After updating to php 7.2, ZoneMinder completely stopped working. Fortunately there were lots of entries in the error_log file

[Fri Aug 10 15:44:19.880809 2018] [php7:error] [pid 5293] [client 127.0.0.1:46958] PHP Fatal error:  Cannot use 'Object' as class name as it is reserved in /usr/share/zoneminder/www        /api/lib/Cake/Core/Object.php on line 30
[Fri Aug 10 15:44:19.889737 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/lib/zoneminder/templogs/cake_error.log) [function.file-put-contents]: failed to open stream: No such file or directory in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on         line 142
[Fri Aug 10 15:44:19.889850 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/log/zonemindererror.log) [function.file-put-contents]: failed to open stream: Permission denied in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on line 142
[Fri Aug 10 15:44:19.890176 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/lib/zoneminder/templogs/cake_error.log) [function.file-put-contents]: failed to open stream: No such file or directory in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on         line 142
[Fri Aug 10 15:44:19.890221 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/log/zonemindererror.log) [function.file-put-contents]: failed to open stream: Permission denied in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on line 142
[Fri Aug 10 15:44:19.890594 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/lib/zoneminder/templogs/cake_error.log) [function.file-put-contents]: failed to open stream: No such file or directory in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on         line 142
[Fri Aug 10 15:44:19.890637 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/log/zonemindererror.log) [function.file-put-contents]: failed to open stream: Permission denied in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on line 142
[Fri Aug 10 15:44:19.890960 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/lib/zoneminder/templogs/cake_error.log) [function.file-put-contents]: failed to open stream: No such file or directory in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on         line 142
[Fri Aug 10 15:44:19.891021 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/log/zonemindererror.log) [function.file-put-contents]: failed to open stream: Permission denied in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on line 142
[Fri Aug 10 15:44:19.892818 2018] [php7:error] [pid 5293] [client 127.0.0.1:46958] PHP Fatal error:  Uncaught Error: Class 'Controller' not found in /usr/share/zoneminder/www/api/li        b/Cake/Error/ExceptionRenderer.php:174\nStack trace:\n#0 /usr/share/zoneminder/www/api/lib/Cake/Error/ExceptionRenderer.php(92): ExceptionRenderer->_getController(Object(InternalErr        orException))\n#1 /usr/share/zoneminder/www/api/lib/Cake/Error/ErrorHandler.php(126): ExceptionRenderer->__construct(Object(InternalErrorException))\n#2 /usr/share/zoneminder/www/ap        i/lib/Cake/Error/ErrorHandler.php(284): ErrorHandler::handleException(Object(InternalErrorException))\n#3 /usr/share/zoneminder/www/api/lib/Cake/Error/ErrorHandler.php(213): ErrorHa        ndler::handleFatalError(64, 'Cannot use 'Obj...', '/usr/share/zone...', 30)\n#4 /usr/share/zoneminder/www/api/lib/Cake/Core/App.php(970): ErrorHandler::handleError(64, 'Cannot use '        Obj...', '/usr/share/zone...', 30, Array)\n#5 /usr/share/zoneminder/www/api/lib/Cake/Core/App.php(943): App::_checkFatalError()\n#6 [internal function]: App::shutdown()\n#7 {main}\n          thrown in /usr/share/zoneminder/www/api/lib/Cake/Error/ExceptionRenderer.php on line 174

Looks like CakePHP used class names that are now reserved words. Unfortunately, you cannot just drop the updated CakePHP files into ZoneMinder (I tried). Until the repository package is updated, you’ve got to build ZoneMinder from source. Or grab the testing RPM from the zoneminder.com repo. 1.31.45-1.13 works. Don’t forget to run “perl /usr/bin/zmupdate.pl” to update the database.

Then I had to throw the database connection into from the config file into /usr/share/zoneminder/www/api/app/Config/database.php (default array) because I do not use the default connection info.

Once I had the updated ZoneMinder along with the newer CakePHP that the ZM folks have in their repo … we’ve got ZoneMinder again.

Sendmail: Giving everyone else a chance

We will occasionally get slammed with thousands of messages from a specific sender (usually one we’ve engaged to e-mail all of our employees, and of course they do it at 11 in the morning instead of some low volume off-hours time) which delays mail from all recipients. You can use the sendmail command line to flush the mail queue for messages other than those from a specific sender or other than those to a specific sender.

Using sendmail -qSdomain.gTLD will attempt to delivery messages where the sender matches domain.gTLD. Adding a ‘not’ in front of the ‘S’ attempts to deliver messages where the sender is not the specified domain.

sendmail -q\!Sbadguy.gTLD -v

That ensures all of the other mail is cleared through the queue even though ten thousand messages from a single sender still need to be processed. In real life, bypassing everything with the ‘mail from’ of @em-sj-77.mktomail.com is just

sendmail -q\!Sem-sj-77.mktomail.com -v

Shell Scripting: “File Exist” Test With Wildcards

Determining if a specific file exists within a shell script is straight-forward:

if [ -f filename.txt ]; then DoSomething; fi

The -f verifies that a regular file exists. You might want -s (exists and size is greater than zero), -w (exists and is writable), -e (a regular or special file exists), etc. But the test comes from the “CONDITIONAL EXPRESSIONS” section of the bash man page and is simply used in an if statement.

What if you don’t know the exact name of the file? Using the text “if [ -f *substring*.xtn ]” seems like it works. If there is no matching file, the condition evaluates to FALSE. If there is one matching file the condition evaluates to TRUE. But when there are multiple matching files, you get an error because there are too many parameters

[lisa@fc test]# ll
total 0
[lisa@fc test]# if [ -f *something*.txt ]; then echo "The file exists"; fi
[lisa@fc test]# touch 1something1.txt
[lisa@fc test]# if [ -f *something*.txt ]; then echo "The file exists"; fi
The file exists
[lisa@fc test]# touch 2something2.txt
[lisa@fc test]# if [ -f *something*.txt ]; then echo "The file exists"; fi
-bash: [: 1something1.txt: binary operator expected

Beyond throwing an error … we are not executing the code-block meant to be run when the condition is TRUE. In a shell script, execution will continue past the block as if the condition evaluated to FALSE (i.e. the script does not just abnormally end on the error, making the failure more obvious).

To test for the existence of possibly multiple files matching a pattern, we can evaluate the number of files returned from ls. I include 2>/dev/null to hide the error which will otherwise be displayed when there are zero matching files.

[lisa@fc test]# ll
total 0
[lisa@fc test]# if [ $(ls *something*.txt 2>/dev/null | wc -l) -gt 0 ]; then echo "Some matching files are found."; fi
[lisa@fc test]# touch 1something1.txt
[lisa@fc test]# if [ $(ls *something*.txt 2>/dev/null | wc -l) -gt 0 ]; then echo "Some matching files are found."; fi
Some matching files are found.
[lisa@fc test]# touch 2something2.txt
[lisa@fc test]# if [ $(ls *something*.txt 2>/dev/null | wc -l) -gt 0 ]; then echo "Some matching files are found."; fi
Some matching files are found.
[lisa@fc test]#

Now we have a test that evaluates to TRUE when there are one or more matching files in the path.