PIP – Where in the world are my binaries!?!?

If I run pip as root (and, yeah, I know you shouldn’t do that), I get happy functional binaries in /usr/bin. When I do the right thing, I (of course) cannot write to that bin folder. I cannot find my binaries! Running updatedb and using locate doesn’t help. Still got no files.

I ended up tracing an operation to find these things. They’re in my user home directory? And, of course, the locate command (and its database via updatedb) often excludes user home directories by default due to privacy and security concerns. So it looked like nothing was getting written to disk during the pip install.

Once I found the missing binaries in my ~/.local/bin folder, I was able to find that this is all well documented:

“Currently if user is installing their own tools with installers like (pip), they will be installed inside ~/.local/bin or ~/bin but if the same CLI tools are installed at system level the user would not be able to use his own tools because the system one would be picked instead. This happens because .bashrc file adds user PATH to the end instead of the top of PATH list variable.”

 

Of course, I go through all this research and learning to actually find the binary … only to discover that home assistant literally has something hard coded to require that you use a virtual environment. Not just do a pip install as your application user. I really wish people would explain why in their documentation. https://fedoraproject.org/wiki/Home_Assistant just says to type these commands. Which is nice if you don’t know what you are doing. But explaining why a virtual environment is being used would have saved me a lot of trouble just to bash up against another brick wall. Sigh. On the up side, I learned something new about Linux.

We didn’t think that should or would happen

“We didn’t think that should or would happen.” Duffy on the air traffic outage.

This seems like a fairly reasonable assertion to anyone who doesn’t actually implement changes to things that impact other people. What dude here fails to appreciate is that there aren’t many changes where we *expect* to take everything down. Most systems have redundancies. I can literally update and reboot one server in the system without anyone noticing. The difference between what we expect to happen and what DOES happen is why work is done during maintenance windows. Publicized so anyone who happens to be working at 2am knows who to call if things don’t look right.

I worked at a company that was planning an electrical upgrade at their data center. The electrician opened a floor panel to look at what he’d be doing. Literally just look. Somehow, dude dropped the quarter inch sheet metal cover into the opening, shorting some stuff out and dropping half of the servers that weren’t high enough priority for redundant power (also identified a few where redundancy power both went to the same place), had to take an ambulance to the hospital (he recovered and was fine). The data center ops guy – the one who declined all changes that weren’t in a maintenance window or accompanied by an emergency change and a critical issue ticket – kept that metal plate in his office as a reminder of why there were no exceptions. No matter how much someone might plead and assure him their change was so trivial.
Worked another place where they had a team removing the unused fiber. Contractors paid hourly, and the rate was higher for nights and weekends. Of course they wanted to save money and do this at 2 in the afternoon instead of 2 in the morning. And they found every poorly documented fiber connection into SAN shelves.
These changes were approved by immature change management structures who took “Nothing should happen, we’re just removing the unused fiber” as a 100% accurate depiction of reality. That took “I’m just opening this panel and looking over some wiring” as  immune to any accidents. In both companies, changes were made. The criterion for approving a change outside of the agreed upon maintenance window became much stricter. The “no impact expected” checkbox meant very little in deciding if the change would be approved or not.

For any organization that is approving non-critical changes during peak usage, hopefully this is a prompt to make some changes without having to take out dozens of your own systems. For the rest of us, though, it’s a good reminder why change management exists. Why it is a good thing someone is rejecting your change and requesting it be rescheduled during a maintenance window. To have some appreciation for the folks to review these changes for impact and handle end user notification. I think most of us who have worked in IT for a while know the feeling the guy who plugged in the Gear of Doom had. Having a couple extra people check your plan is a wonderful contribution to never feeling that again.

None is not 0

W3Schools documents the default number of digits for the python round function as 0

Except that’s not actually how the function works — if you don’t supply a value, you get an integer. No decimal places. If you supply a zero, you get the same value but with a decimal point. The float representation of that rounded value.

The default, per Python’s documentation at https://docs.python.org/3.11/library/functions.html#round :

round(number, ndigits=None)
Return number rounded to ndigits precision after the decimal point. If ndigits is omitted or is None, it returns the nearest integer to its input.

The default value is None, not 0.

Blender – Making a Part Thicker

We are modifying a mount that lets you attach your cell phone to a microscope – one of the problems with the original design is that it’s not particularly sturdy. We lengthened the arms to accommodate larger cell phones, but that made the arms even flimsier. Anya showed us how to use s – z – 0 which flattens the selection to the average z height of the selected faces. Starting with a curved surface, I selected all of the faces. Then typed the letter s, the letter z, and 0

This flattens the object – the difference between the z location of all faces is zero. You can keep the curve but make it smaller by using a fractional number instead of zero. You can exaggerate the difference by using a number greater than 1. A negative number inverts the curve – the high part becomes low, and the low part becomes high.

Once the faces were on a single plane, we used G Z {number} to move the selected faces. The number is an offset from the plane where the faces are currently – so if you are at 4 and want the object to be 5mm thick, you use G Z 1. Using G Z 5 (which makes sense to move them to z of 5) actually makes the object 9mm thick!

The perpendicular faces are magically extruded so the faces you moved are not floating out there in space. The two arms that have already been increased, you can see that the perpendicular faces have been extended to meet the new face height. Obviously, if you want all three arms to be the same height you should select all of the top faces and adjust them simultaneously.

 

One other thing we learned in this adventure is how to select a bunch of faces between the faces you’ve already got selected and where you want to be. Shift will let you individually click faces, but I don’t want to click all of those little triangles along the curved side.

Selecting a few faces and then holding ctrl will select everything along the shortest path from where you already selected to where you click — here, selecting all the faces on the top of this arm. You can also use alt for a loop select — Anya finds that very functional, but it didn’t work for me!

On outages

What causes all the recent outages? I cannot rule out that there were AI driven cyber attacks. I cannot rule out that they were attacks from “enemy” countries. I also cannot rule out space aliens. But there were over a thousand MS patches this month, last month, etc. There are dozens of Linux patches most nights. We’ll get the Oracle patches of the month in a few days. In the rush to get things patched (absolutely the right thing to do), testing is either fully automated or getting less robust.

My expectation is that fully automated regression testing is going to be pushed hard in the near future. I’ve got Selenium scripts that run after the monthly Windows patch & reboot. Get a ping at dark o’clock that something doesn’t look right rather than a flood of calls at 7:30 AM.

Centralization

The potential for noticeable, widespread outages is an unappreciated component of centralization. I started using Exchange in 2000 as a tier 3 sys admin at a Fortune 500 company. Big company everyone’s heard of, but you wouldn’t hear about it when our email system went down for a day. If you were actively looking at buying from us, you might get a call from your sales guy letting you know he’s going to need a little extra time on a quote. Or you might get the email from an “odd” address (the sales guy’s gmail account or whatever). But it wouldn’t be national news. It wouldn’t be anything that everyone knew about.

I worked a major outage every couple of years from 2000 through 2015 when we moved to the hosted Exchange service. I sat next to the network infrastructure team. They’d have site outages that impacted a single office with a couple hundred people working there. Inside the company, even that wasn’t general knowledge. Every couple of years, they’d have something massive melt down. Someone would have an upgrade go sideways and a quick three hour task turned into a full weekend with dozens of people working on restoring service. Even cyber attacks. Worked at a place that’s been hacked. It made the news, but like “third page, under the fold” kind of news.

Fundamentally, there isn’t anything different going on today. It’s just that we don’t have ten thousand companies with ten thousand email systems, ten thousand authentication directories, ten thousand file servers, ten thousand networks. We have ten thousand companies of which at least 8,000 of them have all their stuff in Google, Microsoft, or Amazon. Something goes down, you notice. That’s true of email systems, authentication systems (federated auth, if you “log in with Google!” to all of your favorite sites, you lose access to those sites when Google’s auth framework goes wonky), networks, etc.

Installing Packages from Koji

Quick script to install the version of PHP that’s been patched for the two critical vulns today

dnf install koji

mkdir -p /root/php-8.5.12-koji
cd /root/php-8.5.12-koji

koji download-build --arch=x86_64 --arch=noarch php-8.5.12-1.fc44

# Select only RPMs whose package name and architecture are already installed.
rpms=()
for file in ./*.rpm; do
package=$(rpm -qp --qf '%{NAME}.%{ARCH}' "$file") || exit 1
if rpm -q "$package" >/dev/null 2>&1; then
rpms+=("$file")
fi
done

if ((${#rpms[@]} == 0)); then
echo "No downloaded RPMs match installed packages."
exit 1
fi

printf 'Selected: %s\n' "${rpms[@]}"

dnf install "${rpms[@]}"

Bernette B70 Reminder: Back Up Your Position

The Bernette B70 has a standby mode (Eco) for when you need to pause a job. Very convenient – especially since we’re still at the “babysit it the whole time” level of machine embroidery usage. Except — it appears that (at least for long-term standby), the system loads before the USB drive is mounted. It has cached information for the job preview, but it doesn’t actually have access to the stepper instructions. The machine essentially loses its place.

Which is fine if you know where you were when you stopped. There’s a sequence control section (#3 below)

Which allows you to use the upper and lower knobs to advance through the instructions. Upper knob is a slow advance, lower makes bigger jumps. You can also click on the magnifying glass and type in where you were. If, of course, you had backed up where you were when you paused the job!