Month: May 2018

Isolated Guest Network On Merlin 380.69_2 (Asus RT-AC68R)

We finally got rid of Time Warner Cable / Spectrum / whatever they want to call themselves this week’s overpriced Internet that includes five free outages between 1100 and 1500 each day. But the firmware on the new ISP’s router doesn’t have a facility to back up the config. And if we’re going to have static IPs for all of our speakers, printers, servers … we don’t want to have to re-enter all of that data if the router config gets reset. Same with configuring the WiFi networks. And, and and. So instead of using the snazzy new router, we are using our old router on .2, the new router on .1 … and everything actually connects to the old router, uses the DHCP server on the old router. And only uses the new router as its default gateway. Worked fine until we tried to turn on the guest network.

I found someone in Internet-land who has the exact same configuration and wants to permit guests to use the LAN printer. His post included some ebtables rules to allow guest network clients access to his printer IP. Swapped his printer IP for our router IP and … nada.

And then I realized that the router is not the packet destination IP when the guest client attempts to communicate outside our network. The router is the destination MAC address. So you cannot add an ebtables rule to the router’s IP address and expect traffic to flow.

The first thing you need to do is figure out the upstream router’s MAC address. From the Asus, you can query the arp table. If the command says “No match found in # entries”, ping the router and try again.

root@ASUS-RT-AC68R:/tmp/home/root# arp -a 10.5.5.1
? (10.5.5.1) at a3:5e:c4:17:a3:c0 [ether] on br0

The six pairs of hex numbers separated by colons – that’s the MAC address. You have to allow bidirectional communication from the guest network interface (wl0.2 for us) with the upstream router’s MAC address. You also have to allow broadcast traffic so guest devices are able to ARP for the router’s MAC address.

To have a persistent config, enable jffs and add the config lines to something like services-start:

root@ASUS-RT-AC68R:/tmp/home/root# cat /jffs/scripts/services-start
#!/bin/sh
logger "SERVICES-START: script start"
# Prevent Echo dots from sending multicast traffic to speaker network
ebtables -I FORWARD -o wl0.1 --protocol IPv4 --ip-source 10.0.0.36 --ip-destination 239.255.255.250 -j DROP
# Guest network - allow broadcast traffic so devices can ARP for router MAC
ebtables -I FORWARD -d Broadcast -j ACCEPT
# Guest network - allow communication to and from router MAC
ebtables -I FORWARD -s a3:5e:c4:17:a3:c0 -j ACCEPT
ebtables -I FORWARD -d a3:5e:c4:17:a3:c0 -j ACCEPT
# This should be automatically added for guest network, but it goes missing sometimes so I am adding it again
ebtables -A FORWARD -o wl0.2 -j DROP
ebtables -A FORWARD -i wl0.2 -j DROP

 

Use -L to view your ebtables rules:

root@ASUS-RT-AC68R:/tmp/home/root# ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 16, policy: ACCEPT
-d a3:5e:c4:17:a3:c0 -j ACCEPT
-s a3:5e:c4:17:a3:c0 -j ACCEPT
-d Broadcast -j ACCEPT
-p IPv4 -o wl0.1 --ip-src 10.0.0.36 --ip-dst 239.255.255.250 -j DROP
-o wl0.2 -j DROP
-i wl0.2 -j DROP

Voila, guests who can access the Internet & DNS on the .1 router, but cannot access anything on the internal network. Of course you can add some specific IPs as allowed destinations too – like the printers in the example that started me down this path.

Show Respect

All NFL Players ‘Shall Stand And Show Respect’ For Flag And Anthem – Wow! I wish I was an avid football viewer so *not* watching football would be a state change. Hopefully being permitted to stay in the locker room enables players to make their point. If two guys on a team of, what, fifty people aren’t on the sideline you’d never notice. Half only emerge once the anthem is over? That makes a statement too. And someone’s camera would end up in the locker room to cover the protest. Any takers on how long it is before players aren’t permitted to stay off the field after that protest makes news?

‘Show respect’ is a ill defined term too. I assume this is so you don’t have guys standing backwards, but how is hanging back in the locker room playing candy crush *more* respectful than kneeling during the anthem? And for the guys on the field, some dude who was kneeling last season is a little slouchy in his stance, the team still gets fined to avoid a presidential tantrum?

The whole idea of being forced to stand for the anthem seems anathema to the ideals of our country, even if the long history of private employment shows we can be forced into just about anything if we want to continue receiving a pay cheque. The same could be said for being forced to stand and pledge allegiance to the country 180 days a year for thirteen years. Or more – I was rather dismayed to learn that my daughter’s preschool class was taught the pledge of allegiance so they could recite it at their graduation ceremony. Now I’ve got a bit of an Anabaptist philosophy – I don’t much see the point in having someone repeat words or go through a ceremony without *understanding* what they are doing. I avoided children’s clothing with words on it – overkill, yeah, but a six month old baby doesn’t *mean* to say “I just did 9 months on the inside”, “Grandma’s Drinking Buddy”, or make a boob joke, no matter how many people find the messages cute or silly. Until she knew and understood what the shirt said, she got shirts with pictures. Or patterns. Or plain colours. So I asked my kid if the teacher explained what allegiance *is*, or even explained any of the historic principals of the United States. Of course not; they were just given words to recite. Now we’ve had some discussion of the country’s principals and failings – she votes with me two times a year (primaries and general, this is not some admission of voter fraud), we’ve discussed how to affect local, state, and federal laws (and the diminishing influence an individual has as you move from local to state to federal government). But the principals of the Republic for which the flag stands is pretty abstract to convey to a preschooler. And pledging allegiance to a flag? The essence of a nation is not bound up in its cloth banner.

Forced recitations of pledges and vows do nothing to impart knowledge, develop skills, or promote good citizenship. As an intimidation technique, forced declaration of faith and loyalty are not new, although they are generally the hallmark of an insecure society. People do not become more patriotic through such declarations, but being subject to coercion can have the opposite effect.

On Proceedure

A little more than a year ago, Trump somehow thought that associates being the subject of a judicially approved wiretap somehow exonerated him. This week, the fact the FBI had sufficient evidence that his campaign received and possibly sought the aid of foreign governments to place an informant in the campaign organization is meant to show how the whole investigation is FAKE NEWS. And, hell, for all we know someone who worked for the campaign heard about these meetings and reached out to the FBI to report it.

And he compares an FBI informant in his campaign to Watergate — where burglars broke into the DNC HQ office, installed listening devices in the phones, and then broke in again. Difference is *burglars* broke into the office and planted devices to intercept conversations (and broke in again to ‘repair’ their initial work). When the FBI uses informants, on the other hand, “special care is taken to carefully evaluate and closely supervise their use so the rights of individuals under investigation are not infringed. The FBI can only use informants consistent with specific guidelines issued by the attorney general that control the use of informants”. Which makes Trump’s claim another bit of ‘deep state’ paranoia.

It’s not unreasonable to conclude that evidence of the campaign’s interaction with foreign powers was discovered and prompted the investigation. Have the DoJ look into it and verify the FBI followed their internal policy, although that’s a bit of a stretch. Given the number of meetings with representatives of foreign governments the campaign took looking for campaign assistance, Trump’s assertion is a bit like a meth cook saying the whole system is corrupt as evidenced by the search warrant for his lab being signed off on by a judge.

The strangest bit of the whole assertion is that a deep state conspiracy to undermine Trump’s campaign would have been far more effective if it were announced prior to the election. After the fact, it’s pretty ineffective. Best case for an after-the-fact investigation is they manage to impede the process of governing until the next election cycle. The day before the last debate, publicize (or leak) news of this investigation? A day or two before the election?

It’ll convince the 30% who are out to prove Trump right on one matter — he could shoot someone on 5th Ave and still have their support.

 

The Horrors!

The TL;DR summary of the Trump Tower meeting, by way of the Senate Committee testimony, seems to be “we wanted dirt on our opponent to help win the election, and were right eager to accept said help from Russia but this meeting failed to provide what we wanted to procure”. Which, as far as defenses go … not a great one.

While one is not meant to consider the ramification of a legal decision, Trump Jr’s testimony brings to mind prostitution sting operations. I would love to see the defendant claiming that they had not in fact engaged in an illegal activity. Sure they wanted to exchange money for sex. The sex was never provided; ipso facto the law was not broken. Case dismissed! Sorry to inconvenience you, upstanding citizen.

DSEE 6.3 To OUD 11g Transition

There’s no direct path to replicate data from DSEE6.3 to OUD11g. Not unreasonable since DSEE is the Sun product based on the Netscape Directory Server and OUD is the Oracle product based on OpenLDAP – they weren’t exactly designed to allow easy coexistence that would permit customers to switch from one to the other. Problem is, with Oracle’s acquisition of Sun & axing the DSEE product line … customers *need* to interoperate or do a flash cut.
Since our Identity Management (IDM) platform was not able to prep development work and implement their changes along with the directory replacement, a flash cut was right out. I’ve done flash cuts before — essentially ran two completely different directories in parallel with data fed from the Identity Management platform, tested against the new directory using quick modification to the OS hosts file, then reconfiguring the virtual IP on the load balancer to direct the existing VIP to the new service hosts. Quick/easy fail-back is to set the VIP to the old config and sort out whatever is wrong on the new hosts. A lot lower risk than a traditional ‘flash cut’ approach as long as you trust the IDM system to keep data in sync. But lacking an IDM system, flash cut is typically a non-starter anyway.
There is a migration path. Oracle put some development effort into the DSEE product line prior to discontinuing it. DSEE7 was the Sun distributed “next version”. It was not widely deployed prior to the Oracle acquisition. Oracle took over DSEE7 development but called it DSEE11 (to match the OUD version numbering, I guess?). Regardless of the rational, you’ll see the “next version” DSEE product referred to as both DSEE7 and DSEE11.
There’s not a direct replication between Oracle DSEE11 and Oracle OUD11. Oracle created a “replication gateway” that handles, among other things, schema name mapping (only Netscape would use attribute names like nsAccountLockout, and that nomenclature carried through to the Sun product). Oracle did a decent job of testing DSEE11<=>OUD11 Replication Gateway interoperability. I don’t know if they just assumed DSEE6 would work because DSEE11 did or if they assumed the installation base for DSEE6 was negligible (i.e. didn’t bother to test older revisions) but we found massive bugs in the replication gateway working with DSEE6. “You cannot import the data to initialize the OUD11 directory” type of bugs which I was willing to work around by manually editing the export file, but subsequent “updates do not get from point ‘A’ to point ‘B’ bugs too. The answer from Oracle was essentially “upgrade to DSEE11” … which, if i could flash-cut upgrade DSEE6 to DSEE11 (see: IDM platform couldn’t do that), I could just cut it to OUD11 and be done. Any non-trivial change was a non-starter, but Oracle wasn’t going to dump a bunch of development time into fixing replication for a dead product to their shiny new thing.
I worked out a path that used tested and working components — DSEE6 replicated just fine with DSEE11. DSEE11 replicated just fine with the OUD11g replication gateway, and the OUD11g replication gateway replicated fine with OUD11g. Instead of introducing additional expense and time setting up dedicated replication translation servers, I installed multiple components on the new servers. There is a DSEE11 directory on one of the new OUD servers, the replication gateway on another one of the new OUD servers, and (of course) the OUD11g directory that we actually intended to run on the new servers is on those new OUD servers.
This creates additional monitoring overhead – watching replication between three different directories and ensuring all of the services are running – but allows the IDM platform to continue writing changes to the DSEE6.3 directory until they are able to develop and test changes that allow them to use OUD11g directly.

Seeing Crime Everywhere

There have been a few stories recently about white people ringing up the police because someone with darker skin exists. This most recent story is a grad student who fell asleep while writing a paper in a common room. Not unheard of, there were kids dozing off in the library and residential common areas all.the.time. Kids fell asleep in my computer lab too. Sometimes even the work study kids who were meant to be supervising the area and assisting with computer problems. Almost a decade ago, it was a Harvard professor and his driver forcing a stuck door at his house. I’ve mentioned before that I’ve encountered a police officer while I was breaking into a car in a car park. It wasn’t just a police officer who happened across me. Dozens of people in the plaza didn’t look twice at the white chick forcing her way into a car.

It isn’t the police response that strikes me as much as the person making the report — it’s like we need a beer summit on a national scale. Why not approach the sleeping person, wake them, and suggest their room is going to be a comfier place for a nap. Or if they wake up and want to hang in the commons area, strike up a conversation. Ask their program, tell them about your program. And if you still think the person isn’t a student (doesn’t know the names of teachers in their department or knows totally made up profs, whatever) then call campus security or the police.

Sitting is the New Smoking

Some company official posted an internal article titled “Sitting is the New Smoking” to tell us all how bad sitting for prolonged periods of time can be for your health. While they make suggestions for using your break to do some exercises or suggest cube-exercises … frankly, they’ve designed a job that requires sitting for prolonged periods of time.

Some people have standing desks. Not all. Not most. My understanding is these things were purchased as accommodative equipment the company had to purchase based on medical need. If sitting is as bad for your health as smoking, did the company not just publish its own statement of medical need to support widespread purchase of standing desks?

Beyond near-term costs, though, the assertion brought to mind the Black Lung Benefits Act from nearly two decades ago. While mine operators may have been able to reduce exposure to coal dust, some level of exposure to coal is requisite in mining the stuff. A generally unavoidable environment based on the work being done caused a major medical problem that led to disability and death, and companies ended up shelling out disability payments and survivor benefits. It wasn’t quite the least they could possibly do to quell public outcry, but there are a lot of *’s on qualifying that let reasonable requests be denied or pushed off for years without retroactive payments. Even so, the payout is like eight grand a year per afflicted miner. And there are like 30k recipients (and something like 5k dependents, which can drastically increase the annual payout). That’s minimum two hundred forty million bucks in 2017. And it’s a LOT less now than a decade ago. There are nuances to determining the payer, but it is generally the mine operator most recently employing an affected individual. A significant portion of this money has been payed by mine operators.

Sitting at work is different from exposure to coal whilst mining coal. There’s no reason most jobs require sitting for hours on end. Historically there’s a component of elitism — a hundred plus years ago, low paying jobs were physically intensive, and it was a bit of an elite thing to be able to sit at work. Now the sign of affluence is a few spare hours a week to exercise, and sitting is just a norm no one has sought to change. If a company is aware of how bad sitting is for its employees, seems like said company would have a better defense against liability if they actively attempt to re-design their workplaces and jobs to avoid sitting. Sending out a mass mail telling you how bad something is or having a webinar to tell you how bad it is … but generally employing people to sit for hours at a time isn’t much in the way of due diligence. Routinely deploying standing desks, even in training classrooms, would reduce mandatory sitting among call centre staff. Walking meetings for one-on-one or small group sessions.

 

The History Of War And Peace

As Plato says for Clinias of Scambonidae — “For (as he would say) ‘peace,’ as the term is commonly employed, is nothing more than a name, the truth being that every State is, by a law of nature, engaged perpetually in an informal war with every other State.”, I have seen peace not as the normative state but as a temporary interlude in an ongoing war. What first drew me to study history was observing the chain of treaties to ‘end’ European conflict that extorted and humiliated the defeated parties. The Peace of Westphalia established the supremacy of the nation-state over religious states, but it also begat machinations to maintain a “balance of power” whereby ‘balance’ more or less meant your nation maintained some level of control throughout the continent. The Treaty of Frankfurt, with the indemnity France was forced to pay and territory it was forced to cede, did nothing to establish good will on the Continent. The Anglo-Ottoman Convention allowed British dominance in the Middle East, and the borders created largely ignored ethnic division. The Treaty of Versailles punishment of Germany undermined the Weimar Republic. European nations learned, my professor asserted, and sought to ensure the treaty ending World War II wouldn’t follow the long chain of humiliating, punitive treaties. An assertion ridiculous on its face – border adjustments in the Balkans under the Paris Peace Treaties begat revolution and conflict decades later as ethnically different peoples lumped into the same country broke apart.

Punishing and embarrassing a nation, or lumping people with a long history of conflict into the same country are hardly conducive to lasting peace. I oft wonder if that was the point — see: Eisenhower’s military industrial complex speech. Demanding four billion dollars from King Salman may not be an insurmountable financial burden to the Saudis, but such payment would certainly be seen as a national embarrassment. Violating the US out of the Iranian nuclear deal — and an extrapolation of what the US will ask from North Korea — is just another event in a centuries long chain of “we win, FU” so-called diplomacy.

Viewing North Korea’s summit in light of Iran – either a set of conditions are acceptable in North Korea but not Iran or North Korea will be told to completely eliminate their nuclear capabilities. It’s one thing for Korea to offer to dismantle their testing facilities — frankly, nuclear testing is frightening, and once you’ve got a bomb there’s not much point in repeatedly exploding a nuclear device — but denuclearizing and permitting frequent, invasive inspections to ensure the program is not renewed … that’s a big ask.

Systemd (a.k.a. where did my log files go!?!?!)

A systemd Primer For sysvinit Users

Background:

Starting in Fedora 15 and RHEL 7, systemd replaces sysvinit. This is a touchy subject among Unix folks – some people think it’s a great change, others think Linux has been ruined forever. Our personal opinions of the shift doesn’t matter: vendors are implementing it, WIN Linux servers use it, so we need to know it. Basically, throw “systemd violates the minimalist, modular philosophy at the core of Unix development” on the “but emacs is so awesome, why are we using vim” and “BETA outperforms VHS any day of the week” pile.

Quick terminology – services are now called units. You’ll see that word a lot. A unit is configured in a “unit file”. Additionally, “run levels” (0-6) have been replaced with the concept of “targets” that have friendly names.

What’s the difference?

Sysvinit wasn’t designed to know about your system, it was designed to run scripts on your system. Sysvinit essentially runs scripts, whereas systemd is a service manager. Systemd knows about the system. One place this becomes apparent – if you manually run the run line from a sysvinit script then check the service status, it will show running because the binary has a PID. If you do the same with systemd, it will say the service is down. This is like Windows – if you have a Docker service that runs “”C:\Program Files\Docker\Docker\com.docker.service”” set to run manually, and use start-run to run the exact same string … the service will not show as running.

Systemd manages a lot of different unit types. As application owners, we’ll use ‘service’ units. ‘Mount’ or ‘automount’ type units manage mountpoints. Socket and device unit types manage sockets (which have associated service unit files using the socket) and devices. Because systemd manages sockets, inetd/xinetd have been obsoleted.

Sysvinit scripts could run user-defined commands. If the init script for myapplication has a section called “bob”, you can run “service myapplication bob” and it will do whatever the ‘bob’ part of the script says to do. Systemd has a fixed list of directives – start, stop, restart, reload, status, enable, disable, is-enabled, list-unit-files, list-dependencies, daemon-reload. You cannot just make a new one.

Systemd may also require a system reboot for more than just kernel patches. This is really different, and I expect there will be a learning curve as to what requires a reboot.

Log files have “vanished”. If you are using a default installation, you won’t find /var/log/messages. You can use “journalctl -f” to tail the equivalent of the messages file. The systemd log files are stored in binary format – potentially corruptible, which is another aspect of the change Unix-types don’t care for.

What does systemd give me?

Systemd doesn’t just start/stop a service when run levels change. A unit can be started because it is configured to start on the runlevel (just like sysvinit scripts), if another service requires it, if the service abends, or if dbus triggers it. “If another service requires it” – that’s a dependency chain. Instead of defining an order and hoping everything you need was loaded by the time the init script ran, systemd allows you to include an “After” directive – units started before the current unit or “Before” – units that will not be started until the current unit starts. Additional directives for “Requires” – units which must be activated to activate the current unit and “Wants” – units that will be started in parallel with the current unit but failing to start these units will not fail the current unit.

A directive, “Conflicts”, allows systemd to identify other units that cannot coexist with the current unit. Conflicting units will be stopped to allow the current unit to start. In addition to the base command starting in the unit file (ExecStart), there are pre (ExecStartPre) and post (ExecStartPost) operations that are run before/after the base command. These could be related to the service itself but do not have to be. You could run a mail command line to alert an admin every time the unit starts or stops cleanly.

Another nice feature of systemd is user-level services – using systemctl –user will control unit files located in user-specific directories like /usr/lib/systemd/user/ and ~/.config/system/user/

Using systemd: (Warning: this is going to get odd)

You use systemctl to control units, and you use journalctl to view the binary blobs that have replaced log files. Use the man pages or your favourite search engine if you want details. The general syntax for systemclt is “systemctl operation unit.type” – e.g. “systemctl restart sendmail” would restart sendmail.

Chkconfig has been completely supplanted. Use “systemctl enable unit.type” and “systemctl disable unit.type” to control if a service auto-starts. Instead of using chkconfig –list, you can query the startup state of an individual unit. Use systemctl –is-enabled unit.type

There’s a service shell script that replaces ‘service’ that you used with sysvinit systems. It turns the old “service something-or-other action” into “systemctl action name.service” so it still works.

Here’s the odd part – it is quite easy to define a permitted sudo operation that allows a non-root user to control sysvinit services. Allow “service sendmail” and the user can run “service sendmail start”, “service sendmail stop”, “service sendmail status”, “service sendmail RandomStuffITossedIntoTheFile”. Because the service name and directive are swapped around in systemctl, we would have to enumerate each individual directive that should be permitted. More secure, because RandomStuffITossedIntoTheFile should not make the cut. But we haven’t done this yet. So until we go through and enumerate the reasonable actions (Are there directives beyond start/stop/status that we should be running? Do we have any business enabling and disabling our services?), submit the access request, confirm it’s all functioning as expected, and remove the “sudo service” access … continue using “sudo service something-or-other action”. We will advise you when the systemctl sudo access has been granted so we can start using the “new way” to control services on RHEL7 systems.

Unlike init scripts, changes to systemd unit files are not immediately activated on the system. Running “systemctl daemon-reload” makes systemd aware of the config change.

Using journalctl:

Our Unix team has implemented rsyslogd to output log data to the expected files. This means you can more or less ignore journalctl – tail/grep the log file as usual. I don’t foresee this changing in the near to mid term, but if you use cloud-hosted sandbox servers (i.e. boxes that don’t have the Unix group’s standard config) … journalctl is what happened to all the log files you cannot find.

To view logs specific to an individual unit, use journalctl -u unit.type. Additionally “systemctl unit.type status” will display the last handful of log lines from the unit.

Load Balance and Failover Sendmail Mailertable Relays

A coworker asked me today how to get the mailertable relays to load balance instead of fail over. Trick is to think beyond sendmail. The square brackets around hosts tell sendmail not to check for an MX record (you’re generally using an A record, so this saves a tiny little bit of time … not to mention *if* there is an MX record there, it creates a whole heap-o confusion). *But* the MX lookup is right useful when setting up load balanced or failover relay targets.

Single host relay in the mailertable looks like this:
yourdomain.gTLD      relay:[somehost.mydomain.gTLD]

If you want to fail over between relays (that is try #1, if it is unavailable try #2, and so on), you can stay within the mailertable and use:
yourdomain.gTLD      relay:[somehost.mydomain.gTLD]:[someotherhost.mydomain.gTLD]

Or even try direct delivery and fail back to a smart host:
yourdomain.gTLD      relay:%1:smart-host

But none of this evenly distributes traffic across multiple servers. The trick to load balancing within the mailertable is to create equal weight MX records in your domain to be used as the relay.

In ISC Bind, this looks like:
yourdomainmailrouting.mydomain.gTLD     IN MX 10 somehost.mydomain.gTLD.
yourdomainmailrouting.mydomain.gTLD     IN MX 10 somehost.mydomain.gTLD.

Once you have created the DNS records, simply use the MX record hostname in your mailertable:

yourdomain.gTLD      relay:yourdomainmailrouting.mydomain.gTLD

By leaving out the square brackets, sendmail will resolve an MX record for ‘yourdomainmailrouting.mydomian.gTLD’, find the equal weight MX records, and do the normal sendmail thing to use both.