Category: Coding

Self Driving Cars (or Market Driven Algorithms)

I don’t see much of a future for self-driving passenger vehicles. There are two non-tenable options for crash avoidance algorithms. Either the algorithm prioritizes my life and property (which means it would kill someone else to save my life … good for me, bad for society) or it won’t (great for society, but am I going to pay money for a car that will literally kill me to save someone else?). Does the computer assisted human driving model suffer this flaw? An algorithm that engages the brakes any time there is an obstacle within X feet fails to consider the vehicle that is about to slam into the side of your car if you don’t move it into the shrubbery ahead of you.

Self-driving unoccupied vehicles can simply de-prioritize itself (and the owner needs to accept that financial risk). We may see driving as a service (DaaS?) where a real human is responsible for making these split-second decisions. But allowing people to achieve the metro experience in their own vehicle (i.e. you sit and work for half an hour whilst your conveyance delivers you to your destination) is probably not going to happen.

Web-Accessible History From OpenHAB MySQL Persistence Database

My husband has wanted a quick/easy way to see the data stored in OpenHAB’s MySQL persistence database. He didn’t care for the mysql command line client. He didn’t care for PHPMyAdmin either. I’ve suggested the MyODBC client — which allows you to use MySQL databases as an ODBC data source so you can view the data in MS Access, Excel, etc. Nope – he wanted a web site.

So I put together a very quick (and ugly) PHP page that provides a list of all Items. If you click on an item, you can page through the item’s records. The index.php from the page is available here. You need a web server (I am using Apache on Fedora), PHP (I am using 5.6) and MySQLi (php-mysqlnd package).

This is a bit of paranoia on my part, but even on a page that is ONLY available internally … I don’t like to use an account with read/write access to display data. I create a new user and assign read access:

CREATE USER 'YourUserName'@'localhost' IDENTIFIED BY 'P#ssw0rdH3r3';
GRANT SELECT ON openhabdb.* to 'YourUserName'@'localhost';

Then use *that* user in the php code. This example has a web server running on the database server – and you connect to the MySQL server via localhost. If your web server is located on a different host, you’ll need to create and grant ‘YourUserName’@ the web server hostname.

Custom Password Filter Update (unable to log on after changing password with custom filter in place)

I had written and tested a custom Active Directory password filter – my test included verifying the password actually worked. The automated testing was to select a UID from a pool, select a test category (good password, re-used password, password from dictionary, password that doesn’t meet character requirements, password containing surname, password containing givenName), set the password on the user id. Record the result from the password set, then attempt to use that password and record the result from the bind attempt. Each test category has an expected result, and any operation where the password set or bind didn’t match the expected results were highlighted. I also included a high precision timer to record the time to complete the password set operation (wanted to verify we weren’t adversely impacting the user experience). Published results, documented the installation and configuration of my password filter, and was done.

Until the chap who was installing it in production rang me to say he couldn’t actually log in using the password he set on the account. Which was odd – I set one and then did an LDAP bind and verified the password. But he couldn’t use the same password to log into a workstation in the test domain. Huh?? I actually knew people who wanted *some* users to be able to log in anywhere and others to be restricted to LDAP-only logons (i.e. web portal stuff) and ended up using the userWorkstations attribute to allow logon to DCs only.

We opened a case with Microsoft and it turns out that their Password Filter Programming Considerations didn’t actually mean “Erase all memory used to store passwords by calling the SecureZeroMemory function before freeing memory.” What they meant was “If you have created copies of the password anywhere within your code, make sure you erase memory used to store those copies by calling SecureZeroMemory …”

Which makes SO much more sense … as the comments in the code I used as our base says, why wouldn’t MS handle wiping the memory? Does it not get cleaned well if you don’t have a custom password filter?? Remarked out the call to SecureZeroMemory and you could use the password on NTLM authentications as well as kerberos!

// MS documentation suggests doing this. I honestly don’t know why LSA
// doesn’t just do this for you after we return. But, I’ll do what the
// docs say…
// LJR – 2016-12-15 Per MS, they actually mean to wipe any COPIES you make
// SecureZeroMemory(Password->Buffer, Password->Length);


I’ve updated my version of the filter and opened an issue on the source GitHub project … but if anyone else is working a custom password filter, following MS’s published programming considerations, and finds themselves unable to use the password they set … see if you are zapping your copies of the password or the PUNICODE_STRING that comes in.

Active Directory: Custom Password Filtering

At work, we’ve never used the “normal” way of changing Windows passwords. Historically, this is because computers were not members of the domain … so you couldn’t use Ctrl-Alt-Del to change your domain password. Now that computers are members of the domain, changing Active Directory passwords using an external method creates a lot of account lockouts. The Windows workstation is logged in using the old credentials, the password gets changed without it knowing (although you can use ctrl-alt-del, lock the workstation unlock with the new password and update the local workstation creds), and the workstation continues using the old credentials and locks the account.

This is incredibly disruptive to business, and quite a burden on the help desk … so we are going to hook the AD-initiated password changes and feed them into the Identity Management platform. Except … the password policies don’t match. But AD doesn’t know the policy on the other end … so the AD password gets changed and then the new password fails to be committed into the IDM system. And then the user gets locked out of something else because they keep trying to use their new password (and it isn’t like a user knows which directory is the back-end authentication source for a web app to use password n in AD and n-1 in DSEE).

long time ago, back when I knew some military IT folks who were migrating to Windows 2000 and needed to implement Rainbow series compliant passwords in AD – which was possible using a custom password filter. This meant a custom coded DLL that accepted or rejected the proposed password based on custom-coded rules. Never got into the code behind it – I just knew they would grab the DLL & how to register it on the domain controller.

This functionality was exactly what we needed — and Microsoft still has a provision to use a custom password filter. Now all we needed was, well, a custom password filter. The password rules prohibit the use of your user ID, your name, and a small set of words that are globally applied to all users. Microsoft’s passfilt.dll takes care of the first two — although with subtle differences from the IDM system’s rules. So my requirement became a custom password filter that prohibits passwords containing case insensitive substrings from a list of words.

I based my project on OpenPasswordFilter on GitHub — the source code prohibits exact string matches. Close, but not quite 🙂 I modified the program to check the proposed password for case insensitive substrings. I also changed the application binding to localhost from all IP address since there’s no need for the program to be accessed from outside the box. For troubleshooting purposes, I removed the requirement that the binary be run as a service and instead allowed it to be run from a command prompt or as a service.  I’m still adding some more robust error handling, but we’re ready to test! I’ve asked them to baseline changing passwords without the custom filter, using a custom filter that has the banned word list hard coded into the binary, and using a custom filter that sources its banned words list from a text file. Hopefully we’ll find there isn’t a significant increase in the time it takes a user to change their password.

My updated code is available at

PHP 7.0 and MySQL Libraries

At work, we have some servers running unsupported operating systems. New servers are being built, and applications are being migrated from the old servers to the new. I started with a fairly easy scenario – a PHP web site running on Windows 2008 is moving to 2012. The new web server was handed off to me, and I loaded PHP. With PHP 5.6 active support ending at the end of this year, it made sense to install PHP 7. Copied code, tested site. Umm, massive fail.

Way back in PHP 5.5, the ext/mysql stuff (ext\php_mysql.dll for Windows folks) was deprecated. And if you are like me, you had a lot of old code from back when that was the way to connect to a MySQL database. And as your MySQL was upgraded past 4.1, you had the DBA’s setting old_password on your ID so your code continued to work.  But the old mysql libraries have been removed in PHP 7… and you need to use MySQLi or pdo_mysql to communicate with your database now.

Which one? Depends on what you need – I’ve been using PDO because I don’t need a procedural API (MySQLi provides a procedural API, PDO does not). PDO supports a dozen or so database drivers, MySQLi is just MySQL … so I’ll be able to use the same basic code to connect to MySQL, MS SQL, Oracle, and db2 (plus a handful of others that I don’t anticipate actually using, but who knows what the future holds).

I found a site ( where the individual has benchmarked MySQLi and PDO and doesn’t find much difference on INSERT statements but does see a non-negligible difference on prepared and non-prepared SELECT statements. His post is fairly old, so I ran timed tests on my server using our existing data and found PDO was within a couple of milliseconds. Using either library requires some recoding, but it is fairly straightforward and I was using a script to rebuild my script with the new functions. So I have a nice new server with nice new PHP and nice new MySQL queries using PDO … hit the page to test it and I get a generic error. Add a few lines to my code so I get some sensible errors



Voila – umm, this is gonna be a problem:

Next PDOException: SQLSTATE[HY000][2054] The server requested authentication 
method unknown to the client 
in D:\vhtml\PKIHome\IssuedDeviceCerts\index.php:46

Stack trace:

PDO->__construct('mysql:host=acil...', 'uidsuppressed', 'pwdsuppressed',Array)

The ‘server requested authentication method unknown to the client” means that the new PDO and MySQLi (yes, I’ve tried both) cannot use the password as required for the currently running production code. And the library used in the currently running production code cannot use the password as required for PDO or MySQLi. I cannot just convert the code to the new method, drop it on the new server, cut over, and decommission the old box. There are two approaches that can be used:


#1 Recode against a development MySQL database

#2 Get new IDs using the new storage scheme


#1 If you have a development MySQL database, you can add a hosts file entry (or have your OS support team do so) to point your production database host to the development server. The development server should be refreshed with data from the production databases. The existing IDs that use the old password storage schema can be updated with the new password storage scheme (either you provide the current password or a new password can be set). You will then need to update your PHP code to use either PDO or MySQLi. The implementation CRQ to move to your new server then involves (a) having the DBAs update the production user ID to the new password storage scheme, (b) removing the hosts file from your server.


Advantage – You don’t need to change to a new user ID in your code.

Disadvantage – anything that uses this ID needs to be updated simultaneously. When the new password storage schema is used on the account, any client requiring the old password storage scheme will fail. If your ID is used for one specific application on one server, then this isn’t a big deal. If you
use the ID to write data from a batch server or middleware platform, and then read the data from a PHP site … you need to recode both to use a library that understands the new password storage scheme.


#2 The other option is to get a new ID created that uses the new password storage scheme and have the same permissions granted for that ID. You can then recode individual pages as they are moved to the new server, and the old ID can be removed when all of the sites using it have been moved.


Advantage –You don’t need to move everything at once.

Disadvantage – you are making more changes to your code and replacing all of your user IDs (if you have a MyODBC driver to link an Access table into the database or if you use the MyPHPAdmin site … you’ll need to remember the new account now).


This isn’t a fatal error that prevents the upgrades from being done, but it sure turned into more of an undertaking than I had originally anticipated! If you should happen to use PHP and MySQL using the old libraries and have a PHP 7 installation planned … it really isn’t just copy some files & update some function calls.


Home Automation Lagering

We are about to make mead (we got near 30 pounds of local honey!). In researching mead-making, different yeasts have different alcohol tolerances … so you make a dry mead by using a yeast with an alcohol tolerance at or above the level your starting gravity would yield if it were fully fermented. A sweeter mead means you have a yeast whose tolerance is lower than that value … the greater the difference, the sweeter the mead. We are going to make a dry mead with Lalvin 71b-1122, a just slightly sweet mead by adding a little more honey but still using Lalvin 71b-1122, and a sweeter mead using Lalvin D-47.

71b-1122 has a very broad temperature range (59-86 F – and how cool is it that Google returns a yeast profile summary if you search for “71b-1122 temperature range”). D-47 is more particular — a published range of 59-68 F, but reading through homebrew sites has us wanting to stay around 63 degrees. Our sub-grade level is cool, but not that cool. Especially as fermentation warms up the fluid.

Scott is developing a home automation controlled fermentation “chamber”. The beer refrigerator is now plugged into a smart outlet. One of the Arduino kits we got has a temperature sensor. We can have a temperature probe monitoring the must and cycle the refrigerator’s power to keep it within a degree or two of our target.

Programming in Unknown Languages

I’ve often thought that the immersion method of learning a language was setting yourself up for failure – it isn’t like knowing the fundamentals of grammar and pronunciation in English helps you in any way when you find yourself in Karnataka trying to communicate in Sanskrit. There are rather complex algorithms that attempt to derive meaning from an unknown language, but apart from body language, pointing, and gesturing … that’s not something I can manage in real-time as someone speaks to me.

*Programming* languages, on the other hand, I am finding are rather easily learnt by immersion. I know several programming languages quite well – C/C++, F77/F90, perl, and php. I know a dozen or so other languages well enough to get by.

Some of our home automation scripts are written in CoffeeScript (which is evidently a way to write JavaScript without *actually* knowing JavaScript) – and I would never be able to write the program. But to come into the middle of the conversation (i.e. to take someone else’s non-functional code and try to fix it), I can glean enough of the language to debug and fix code. And there’s always Google for any syntax I cannot guess.

I wonder if someone who is fluent in multiple disparate languages (knowing half a dozen Romance languages doesn’t really give you a good base of knowledge – I mean someone who speaks Italian, Hindi, Cantonese, Swahili, and some Levantine dialect of Arabic) is able to do something similar — they know enough words to pretty much guess what words mean & enough different language structures to guess words in their context.

Using BC And Command Substitution In OpenHAB’s Exec Binding

My husband has been setting up OpenHAB to control our home automation. Our dimmers are very direct – there’s a z-Wave binding that you set to 100 if you want it at 100%, set it to 18 if you want it at 18%, and so on. We have a handful of Zigbee bulbs, though, which are not so direct. We are controlling these bulbs through a Wink hub by running a curl command with the exec binding.

The OpenHAB exec binding runs a shell with a command string passed in from the -c parameter. Thus far, I have not found anything that runs within a shell not work in the exec binding. This includes command substitution {I personally use the backtick format instead of the $(command) format, but I expect the later to be equally functional}.

What is command substitution (without having to read the Open Group Base Specifications linked above)? If you run

kill `pidof java`

the shell takes the component within the backticks, evaluates it, and then takes the standard output and places that into the command. When “pidof java” returns “938 984 1038”, the command above becomes “kill 938 984 1038”.

We want to set the value to the OpenHab value (0-100) scaled to the Wink value (0-255 for GE Link bulbs) using command substitution with bc (an arbitrary precision calculator language). To evaluate a mathematical expression, echo the expression text and pipe it to bc. To set a bulb to 75% of its maximum brightness, our post data is “nodeId=a&attrId=aprontest -u -m9 -t2 -v`echo 2.55*75/1|bc`”.

Notice the divide by 1 at the end — that’s to turn a decimal value into an integer. If you use just 2.55*75, you post a value of 191.25 which throws an error. In bc’s language, / returns the quotient — this isn’t *rounding* but rather truncating the decimal portion( i.e. bc 9.99999/1 = 9).

We configure the OpenHAB item to take the selected value (the %2$s below), scale the value with bc, and insert the result into the curl command. We use a similar technique to read the data from Wink and present the scaled value through OpenHAB.

The item entry in our sitemap.items file:

Dimmer  DS_Pantry_Bulb_Level                                            “Bulb (Pantry Downstairs) [%d]”                                 <slider>        (gZigbeeBulb,gDS_Pantry,gLight)                                                                                 { exec=”<[/bin/sh@@-c@@echo `/usr/bin/curl \”http://wink.hub.address/set_dev_value.php\” -s -d \”nodeId=a&attrId=aprontest -l -m9;\”|grep Level|grep -oP \”\\d+\\D+\\K\\d+\\D+\\K\\d+\”` /2.55|bc:3600000:] >[*:/bin/sh@@-c@@/usr/bin/curl \”http://wink.hub.address/set_dev_value.php\” -s -d \”nodeId=a&attrId=aprontest -u -m9 -t2 -v`echo 2.55*%2$s/1|bc`;\”]”}

Parsing JSON In JavaScript

We’ve been trying to get our BloomSky data parsed and reflected in OpenHAB — we can automatically turn the lights on when there is motion *and* the luminescence is lower than some desired value.  Bloomsky has an API which allows us to retrieve JSON formatted data from our weather station. I never worked with JSON before – I’d heard the term, but didn’t actually know what it was … but I needed to parse it in a JavaScript transform. Does JavaScript do JSON? D’oh! Turns out JSON is an abbreviation for JavaScript Object Notation, and JavaScript parses JSON data really well.

Still need to turn my example web code into a transform that runs from OpenHAB, but getting values out of a JSON formatted string is as easy as using the “parse” function:

	      function parseMyData() {
		var input = '{"DeviceID":"83237E","LAT":41.226644299999997,"LON":-81.7224322,"ALT":292.78720092773438,"UTC":-4,"DST":1,"Searchable":true,"RegisterTime":1464494138,"CityName":"Hinckley","StreetName":"Bellus Road","FullAddress":"Bellus Road, Hinckley, Ohio, US","DeviceName":"Buzzard Cam 01","BoundedPoint":null,"NumOfFollowers":5,"Data":{"Temperature":80.528000000000006,"ImageURL":"","Humidity":50,"Night":false,"ImageTS":1465938980,"Luminance":3445,"TS":1465938980,"Rain":false,"Pressure":29.087148500000001,"Voltage":2613,"UVIndex":"1"},"Point":{},"VideoList":["","","","",""],"NumOfFavorites":0}'

		var jsonOfInput = JSON.parse(input);

		document.write("<P>Device ID is: " + jsonOfInput.DeviceID + "</P>");
		document.write("<P>Temp is: " + jsonOfInput.Data.Temperature + "</P>");
		document.write("<P>Luminance is: " + jsonOfInput.Data.Luminance + "</P>");
	  <h2>Press the button to start</h2>
	    <input type="button" onclick="parseMyData()" value="Parse"/>

Tracking Electrical Usage With SmartThings and AeonLabs Home Energy Meters

When we started shopping for solar generation installations, how much electricity we can consume was a challenging question. We were replacing our HVAC system, so “look at the last 12 months of electric bills” wasn’t an approach that would yield valid data. What we needed was a way to see electrical consumption for the next two or three weeks once our new HVAC system was installed.

We purchased several AeonLabs Home Energy Meters (HEM) and have been using them to track our power consumption for almost six months now. The HEM’s are set up in SmartThings & have a SmartApp-HEMLogger “SmartApp” attached to them that posts data to a MySQL table on our server via a web form (myURL needs to be … well, your URL).

Install a quick MySQL server (does not need to be Internet accessible) and a web server / programming language of your choice combination (*does* need to be Internet accessible – we are using Apache and PHP).

Create the database and a table to hold your energy data:

mysql> describe EnergyMonitors;
| Field | Type | Null | Key | Default | Extra |
| keyID | int(11) | NO | PRI | NULL | auto_increment |
| monitorID | varchar(50) | YES | | NULL | |
| clampID | varchar(50) | YES | | NULL | |
| eventTime | timestamp | NO | | CURRENT_TIMESTAMP | |
| kwatts | double | YES | | NULL | |
| kwhours | double | YES | | NULL | |
6 rows in set (0.00 sec)

Create an ID within your database that has read/write permission to this table. I create another ID that has read-only access (pages displaying data use this ID, the page to post data uses the read/write ID).

We also track temperature — ideally, we’d be able to compare power consumption based on temperature *and* sunlight (we use a lot less power on a cold sunny day than I expect … just don’t know how much less) … but I’m not there yet. Currently, the weather database only holds temperature:

mysql> describe weather;
| Field | Type | Null | Key | Default | Extra |
| recordedTime | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| temperature | int(11) | YES | | NULL | |
| id | int(11) | NO | PRI | NULL | auto_increment |
3 rows in set (0.01 sec)

Since we brought our Bloomsky online, I use the Bloomsky API to record temperature. Prior to that, I was parsing the XML from NWS’s closest reporting station’s – for Cleveland, that is … there’s probably one near you. I grabbed both observation_time_rfc822 and temp_f in case observations were not updated regularly – the observation time was stored as the recordedTime. The temperature recording script is in cron as an hourly task.

You also need a small web page where SmartThings can post data — in PHP, I have

if(!$_POST["monitorID"] || !$_POST["clampID"] ){
echo "<html><body>\n";
echo "<form action=\"";
echo $_SERVER['PHP_SELF'];
echo "\" method=\"post\">\n";
echo "Monitor ID: <input type=\"text\" name=\"monitorID\"><br>\n";
echo "Clamp ID: <input type=\"text\" name=\"clampID\"><br>\n";
echo "KW Value: <input type=\"text\" name=\"kwatts\"><br>\n";
echo "KWH Value: <input type=\"text\" name=\"kWHours\"><br>\n";
echo "<input type=\"submit\">\n";
echo "</form>\n";
echo "</body></html>\n";
$strMeter = $_POST["monitorID"];
$strClamp = $_POST["clampID"];
$strKWValue = $_POST["kwatts"];
$strKWHValue = $_POST["kWHours"];
print "<pre>Meter: $strMeter\nClamp: $strClamp\nKW: $strKWValue\nKWH: $strKWHValue\n</pre>\n";
$servername = "DatabaseHostname";
$username = "MySQLUID";
$password = 'MySQLPassword';
$dbname = "HomeAutomation";
$conn = new mysqli($servername, $username, $password, $dbname);
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
if($strKWHValue && $$strKWValue){
$sql = "INSERT INTO EnergyMonitors (monitorID, clampID, kwatts, kwhours) VALUES ('$strMeter', '$strClamp', '$strKWValue', '$strKWHValue')";
$sql = "INSERT INTO EnergyMonitors (monitorID, clampID, kwhours) VALUES ('$strMeter', '$strClamp', '$strKWHValue')";
$sql = "INSERT INTO EnergyMonitors (monitorID, clampID, kwatts) VALUES ('$strMeter', '$strClamp', '$strKWValue')";
if ($conn->query($sql) === TRUE) {
echo "New record created successfully";
else {
echo "Error: " . $sql . "<br>" . $conn->error;


DatabaseHostname, MySQLUID and MySQLPassword need to be yours too. Since posted data is meant to come from a trusted source (a source where I’ve supplied the URL) and in a known good format, these are quick INSERT statements *not* the safest.

Check in your database that you are getting data, then let it run for a few hours. Once you have a little bit of history, you can start viewing your energy usage. Link it into Excel/Access using MyODBC for ad-hoc reporting, write your own code to do exactly what you want, use a generic charting package capable of reading MySQL data …

I am using pChart to display data – the chart I use most frequently is the stacked bar chart for kWH and a line chart for temperature. Here is the PHP code which generates this PNG: energyUsage-StackedBarChart-Flat – again, DatabaseHostname, MySQLUID, and MySQLPassword need to be yours.


We no longer have our heat pump, air handler, and heat strips being monitored – but for periods where there is data from the other sources, we had several segments to our energy usage report (“other” is the report from the HEM on our mains MINUS all of the other reporting segments). You can yank all of the non-mains segments (or change them to be whatever sub-segments you are monitoring. The monitor ID comes from the HEM name in SmartThings – since those are user-configured, I have the names hard coded. You *could* hard code the Mains and then use “select distinct” to get a list of all the others and make the code more flexible.).

Short term charts (past 24 hours or so) can render out real-time, but longer term views take a long time to load. Since we’re looking more for trends and totals – being up to the second isn’t critical. I’ve got cron tasks that generate out PNG files of the charts.

Two enhancements for a nothing-doing rainy day – adding authentication to the HEM Logger (SmartThings posts from multiple netblocks, so there isn’t a good way to IP source restrict access to the post data page. Anyone with your URL could post rogue energy usage info into your database – or more likely try to hack your servers.) and add lumen to the weather data / graph displays.