Tag: MySQL

Building LIB_MYSQLUDF_SYS On Fedora 31

I moved my MariaDB server to a new host and could not follow my previously working instructions to build lib_mysqludf_sys. The error indicated that my_atomic.h was not found.

[lisa@server03 lib_mysqludf_sys]# make
gcc -fPIC -Wall -I/usr/include/mysql/server -I. -shared lib_mysqludf_sys.c -o /usr/lib64/mariadb/plugin//lib_mysqludf_sys.so
In file included from /usr/include/mysql/server/my_sys.h:34,
from lib_mysqludf_sys.c:41:
/usr/include/mysql/server/my_pthread.h:26:10: fatal error: my_atomic.h: No such file or directory
26 | #include <my_atomic.h>
| ^~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:4: install] Error 1

The missing file is located in /usr/include/mysql/server/private … so I had to include that file in the gcc command as well. My new Makefile reads as follows:

[lisa@server03 lib_mysqludf_sys]# cat Makefile
LIBDIR=/usr/lib64/mariadb/plugin/

install:
gcc -fPIC -Wall -I/usr/include/mysql/server -I/usr/include/mysql/server/private -I. -shared lib_mysqludf_sys.c -o $(LIBDIR)/lib_mysqludf_sys.so

I was then able to make and use install.sh to load it into MariaDB.

MySQL: Moving Data From One Table To Another

Our OpenHAB persistence data is stored in MySQL. There’s an “items” table which correlates each ItemName string to an ItemID integer. There are then Item#### tables that store persistence data for each item. If you rename an item, this means a new table is created and previous persistence data is no longer associated with the item. For some items, that’s fine — I don’t really care when the office light was on last month. But there’s persistence data that we use over a long term — outdoor temperature, luminance, electrical usage. In these cases, we want to pull the old data into the new table. There’s a quick one-liner SQL command that accomplishes this:

INSERT INTO NewTable SELECT * from OldTable;
e.g. INSERT INTO Item3857 SELECT * FROM Item3854;

You can drop the old table too:

DROP OldTable;

But I run a cleanup script against the item list so often don’t bother to remove tables one-off.

Cleaning Up Old OpenHAB Persistence Tables

So my husband asked for a program that would go out to the OpenHAB persistence database and identify all of the item tables that are no longer associated with active items. If you rename or delete an item from OpenHAB, the associated data is retained in the persistence database. Might be a good thing – maybe you wanted that data. But if it’s useless fluff … well, no need to keep the state changes from a door sensor that’s no longer around.

Wrote the code, and asked him how many days old he wanted the last update to be before the item table got dropped … and he told me this was a useless way to do it and maybe something really hadn’t updated in six months or three years and age of last update is no way to be identifying tables to be removed. Which, yeah, then why ask for it!? Then I needed to write something that takes a list of items from OpenHAB and identifies everything in the items table that does not appear in the OpenHAB list so those tables can be deleted. But I figured I’d post the original code too in case anyone else could use it. Both in perl, and neither in particularly well written perl. I trust the data and don’t want to protect against insertion attacks.

Drop tables for items that no longer appear in OpenHAB:

use strict;
use DBI;

my %strItemsFromOpenHAB = ();
open(INPUT,"./openhabItemList.txt");
while(<INPUT>){
        chomp();
        my $strCurrentItem = $_;
        $strItemsFromOpenHAB{$strCurrentItem}++;
}
close INPUT;

my $dbh = DBI->connect('DBI:mysql:openhabdb;host=DBHOST', 'DBUID', 'DBPassword', { RaiseError => 1 } );

my $sth = $dbh->prepare("SELECT * FROM items");
$sth->execute();
while (my @row = $sth->fetchrow_array) {
        my $strItemID = $row[0];
        my $strItemName = $row[1];
        if(! $strItemsFromOpenHAB{$strItemName} ){              # If the current item name is not in the list of items from OpenHAB
#               print "DELETE FROM items where ItemID = $strItemID\n";
                print "DROP TABLE Item$strItemID;  # $strItemName \n";
        }
}
$sth->finish();

$dbh->disconnect();
close OUTPUT;

 

Identify tables that have not been updated in iTooOldInDays days:

use strict;
use DBI;
use Date::Parse;
use Time::Local;

my $iTooOldInDays = 365;

my $iCurrentEpochTime = time();

my @strItems = ();
my $iItems = 0;

my $dbh = DBI->connect('DBI:mysql:openhabdb;host=DBHOST', 'DBUID', 'DBPassword', { RaiseError => 1 } );

my $sth = $dbh->prepare("SELECT * FROM Items");
$sth->execute();
while (my @row = $sth->fetchrow_array) {
        $strItems[$iItems++] = $row[0];
}
$sth->finish();

for(my $i = 0; $i < $iItems; $i++){ my $strTableName = 'Item' . $strItems[$i]; my $sth = $dbh->prepare("SELECT * FROM $strTableName ORDER BY Time DESC LIMIT 1");
        $sth->execute();
        while (my @row = $sth->fetchrow_array) {
                my $strUpdateTime = $row[0];
                my @strDateTimeBreakout = split(/ /,$strUpdateTime);
                my $strDate = $strDateTimeBreakout[0];
                my $strTime = $strDateTimeBreakout[1];

                my @strDateBreakout = split(/-/,$strDate);
                my @strTimeBreakout = split(/:/,$strTime);

                my $iUpdateEpochTime = timelocal($strTimeBreakout[2],$strTimeBreakout[1],$strTimeBreakout[0], $strDateBreakout[2],$strDateBreakout[1]-1,$strDateBreakout[0]);
                my $iTableAge = $iCurrentEpochTime - $iUpdateEpochTime;

                if($iTableAge > ($iTooOldInDays * 86400) ){
                        print "$strTableName last updated $strUpdateTime - $iUpdateEpochTime\n";
                }
        }
        $sth->finish();
}

$dbh->disconnect();
close OUTPUT;

Web-Accessible History From OpenHAB MySQL Persistence Database

My husband has wanted a quick/easy way to see the data stored in OpenHAB’s MySQL persistence database, so I put together a very quick (and ugly) PHP page that provides a list of all Items. If you click on an item, you can page through the item’s records. The index.php from the page is available here. You need a web server (I am using Apache on Fedora), PHP (I am using 5.6) and MySQLi (php-mysqlnd package).

This is a bit of paranoia on my part, but even on a page that is ONLY available internally … I don’t like to use an account with read/write access to display data. I create a new user and assign read access:

CREATE USER 'YourUserName'@'localhost' IDENTIFIED BY 'P#ssw0rdH3r3';
GRANT SELECT ON openhabdb.* to 'YourUserName'@'localhost';
FLUSH PRIVILEGES;

Then use *that* user in the php code. This example has a web server running on the database server – and you connect to the MySQL server via localhost. If your web server is located on a different host, you’ll need to create and grant ‘YourUserName’@ the web server hostname.

PHP 7.0 and MySQL Libraries

At work, we have some servers running unsupported operating systems. New servers are being built, and applications are being migrated from the old servers to the new. I started with a fairly easy scenario – a PHP web site running on Windows 2008 is moving to 2012. The new web server was handed off to me, and I loaded PHP. With PHP 5.6 active support ending at the end of this year, it made sense to install PHP 7. Copied code, tested site. Umm, massive fail.

Way back in PHP 5.5, the ext/mysql stuff (ext\php_mysql.dll for Windows folks) was deprecated. And if you are like me, you had a lot of old code from back when that was the way to connect to a MySQL database. And as your MySQL was upgraded past 4.1, you had the DBA’s setting old_password on your ID so your code continued to work.  But the old mysql libraries have been removed in PHP 7… and you need to use MySQLi or pdo_mysql to communicate with your database now.

Which one? Depends on what you need – I’ve been using PDO because I don’t need a procedural API (MySQLi provides a procedural API, PDO does not). PDO supports a dozen or so database drivers, MySQLi is just MySQL … so I’ll be able to use the same basic code to connect to MySQL, MS SQL, Oracle, and db2 (plus a handful of others that I don’t anticipate actually using, but who knows what the future holds).

I found a site (http://archive.jnrbsn.com/2010/06/mysqli-vs-pdo-benchmarks) where the individual has benchmarked MySQLi and PDO and doesn’t find much difference on INSERT statements but does see a non-negligible difference on prepared and non-prepared SELECT statements. His post is fairly old, so I ran timed tests on my server using our existing data and found PDO was within a couple of milliseconds. Using either library requires some recoding, but it is fairly straightforward and I was using a script to rebuild my script with the new functions. So I have a nice new server with nice new PHP and nice new MySQL queries using PDO … hit the page to test it and I get a generic error. Add a few lines to my code so I get some sensible errors

     error_reporting(E_ALL);
     ini_set('display_errors',1);

 

Voila – umm, this is gonna be a problem:

Next PDOException: SQLSTATE[HY000][2054] The server requested authentication 
method unknown to the client 
in D:\vhtml\PKIHome\IssuedDeviceCerts\index.php:46

Stack trace:

#0
D:\vhtml\PKIHome\IssuedDeviceCerts\index.php(46):
PDO->__construct('mysql:host=acil...', 'uidsuppressed', 'pwdsuppressed',Array)

The ‘server requested authentication method unknown to the client” means that the new PDO and MySQLi (yes, I’ve tried both) cannot use the password as required for the currently running production code. And the library used in the currently running production code cannot use the password as required for PDO or MySQLi. I cannot just convert the code to the new method, drop it on the new server, cut over, and decommission the old box. There are two approaches that can be used:

**************************************************

#1 Recode against a development MySQL database

#2 Get new IDs using the new storage scheme

**************************************************

#1 If you have a development MySQL database, you can add a hosts file entry (or have your OS support team do so) to point your production database host to the development server. The development server should be refreshed with data from the production databases. The existing IDs that use the old password storage schema can be updated with the new password storage scheme (either you provide the current password or a new password can be set). You will then need to update your PHP code to use either PDO or MySQLi. The implementation CRQ to move to your new server then involves (a) having the DBAs update the production user ID to the new password storage scheme, (b) removing the hosts file from your server.

 

Advantage – You don’t need to change to a new user ID in your code.

Disadvantage – anything that uses this ID needs to be updated simultaneously. When the new password storage schema is used on the account, any client requiring the old password storage scheme will fail. If your ID is used for one specific application on one server, then this isn’t a big deal. If you
use the ID to write data from a batch server or middleware platform, and then read the data from a PHP site … you need to recode both to use a library that understands the new password storage scheme.

**************************************************

#2 The other option is to get a new ID created that uses the new password storage scheme and have the same permissions granted for that ID. You can then recode individual pages as they are moved to the new server, and the old ID can be removed when all of the sites using it have been moved.

 

Advantage –You don’t need to move everything at once.

Disadvantage – you are making more changes to your code and replacing all of your user IDs (if you have a MyODBC driver to link an Access table into the database or if you use the MyPHPAdmin site … you’ll need to remember the new account now).

**************************************************

This isn’t a fatal error that prevents the upgrades from being done, but it sure turned into more of an undertaking than I had originally anticipated! If you should happen to use PHP and MySQL using the old libraries and have a PHP 7 installation planned … it really isn’t just copy some files & update some function calls.