Category: System Administration

Cleaning Up Old OpenHAB Persistence Tables

So my husband asked for a program that would go out to the OpenHAB persistence database and identify all of the item tables that are no longer associated with active items. If you rename or delete an item from OpenHAB, the associated data is retained in the persistence database. Might be a good thing – maybe you wanted that data. But if it’s useless fluff … well, no need to keep the state changes from a door sensor that’s no longer around.

Wrote the code, and asked him how many days old he wanted the last update to be before the item table got dropped … and he told me this was a useless way to do it and maybe something really hadn’t updated in six months or three years and age of last update is no way to be identifying tables to be removed. Which, yeah, then why ask for it!? Then I needed to write something that takes a list of items from OpenHAB and identifies everything in the items table that does not appear in the OpenHAB list so those tables can be deleted. But I figured I’d post the original code too in case anyone else could use it. Both in perl, and neither in particularly well written perl. I trust the data and don’t want to protect against insertion attacks.

Drop tables for items that no longer appear in OpenHAB:

use strict;
use DBI;

my %strItemsFromOpenHAB = ();
open(INPUT,"./openhabItemList.txt");
while(<INPUT>){
        chomp();
        my $strCurrentItem = $_;
        $strItemsFromOpenHAB{$strCurrentItem}++;
}
close INPUT;

my $dbh = DBI->connect('DBI:mysql:openhabdb;host=DBHOST', 'DBUID', 'DBPassword', { RaiseError => 1 } );

my $sth = $dbh->prepare("SELECT * FROM items");
$sth->execute();
while (my @row = $sth->fetchrow_array) {
        my $strItemID = $row[0];
        my $strItemName = $row[1];
        if(! $strItemsFromOpenHAB{$strItemName} ){              # If the current item name is not in the list of items from OpenHAB
#               print "DELETE FROM items where ItemID = $strItemID\n";
                print "DROP TABLE Item$strItemID;  # $strItemName \n";
        }
}
$sth->finish();

$dbh->disconnect();
close OUTPUT;

 

Identify tables that have not been updated in iTooOldInDays days:

use strict;
use DBI;
use Date::Parse;
use Time::Local;

my $iTooOldInDays = 365;

my $iCurrentEpochTime = time();

my @strItems = ();
my $iItems = 0;

my $dbh = DBI->connect('DBI:mysql:openhabdb;host=DBHOST', 'DBUID', 'DBPassword', { RaiseError => 1 } );

my $sth = $dbh->prepare("SELECT * FROM Items");
$sth->execute();
while (my @row = $sth->fetchrow_array) {
        $strItems[$iItems++] = $row[0];
}
$sth->finish();

for(my $i = 0; $i < $iItems; $i++){ my $strTableName = 'Item' . $strItems[$i]; my $sth = $dbh->prepare("SELECT * FROM $strTableName ORDER BY Time DESC LIMIT 1");
        $sth->execute();
        while (my @row = $sth->fetchrow_array) {
                my $strUpdateTime = $row[0];
                my @strDateTimeBreakout = split(/ /,$strUpdateTime);
                my $strDate = $strDateTimeBreakout[0];
                my $strTime = $strDateTimeBreakout[1];

                my @strDateBreakout = split(/-/,$strDate);
                my @strTimeBreakout = split(/:/,$strTime);

                my $iUpdateEpochTime = timelocal($strTimeBreakout[2],$strTimeBreakout[1],$strTimeBreakout[0], $strDateBreakout[2],$strDateBreakout[1]-1,$strDateBreakout[0]);
                my $iTableAge = $iCurrentEpochTime - $iUpdateEpochTime;

                if($iTableAge > ($iTooOldInDays * 86400) ){
                        print "$strTableName last updated $strUpdateTime - $iUpdateEpochTime\n";
                }
        }
        $sth->finish();
}

$dbh->disconnect();
close OUTPUT;

Configuring and Using RPZ

I realized today what, while I had written about why response policy zones are useful, I never indicated how to configure one! So here’s a quick document outlining how to set it up in ISC Bind. In your named.conf file, add a response policy to your options section:

        response-policy {
                zone “rpz”;
        };
Then add the correspondingly named zone at the end of the file. For purposes of testing, I added a zone as a forward only zone so I could perform a network capture to see what exactly transpires when a name in the RPZ is resolved.
zone “rpz” {
      type master;
      file “rpz.db”;
      allow-query { none; };
      allow-transfer { none; };
};
zone “windstream.com” {
    type forward;
    forward only;
    forwarders { 8.8.8.8; };
};
Then you just have to make a rpz.db where you store your named files:
$TTL 60
$ORIGIN rpz.
@            IN    SOA  localhost. root.localhost.  (
                          2   ; serial
                          3H  ; refresh
                          1H  ; retry
                          1W  ; expiry
                          1H) ; minimum
                  IN    NS    localhost.

www.windstream.com    CNAME    www.yahoo.com.
Restarted named and ran “rndc flush” to avoid serving cached content instead of the RPZ host data. Then ran a few tests and confirmed that the resolution configured in the rpz zone:
[lisa@fedora02 named]# dig +short www.windstream.com @localhost
www.yahoo.com.
atsv2-fp.wg1.b.yahoo.com.
98.139.183.24
98.138.252.30
98.139.180.149
98.138.253.109
[lisa@fedora02 named]# dig +short dell905.windstream.com @localhost
ns4.windstream.com.
173.186.244.139
[lisa@fedora02 named]# dig +short www.google.com @localhost
216.58.218.228
In this process, I learnt something interesting about ICS’s implementation of RPZ: it still performs the query and then overrides the results. Odd waste of cycles, but the resolution that was subsequently turned into yahoo’s address from the rpz zone. Looking up a windstream.com host that isn’t in my RPZ and I got another query out to 8.8.8.8 which was expected. Query to something not in the forward zone and not in the rpz zone and I get no traffic to 8.8.8.8 (because it follows my normal forwarding which is to our ISP’s DNS).
I was curious if this meant rpz could not be used to publish a bad hostname locally – but attempting to resolve a bad hostname (added abadhost.windstream.com with the same CNAME to Yahoo and reloaded my zone) worked just fine.

[root@fedora02 ~]# dig abadhost.windstream.com @localhost

; <<>> DiG 9.11.1-P2-RedHat-9.11.1-2.P2.fc26 <<>> abadhost.windstream.com @localhost
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8382
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 4, ADDITIONAL: 3

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 1aa34751c5df7f78857a921259a8706fb5e1741a46eb5352 (good)
;; QUESTION SECTION:
;abadhost.windstream.com. IN A

;; ANSWER SECTION:
abadhost.windstream.com. 5 IN CNAME www.yahoo.com.
www.yahoo.com. 1800 IN CNAME atsv2-fp.wg1.b.yahoo.com.
atsv2-fp.wg1.b.yahoo.com. 60 IN A 98.139.180.149
atsv2-fp.wg1.b.yahoo.com. 60 IN A 98.138.253.109
atsv2-fp.wg1.b.yahoo.com. 60 IN A 98.139.183.24
atsv2-fp.wg1.b.yahoo.com. 60 IN A 98.138.252.30

;; AUTHORITY SECTION:
wg1.b.yahoo.com. 172800 IN NS yf3.a1.b.yahoo.net.
wg1.b.yahoo.com. 172800 IN NS yf4.a1.b.yahoo.net.
wg1.b.yahoo.com. 172800 IN NS yf1.yahoo.com.
wg1.b.yahoo.com. 172800 IN NS yf2.yahoo.com.

;; ADDITIONAL SECTION:
yf1.yahoo.com. 86400 IN A 68.142.254.15
yf2.yahoo.com. 86400 IN A 68.180.130.15

;; Query time: 1204 msec
;; SERVER: ::1#53(::1)
;; WHEN: Thu Aug 31 16:24:15 EDT 2017
;; MSG SIZE rcvd: 315

But there is a query that goes out to the name server and a ‘no such name’ result returned. Odd.

DMARC and DKIM

Microsoft’s latest security newsletter included the fact that more than 90% of Fortune 500 companies have not fully implemented DMARC. Wow — that’s something I do at home! Worse still, the Fortune 500 company for which I work is in that 90% … a fact I hope to rectify this week. SPF is just some DNS entries that indicate the source IPs that are expected to be sending email from your domain. Lots of SPF record generators online.

DKIM is a little more involved, but it’s a lot easier now that packages for DKIM are available on Linux distro repositories. You still *can* build it from source, but it’s easier to install the OpenDKIM package.

Once the package is installed, generate the key(s) to be used with your domain(s).

cd /etc/opendkim/keys/
openssl genrsa -out dkim.private 2048
openssl rsa -in dkim.private -out dkim.public -pubout -outform PEM
# secure private key file
chown opendkim:opendkim dkim.private
chmod go-r dkim.private

Decide on the selector you are using — I use ‘mail’ as my selector. At work, I use ‘2017Q3Key’ — this allows us to change to a new key without in-transit mail being impacted. Old mail was sent with the 2017Q2 selector and *that* public key is in DNS. New mail comes across with 2017Q3 and uses the new DNS record to verify. I do *not* share these keys – anyone else sending mail from our domain needs to generate their own key (or I make one for them), use their own unique selector, and I will create the DNS records for their selector. When marketing engages a third party to send e-mails on our behalf, we have a 2017VendorName selector too.

Edit /etc/opendkim.conf. The socket line is not necessary – I just tend away from default ports as a habit. Since it’s bound to localhost, not such a big deal.

Mode sv
Socket   inet:8895@localhost
Selector mail
KeyFile /etc/opendkim/keys/dkim.private
KeyTable /etc/opendkim/KeyTable
SigningTable refile:/etc/opendkim/SigningTable
InternalHosts refile:/etc/opendkim/TrustedHosts

There’s a config option to “SendReports” — it’s a boolean that indicates if you want your system to send failure reports when the sender indicates they want such reports and provide a reporting address. Especially for testing purposes, I recommend indicating your domain wants reports — it is helpful in case you’ve got something configured not quite right and are failing delivery on some messages. As such, configure my installation to send reports. It’s additional overhead in cases where verification fails; I don’t see all that many failures, and it isn’t a lot of extra load. Since I know my installation will send detailed failure information, I can use my domain when testing new implementations.

Once you have the base configuration set, edit /etc/opendkim/SigningTable and add your domain(s) and the appropriate selector

*@rushworth.us mail._domainkey.rushworth.us
*@lisa.rushworth.us mail._domainkey.lisa.rushworth.us
*@scott.rushworth.us mail._domainkey.scott.rushworth.us
*@anya.rushworth.us mail._domainkey.anya.rushworth.us

Edit /etc/opendkim/KeyTable and map each selector from the SigningTable to a key file

mail._domainkey.rushworth.us rushworth.us:default:/etc/opendkim/keys/dkim.private
mail._domainkey.lisa.rushworth.us lisa.rushworth.us:default:/etc/opendkim/keys/lisa.dkim.private
mail._domainkey.scott.rushworth.us scott.rushworth.us:default:/etc/opendkim/keys/scott.dkim.private
mail._domainkey.anya.rushworth.us anya.rushworth.us:default:/etc/opendkim/keys/anya.dkim.private

Edit /etc/opendkim/TrustedHosts and add the internal IPs that relay your domain’s mail through the server (IP addresses or subnets)

Create DNS TXT records – the part after p= is the content of the public key file for that selector. When you are first setting up DKIM, use t=y (yes, we are just testing this). Once you confirm everything is functional, you can change to y=n (nope, really pay attention to our DKIM signature and policy). The policy is an individual preference. I use ‘all’ (all mail from my domain will be signed) and “o=-” (again all mail from my domain will be signed). You can use “o=~” (some mail from my domain is signed, some isn’t … who knows) and “dkim=unknown” (again, some is signed). You can use “dkim=discardable” (don’t just consider the message as more likely to be spam if it is not signed … you can outright drop the message). As a business, I don’t use this *just in case*. Something crazy happens – the dkim service falls over, your key gets mangled – and receiving parties can start dropping your messages. Using “dkim=all” means they are more apt to quarantine them as spam, but someone can go and get the messages. And hopefully notice something odd is happening.

mail._domainkey.domain.tld  TXT k=rsa;t=y;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzTnpc7tHfyH1zgT3Jx/JHmGSz8WCy1jvzu5QsYvDBmimKEHRY4Kz4mya5bOYsDQuJ/sz+BJo6xDwsUXCuyEkykIlgqP+7E9oK2EcW0dZms87SGmNEnNBN5iTe0pdzk1lXx2js3QdOWswO+cmA9F1Z8OzSR+2u79huugPFBHl79zFvOEHbigrmeHEfo0KHWpeNomf/xKx+wyYr1n3R5gS+28CeC3abSyKgmaYYRLoZsjrCLbEM0m2YPJRKd1ZGOObBMa4PZWj7pT07ISEjoNnXQ27BtcL/QjKKeLkbJ0UGEOSdPEJKuEpAUvYU9lA5hbtzrqiwdlPxWYocDVPrcqAHwIDAQAB
_adsp._domiankey.domain.tld TXT dkim=all
_domainkey.rushworth.us TXT t=y;o=-;r=dkim@lisa.rushworth.us
_ssp._domainkey.rushworth.us TXT t=y;dkim=all

 

Edit /etc/mail/sendmail.mc (using the port defined in /etc/opendkim.conf

INPUT_MAIL_FILTER(`dkim-filter’, `S=inet:8895@localhost’)

Make your sendmail.mc to sendmail.cf and verify that you’ve got the dkim-filter line

Xdkim-filter, S=inet:8895@localhost

Start opendkim, then restart sendmail. Now test it — inbound mail should have *their* DKIM signatures verified, outbound mail should be signed with the appropriate key.

Once you have verified your DKIM is functioning properly — well, first of all you can update your DNS records to remove testing mode. Then create your DMARC record:

_dmarc.rushworth.us     v=DMARC1; p=quarantine; sp=quarantine; rua=mailto:dmarc-rua@lisa.rushworth.us!10m; ruf=mailto:dmarc-ruf@lisa.rushworth.us!10m; rf=afrf; pct=100; ri=172800
 Again, you don’t need to use quarantine — ‘reject’ would recommend mail be dropped or ‘none’ recommends no action (good for testing). The rua (aggregate reporting email address) and ruf (address to recieve failing samples for analysis) should be in your domain.
You could add either/both “adkim=s” or “aspf=s” to indicate your DKIM or SPF adhere to strict standards. I use relaxed (default, do not need to specify it in the TXT record).
If you want the reports delivered to an address outside of your domain, that domain needs to publish a DNS record authorizing receipt of the reports:
     rushworth.us._report._dmarc.lisa.rushworth.us     v=DMARC1

Sendmail VirtUserTable

Some mail systems support sub-addressing (i.e. user+ignoredstring@domain.tld), but Exchange is not one of them. Instead of trying to sort out sub-addressing through sendmail or Exchange, we use sendmail’s virtusertable to map entire subdomains (i.e. @lisa.rushworth.us) over to our primary e-mail addresses. If an address becomes compromised, we can blacklist the particular something@subdomain.rushworth.us address in the access table).

Virtual Domain Aliases

These aliases allow changes to be made to intended recipient addresses.  There are two files required for an address to be aliased.  An entry for “VIRTUSER_DOMAIN_FILE” will exist in the sendmail.mc specifying the file listing the domains to be included for aliasing.  For us, this is /etc/mail/virtuser-domains.  This is a text file containing the name of each domain to be virtualized for aliasing, one domain per line.  Please note, the domains included herein need only be the recipient domains, not the domains to which aliases are mapped.  E.G. our virtuser-domains file contains just:

rushworth.us

And yet we can alias test.addy@rushworth.us to lisa.landers@vibiant.dnsalias.com … it is only the source address that needs to be defined in virtuser-domains.

Aliases for the virtual domains are contained in /etc/mail/virtusertable.  The left-hand entry is the recipient address and the right-hand entry is what that recipient will be translated to.  Left-hand entries can be an email address (testaddy@rushworth.us) or a domain (@lisa.rushworth.us)

Right-hand entries can be an alternate address.  If the address should remain the same, an exclamation point can be used:

myfakeaddress@rushworth.us        external.email@domain.tld
myaddress@rushworth.us            !

The right-hand entry can also be an action, like error which will return an error code

compromised.address@lisa.rushworth.us            error:nouser User unknown

 

To commit changes to the virtusertable:

makemap hash /etc/mail/virtusertable.db < /etc/mail/virtusertable

 

Testing Virtual Aliases:

You can test the results of the virtual address space aliasing using sendmail –bt.  From within the new prompt (a greater than sign on a blank line) type3,0 followed by the address you would like to test.  E.G.:

[uid@NEOHTWNLX821 ~]# sendmail -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 3,0 llanders@windstream.com
canonify           input: llanders @ windstream . com
Canonify2          input: llanders < @ windstream . com >
Canonify2        returns: llanders < @ windstream . com . >
canonify         returns: llanders < @ windstream . com . >
parse              input: llanders < @ windstream . com . >
Parse0             input: llanders < @ windstream . com . >
Parse0           returns: llanders < @ windstream . com . >
ParseLocal         input: llanders < @ windstream . com . >
ParseLocal       returns: llanders < @ windstream . com . >
Parse1             input: llanders < @ windstream . com . >
Recurse            input: llanders @ valortelecom . com
canonify           input: llanders @ valortelecom . com
Canonify2          input: llanders < @ valortelecom . com >
Canonify2        returns: llanders < @ valortelecom . com . >
canonify         returns: llanders < @ valortelecom . com . >
parse              input: llanders < @ valortelecom . com . >
Parse0             input: llanders < @ valortelecom . com . >
Parse0           returns: llanders < @ valortelecom . com . >
ParseLocal         input: llanders < @ valortelecom . com . >
ParseLocal       returns: llanders < @ valortelecom . com . >
Parse1             input: llanders < @ valortelecom . com . >
Mailertable        input: < valortelecom . com > llanders < @ valortelecom . com . >
Mailertable        input: valortelecom . < com > llanders < @ valortelecom . com . >
Mailertable      returns: llanders < @ valortelecom . com . >
Mailertable      returns: llanders < @ valortelecom . com . >
MailerToTriple     input: < > llanders < @ valortelecom . com . >
MailerToTriple   returns: llanders < @ valortelecom . com . >
Parse1           returns: $# esmtp $@ valortelecom . com . $: llanders < @ valortelecom . com . >
parse            returns: $# esmtp $@ valortelecom . com . $: llanders < @ valortelecom . com . >
Recurse          returns: $# esmtp $@ valortelecom . com . $: llanders < @ valortelecom . com . >
Parse1           returns: $# esmtp $@ valortelecom . com . $: llanders < @ valortelecom . com . >
parse            returns: $# esmtp $@ valortelecom . com . $: llanders < @ valortelecom . com . >

Sendmail Mailertable

Mailertable (/etc/mail/mailertable)

Routing information for external delivery. Functionally, these are like the SMTP Connectors within Exchange. The mailertable entries can override everything including smarthost definitions. This is required for internal mail routing – our sendmail servers should not transmit email for @windstream.com to the MX records but rather the destination we intend. We also use mailertable entries to force B2B communication over internal secured channels.

If a server is unable to deliver mail to a specific domain (e.g. one of our public IP addresses gets blacklisted), a mailertable entry can be used to direct all mail destined for the domain through one of our servers still able to make delivery.

The file contains two columns, domains and actions. Domains can be ends-with substring matches:
.anythingfromthisdomain.com

Will match @thishost.anythingfromthisdomain.com as well as @thathost.anythingfromthisdomain.com. Domains can also be a full match of the right-hand side of the email address:
justthisemaildomain.com

Which will match @justthisemaildomain.com. The most “accurate” match will win, not just the first match in line. So if your file contains the following:
.mysampledomain.com relay:[10.10.10.10]
thishost.mysampledomain.com relay:[20.20.20.20]

Mail destined for thishost.mysampledomain.com will be sent to 20.20.20.20

Actions contain both a mailer and a host. The mailer can redirect messages to local users:
.egforlocaldelivery.com local:username

Or it can force an error response:
Baddomain.com error:5.3.0:Unknown User

Our use of the mailertable, though, is to redirect mail destined for the domain:
windstream.com relay:[twnexchinbound.windstream.com]
newacquisition.com relay:[theirinternalhost.theirdomain.com]

In these cases, the square brackets around the destination override the MX record. To reroute a domain’s delivery destination, then, it is imperative that the host be enclosed in square brackets.

To commit changes to the file, either use “make” from within /etc/mail to commit all changes or the following command to commit just the changes to mailertable:
makemap hash /etc/mail/mailertable < /etc/mail/mailertable

Sendmail Configuration

Sendmail Configuration Files – sendmail.cf and sendmail.mc

 

Sendmail configuration files are located by default in /etc/mail/.  PureMessage uses /opt/pmx4/sendmail/etc/mail/

 

The main configuration file is sendmail.cf.  This is a rather cryptic file which we will not configure directly.  If you want to know the syntax for sendmail.cf, read the doc at http://www.sendmail.org or get the O’Reily book.  This information is specific to the MC file from which a macro builds the CF file..

 

sendmail.mc contains instructions to allow the M4 macro processor to build sendmail.cf.  Very important, before you can use a macro to create a sendmail.cf file, you need to have the macro installed.  This is the sendmail package sendmail-cf.  To ascertain if the package has been installed on RedHat:

 

[root@LJLLX001 mail]# rpm -qa | grep sendmail

sendmail-8.13.1-2

sendmail-cf-8.13.1-2

 

Both sendmail and sendmail-cf packages should appear in the results.  If you do not have the CF package, install it.

 

The text “dnl” within sendmail.mc denotes a comment – like a tic in VisualBasic or a hash in perl.  Many lines end with dnl, or dnl with some type of commentary.  Lines beginning with dnl are not processed.

 

Common instructions within a sendmail.mc file:

 

include(`/usr/share/sendmail-cf/m4/cf.m4′)dnl

This line refers the m4 utility to the correct “translation” to build the sendmail.cf file.  Important that the line is at the top of the mc file, but nothing to do with sendmail configuration specifically

 

VERSIONID(`setup for Red Hat Linux’)dnl

This line is not required, and we have ‘junk’ in it frequently.  It records the version of sendmail in the cf file for administrative reference.

 

OSTYPE(`linux’)dnl

More instructions for m4, different OS’s have different locations for sendmail files and the OS defined here identifies which parameters to use.  This line again needs to be at the top of the mc file

 

define(`confDEF_USER_ID’,“8:12”)dnl

Defines which user and group sendmail will run as – do NOT pick root here.  User id 8 (mail) and group id 12 (mail) from /etc/passwd and /etc/groups respectively.

 

define(`confTO_CONNECT’, `1m’)dnl

Time limit for SMTP connection timeout, set to one minute normally.  This is how long your server will wait for an initial connect() to complete.

 

define(`confTRY_NULL_MX_LIST’,true)dnl

Email is normally routed by MX records.  This instruction means the ‘domain’ can also be a host name with no MX defined.  E.G.  sending email to @windstream.com will return the MX records, as they exist.  Attempting to email @neohtwnlx810.windstream.com will return no MX records, but LX810 will be contacted directly to attempt delivery.  This is a most useful instruction for return delivery to system mailers.

 

define(`confDONT_PROBE_INTERFACES’,true)dnl

The sendmail class w lists the host and IP addresses for which sendmail accepts and takes local delivery.  This class can be automatically populated, or using this directive not automatically populated.  We configure this information manually in other files.

 

You can use a sendmail command line to determine what is set to various system variables:

[root@LJLLX001 ~]# sendmail -d0.1 -bv

Version 8.13.1

Compiled with: DNSMAP HESIOD HES_GETMAILHOST LDAPMAP LOG MAP_REGEX

MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6

NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS

USERDB USE_LDAP_INIT

============ SYSTEM IDENTITY (after readcf) ============

(short domain name) $w             =      LJLLX001

(canonical domain name) $j        =      LJLLX001.vibiant.dnsalias.com

(subdomain name) $m                 =      vibiant.dnsalias.com

(node name) $k                             =      LJLLX001.vibiant.com

========================================================

 

define(`PROCMAIL_MAILER_PATH’,`/usr/bin/procmail’)dnl

Exactly what it says – the location of procmail

 

define(`ALIAS_FILE’, `/etc/aliases’)dnl

Location of the file for local delivery aliases – not something we use often as there are few local delivery accounts.  In the ISP, this file can be used to give someone additional addresses which deliver to the same mailbox.  This file can also be used to direct delivery of a local account to a program – in PureMessage for example, /opt/pmx4/sendmail/etc/mail/aliases directs the pmx-auto-approve address to the application which releases user messages.

 

define(`confBIND_OPTS’, `WorkAroundBrokenAAAA’)dnl

This is a resolver option, it instructs sendmail to ignore SERVFAIL errors during an IPv6 lookup.  We had a few domains for which we could not deliver mail without this directive.

 

define(`SMART_HOST’, `[192.168.1.53]’)

A smart host can be used instead of direct mail delivery.  For a server which is not meant to deliver mail to the internet (neohtwnlx824 for instance) the smart_host directive sends all mail to the defined destination.  The destination can be a hostname or an IP address.  Note, the mailertable will override the smarthost.

 

define(`STATUS_FILE’, `/var/log/mail/statistics’)dnl

Retains statistical information on server – use the command mailstats to output the statistics, the file created here is not text

 

define(`UUCP_MAILER_MAX’, `2000000′)dnl

Maximum size for messages relayed by UUCP mailers

 

define(`confPRIVACY_FLAGS’, `authwarnings,novrfy,noexpn,restrictqrun’)dnl

Disables unwanted commands – usually for security reasons.  EXPN expands groups into component members, for instance, so NOVRFY is used to disable the command.  Some of these are more important if local delivery is handled by the sendmail server.

 

define(`confAUTH_OPTIONS’, `A’)dnl

What kinds of authentication are supported by the server.  Useful if you are requiring authentication to relay mail, we do not do this.  Some UNIX hosts get confused if AUTH is an option made available, and you need to remark this line out of the mc file.

 

define(`confTO_QUEUEWARN’, `6d’)dnl

If you ever see an email from a destination mail server saying it is still trying to deliver your message and just wanted to let you know – that is what this interval defines.  To truly adhere to RFC specifications, a sendmail server should continue to attempt delivery for at least four to five days.  As a “nice” feature, the server can send periodic notifications to the sender that delivery has been delayed.  This standard comes from a time when circuits were smaller and quite lossy.  It could reasonably take days to establish a connection to the destination and transmit a message.

We are rogue and just return mail as undeliverable after a shorter period.  No reason to notify users, but to ensure that a notification is not sent, we put the warning interval at something higher than the expiration interval.

 

define(`confTO_QUEUERETURN’, `12h’)dnl

Related to the QUEUEWARN interval – this is the period after which the sendmail server considers the message undeliverable and returns it to the sender.  By default, this is five days so we make sure to define something more reasonable.  Otherwise there would be no way to identify “high” mail queue counts for alerting.

 

define(`confQUEUE_LA’, `16′)dnl

Load average at which queue only functionality is engaged

 

define(`confREFUSE_LA’, `48′)dnl

Load average at which SMTP connections are refused

 

define(`confDELAY_LA’, `30′)dnl

Load average at which sendmail will delay one second on SMTP commands

 

define(`confMIN_QUEUE_AGE’, `5m’)dnl

Minimum time a message has to sit in the queue before it is retried

 

define(`confTO_HOSTSTATUS’, `2m’)dnl

If a host has been denoted as unavailable, the status will be cached for this duration.  After the interval expires, connection to the host will be retried

 

define(`confMAX_DAEMON_CHILDREN’, 2000)

Maximum number of children processes permitted.  Sendmail will reject subsequent connections once this number has been reached.  Very important to have something defined on the DMZ servers.  Default is infinite and it is possible for a server to become unresponsive and need to be rebooted with out of memory errors when too many processes are spawned.

 

define(`confTO_IDENT’, `0′)dnl

Timeout for responses to IDENT

 

FEATURE(`no_default_msa’,`dnl’)dnl

The default MSA options are not used, but rather explicitly defined in the DAEMON_OPTIONS directive

 

FEATURE(`smrsh’,`/usr/sbin/smrsh’)dnl

Shell used for command line mailing programs, not really pertinent in our case

 

FEATURE(`mailertable’,`hash -o /etc/mail/mailertable.db’)dnl

This file will be discussed in more detail later, this directive specifies the use of a mailertable and the location of the file.

 

VIRTUSER_DOMAIN_FILE(/etc/mail/virtuser-domains)dnl

This file will be discussed in more detail later, this directive specifies the location of the file containing virtualised domains

 

FEATURE(`virtusertable’,`hash -o /etc/mail/virtusertable.db’)dnl

This file will be discussed in more detail later, this directive specifies the use of virtual user mapping and the location of the file containing said mappings

 

FEATURE(always_add_domain)dnl

Appends the local host domain to even locally delivered mail.

 

FEATURE(use_cw_file)dnl

Alternate host names are in /etc/mail/local-host-names – machine aliases

 

FEATURE(use_ct_file)dnl

Users who can set alternate envelope from addresses without generating a warning message.  File is /etc/mail/trusted-users

 

FEATURE(local_procmail,`’,`procmail -t -Y -a $h -d $u’)dnl

Specifies program to use as the local mailer, and command options

 

FEATURE(`access_db’,`hash -T<TMPF> -o /etc/mail/access.db’)dnl

This file will be discussed in more detail later, this directive specifies the use of an access restriction table and the location of the file.

 

EXPOSED_USER(`root’)dnl

 

 

DAEMON_OPTIONS(`Port=smtp, Name=MTA’)dnl

This is where the settings for the MSA are defined.  Port=smtp uses the default port of 25, or an alternate port can be used.  Addr=# can be included to bind sendmail to a specific address (including 127.0.0.1 for localhost access only).

 

INPUT_MAIL_FILTER(`vamilter’,`S=inet:3333@localhost,F=R,T=S:10m;R:10m;E:10m’)

Defines a “milter” – mail filter.  The port and destination of the milter must be included with S=.  S=inet is a IPv4 socket, S=inet6 is an IPv6 socket, and S=local is a Unix-domain socket (/var/run/)

F= defines an action to take on failure, R (reject), T (tempfail), or if no option is included just pass the message through sendmail and ignore the milter

T= defines timeouts for sendmail’s communication with the milter:

C         Connect timeout

S          Sending timeout (sendmail transmission of data to milter)

R          Reading timeout (for reply from milter)

E          Overall timeout (between sending end of message and final ack)

 

MASQUERADE_AS(`vibiant.dnsaliascom’)dnl

FEATURE(`masquerade_envelope’)dnl

FEATURE(`allmasquerade’)dnl

MASQUERADE_DOMAIN(`arlitljl.com’)dnl

MASQUERADE_DOMAIN(`homedomain.local’)dnl

This group of directives are all interrelated.  Masquerading is basically replacement – MASQUERADE_AS is the domain which will be used in place of the domains identified in MASQUERADE_DOMAIN lines.  In this case, both @arlitljl.com and @homedomain.local will be overwritten with @vibiant.dnsalias.com.  The directive FEATURE(masquerade_entire_domain) could be included to replace any subdomain of the masquerade domains (e.g. @secured.arlitljl.com, @public.arlitljl.com, and @restricted.arlitljl.com in addition to @arlitljl.com)

Masquerade envelope applies the masquerade to the envelope information and allmasquerade applies the masquerade to everything in the envelope, including cc:, from: and to: — this directive is important when we mask an acquired company’s email domain with our own.

 

FEATURE(`accept_unresolvable_domains’)dnl

Allows the use of domains in the MAIL FROM command to be invalid network and sender domains.  Since some people do not manage to configure their mail servers properly, we are less restrictive here to avoid complaints.

 

LOCAL_DOMAIN(`localhost.localdomain’)dnl

Domain(s) for which the server will accept local delivery – since our servers do not really deliver mail the domain should include the localdomain to prevent accidental misdirection of mail

 

MAILER(smtp)dnl

MAILER(procmail)dnl

Defines mailers to be used in addition to local – these should be the last lines of the mc file

 

 

 

When you make changes to the sendmail.mc file, you will need to run the macro processor to update the CF file.  You can see the results by running:

m4 sendmail.mc | less

 

The text which will be used in sendmail.cf will be displayed on the screen.  To actually commit the changes, use:

m4 sendmail.mc > sendmail.cf

or just type

make

 

Make will update all of the files in /etc/mail, so ensure you like all the changes you have made, not just the changes to sendmail.mc

Sendmail

 

Sendmail is an OpenSource SMTP mail transfer agent implemented on many different Unix platforms. The original version of sendmail, written in the early 1980’s, was written by Eric Allman at Berkeley.  The release code base of sendmail is version 8.  The packages and source can be found at http://www.sendmail.org.

 

Sendmail in its current iteration is configured by many individual files.  All of the configuration options available within the product are well documented at http://www.sendmail.org/doc and http://www.sendmail.org/m4/readme.html.

Future Releases:

There is not a code base 9, but rather SendmailX which has now become MeTA1 (http://www.meta1.org/).  MeTA1 does not include a local delivery agent or mail submission program – it is intended as a conduit for email only.  It will use a single configuration file with a radically different syntax.  Currently, the summer of 2007, the code is in a pre-alpha release.  So, it will be a while.

 

Practical Information:

We back the sendmail configuration files up nightly to NEOHTWNLX810 (/home/NDSSupport/Backups/).  You can restore the files from /etc/mail (or /opt/pmx4/sendmail/etc/mail as appropriate) to a rebuilt server and return the server’s complete configuration.

 

Mail Queues:

Sendmail stores unsent relayed messages in /var/spool/mqueue.  Unsent locally submitted messages will first be in /var/spool/clientmqueue.  Within the mqueue folder, each message has two separate files, one for the header information and a second for the message data.  To count the number of messages queued for delivery, then, you need to divide the number of files within /var/spool/mqueue in half:

echo `ls -al /var/spool/mqueue | wc -l` / 2 | bc

 

New Email Domain Configuration:

We have all of the resources required to establish a new email domain.  (Registration may well be required for a new DNS zone).  To establish a new publicly functional email domain from an existing DNS zone:

  • Create MX records within the DNS zone.  The 10 weight record should point to neohtwnlx821.windstream.com. and the 20 weight record should point to neohtwnlx823.windstream.com.  It is important in these MX records to include the period trailing the hostname
  • On NEOHTWNLX821 and NEOHTWNLX823, edit /etc/mail/access to include the new domain with RELAY
  • On NEOHTWNLX821 and NEOHTWNLX823, edit /etc/mail/mailertable to direct mail to the appropriate destination (unix host, Exchange server, etc)
    • The destination must be configured to accept email from LX821/LX823
  • If internal mail routing needs to be established, an SMTP connector needs to be added to the Exchange organization.  Additionally, mailertable entries should be created on at minimum LX825, LX830, LX833, and LX828
  • If mail should be delivered to mailboxes in the Exchange organization, the new domain should be added to the “Additional Mail Domains” recipient policy.  In this case, the SMTP connector would not be created with Exchange.

 

Sendmail Troubleshooting:

To display information about queued messages:

sendmail –bp

Or to obtain analysis of the domains and addresses within the mail queue, use the perl scripts located in /root/bin:

frombydomain.pl                       Ascending count of sender domains

frombyemail.pl                          Ascending count of sender email addresses

tobydomain.pl                           Ascending count of recipient domains

tobyemail.pl                              Ascending count of recipient email addresses

 

To retry the queues messages with output to the terminal:

sendmail –v –q –C/etc/mail/sendmail.cf &

 

To retry a specific recipient domain’s queue:

sendmail –v –qRthedomain.com –C/etc/mail/sendmail.cf &

Or a specific sender domain’s queue:

sendmail –v –qSthedomain.com –C/etc/mail/sendmail.cf &

 

To retry a specific message ID:

sendmail –v –qImsgidgoeshere –C/etc/mail/sendmail.cf &

 

Add “-d8.11” to the queue retry commands to output debug level diagnostic information to the terminal.  E.G.

sendmail –v -qIl6UJtCE3021014 –C/etc/mail/sendmail.cf –d 8.11

 

Linux Authentication Over Key Exchange

On Linux, you can log in without logging in (essential for non-interactive processes that run commands on remote hosts, but also nice accessing hosts when you get paged at 2AM to look into an issue). The first thing you need is a key. You can use the openssh installation on a server to generate the key:

ssh-keygen -t rsa -b 2048

Or you can run puttygen.exe (www.chiark.greenend.org.uk/~sgtatham/putty/download.html or our Samba share on neohtwnlx810, or cwwapp556, or twwapp081) for a GUI key generator.

Click the “Generate” button & then move the mouse around over the blank area of the PuttyGen window – your coordinates are used as random data for the key seed.

Once the key is generated, click “save public key” and store it somewhere safe. Click “save private key” and store it somewhere safe. ** Key recovery isn’t a big deal – you can always generate a new public/private key pair and set it up. Time consuming if you out your key is a lot of places, but it isn’t a data loss kind if thing. *** Anyone who gets your private key can log in as you anywhere you set up this key exchange. You can add a passphrase to your key for additional security.

Once you’ve saved your keys, copy the public key at the top of the window. You don’t have to – you can drop the newline characters from the saved public key file, but this saves time.

Go to whatever box you want to log into using the key exchange. ** I have a key exchange set up from my Windows boxes (laptop, cwwapp556, and twwapp081) to myid@neohtwnlx810. I then have a different key used from myid@neohtwnlx810 to all of our other boxes. This allows me to change my on laptop key (i.e. the one more likely to get lost) out more frequently without having to get a new public key on dozens of hosts.

Once you are on the box you want as the ID you want (you can do a key exchange to any id for which you know the password – so you can log into ldap@vml106 or sendmail@vml905 and do this). Run “cd $HOME/.ssh” – if it says no such file, run “ssh localhost” – it will ask you if you want to store the server public key – say yes, that creates the .ssh folder with proper permissions. Ctrl-c and cd .ssh again. Now Check if there is an authorized_keys, authorized_keys2, or both. Vi whatever ones you find – if there aren’t any, try “vi authorized_keys” first– go into edit mode and paste in the public key line we copied earlier. Save the file. If you get an error like “The server refused our key”, you can “mv authorized_keys authorized_keys2” (what I need to do on RHEL <=6.7 although we probably should look into that!).

In putty, load in your configuration for whatever host we just pasted the public key into. Under Connection -> Data, find the “Auto-login username” section. Put in whatever ID you used when you added the public key (my use case is me e0082643 … but if you were using ldap@vml106, you would put ldap in here)

Then under Connection ->SSH->Auth, find the “private key file for authentication” section and put in your private key location. Go back to the Session section and save the configuration changes.

Now connect & you shouldn’t need to supply a password (or you only need to supply your key passphrase).

** OpenSSH automatically uses the id_dsa or id_rsa (private keys) from $HOME/.ssh/ when you attempt to authenticate to other hosts. If the destination id@host has your public key in its $HOME/.ssh/authorized_keys (or $HOME/.ssh/authorized_keys2 if you happen to be using a deprecated version), then you’ll get magic key based authentication too. Caveat: on the source Linux host, your private key cannot be group or other readable. Run “chmod go-rw $HOME/.ssh/id_rsa” to ensure it is sufficiently private, otherwise auth will fail due to permissive access.

** Once you have a key exchange in place, it is fairly easy to update your key. Create a new one but do not yet replace your old one. You can make a shell script that updates all remote hosts with your new public key – per host, run:

ssh user@remoteHost “echo \”`cat $HOME/.ssh/new_id_rsa.pub`\” >> $HOME/.ssh/authorized_keys”

Once the new public key info has been pushed out, test it using “ssh -i new_id_rsa user@remoteHost” and verify the key authentication works. Once confirmed, rename your old id_rsa and id_rsa.pub files to something else. Then rename your new_id_rsa to id_rsa and new_id_rsa.pub to id_rsa.pub

Role Based Provisioning

I had done a good bit of data mining research to build out an role based provisioning analysis engine. A decade ago, at University. I had a friend in high-school who had just completed her PhD project on using technology to enhance education in K-12 education. She performed paid consulting services to implement a technology approach package in school systems. Except for the one which employed her — even though she offered her package and guidance for free as part of her employment. I remember thinking that seemed a bit insulting. Here all sorts of people are handing over taxpayer/tuition money for your expertise, but the people who employ you won’t even take it for free.

Well, my company was never much for role based provisioning. Even algorithms I’d built as part of my own research projects. Ohhh, they were all for it in theory. Get the data mining in place, figure out what everyone has, build the templates. Now who signs off on all customer service reps getting access to the billing system and the entire corporate finance group getting access to the financial record system? Anyone? Hello?

Because, in the real world, some finance flunky is going to embezzle some money. And some customer service kid is going to credit back his friends accounts. At which point who said so-and-so could access such-and-such. And no one wants their name associated with that decision. Which makes sense — the individual manager hired the person. Trusted the person. Is responsible for ensuring that trust was warranted.

I am proposing a new approach to role based provisioning. We retain the data mining component. We have access templates built on that data. But we use the template to form a provisioning request. On hire or job transfer, the manager receives a notice to go review the access request form. They can add/remove items at will. They can click to compare access with another specific individual on their team. But before any of this access is granted they click the “I say this person can have this access” button. Voila, no single person responsible for all electronic malfeasance within the company.

Visual Studio Code

We found a free, open source code editor from Microsoft called Visual Studio Code — there are downloadable modules that include formatting for a variety of programming languages (c#, cpp, fortran), scripts (perl, php), and other useful formats like MySQL, Apache httpd config files. It also serves as a GUI front end to git. And that is something I’ve been trying to find since I inherited a git server at work — a way for people to avoid having to remember a dozen different git commands.