Tag: System Administration

Sorting Grep Results When Log File Names Are Incremented Integers

While rotated log files can have a timestamp like YYYYMMDDHHmmss appended, a lot of log files are rotated with incremented integers (i.e. file.3 is removed, file.2 becomes file.3, file.1 becomes file.2, file becomes file.1, and file is a new file for current log ‘stuff’). We had some … challenges changing the log4j settings to use the timestamp format. When you grep all of the log files, the results are organized by alpha-sorted file names. Problem is that alpha sorted file names are 1, 10, 11, 12, …, 19, 2, 20, 21 … which doesn’t produce results in ascending or descending date order. You get:

zwave.log.1:2018-07-01 22:51:21.450 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 58: Node not awake!
zwave.log.1:2018-07-01 22:51:21.450 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 55: Node not awake!
zwave.log.11:2018-07-02 00:47:05.375 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 70: Node not awake!
zwave.log.11:2018-07-02 00:47:05.376 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 46: Node not awake!
zwave.log.12:2018-07-02 01:01:12.850 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 82: Node not awake!
zwave.log.13:2018-07-02 01:11:22.465 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 48: Node not awake!
zwave.log.13:2018-07-02 01:11:22.478 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 82: Node not awake!
zwave.log.13:2018-07-02 01:11:22.478 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 49: Node not awake!
zwave.log.14:2018-07-02 01:25:22.632 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 112: Node not awake!
zwave.log.14:2018-07-02 01:25:22.632 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 212: Node not awake!
zwave.log.15:2018-07-02 01:40:29.053 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 58: Node not awake!
zwave.log.15:2018-07-02 01:40:29.053 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 70: Node not awake!
zwave.log.16:2018-07-02 01:55:01.702 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 50: Node not awake!
zwave.log.16:2018-07-02 01:55:01.702 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 122: Node not awake!
zwave.log.17:2018-07-02 02:08:48.818 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 67: Node not awake!
zwave.log.17:2018-07-02 02:08:48.818 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 212: Node not awake!
zwave.log.18:2018-07-02 02:23:53.281 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 55: Node not awake!
zwave.log.18:2018-07-02 02:23:53.281 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 46: Node not awake!
zwave.log.19:2018-07-02 02:44:23.567 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 48: Node not awake!
zwave.log.19:2018-07-02 02:44:23.567 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 86: Node not awake!
zwave.log.19:2018-07-02 02:44:23.567 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 50: Node not awake!
zwave.log.2:2018-07-01 23:03:36.704 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 57: Node not awake!
zwave.log.2:2018-07-01 23:03:36.704 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 64: Node not awake!
zwave.log.2:2018-07-01 23:03:36.704 [DEBUG] [ng.zwave.internal.protocol.ZWaveTransactionManager] – NODE 68: Node not awake!

So I put together a quick sequence of commands that will produce results sorted by date. This process strips off the file names (I could create a more complex command set to shuffle the file name off to the end, but I didn’t need the file name. I just needed to scroll through the times at which an event occurred).

grep “Node not awake” zwave.log* | sed -r ‘s/^zwave.log.*:2018/2018/’ | sort

The grep results are piped to sed to have the file name prefix removed. The subsequent results are then piped to sort to be, well, sorted.

Ransomware

My company held a ransomware response through experiment recently – and, honestly, every ransomware response I’ve seen has been some iteration of “walk through backups until we find good files”. Maybe use something like the SharePoint versioning to help identify a good target date (although that date may be different for different files … who knows!). But why wouldn’t you attempt a proactive identification of compromised files?

The basis of ransomware is that it encrypts data and you get the password after paying so-and-so a bitcoin or three. Considering that NGO virus authors (e.g. those who aren’t trying to slow down Iran’s centrifuges) are generally interested in creating mayhem. There’s not a lot of disincentive to creating mayhem and making a couple of bucks. I don’t anticipate ransomware to become less prevalent in the future; in fact I anticipate seeing it in vigilante hacking: EntityX gets their files back after they publicly donate 100k to their antithesis organisation.

Since it’s probably not going away, it seems worthwhile to immediately identify the malicious data scrambling. Reverting to yesterday’s backups sucks, but not as much as finding that your daily backups have aged out and you’re stuck with the monthly backup from 01 Nov as your last “good” data set. It would also be good to merge whatever your last good backup is into the non-encrypted files so the only ‘stuff’ that reverts is a worthless scramble of data anyway. Sure someone may have worked on the file this morning and sucks for them to find their work back-rev’d to last night … but again that’s better than everyone having to reproduce their last two and a half months of work.

Promptly identifying the attack: There are routine processes that read changed files. Windows Search indexing, antivirus scanner, SharePoint indexing. Running against the Windows Search index log on every computer in the organisation is logistically challenging. Not impossible, but not ideal either. A central log for enterprise AV software or the SharePoint indexing log, however, can be parsed from the data centre. Scrape the log files for “unable to read this encrypted file” events. Then there are a myriad of actions that can be taken. Alert the file owner and have them confirm the file should be encrypted. Alert the IT staff when more than x encrypted files are identified in a unit time. Check the create time-stamp and alert the file owner for any files that were created prior to encountering them as encrypted.

Restoring only scrambled files: Since you have a list of encrypted files, you have a scope for the restore job. Instead of restoring everything in place (because who has 2x the storage space to restore to an alternate location?!). Restore just the recently identified as encrypted files – to an alternate location or in place. Ideally you’ve gotten user input on the encrypted files and can omit any the user indicated they encrypted too.

Checking Supported TLS Versions and Ciphers

There have been a number of ssl vulnerabilities (and deprecated ciphers that should be unavailable, especially when transiting particularly sensitive information). On Linux distributions, nmap includes a script that enumerates ssl versions and, per version, the supported ciphers.

[lisa@linuxbox ~]# nmap -P0 -p 25 –script +ssl-enum-ciphers myhost.domain.ccTLD

Starting Nmap 7.40 ( https://nmap.org ) at 2017-10-13 11:36 EDT
Nmap scan report for myhost.domain.ccTLD (#.#.#.#)
Host is up (0.00012s latency).
Other addresses for localhost (not scanned): ::1
PORT STATE SERVICE
25/tcp open smtp
| ssl-enum-ciphers:
| TLSv1.0:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (rsa 2048) – A
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) – A
| TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (dh 2048) – A
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (rsa 2048) – A
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) – A
| TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (dh 2048) – A
| TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) – A
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) – A
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) – A
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) – A
| compressors:
| NULL
| cipher preference: server
| TLSv1.1:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (rsa 2048) – A
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) – A
| TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (dh 2048) – A
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (rsa 2048) – A
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) – A
| TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (dh 2048) – A
| TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) – A
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) – A
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) – A
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) – A
| compressors:
| NULL
| cipher preference: server
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) – A
| TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 2048) – A
| TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (rsa 2048) – A
| TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (dh 2048) – A
| TLS_DHE_RSA_WITH_AES_256_CCM_8 (dh 2048) – A
| TLS_DHE_RSA_WITH_AES_256_CCM (dh 2048) – A
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) – A
| TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) – A
| TLS_DHE_RSA_WITH_AES_128_CCM_8 (dh 2048) – A
| TLS_DHE_RSA_WITH_AES_128_CCM (dh 2048) – A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (rsa 2048) – A
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 2048) – A
| TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384 (rsa 2048) – A
| TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256 (dh 2048) – A
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) – A
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (dh 2048) – A
| TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 (rsa 2048) – A
| TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 (dh 2048) – A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (rsa 2048) – A
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) – A
| TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (dh 2048) – A
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (rsa 2048) – A
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) – A
| TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (dh 2048) – A
| TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) – A
| TLS_RSA_WITH_AES_256_CCM_8 (rsa 2048) – A
| TLS_RSA_WITH_AES_256_CCM (rsa 2048) – A
| TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) – A
| TLS_RSA_WITH_AES_128_CCM_8 (rsa 2048) – A
| TLS_RSA_WITH_AES_128_CCM (rsa 2048) – A
| TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) – A
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256 (rsa 2048) – A
| TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) – A
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256 (rsa 2048) – A
| TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) – A
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) – A
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) – A
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) – A
| compressors:
| NULL
| cipher preference: server
|_ least strength: A

Nmap done: 1 IP address (1 host up) scanned in 144.67 seconds