smart_

#!/usr/bin/python3
"""

=head1 NAME

smart_ - Munin wildcard-plugin to monitor S.M.A.R.T. attribute values through smartctl

=head1 APPLICABLE SYSTEMS

Node with B<Python> interpreter and B<smartmontools> (http://smartmontools.sourceforge.net/)
installed and in function.

=head1 CONFIGURATION

=head2 Create link in service directory

To monitor a S.M.A.R.T device, create a link in the service directory
of the munin-node named smart_<device>, which is pointing to this file.

E.g.

    ln -s /usr/share/munin/plugins/smart_ /etc/munin/plugins/smart_hda

...will monitor /dev/hda.

=head2 Grant privileges in munin-node

The plugin must be run under high privileged user B<root>, to get access to the raw device.

So following minimal configuration in plugin-conf.d/munin-node is needed.

=over 2

  [smart_*]
  user root
  group disk

=back

=head2 Set Parameter if needed

  smartpath     - Specify path to smartctl program (Default: /usr/sbin/smartctl)
  smartargs     - Override '-a' argument passed to smartctl with '-A -i'+smartargs
  ignorestandby - Ignore the standby state of the drive and perform SMART query. Default: False
  ignoreexit    - Bits in smartctl exit code to ignore, e.g. 64. Default: 0

Parameters can be specified on a per-drive basis, eg:

=over 2

  [smart_hda]
  user root
  env.smartargs -H -c -l error -l selftest -l selective -d ata
  env.smartpath /usr/local/sbin/smartctl

=back

In particular, for SATA drives, with older versions of smartctl:

=over 2

  [smart_sda]
  user root
  env.smartargs -d ata -a

  [smart_twa0-1]
  user root
  env.smartargs -H -l error -d 3ware,1
  env.ignorestandby True

  [smart_twa0-2]
  user root
  env.smartargs -H -l error -d 3ware,2

=back

The C<ignoreexit> parameter can be useful to exclude some bits in smartctl exit
code, which is a bit mask described in its main page, from consideration. For
example, if the drive had any errors in the past, the exit code would always
have its bit 6 ("The device error log contains records of errors.") set, even if
the errors happened a long time ago and are not relevant any more. To avoid
getting munin warnings about this you can use

=over 2

  [smart_sda]
  env.ignoreexit 64

=back


=head1 INTERPRETATION

If a device supports the B<Self-Monitoring, Analysis
and Reporting Technology (S.M.A.R.T.)> it offers readable
access to the attribute table. There you find the B<raw value>,
a B<normalised value> and a B<threshold> (set by the vendor)
for each attribute, that is supported by that device.

The meaning and handling of the raw value is a secret of the
vendors embedded S.M.A.R.T.-Software on the disk. The only
relevant info from our external view is the B<normalised value>
in comparison with the B<threshold>. If the attributes value is
equal or below the threshold, it signals its failure and
the B<health status> of the device will switch from B<passed> to B<failed>.

This plugin fetches the B<normalised values of all SMART-Attributes>
and draw a curve for each of them.
It takes the vendors threshold as critical limit for the munin datafield.
So you will see an alarm, if the value reaches the vendors threshold.

Looking at the graph: It is a bad sign, if the curve starts
to curl or to meander. The more horizontal it runs,
the better. Of course it is normal, that the temperatures
curve swings a bit. But the others should stay steady on
their level if everything is ok.

S.M.A.R.T. distinguishes between B<Pre-fail> and B<Old-age>
Attributes. An old disk will have more curling curves
because of degradation, especially for the B<Old-age> Attributes.
You should then backup more often, run more selftests[1] and prepare
the disks replacement.

B<Act directly>, if a <Pre-Fail> Attribute goes below threshold.
Immediately back-up your data and replace your hard disk drive.
A failure may be imminent..

[1] Consult the smartmontools manpages to learn about
offline tests and automated selftests with smartd.
Only with both activated, the values of the SMART-Attributes
reflect the all over state of the device.

Tutorials and articles about S.M.A.R.T. and smartmontools:
http://smartmontools.sourceforge.net/doc.html#tutorials

=head1 MAGIC MARKERS

 #%# family=auto
 #%# capabilities=autoconf suggest

=head1 CALL OPTIONS

B<none>

=over 2

Fetches values if called without arguments:

E.g.: munin-run smart_hda

=back

B<config>

=over 2

Prints plugins configuration.

E.g.: munin-run smart_hda config

=back

B<autoconf>

=over 2

Tries to find smartctl and outputs value 'yes' for success, 'no' if not.

It's used by B<munin-node-configure> to see whether autoconfiguration is possible.

=back

B<suggest>

=over 2

Outputs the list of device names, that it found plugged to the system.

B<munin-node-configure> use this to build the service links for this wildcard-plugin.

=back

=head1 VERSION

Version 2.2

=head1 CHANGELOG

v1.0  22/08/2004

    - First draft

v1.2  28/08/2004

    - Clean up the code, add a verbose option

v1.3  14/11/2004

    - Compatibility with python<2.2. See comments in the code

v1.4  17/11/2004

    - Deal with non zero exit codes of smartctl
    - config now prints the critical thresholds, as reported by smartctl

v1.5  18/11/2004

    - Plot smartctl_exit_code bitmask

v1.6  21/11/2004

    - Add autoconf and suggest capabilities
    - smartctl path can be passed through "smartpath" environment variable
    - Additional smartctl args can be passed through "smartargs" environment variable

v1.7  29/11/2004

    - Add suggest capabilities for NetBSD, OpenBSD, FreeBSD and SunOS.
    - Allow one to override completely the smartctl arguments with "smartargs"

v1.8  16/02/2005

    - Exit status field now only triggers warnings, not criticals.

v1.9  07/07/2005

    - Allow to query several drives on the same 3ware card.
    - Correct a bug when '-i' was not listed in smartargs
    - Don't fail if no value was obtained for hard drive model

v1.10 19/08/2005

    - smartctl_exit_code is now a numerical value

v2.0  08/05/2009

    - Correct bug in the interpretation of smartctl_exit_code
    - New option to suppress SMART warnings in munin
    - Temporary lack of output for previously existing drive now reports U
    - The plugin now contains its own documentation for use with munindoc
    - Removed python<2.2 compatibility comments
    - Better autodetection of drives
    - Don't spin up devices in a low-power mode.

v2.1  2012-02-14

    - Add support for Darwin (Mac OS X).
    - Print the last line of smartctl output to verbose log to aid in
      understanding why smartctl exited with a nonzero status.

v2.2  2014-08-22

    - Add "ignoreexit" environment variable.

v2.3 2018-08-10

    - Improve readability, reduce code duplication, avoid shell expansion.

v2.4 2018-12-19

    - Ignore invalid threshold data

=head1 BUGS

None known

=head1 AUTHOR

  Copyright (C) 2004-2009 Nicolas Stransky <Nico@stransky.cx>
  Copyright (C) 2008 Gabriele Pohl <contact@dipohl.de>

=head1 LICENSE

GPLv2 (http://www.gnu.org/licenses/gpl-2.0.txt)

=cut
"""

import collections
import os
import sys
import string
import pickle
import subprocess

# Increase verbosity (True/False) -> use munin-run --pidebug
verbose = True if os.getenv('MUNIN_DEBUG') == '1' else False

# collect configuration from environment
smartctl_bin = os.getenv('smartpath', '/usr/sbin/smartctl')
smartctl_args = os.getenv('smartargs', '-a')
smartctl_ignore_standby = bool(os.getenv('ignorestandby', False))
smartctl_ignore_exitcode_bitmask = int(os.getenv('ignoreexit', 0))

# You may edit the following 3 variables
# Suppress SMART warnings (True/False)
report_warnings = True
# You may not modify anything below this line

plugin_version = "2.4"

# some disks report invalid threshold values
INVALID_THRESHOLDS_BLACKLIST = {"---"}


SmartCtlParseResult = collections.namedtuple("SmartCtlParseResult",
                                             ("has_failed", "smart_data", "model", "is_empty"))


def verboselog(s):
    if verbose:
        sys.stderr.write('{}: {}\n'.format(plugin_name, s))


def guess_full_path(hard_drive):
    """ try to find the full path for a given hard disk name

    None is returned if no device node was found.
    """
    for dev_dir in ('/dev', '/dev/disk/by-id'):
        full_path = os.path.join(dev_dir, hard_drive)
        if os.path.exists(full_path):
            return full_path
    else:
        return None


def is_fatal_exitcode(exit_code):
    # The exit code represents a bitmask.
    # Bits 0/1/2 belong to fatal errors (see smartctl's man page). Check if one of these is set.
    return (exit_code & 0b111) > 0


def read_values(hard_drive):
    smart_values = {}
    try:
        verboselog('Reading S.M.A.R.T values')
        os.putenv('LC_ALL', 'C')
        device = guess_full_path(hard_drive)
        command_tokens = [smartctl_bin] + smartctl_args.split()
        if not smartctl_ignore_standby:
            command_tokens.extend(('-n', 'standby'))
        command_tokens.extend(('-A', '-i', device))
        proc = subprocess.Popen(command_tokens, stdout=subprocess.PIPE)
        stdout, stderr = proc.communicate()
        in_table_data = False
        last_output_line = None
        model = "unknown"
        for line in stdout.decode().splitlines():
            if not line:
                # the table is finished
                in_table_data = False
            elif not in_table_data:
                # process header data
                if line.startswith('Device Model:') or line.startswith('Device:'):
                    value = line.split(':', 1)[1].strip()
                    # ignore the "Version" string
                    model = ' '.join(token for token in value.split() if token != 'Version')
                elif line.startswith('ID# ATTRIBUTE_NAME'):
                    # Start reading the Attributes block
                    in_table_data = True
                else:
                    # we can ignore other header lines
                    pass
            else:
                # this is a data table row
                tokens = line.split()
                key = tokens[1].replace('-', '_')
                value = tokens[3]
                threshold = None if tokens[5] in INVALID_THRESHOLDS_BLACKLIST else tokens[5]
                smart_values[key] = {"value": value, "threshold": threshold}
            last_output_line = line
        real_exit_code = proc.returncode
        if real_exit_code > 0:
            # Allow to turn off warnings for some bits
            num_exit_status = real_exit_code & ~smartctl_ignore_exitcode_bitmask
        else:
            num_exit_status = 0
        if num_exit_status != 0:
            if is_fatal_exitcode(num_exit_status):
                verboselog('smartctl cannot access S.M.A.R.T values on drive {}. Command exited '
                           'with code {}'.format(hard_drive, num_exit_status))
                verboselog(last_output_line)
            else:
                # the error is not fatal, but we should announce a warning
                verboselog('smartctl exited with code {}. {} may be FAILING RIGHT NOW!'
                           .format(num_exit_status, hard_drive))
    except Exception as exc:
        verboselog('Cannot access S.M.A.R.T values ({})! Check user rights or proper '
                   'smartmontools installation/arguments.'.format(exc))
        sys.exit(1)
    if not smart_values:
        verboselog("Can't find any S.M.A.R.T values in smartctl output!")
        is_empty = True
    else:
        is_empty = False
    smart_values["smartctl_exit_status"] = {"value": str(num_exit_status), "threshold": "1"}
    return SmartCtlParseResult(is_fatal_exitcode(real_exit_code), smart_values, model, is_empty)


def get_state_filename(hard_drive):
    statefiledir = os.environ['MUNIN_PLUGSTATE']
    return os.path.join(statefiledir, "smart-{}.state".format(hard_drive_identifier))


def open_state_file(hard_drive, mode):
    return open(get_state_filename(hard_drive), mode)


def update_state_file(hard_drive, model, smart_values):
    data_storage = dict(smart_values)
    data_storage["model"] = model
    try:
        verboselog('Saving statefile')
        pickle.dump(data_storage, open_state_file(hard_drive, "wb"))
    except Exception as exc:
        verboselog('Error trying to save state file ({})! Check access rights'.format(exc))


def parse_state_file(hard_drive):
    data = pickle.load(open_state_file(hard_drive, "rb"))
    model = data.pop("model", "unknown")
    return model, data


def print_plugin_values(hard_drive, is_empty, smart_values):
    if not is_empty:
        verboselog('Printing S.M.A.R.T values')
        for key, content in smart_values.items():
            print("{}.value {}".format(key, content["value"]))
    else:
        print_unknown_from_statefile(hard_drive)


def print_config(hard_drive):
    if os.path.exists(get_state_filename(hard_drive)):
        try:
            verboselog('Try to recall previous S.M.A.R.T attributes for {}'
                       .format(hard_drive))
            model, smart_values_state = parse_state_file(hard_drive)
        except Exception as exc:
            verboselog('Error opening existing state file ({})!'.format(exc))
            sys.exit(1)
    else:
        verboselog('No state file, reading S.M.A.R.T values for the first time')
        parsed_data = read_values(hard_drive)
        update_state_file(hard_drive, parsed_data.model, parsed_data.smart_data)
        model = parsed_data.model
        smart_values_state = parsed_data.smart_data

    verboselog('Printing configuration')
    print('graph_title S.M.A.R.T values for drive '+','.join(hard_drive_identifier))
    print('graph_vlabel Attribute S.M.A.R.T value')
    print('graph_args --base 1000 --lower-limit 0')
    print('graph_category disk')
    print('graph_info This graph shows the value of all S.M.A.R.T attributes of drive {} ({}). '
          'smartctl_exit_status is the return value of smartctl. A non-zero return value '
          'indicates an error, a potential error, or a fault on the drive.'
          .format(hard_drive, model))
    for key, content in smart_values_state.items():
        print('{}.label {}'.format(key, key))
        if report_warnings:
            if content["threshold"] is None:
                # we did not parse a valid threshold
                pass
            elif key == 'smartctl_exit_status':
                level = 'warning'
                print('{}.{} {}'.format(key, level, content["threshold"]))
            else:
                level = "critical"
                print('{}.{} {}:'.format(key, level, content["threshold"]))


def print_unknown_from_statefile(hard_drive):
    if os.path.exists(get_state_filename(hard_drive)):
        try:
            verboselog('Failed to get S.M.A.R.T values from drive. Try to recall previous '
                       'S.M.A.R.T attributes for {}'.format(hard_drive))
            model, smart_values_state = parse_state_file(hard_drive)
        except Exception as exc:
            verboselog('Error opening existing state file ({})!'.format(exc))
            sys.exit(1)
    else:
        verboselog('No state file, reading S.M.A.R.T values for the first time')
        sys.exit(1)

    verboselog('Printing unknown values for all attributes in state file')
    for key in smart_values_state.keys():
        print('{}.value U'.format(key))


def get_hard_drive_name(plugin_name):
    try:
        name = plugin_name.split('_', 1)[1]
        if os.uname()[0] == "SunOS":
            # Special handling of hard_drive names starting with "rdsk" or "rmt".
            # These are changed from "rdsk0" to "rdsk/0".
            for prefix in ('rdsk', 'rmt'):
                if name.startswith(prefix):
                    name = os.path.join(prefix, name[len(prefix):])
        if guess_full_path(name) is None:
            # For 3ware cards, we have to set multiple plugins for the same hard drive name.
            # Let's see if we find a '-' in the drive name.
            if '-' in name:
                name = name.split('-')[0]
        # Check that the drive exists in /dev
        if guess_full_path(name) is None:
            verboselog('/dev/(disk/by-id/)? {} not found!'.format(name))
            sys.exit(1)
        return name
    except Exception as exc:
        verboselog("No S.M.A.R.T device name found in plugin's symlink ({})!".format(exc))
        sys.exit(1)


def find_smart_drives():
    # Try to autodetect Linux, *BSD, SunOS drives. Don't try to autodetect drives on a 3Ware card.
    drives = []
    if os.uname()[0] == "Linux":
        if os.path.exists('/sys/block/'):
            # Running 2.6
            try:
                for drive in os.listdir('/sys/block/'):
                    if drive[:2] in ['md', 'fd', 'lo', 'ra', 'dm']:
                        continue  # Ignore MD, Floppy, loop , RAM and LVM devices.
                    try:
                        verboselog('Trying {} ...'.format(drive))
                        parsed_data = read_values(drive)
                        if not parsed_data.has_failed and not parsed_data.is_empty:
                            drives.append(drive)
                    except Exception:
                        continue
            except Exception:
                verboselog('Failed to list devices in /sys/block')
        else:
            verboselog('Not running linux2.6, failing back to /proc/partitions')
            try:
                partitions = open('/proc/partitions', 'r')
                lines = partitions.readlines()
                for line in lines:
                    words = line.split()
                    if len(words) == 0 or words[0][0] not in string.digits:
                        continue
                    if words[0] in ['1', '9', '58', '254']:
                        # Ignore RAM, md, LVM and LVM2 devices
                        continue
                    if words[-1][-1] not in string.digits:
                        try:
                            verboselog('Trying '+words[-1]+'...')
                            parsed_data = read_values(words[-1])
                            if not parsed_data.has_failed and not parsed_data.is_empty:
                                drives.append(words[-1])
                        except Exception:
                            continue
                verboselog('Found drives in /proc/partitions ! '+str(drives))
            except Exception as exc:
                verboselog('Failed to list devices in /proc/partitions: {}'.format(exc))
    elif os.uname()[0] == "OpenBSD":
        try:
            sysctl_kerndisks = os.popen('sysctl hw.disknames')
            kerndisks = sysctl_kerndisks.readline().strip()
            for drive in kerndisks[kerndisks.rindex('=')+1:].split(','):
                if drive[:2] in ['md', 'cd', 'fd']:
                    # Ignore Memory Disks, CD-ROM drives and Floppy
                    continue
                try:
                    verboselog('Trying '+drive+'c...')
                    parsed_data = read_values(drive + 'c')
                    if not parsed_data.has_failed and not parsed_data.is_empty:
                        drives.append(drive+'c')
                except Exception:
                    continue
        except Exception as exc:
            verboselog('Failed to list OpenBSD disks: {}'.format(exc))
    elif os.uname()[0] == "FreeBSD":
        try:
            sysctl_kerndisks = os.popen('sysctl kern.disks')
            kerndisks = sysctl_kerndisks.readline().strip()
            for drive in kerndisks.split()[1:]:
                if drive[:2] in ['md', 'cd', 'fd']:
                    # Ignore Memory Disks, CD-ROM drives and Floppy
                    continue
                try:
                    verboselog('Trying '+drive+'...')
                    parsed_data = read_values(drive)
                    if not parsed_data.has_failed and not parsed_data.is_empty:
                        drives.append(drive)
                except Exception:
                    continue
        except Exception as exc:
            verboselog('Failed to list FreeBSD disks: {}'.format(exc))
    elif os.uname()[0] == "Darwin":
        try:
            from glob import glob
            for drivepath in glob('/dev/disk[0-9]'):
                try:
                    drive = os.path.basename(drivepath)
                    verboselog('Trying '+drive+'...')
                    parsed_data = read_values(drive)
                    if not parsed_data.has_failed and not parsed_data.is_empty:
                        drives.append(drive)
                except Exception:
                    continue
        except Exception as exc:
            verboselog('Failed to list Darwin disks: {}'.format(exc))
    elif os.uname()[0] == "NetBSD":
        try:
            sysctl_kerndisks = os.popen('sysctl hw.disknames')
            kerndisks = sysctl_kerndisks.readline().strip()
            for drive in kerndisks.split()[2:]:
                if drive[:2] in ['md', 'cd', 'fd']:
                    # Ignore Memory Disks, CD-ROM drives and Floppy
                    continue
                try:
                    verboselog('Trying {} ...'.format(drive))
                    parsed_data = read_values(drive + 'c')
                    if not parsed_data.has_failed and not parsed_data.is_empty:
                        drives.append(drive + 'c')
                except Exception:
                    continue
        except Exception as exc:
            verboselog('Failed to list NetBSD disks: {}'.format(exc))
    elif os.uname()[0] == "SunOS":
        try:
            from glob import glob
            for drivepath in glob('/dev/rdsk/*s2'):
                try:
                    drive = os.path.basename(drivepath)
                    verboselog('Trying rdsk {} ...'.format(drive))
                    parsed_data = read_values('rdsk' + drive)
                    if not parsed_data.has_failed and not parsed_data.is_empty:
                        drives.append('rdsk' + drive)
                except Exception:
                    continue
            for drivepath in glob('/dev/rmt/*'):
                try:
                    drive = os.path.basename(drivepath)
                    verboselog('Trying rmt {} ...'.format(drive))
                    parsed_data = read_values('rmt' + drive)
                    if not parsed_data.has_failed and not parsed_data.is_empty:
                        drives.append('rmt' + drive)
                except Exception:
                    continue
        except Exception:
            verboselog('Failed to list SunOS disks')
    return drives


""" Main part """

plugin_name = list(os.path.split(sys.argv[0]))[1]
verboselog("plugins' UID: {:d} / plugins' GID: {:d}".format(os.geteuid(), os.getegid()))

hard_drive_identifier = plugin_name.split('_', 1)[1]

# Parse arguments
if len(sys.argv) > 1:
    if sys.argv[1] == "config":
        hard_drive = get_hard_drive_name(plugin_name)
        print_config(hard_drive)
        sys.exit(0)
    elif sys.argv[1] == "autoconf":
        if os.path.exists(smartctl_bin):
            if not find_smart_drives():
                print('no (no drives accessible)')
            else:
                print('yes')
            sys.exit(0)
        else:
            print('no (smartmontools not found)')
            sys.exit(0)
    elif sys.argv[1] == "suggest":
        for drive in find_smart_drives():
            print(drive)
        sys.exit(0)
    elif sys.argv[1] == "version":
        print('smart_ Munin plugin, version '+plugin_version)
        sys.exit(0)
    elif sys.argv[1] != "":
        verboselog('unknown argument "'+sys.argv[1]+'"')
        sys.exit(1)

# No argument given, doing the real job:
hard_drive = get_hard_drive_name(plugin_name)
parsed_data = read_values(hard_drive)
if not parsed_data.is_empty:
    update_state_file(hard_drive, parsed_data.model, parsed_data.smart_data)
print_plugin_values(hard_drive, parsed_data.is_empty, parsed_data.smart_data)
sys.exit(0)