Syslog, Cron & Software RAID

Syslog

Syslog architecture

Typical log files

This can vary depending on the syslog config file

File Contents
auth.log/secure Auth, sudo, sshd, user adds
boot.log Output from init scripts
cron.log/cron Cron runs and errors
dmesg Dump of kernel messages
lastlog Last login time per user (binary)
mail.log/maillog All mail logs
messages Main system logs (i.e. catch all typically)
wtmp Login records (binary)
yum.log package management log

Syslog facilities

Categories and levels defined in the kernel

Facility Description
* Everything
authpriv Sensitive and private messages (i.e. /var/log/secure)
cron Cron daemon messages
daemon System daemons
kern Kernel messages
local0-7 Various local messages

Syslog severity levels

In descending severity..

Level Description
emerg Panic situations
alert Urgent situations
crit Critical conditions
err Other error conditions
warning Warning messages
notice Things that might merit investigation
info Information messages
debug Debug messages

Remote logging

Rsyslog

Rsyslog config

# GENERAL CONFIG
$ModLoad imuxsock # provides support for local system logging (e.g. via logger
                  # command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# RULES
# Log anything (except mail) of level info or higher. Don't log private
# authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure
# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog
# Log cron stuff
cron.*                                                  /var/log/cron
# Everybody gets emergency messages
*.emerg                                                 *
# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler
# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

Rsyslog remote logging

# Client config
# Send all logs to remote loghost using TCP
*.* @@loghost.example.org:10514

# Server (loghost) config
# Use TCP
module(load="imtcp")
input(type="imtcp" port="10514")
# Define a template for where to put the logs
$template DailyPerHostLogs,"/var/log/HOSTS/%HOSTNAME%/%YEAR%-%MONTH%-%DAY%.log"
# Send all logs using the template
*.* -?DailyPerHostLogs

Rsyslog also supports TLS/SSL over TCP.

Accessing logs with systemd

# Tail the log and watch it live
$ journalctl -f
# Filter by priority
$ journalctl -p err
# Filter by time
$ journalctl --since="2016-01-20 05:00:00"
# Filter by unit (service)
$ journalctl -u crond

RedHat journalctl Documentation

Userspace tools: logger

$ logger -t mirror "trigger set centos"

# output will be:
# Jan 21 18:55:38 hostname.example.org mirror: trigger set centos

# Send a message to the auth facility using the info severity level
$ logger -p auth.info "Set user john locked"

Cron

Cron: Schedule commands

Other cron-like services

Crontab fields

minute hour dom month weekday command

Field Description Range
minute Minute of the hour 0 to 59
hour Hour of the day 0 to 23
dom Day of the month 1 to 31
month Month of the year 1 to 12
weekday Day of the week 0 to 6 (0= Sunday)

Crontab Time Fields

A star, which matches everything * * * * *
A single integer, which matches exactly 10 * * * *
Two integers separated by a dash, matching a range of values 0 0 * * 1-5
A range followed by a slash and a step value 23 0-23/2 * * *
A comma-separated list of integers or ranges, matching any value 15,45 * * * *

Crontab format

Taken from man 5 crontab

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed

# minute hour dom month weekday command

# run five minutes after midnight, every day
5 0 * * *       $HOME/bin/daily.job >> $HOME/tmp/out 2>&1
# run at 2:15pm on the first of every month -- output mailed to paul
15 14 1 * *     $HOME/bin/monthly
# run at 10 pm on weekdays, annoy Joe
0 22 * * 1-5    mail -s "It’s 10pm" joe%Joe,%%Where are your kids?%
23 0-23/2 * * * echo "run 23 minutes after midn, 2am, 4am ..., everyday"
5 4 * * sun     echo "run at 5 after 4 every sunday"

Managing user crontabs

Never edit the user files directly in /var/spool/cron

# Edit the current user crontab
$ crontab -e

# Edit user john's crontab
$ crontab -e -u john

Other Crontab files

File/Directory Description
/etc/crontab Primary system crontab file
/etc/cron.d/ Arbitrary crontab formatted files
/etc/anacrontab system crontab that manages cron.daily, weekly, hourly and monthly
/etc/cron.daily/ Scripts that will run daily
/etc/cron.hourly/ Scripts that will run hourly
/etc/cron.monthly/ Scripts that will run monthly
/etc/cron.weekly/ Scripts that will run weekly

Crontab environment variables

Can set any arbitrary environment variables in crontab

Variable Description
MAILTO Email address to send stdout/stderr output to
SHELL Default shell to use

Crontab Tips

(From a 'seasoned' sysadmin)

Software RAID (mdadm)

mdadm

When should you use mdadm?

Formatting and Booting

Creating a RAID1

$ yum install mdadm

# Note: I created loop0/1 using dd and losetup
$ fdisk /dev/loop0

$ mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/loop0 /dev/loop1
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

$ cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 loop1[1] loop0[0]
      20416 blocks super 1.2 [2/2] [UU]

unused devices: <none>

/etc/mdadm.conf

If the partition is set to fd, the kernel should automatically detect it and build the array based on the metadata on the partition.

# Show metadata about arrays using md devices
$ mdadm --detail --scan
ARRAY /dev/md0 metadata=1.2 name=mdadm:0 UUID=ead812c6:ee734fb3:fcb6264d:e3a00c40

# Add it to the config file (not required, but useful)
$ mdadm --detail --scan >> /etc/mdadm.conf

# Stop the array
$ mdadm --stop /dev/md0
mdadm: stopped /dev/md0

# Start (assemble) the array
$ mdadm --assemble /dev/md0
mdadm: /dev/md0 has been started with 2 drives.

Monitoring mdadm

Dealing with failures

# Simulate a disk failure
$ mdadm /dev/md0 -f /dev/loop1
mdadm: set /dev/loop1 faulty in /dev/md0

$ journalctl -n 10 -k
Jan 20 21:52:33 kernel: md0: detected capacity change from 0 to 4
Jan 20 21:52:33 kernel:  md0: unknown partition table
Jan 20 21:53:29 kernel: md/raid1:md0: Disk failure on loop1, disa
                        md/raid1:md0: Operation continuing on 1 d
Jan 20 21:53:29 kernel: RAID1 conf printout:
Jan 20 21:53:29 kernel:  --- wd:1 rd:2
Jan 20 21:53:29 kernel:  disk 0, wo:0, o:1, dev:loop0
Jan 20 21:53:29 kernel:  disk 1, wo:1, o:0, dev:loop1
Jan 20 21:53:29 kernel: RAID1 conf printout:
Jan 20 21:53:29 kernel:  --- wd:1 rd:2
Jan 20 21:53:29 kernel:  disk 0, wo:0, o:1, dev:loop0

Dealing with failures

# Hot remove the disk
$ mdadm /dev/md0 -r /dev/loop1
mdadm: hot removed /dev/loop1 from /dev/md0

# Check the status of the array
$ cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 loop0[0]
      20416 blocks super 1.2 [2/1] [_U]

unused devices: <none>

# Hot add the drive back
$ mdadm /dev/md0 -a /dev/loop1
mdadm: added /dev/loop1

More information about an md device

$ mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Wed Jan 20 16:56:25 2016
     Raid Level : raid1
     Array Size : 409024 (399.50 MiB 418.84 MB)
  Used Dev Size : 409024 (399.50 MiB 418.84 MB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent
    Update Time : Wed Jan 20 22:01:02 2016
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
           Name : mdadm:0  (local to host mdadm)
           UUID : 87f67b6c:622ca752:4dd25200:6b3f23c5
         Events : 39

    Number   Major   Minor   RaidDevice State
       0       7        0        0      active sync   /dev/loop0
       2       7        1        1      active sync   /dev/loop1

Block device metadata

$ mdadm -E /dev/loop1
/dev/loop1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : ead812c6:ee734fb3:fcb6264d:e3a00c40
           Name : mdadm:0  (local to host mdadm)
  Creation Time : Wed Jan 21 22:13:57 2015
     Raid Level : raid1
   Raid Devices : 2
 Avail Dev Size : 40896 (19.97 MiB 20.94 MB)
     Array Size : 20416 (19.94 MiB 20.91 MB)
  Used Dev Size : 40832 (19.94 MiB 20.91 MB)
    Data Offset : 64 sectors
   Super Offset : 8 sectors
   Unused Space : before=0 sectors, after=64 sectors
          State : clean
    Device UUID : bac67523:e1f44d96:a64c1322:50135cf9
    Update Time : Wed Jan 21 22:28:43 2015
  Bad Block Log : 512 entries available at offset 48 sectors
       Checksum : 92d13b09 - correct
         Events : 39
   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

Class Announcements