Differences

This shows you the differences between two versions of the page.

--- vms:debian [2018/11/24 00:46]
admin [SMART]
+++ vms:debian [2019/04/09 12:25] (current)
admin [Swap File]
@@ Line 269: / Line 269: @@
 > systemctl reload sshd
 </Code>
+===== Swap File =====
+As stated above, using a swap partition isn't always the best option when using a VM since it might become necessary to resize the swap space when the VM's memory configuration is changed based on its utilization. When a swap partition is used, it becomes necessary to "play" with the partitions sizes, which means changing the partitions scheme and oftentimes even the virtual disk's size, which is tedious...
+Therefore, in the case of VMs, it is often much easier to create the swap space using a system file, you can learn more about this in [[https://www.digitalocean.com/community/tutorials/how-to-configure-virtual-memory-swap-file-on-a-vps|this Digital Ocean's article]]. Here are the steps to create and use a swap file.
+First to make sure swap is not already activated on the system use the ''free'' command:
+<Code:bash>
+> free
+              total        used        free      shared  buff/cache   available
+Mem:        2058304       37680     1759884        2968      260740     1872108
+Swap:             0           0           0
+</Code>
+The ''free : 0'' value on the ''Swap:'' line means swap is indeed **not activated**
+We'll create the partition file under ''/var'' and name it ''swap.img'', then change its permissions to ''600'' so no user will be able to access it:
+<Code:bash>
+> touch /var/swap.img
+> chmod 600 /var/swap.img
+</Code>
+=== Sizing ===
+Deciding on the swap size is really case dependent. In general, it is recommended recommend to set it to 1-2x the available system RAM. So, if you have a 512mb RAM VM, use 512mb-1gb swap. If you have a 1gb RAM VM use 1gb-2gb swap, etc. This is not a hard and fast rule, for example if you have a 4gb RAM VM it may be best to use little (512mb) or no swap at all.\\
+We use the ''dd'' command to stretch our swap file size, filling it with zeroes to the size we need (here 2Gb):
+<Code:bash>
+> dd if=/dev/zero of=/var/swap.img bs=1024k count=2048
++0 records in
++0 records out
+2147483648 bytes (2.1 GB, 2.0 GiB) copied, 1.57258 s, 1.4 GB/s
+</Code>
+Next, we prepare the file to be usable as the swap file:
+<Code:bash>
+> mkswap /var/swap.img
+Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
+no label, UUID=f8d9bdfe-4090-4bc6-8f7a-fc74b64946ad
+</Code>
+Then we turn on swapping:
+<Code:bash>
+> swapon /var/swap.img
+> free
+              total        used        free      shared  buff/cache   available
+Mem:        2058304       38784       67972        2968     1951548     1847628
+Swap:       2097148           0     2097148
+</Code>
+We now see that the swap is active. We could turn off swapping with the ''swapoff /var/swap.img'' command.\\
+Turning the swap on this way will **not activate swapping on the next boot**, so we need to modify ''/etc/fstab'' to have the swap file activated at boot time:
+<Code>
+> nano /etc/fstab
+ADD THOSE LINES
+# swap file
+/var/swap.img    none    swap    sw    0    0
+</Code>
+Finally, one could define the system's //swappiness//, which tells the Linux kernel/VM handler how likely it should be to use VM. It is a percent value, between 0 & 100. A usual recommendation for VMs is 30:
+<Code:bash>
+> sysctl -w vm.swappiness=30
+vm.swappiness = 30
+</Code>
+To make swappiness persistent:
+<Code:bash>
+> nano /etc/sysctl.conf
+...
+###################################################################
+# Swap usage
+#
+# vm.swappiness = 0         The kernel will swap only to avoid an out of memory condition
+# vm.swappiness = 1         Minimum amount of swapping without disabling it entirely.
+# vm.swappiness = 10        This value is sometimes recommended to improve performance when sufficient memory exists in a system.
+# vm.swappiness = 60        The default value.
+# vm.swappiness = 100       The kernel will swap aggressively.
+###################################################################
+vm.swappiness = 10
+</Code>
+That's about it !\\
+If you want to know what your current system's swappiness is, use ''sysctl -a | grep swappiness''.
 ===== Shell Customization & Utilities =====
@@ Line 421: / Line 505: @@
 ==== Time Sync ====
-One important aspect when setting up a new system is time synchronization, to achieve this we'll use ''ntp''. Debian 9 has time synchronization built in and activated by default using the standard ''ntpd'' time server, provided by the ''ntp'' package.
+One important aspect when setting up a new system is time synchronization, to achieve this we'll use ''ntp''. Debian 9 should have time synchronization built in and activated by default using the standard ''ntpd'' time server, provided by the ''ntp'' package.
 First lets check whether the ''ntp'' time server is running:
@@ Line 453: / Line 537: @@
 You can refer to [[https://blog.sleeplessbeastie.eu/2015/04/27/how-to-manage-system-services-on-debian-jessie/|this list of systemctl commands]] to see how to manage services.
+To configure your system's time zone use the ''dpkg-reconfigure tzdata''
 ==== SMART ====
+<wrap round info 60%>The following **only applies to physical servers**</wrap>
 ''smartmontools'' is a set of applications that can test hard drives, automatically notify you when the failure rate rises and read the harddisk SMART statistics to detect failures early. For a complete //How To// check [[https://www.howtoforge.com/tutorial/monitor-harddisk-with-smartmon-on-ubuntu/|this article @howtoforge.com]].
+The smartmontools package contains two utility programs (smartctl and smartd) to control and monitor storage systems using the Self-Monitoring, Analysis and Reporting Technology System (SMART).
+<Code:bash>
+> apt-get install smartmontools
+> lsblk
+NAME    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
+sda       8:0    0  1.8T  0 disk
+├─sda1    8:1    0  511M  0 part
+sdb       8:16   0  1.8T  0 disk
+├─sdb1    8:17   0  511M  0 part
+sdc       8:32   1  1.8T  0 disk
+├─sdc1    8:33   1  511M  0 part
+> smartctl -i /dev/sda
+smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.138-xxxx-std-ipv6-64] (local build)
+Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
+=== START OF INFORMATION SECTION ===
+Device Model:     HGST HUS726020ALA610
+Serial Number:    N4G3M6DY
+...
+SMART support is: Available - device has SMART capability.
+SMART support is: Enabled
+</Code>
+Check the last 2 lines to make sure SMART is enabled on each device you want to monitor, if you need to enable it (it won't do any harm if already enabled):
+<Code:bash>
+> smartctl -s on /dev/sda
+</Code>
+To check the complete SMART information for an IDE drive and SATA drive type:
+<Code:bash>
+#  (for IDE drives)
+> sudo smartctl -a /dev/sda
+# (for SATA drives)
+> sudo smartctl -a -d ata /dev/sda
+</Code>
+You can check your hard drive's status by typing:
+<Code:bash>
+> sudo smartctl -H /dev/sda
+</Code>
+The tool used to test your hard drive has three types of tests to perform, Short, Extended & Conveyance.\\
+You can check which is available by typing:
+<Code:bash>
+> sudo smartctl -c /dev/sda
+...
+                                        Self-test supported.
+                                        No Conveyance Self-test supported.
+...
+Short self-test routine
+recommended polling time:        (   2) minutes.
+Extended self-test routine
+recommended polling time:        ( 288) minutes.
+...
+</Code>
+To effectively launch a test:
+<Code:bash>
+> sudo smartctl -t short /dev/sda
+> sudo smartctl -t long /dev/sda
+> sudo smartctl -t conveyance /dev/sda
+=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
+Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
+Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
+Testing has begun.
+Please wait 2 minutes for test to complete.
+Test will complete after Sat Nov 24 00:12:47 2018
+Use smartctl -X to abort test.
+</Code>
+To get the test results you will have to type:
+<Code:bash>
+> sudo smartctl -l selftest /dev/sda
+</Code>
+=== Sending Mail ===
+It would be nice to have ''smartd'' automatically sending us emails whenever SMART detects something worth notifying.\\
+This could be achieved by installing and configuring a MTA like a ''postfix'' server.
+Although, as we don't need (want) this server to become an smtp relay, we'll rather install the ''heirloom-mailx / s-nail'' package and configure it to use an external smtp relay, see the [[https://www.systutorials.com/1411/sending-email-from-mailx-command-in-linux-using-gmails-smtp/|this article @systurials.com]].
+<Code:bash>
+> apt-get install heirloom-mailx
+> cd /usr/bin/
+> ln -s heirloom-mailx mailx
+</Code>
+We need to create the ''mailx'' symbolic link in order to have it available as such under the command line.
+As is stated in the [[https://linux.die.net/man/1/mailx|mailx man page]]:
+<WRAP box 90%>
+Normally, mailx invokes ''sendmail(8)'' directly to transfer messages. If the ''smtp'' variable is set, a SMTP connection to the server specified by the value of this variable is used instead. If the SMTP server does not use the standard port, a value of ''server:port'' can be given, with port as a name or as a number.
+</WRAP>
+Let's test an external smtp server using an all-in-one command:
+<Code:bash>
+> echo "My message body" | mailx -v \
+-s "Server Test Mail" \
+-S smtp-use-starttls \
+-S ssl-verify=ignore \
+-S smtp-auth=login \
+-S smtp=smtp.gmail.com:587 \
+-S from="*******@gmail.com(System Watch)" \
+-S smtp-auth-user=*******@gmail.com \
+-S smtp-auth-password=******* \
+recipient@domain.com
+Resolving host "smtp.gmail.com:587" ... done
+Connecting to "2a00:1450:400c:c06::6c:587" ...connected.
+</Code>
+You'll probably need to allow //Less secured apps access// in you gmail account for this to work ([[https://myaccount.google.com/lesssecureapps|here]])
+Once this works, and you effectively receive the email at the ''recipient@domain.com'' address, it's time to configure a system wide smtp configuration for ''mailx'' and send a last test.\\
+Note that, although the usage of the user specific ''~/.mailrc'' config file was quite obvious, it was much more tricky to determine the correct location (and name) of the global configuration file used by the ''mailx'' command. Having seen a lot of references to ''/etc/mail.rc'', it took a peak at the source code of ''s-nail'' to finally establish that, for v14.8.16, the correct location for the general configuration file was ''/etc/s-nail.rc''!
+<Code:bash>
+> nano /etc/s-nail.rc
+TYPE
+account gmail {
+  set smtp-use-starttls
+  set ssl-verify=ignore
+  set smtp=smtp.gmail.com:587
+  set smtp-auth=login
+  set smtp-auth-user=*******@gmail.com
+  set smtp-auth-password=*******
+  set from="*******@gmail.com(System Watch)"
+}
+> echo "Global mailx configuration file was used here." | mailx -v -A gmail -s "Sent with global configuration" recipient@domain.com
+Resolving host "smtp.gmail.com:587" ... done
+Connecting to "2a00:1450:400c:c0b::6d:587" ...connected.
+</Code>
+When this works, we're ready to automate SMART reports...
+=== Automating SMART Reports ===
+First we'll need to run ''smartmontools'' as a system Deamon in order to have SMART tests run automatically, to do this we'll edit the ''smartmontools'' config file:
+<Code:bash>
+> sudo nano /etc/default/smartmontools
+...
+# uncomment to start smartd on system startup
+start_smartd=yes
+...
+</Code>
+Now, ''smartd'' will be launched at boot time.
+Next, to define how SMART will scan the disk and what actions are to be taken if SMART returns any error, we'll configure ''smartd''.\\
+By default, ''smartd'' will run ''/usr/share/smartmontools/smartd-runner'', which will create a temporary report file, and in turn, will run scripts located in ''/etc/smartmontools/run.d/''. As we'd like to modify this behavior, well create our own copy of those scripts:
+<Code:bash|As root do:>
+> mkdir -p ~/.smartd/run.d
+> cp /usr/share/smartmontools/smartd-runner ~/.smartd/
+> cp /etc/smartmontools/run.d/10mail ~/.smartd/run.d/10mailx
+> nano ~/.smartd/smartd-runner
+#!/bin/bash -e
+run-parts --report --lsbsysinit --arg="$1" \
+    --arg="$2" --arg="$3" -- /root/.smartd/run.d
+> nano ~/.smartd/run.d/10mailx
+#!/bin/bash -e
+# Send mail if /usr/bin/mailx exists
+if ! [ -x /usr/bin/mailx ]; then
+        echo "Your system does not have /usr/bin/mailx.  Install the mailx package"
+        exit 1
+fi
+echo "$SMARTD_FULLMESSAGE" | /usr/bin/mailx -A gmail -s "$SMARTD_FAILTYPE - $SMARTD_MESSAGE" $SMARTD_ADDRESS
+</Code>
+These two scripts should now work together and generate a SMART report that gets emailed to a defined email address. In order to achieve this, we still need to configure the ''smartd'' service through the ''/etc/smartd.conf file''. Open it and uncomment the first line starting with ''DEVICESCAN'', replacing it as follow:
+<Code:bash>
+> sudo nano /etc/smartd.conf
+...
+DEVICESCAN -M test -a -H -l error -l selftest -f -s (S/../..././02|L/../../1/00) -m tech@tacticz.com -M exec /root/.smartd/smartd-runner
+...
+</Code>
+OPTIONS:\\
+  * -M test : specifies that a test run should be executed the next time the ''smartd'' service is restarted
+  * -a : This enables some common options. You almost certainly want to use it as it checks the SMART health status (-H). Reports increases in both SMART error logs (-l). To check for failure of any Usage Attributes (-f) .
+  * -s (S/../.././02|L/../../1/00) : This schedules the short and long self-tests. In this example, the short self-test will run daily at 2:00 A.M. The long test will run on every Monday at 0:00.
+For more information, see [[https://www.freebsd.org/cgi/man.cgi?query=smartd.conf&manpath=ports&sektion=5|the smartd.conf man page]].
+To test that everything works as expected, specifically that emails get sent, and since we have set the ''-M test'' option, we'll restart ''smartd''. As we restart the service, it is a good idea to have an eye on the log files so to get a feedback of the operations:
+<Code:bash: Watch logs in one terminal>
+> lnav /var/log/
+</Code>
+<Code:bash| Restart smartd from another terminal>
+> systemctl restart smartd
+</Code>
+You should see something like this in the logs:
+<Code>
+...
+Nov 25 03:06:09 cloud smartd[433]: Opened configuration file /etc/smartd.conf
+...
+Nov 25 03:06:09 cloud smartd[433]: Device: /dev/sda [SAT], state read from /var/lib/smartmontools/smartd.HGST_HUS726020ALA610-N4G3M6DY.ata.state
+...
+Nov 25 03:06:09 cloud smartd[433]: Monitoring 3 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices
+Nov 25 03:06:09 cloud smartd[433]: Executing test of /root/.smartd/smartd-runner to recipient@domain.com
+Nov 25 03:06:09 cloud smartd[433]: Test of /root/.smartd/smartd-runner to recipient@domain.com: successful
+...
+</Code>
+Test emails should be delivered to the target mailbox.\\
+When it works, remove the ''-M test'' option in ''/etc/smartd.conf'' and restart the deamon with ''systemctl restart smartd''.
+=== FAIL2BAN ===
+On a physical server it might be a good idea to install ''fail2ban'' to establish a minimal protection.
+[[https://upcloud.com/community/tutorials/install-fail2ban-debian/]]
 ===== Install VirtualBox guest additions =====