Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
vms:debian [2018/11/24 00:46] admin [SMART] |
vms:debian [2019/04/09 12:25] (current) admin [Swap File] |
||
---|---|---|---|
Line 269: | Line 269: | ||
> systemctl reload sshd | > systemctl reload sshd | ||
</Code> | </Code> | ||
+ | |||
+ | ===== Swap File ===== | ||
+ | |||
+ | As stated above, using a swap partition isn't always the best option when using a VM since it might become necessary to resize the swap space when the VM's memory configuration is changed based on its utilization. When a swap partition is used, it becomes necessary to "play" with the partitions sizes, which means changing the partitions scheme and oftentimes even the virtual disk's size, which is tedious... | ||
+ | |||
+ | Therefore, in the case of VMs, it is often much easier to create the swap space using a system file, you can learn more about this in [[https://www.digitalocean.com/community/tutorials/how-to-configure-virtual-memory-swap-file-on-a-vps|this Digital Ocean's article]]. Here are the steps to create and use a swap file. | ||
+ | |||
+ | First to make sure swap is not already activated on the system use the ''free'' command: | ||
+ | <Code:bash> | ||
+ | > free | ||
+ | total used free shared buff/cache available | ||
+ | Mem: 2058304 37680 1759884 2968 260740 1872108 | ||
+ | Swap: 0 0 0 | ||
+ | </Code> | ||
+ | |||
+ | The ''free : 0'' value on the ''Swap:'' line means swap is indeed **not activated** | ||
+ | |||
+ | We'll create the partition file under ''/var'' and name it ''swap.img'', then change its permissions to ''600'' so no user will be able to access it: | ||
+ | <Code:bash> | ||
+ | > touch /var/swap.img | ||
+ | > chmod 600 /var/swap.img | ||
+ | </Code> | ||
+ | |||
+ | === Sizing === | ||
+ | |||
+ | Deciding on the swap size is really case dependent. In general, it is recommended recommend to set it to 1-2x the available system RAM. So, if you have a 512mb RAM VM, use 512mb-1gb swap. If you have a 1gb RAM VM use 1gb-2gb swap, etc. This is not a hard and fast rule, for example if you have a 4gb RAM VM it may be best to use little (512mb) or no swap at all.\\ | ||
+ | We use the ''dd'' command to stretch our swap file size, filling it with zeroes to the size we need (here 2Gb): | ||
+ | <Code:bash> | ||
+ | > dd if=/dev/zero of=/var/swap.img bs=1024k count=2048 | ||
+ | 2048+0 records in | ||
+ | 2048+0 records out | ||
+ | 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 1.57258 s, 1.4 GB/s | ||
+ | </Code> | ||
+ | |||
+ | Next, we prepare the file to be usable as the swap file: | ||
+ | <Code:bash> | ||
+ | > mkswap /var/swap.img | ||
+ | Setting up swapspace version 1, size = 2 GiB (2147479552 bytes) | ||
+ | no label, UUID=f8d9bdfe-4090-4bc6-8f7a-fc74b64946ad | ||
+ | </Code> | ||
+ | |||
+ | Then we turn on swapping: | ||
+ | <Code:bash> | ||
+ | > swapon /var/swap.img | ||
+ | > free | ||
+ | total used free shared buff/cache available | ||
+ | Mem: 2058304 38784 67972 2968 1951548 1847628 | ||
+ | Swap: 2097148 0 2097148 | ||
+ | </Code> | ||
+ | |||
+ | We now see that the swap is active. We could turn off swapping with the ''swapoff /var/swap.img'' command.\\ | ||
+ | Turning the swap on this way will **not activate swapping on the next boot**, so we need to modify ''/etc/fstab'' to have the swap file activated at boot time: | ||
+ | |||
+ | <Code> | ||
+ | > nano /etc/fstab | ||
+ | ADD THOSE LINES | ||
+ | # swap file | ||
+ | /var/swap.img none swap sw 0 0 | ||
+ | </Code> | ||
+ | |||
+ | Finally, one could define the system's //swappiness//, which tells the Linux kernel/VM handler how likely it should be to use VM. It is a percent value, between 0 & 100. A usual recommendation for VMs is 30: | ||
+ | <Code:bash> | ||
+ | > sysctl -w vm.swappiness=30 | ||
+ | vm.swappiness = 30 | ||
+ | </Code> | ||
+ | |||
+ | To make swappiness persistent: | ||
+ | <Code:bash> | ||
+ | > nano /etc/sysctl.conf | ||
+ | ... | ||
+ | ################################################################### | ||
+ | # Swap usage | ||
+ | # | ||
+ | # vm.swappiness = 0 The kernel will swap only to avoid an out of memory condition | ||
+ | # vm.swappiness = 1 Minimum amount of swapping without disabling it entirely. | ||
+ | # vm.swappiness = 10 This value is sometimes recommended to improve performance when sufficient memory exists in a system. | ||
+ | # vm.swappiness = 60 The default value. | ||
+ | # vm.swappiness = 100 The kernel will swap aggressively. | ||
+ | ################################################################### | ||
+ | vm.swappiness = 10 | ||
+ | </Code> | ||
+ | |||
+ | That's about it !\\ | ||
+ | If you want to know what your current system's swappiness is, use ''sysctl -a | grep swappiness''. | ||
===== Shell Customization & Utilities ===== | ===== Shell Customization & Utilities ===== | ||
Line 421: | Line 505: | ||
==== Time Sync ==== | ==== Time Sync ==== | ||
- | One important aspect when setting up a new system is time synchronization, to achieve this we'll use ''ntp''. Debian 9 has time synchronization built in and activated by default using the standard ''ntpd'' time server, provided by the ''ntp'' package. | + | One important aspect when setting up a new system is time synchronization, to achieve this we'll use ''ntp''. Debian 9 should have time synchronization built in and activated by default using the standard ''ntpd'' time server, provided by the ''ntp'' package. |
First lets check whether the ''ntp'' time server is running: | First lets check whether the ''ntp'' time server is running: | ||
Line 453: | Line 537: | ||
You can refer to [[https://blog.sleeplessbeastie.eu/2015/04/27/how-to-manage-system-services-on-debian-jessie/|this list of systemctl commands]] to see how to manage services. | You can refer to [[https://blog.sleeplessbeastie.eu/2015/04/27/how-to-manage-system-services-on-debian-jessie/|this list of systemctl commands]] to see how to manage services. | ||
+ | |||
+ | To configure your system's time zone use the ''dpkg-reconfigure tzdata'' | ||
==== SMART ==== | ==== SMART ==== | ||
+ | <wrap round info 60%>The following **only applies to physical servers**</wrap> | ||
+ | |||
''smartmontools'' is a set of applications that can test hard drives, automatically notify you when the failure rate rises and read the harddisk SMART statistics to detect failures early. For a complete //How To// check [[https://www.howtoforge.com/tutorial/monitor-harddisk-with-smartmon-on-ubuntu/|this article @howtoforge.com]]. | ''smartmontools'' is a set of applications that can test hard drives, automatically notify you when the failure rate rises and read the harddisk SMART statistics to detect failures early. For a complete //How To// check [[https://www.howtoforge.com/tutorial/monitor-harddisk-with-smartmon-on-ubuntu/|this article @howtoforge.com]]. | ||
+ | |||
+ | The smartmontools package contains two utility programs (smartctl and smartd) to control and monitor storage systems using the Self-Monitoring, Analysis and Reporting Technology System (SMART). | ||
+ | |||
+ | <Code:bash> | ||
+ | > apt-get install smartmontools | ||
+ | |||
+ | > lsblk | ||
+ | NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT | ||
+ | sda 8:0 0 1.8T 0 disk | ||
+ | ├─sda1 8:1 0 511M 0 part | ||
+ | sdb 8:16 0 1.8T 0 disk | ||
+ | ├─sdb1 8:17 0 511M 0 part | ||
+ | sdc 8:32 1 1.8T 0 disk | ||
+ | ├─sdc1 8:33 1 511M 0 part | ||
+ | |||
+ | > smartctl -i /dev/sda | ||
+ | smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.138-xxxx-std-ipv6-64] (local build) | ||
+ | Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org | ||
+ | |||
+ | === START OF INFORMATION SECTION === | ||
+ | Device Model: HGST HUS726020ALA610 | ||
+ | Serial Number: N4G3M6DY | ||
+ | ... | ||
+ | SMART support is: Available - device has SMART capability. | ||
+ | SMART support is: Enabled | ||
+ | </Code> | ||
+ | |||
+ | Check the last 2 lines to make sure SMART is enabled on each device you want to monitor, if you need to enable it (it won't do any harm if already enabled): | ||
+ | <Code:bash> | ||
+ | > smartctl -s on /dev/sda | ||
+ | </Code> | ||
+ | |||
+ | To check the complete SMART information for an IDE drive and SATA drive type: | ||
+ | <Code:bash> | ||
+ | # (for IDE drives) | ||
+ | > sudo smartctl -a /dev/sda | ||
+ | # (for SATA drives) | ||
+ | > sudo smartctl -a -d ata /dev/sda | ||
+ | </Code> | ||
+ | |||
+ | You can check your hard drive's status by typing: | ||
+ | <Code:bash> | ||
+ | > sudo smartctl -H /dev/sda | ||
+ | </Code> | ||
+ | |||
+ | The tool used to test your hard drive has three types of tests to perform, Short, Extended & Conveyance.\\ | ||
+ | You can check which is available by typing: | ||
+ | <Code:bash> | ||
+ | > sudo smartctl -c /dev/sda | ||
+ | ... | ||
+ | Self-test supported. | ||
+ | No Conveyance Self-test supported. | ||
+ | ... | ||
+ | Short self-test routine | ||
+ | recommended polling time: ( 2) minutes. | ||
+ | Extended self-test routine | ||
+ | recommended polling time: ( 288) minutes. | ||
+ | ... | ||
+ | </Code> | ||
+ | |||
+ | To effectively launch a test: | ||
+ | <Code:bash> | ||
+ | > sudo smartctl -t short /dev/sda | ||
+ | > sudo smartctl -t long /dev/sda | ||
+ | > sudo smartctl -t conveyance /dev/sda | ||
+ | |||
+ | === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION === | ||
+ | Sending command: "Execute SMART Short self-test routine immediately in off-line mode". | ||
+ | Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful. | ||
+ | Testing has begun. | ||
+ | Please wait 2 minutes for test to complete. | ||
+ | Test will complete after Sat Nov 24 00:12:47 2018 | ||
+ | |||
+ | Use smartctl -X to abort test. | ||
+ | </Code> | ||
+ | |||
+ | To get the test results you will have to type: | ||
+ | <Code:bash> | ||
+ | > sudo smartctl -l selftest /dev/sda | ||
+ | </Code> | ||
+ | |||
+ | |||
+ | === Sending Mail === | ||
+ | It would be nice to have ''smartd'' automatically sending us emails whenever SMART detects something worth notifying.\\ | ||
+ | This could be achieved by installing and configuring a MTA like a ''postfix'' server. | ||
+ | |||
+ | Although, as we don't need (want) this server to become an smtp relay, we'll rather install the ''heirloom-mailx / s-nail'' package and configure it to use an external smtp relay, see the [[https://www.systutorials.com/1411/sending-email-from-mailx-command-in-linux-using-gmails-smtp/|this article @systurials.com]]. | ||
+ | |||
+ | <Code:bash> | ||
+ | > apt-get install heirloom-mailx | ||
+ | > cd /usr/bin/ | ||
+ | > ln -s heirloom-mailx mailx | ||
+ | </Code> | ||
+ | We need to create the ''mailx'' symbolic link in order to have it available as such under the command line. | ||
+ | |||
+ | As is stated in the [[https://linux.die.net/man/1/mailx|mailx man page]]: | ||
+ | <WRAP box 90%> | ||
+ | Normally, mailx invokes ''sendmail(8)'' directly to transfer messages. If the ''smtp'' variable is set, a SMTP connection to the server specified by the value of this variable is used instead. If the SMTP server does not use the standard port, a value of ''server:port'' can be given, with port as a name or as a number. | ||
+ | </WRAP> | ||
+ | |||
+ | Let's test an external smtp server using an all-in-one command: | ||
+ | <Code:bash> | ||
+ | > echo "My message body" | mailx -v \ | ||
+ | -s "Server Test Mail" \ | ||
+ | -S smtp-use-starttls \ | ||
+ | -S ssl-verify=ignore \ | ||
+ | -S smtp-auth=login \ | ||
+ | -S smtp=smtp.gmail.com:587 \ | ||
+ | -S from="*******@gmail.com(System Watch)" \ | ||
+ | -S smtp-auth-user=*******@gmail.com \ | ||
+ | -S smtp-auth-password=******* \ | ||
+ | recipient@domain.com | ||
+ | |||
+ | Resolving host "smtp.gmail.com:587" ... done | ||
+ | Connecting to "2a00:1450:400c:c06::6c:587" ...connected. | ||
+ | </Code> | ||
+ | |||
+ | You'll probably need to allow //Less secured apps access// in you gmail account for this to work ([[https://myaccount.google.com/lesssecureapps|here]]) | ||
+ | |||
+ | Once this works, and you effectively receive the email at the ''recipient@domain.com'' address, it's time to configure a system wide smtp configuration for ''mailx'' and send a last test.\\ | ||
+ | Note that, although the usage of the user specific ''~/.mailrc'' config file was quite obvious, it was much more tricky to determine the correct location (and name) of the global configuration file used by the ''mailx'' command. Having seen a lot of references to ''/etc/mail.rc'', it took a peak at the source code of ''s-nail'' to finally establish that, for v14.8.16, the correct location for the general configuration file was ''/etc/s-nail.rc''! | ||
+ | <Code:bash> | ||
+ | > nano /etc/s-nail.rc | ||
+ | TYPE | ||
+ | account gmail { | ||
+ | set smtp-use-starttls | ||
+ | set ssl-verify=ignore | ||
+ | set smtp=smtp.gmail.com:587 | ||
+ | set smtp-auth=login | ||
+ | set smtp-auth-user=*******@gmail.com | ||
+ | set smtp-auth-password=******* | ||
+ | set from="*******@gmail.com(System Watch)" | ||
+ | } | ||
+ | |||
+ | > echo "Global mailx configuration file was used here." | mailx -v -A gmail -s "Sent with global configuration" recipient@domain.com | ||
+ | Resolving host "smtp.gmail.com:587" ... done | ||
+ | Connecting to "2a00:1450:400c:c0b::6d:587" ...connected. | ||
+ | </Code> | ||
+ | |||
+ | When this works, we're ready to automate SMART reports... | ||
+ | |||
+ | |||
+ | === Automating SMART Reports === | ||
+ | |||
+ | First we'll need to run ''smartmontools'' as a system Deamon in order to have SMART tests run automatically, to do this we'll edit the ''smartmontools'' config file: | ||
+ | <Code:bash> | ||
+ | > sudo nano /etc/default/smartmontools | ||
+ | ... | ||
+ | # uncomment to start smartd on system startup | ||
+ | start_smartd=yes | ||
+ | ... | ||
+ | </Code> | ||
+ | Now, ''smartd'' will be launched at boot time. | ||
+ | |||
+ | Next, to define how SMART will scan the disk and what actions are to be taken if SMART returns any error, we'll configure ''smartd''.\\ | ||
+ | By default, ''smartd'' will run ''/usr/share/smartmontools/smartd-runner'', which will create a temporary report file, and in turn, will run scripts located in ''/etc/smartmontools/run.d/''. As we'd like to modify this behavior, well create our own copy of those scripts: | ||
+ | |||
+ | <Code:bash|As root do:> | ||
+ | > mkdir -p ~/.smartd/run.d | ||
+ | > cp /usr/share/smartmontools/smartd-runner ~/.smartd/ | ||
+ | > cp /etc/smartmontools/run.d/10mail ~/.smartd/run.d/10mailx | ||
+ | |||
+ | > nano ~/.smartd/smartd-runner | ||
+ | #!/bin/bash -e | ||
+ | |||
+ | run-parts --report --lsbsysinit --arg="$1" \ | ||
+ | --arg="$2" --arg="$3" -- /root/.smartd/run.d | ||
+ | |||
+ | > nano ~/.smartd/run.d/10mailx | ||
+ | #!/bin/bash -e | ||
+ | |||
+ | # Send mail if /usr/bin/mailx exists | ||
+ | if ! [ -x /usr/bin/mailx ]; then | ||
+ | echo "Your system does not have /usr/bin/mailx. Install the mailx package" | ||
+ | exit 1 | ||
+ | fi | ||
+ | |||
+ | echo "$SMARTD_FULLMESSAGE" | /usr/bin/mailx -A gmail -s "$SMARTD_FAILTYPE - $SMARTD_MESSAGE" $SMARTD_ADDRESS | ||
+ | </Code> | ||
+ | |||
+ | These two scripts should now work together and generate a SMART report that gets emailed to a defined email address. In order to achieve this, we still need to configure the ''smartd'' service through the ''/etc/smartd.conf file''. Open it and uncomment the first line starting with ''DEVICESCAN'', replacing it as follow: | ||
+ | |||
+ | <Code:bash> | ||
+ | > sudo nano /etc/smartd.conf | ||
+ | ... | ||
+ | DEVICESCAN -M test -a -H -l error -l selftest -f -s (S/../..././02|L/../../1/00) -m tech@tacticz.com -M exec /root/.smartd/smartd-runner | ||
+ | ... | ||
+ | </Code> | ||
+ | |||
+ | OPTIONS:\\ | ||
+ | * -M test : specifies that a test run should be executed the next time the ''smartd'' service is restarted | ||
+ | * -a : This enables some common options. You almost certainly want to use it as it checks the SMART health status (-H). Reports increases in both SMART error logs (-l). To check for failure of any Usage Attributes (-f) . | ||
+ | * -s (S/../.././02|L/../../1/00) : This schedules the short and long self-tests. In this example, the short self-test will run daily at 2:00 A.M. The long test will run on every Monday at 0:00. | ||
+ | |||
+ | For more information, see [[https://www.freebsd.org/cgi/man.cgi?query=smartd.conf&manpath=ports&sektion=5|the smartd.conf man page]]. | ||
+ | |||
+ | |||
+ | To test that everything works as expected, specifically that emails get sent, and since we have set the ''-M test'' option, we'll restart ''smartd''. As we restart the service, it is a good idea to have an eye on the log files so to get a feedback of the operations: | ||
+ | |||
+ | <Code:bash: Watch logs in one terminal> | ||
+ | > lnav /var/log/ | ||
+ | </Code> | ||
+ | |||
+ | <Code:bash| Restart smartd from another terminal> | ||
+ | > systemctl restart smartd | ||
+ | </Code> | ||
+ | |||
+ | You should see something like this in the logs: | ||
+ | <Code> | ||
+ | ... | ||
+ | Nov 25 03:06:09 cloud smartd[433]: Opened configuration file /etc/smartd.conf | ||
+ | ... | ||
+ | Nov 25 03:06:09 cloud smartd[433]: Device: /dev/sda [SAT], state read from /var/lib/smartmontools/smartd.HGST_HUS726020ALA610-N4G3M6DY.ata.state | ||
+ | ... | ||
+ | Nov 25 03:06:09 cloud smartd[433]: Monitoring 3 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices | ||
+ | Nov 25 03:06:09 cloud smartd[433]: Executing test of /root/.smartd/smartd-runner to recipient@domain.com | ||
+ | Nov 25 03:06:09 cloud smartd[433]: Test of /root/.smartd/smartd-runner to recipient@domain.com: successful | ||
+ | ... | ||
+ | </Code> | ||
+ | |||
+ | Test emails should be delivered to the target mailbox.\\ | ||
+ | When it works, remove the ''-M test'' option in ''/etc/smartd.conf'' and restart the deamon with ''systemctl restart smartd''. | ||
+ | |||
+ | |||
+ | === FAIL2BAN === | ||
+ | On a physical server it might be a good idea to install ''fail2ban'' to establish a minimal protection. | ||
+ | |||
+ | [[https://upcloud.com/community/tutorials/install-fail2ban-debian/]] | ||
+ | |||
===== Install VirtualBox guest additions ===== | ===== Install VirtualBox guest additions ===== | ||