Friday, November 12, 2021

Gentoo with EFIStub, encrypted BTRFS, swap, dracut or genkernel initramfs, open-rc.

Hello!  It's been a while since I wrote a post; life has been busy I'm working on a better blog management system and I finally got a desktop!  Hopefully more updates to come both here and on!  Stay tuned for more updates.

Also, I must apologize for the lack of wrapping on this blog entry.  This blog will be 11 years old next month, and I plan on updating the theme soon.  Please bear with me.

Recently, I built a Gentoo system with an efistub (no bootloader) and ZFS natively-encrypted rootfs.  It was great until I ran into a bug on kernels below 5.14 where the zfs system would get currupted.  OpenZFS's answer was "Update to 5.14" (apparently Gentoo's stable was 5.10), and if I'm updating out of whatever is stable I'm going all the way and trying 5.15...  which isn't supported yet for ZFS.  Result? I scrapped the whole thing and decided to just use btrfs!

This is heavily inspired by this post (shout out to William for putting that together) except I plan on using btrfs with zlib compression instead of ext4, I plan on having a module-free kernel, I want an encrypted swap partition, and we can no longer use eudev (because they're retiring it).  I also don't want to use lvm, it's just another layer of complexity that I don't need, and I'll be showing how to use dracut as well as genkernel depending on the route you wanted to go.  We'll be extracting the resulting initramfs, and building it into the kernel.

Boot into a Linux environment

 I decided to use an ubuntu live CD, but anything with the standard tools will work. I also plugged as many devices into the computer as I thought I would need, so I can take advantage of localyesconfig.

Bring up a terminal and find your HDD/SSD

This is a good way to do it:

root@ubuntu:~$ sudo fdisk -l | grep GiB
Disk /dev/loop0: 2.3 GiB, 2470006784 bytes, 4824232 sectors
Disk /dev/nvme0n1: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/sda: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/sdb: 111.79 GiB, 120034123776 bytes, 234441648 sectors
Disk /dev/sdd: 14.32 GiB, 15376000000 bytes, 30031250 sectors

I'm using the nvme, but others are probably using sdX.

 Partition out the drive

We'll be setting the EFI partition with about 1GB of space (more than we'll ever need), along with plenty of swap and leave the rest for the rootfs:

root@ubuntu:~# parted -a optimal /dev/nvme0n1
GNU Parted 3.4
Using /dev/nvme0n1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit mib
(parted) mklabel gpt
(parted) mkpart primary 1 1025
(parted) name 1 "EFI System Partition"                                    
(parted) set 1 esp on                                                     
(parted) set 1 boot on                                                    
(parted) mkpart primary 1025 33280                                        
(parted) name 2 "swap"
(parted) mkpart primary 33280 100%
(parted) name 3 "rootfs"
(parted) print                                                            
Model: WD Green SN350 1TB (nvme)
Disk /dev/nvme0n1: 953870MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start     End        Size       File system  Name                  Flags
 1      1.00MiB   1025MiB    1024MiB                 EFI System Partition  boot, esp
 2      1025MiB   33280MiB   32255MiB                swap
 3      33280MiB  953869MiB  920589MiB               rootfs

(parted) quit                                                             
Information: You may need to update /etc/fstab.

root@ubuntu:~# fdisk -l | grep nvme
Disk /dev/nvme0n1: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
/dev/nvme0n1p1     2048    2099199    2097152    1G EFI System
/dev/nvme0n1p2  2099200   68157439   66058240 31.5G Linux filesystem
/dev/nvme0n1p3 68157440 1953523711 1885366272  899G Linux filesystem

Encrypt / format the root partition

 I'm taking a page out of Sakaki's handbook and choosing the serpent cipher with the whirlpool cipher.  The reason being is that serpent is a technically better cipher and has actually managed to outperform AES when using xts.  While I do have an intel processor that could take advantage of AES-NI, not everyone does.  As for whirlpool, it has been shown to slow brute force attacks.

root@ubuntu:~# cryptsetup --cipher serpent-xts-plain64 --key-size 512 --hash whirlpool luksFormat /dev/nvme0n1p3

Since I use dvorak, I then switched my layout to US, and added an additional key, re-typing the same passphrase that I used on dvorak (in case I need to unlock this partition on a qwerty layout):

root@ubuntu:~# cryptsetup luksAddKey /dev/nvme0n1p3

What about swap?  Don't worry, we'll get there.  That's later down the road.  Let's open, map, and format the root partition:

root@ubuntu:~# cryptsetup luksOpen /dev/nvme0n1p1 root
:~# mkfs.btrfs -L "root" /dev/mapper/root root@ubuntu:~# mkdir /mnt/gentoo root@ubuntu:~# mount -o noatime,nodiratime,ssd,compress=zlib /dev/mapper/root /mnt/gentoo

Now that the root partition is added, don't forget the EFI partition!

root@ubuntu:~# mkfs.vfat -F32 /dev/nvme0n1p1 
root@ubuntu:~# mkdir -p /mnt/gentoo/boot
root@ubuntu:~# mount /dev/nvme0n1p1 /mnt/gentoo/boot
root@ubuntu:~# mkdir /mnt/gentoo/boot/efi

At this point, go ahead and continue following the Gentoo HandBook through Installing the Gentoo installation files and come back here after you finish that page and extract the stage3.

Encrypt swap

We're going to be handling swap a little differently, by encrypting it with a keyfile.  First, let's generate that file:

root@ubuntu:~# mkdir /mnt/gentoo/etc/keys
root@ubuntu:~# dd if=/dev/random of=/mnt/gentoo/etc/keys/swap.key bs=8388607 count=1
1+0 records in
1+0 records out
8388607 bytes (8.4 MB, 8.0 MiB) copied, 0.0694414 s, 121 MB/s
Now we'll use that file to encrypt and create the swap partition. Make sure you're using the right partition!
root@ubuntu:~# cryptsetup --cipher serpent-xts-plain64 --key-size 512 --hash whirlpool luksFormat /dev/nvme0n1p2 /mnt/gentoo/etc/keys/swap.key

This will overwrite data on /dev/nvme0n1p2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
root@ubuntu:~# cryptsetup luksOpen -d /mnt/gentoo/etc/keys/swap.key /dev/nvme0n1p2 swap root@ubuntu:~# mkswap /dev/mapper/swap Setting up swapspace version 1, size = 31.5 GiB (33805037568 bytes) root@ubuntu:~# swapon /dev/mapper/swap

Finally, get the UUID for the crypt root/swap partitions, /boot, mapper/root, and mapper/swap. We'll need them later:

root@ubuntu:~# blkid | grep '/dev/mapper\|nvme'
/dev/mapper/root: LABEL="root" UUID="2c98ff1c-a40a-4b8d-9131-e5fbc0446091" UUID_SUB="0927d47e-97c0-491b-8654-d4c3f939d1d7" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/nvme0n1p3: UUID="47e7fa0b-2472-4015-96fb-e92ff1c21df7" TYPE="crypto_LUKS" PARTLABEL="rootfs" PARTUUID="aa3322ba-70fd-447a-a353-642a5ad9d1a9"
/dev/nvme0n1p1: UUID="3B87-3ED8" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="e26dee0c-13fe-4b9b-95f7-4001547424a9"
/dev/mapper/swap: UUID="3303e028-df54-424a-876d-1f0c9197d965" TYPE="swap"
/dev/nvme0n1p2: UUID="87833367-6fe9-4118-87b8-9accbe4a09e6" TYPE="crypto_LUKS" PARTLABEL="swap" PARTUUID="d74d55e7-f97f-4007-8d97-181d248ce233"

Now we can proceed with Installing the Gentoo base system up to the Configuring the Kernel: Manual configuration step.

Sidenote: Configuring /etc/portage/make.conf

Here's some special stuff that I'm doing with my make.conf.  First, I installed app-portage/cpuid2cpuflags, to get my supported CPU flags:

(chroot) ubuntu /usr/src/linux # emerge --quiet app-portage/cpuid2cpuflags >>> Verifying ebuild manifests >>> Emerging (1 of 1) app-portage/cpuid2cpuflags-11::gentoo >>> Installing (1 of 1) app-portage/cpuid2cpuflags-11::gentoo >>> Recording app-portage/cpuid2cpuflags in "world" favorites file... (chroot) ubuntu /usr/src/linux # cpuid2cpuflags CPU_FLAGS_X86: aes avx avx2 avx512f avx512dq avx512cd avx512bw avx512vl f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3 (chroot) ubuntu /usr/src/linux #

Then I added them toto my make.conf like so:

CPU_FLAGS_X86="aes . . .ssse3"

 Then, enter the command `nproc` to find out how many cores you have.

(chroot) ubuntu /usr/src/linux # nproc
(chroot) ubuntu /usr/src/linux # 
I recommend setting MAKEOPTS="-j<nproc>/2" and EMERGE_DEFAULT_OPTS to have "--jobs <nproc> --load-average <nproc+1>", so we'll end up with in the make.conf something like:
COMMON_FLAGS="-O2 -pipe -march=native -fomit-frame-pointer"


# This sets the language of build output to English.
LINGUAS="en en_US"
# Please keep this setting intact when reporting bugs.

EMERGE_DEFAULT_OPTS="--with-bdeps y --complete-graph y --jobs 40 --load-average 41"
# This is where the real fun begins USE="offensive"

People probably won't like that I also do this:

# Get off my lawn

On fstab

We'll be more heavily modifying /etc/fstab later, but for now, be sure to add the boot entry:
(chroot) ubuntu /usr/src/linux # echo 'UUID="3B87-3ED8" /boot vfat defaults 0 2' >> /etc/fstab

This will come in handy later so we can just run "mount /boot" if it isn't mounted (though for right now it already is).

Configuring the kernel

The change I'm going to make from the Gentoo Handbook is that we're going to run a couple extra config commands.  Remember when I said to plug everything you think you'll need into the computer when you boot up?  We're going to leverage that now:

(chroot) ubuntu / # eselect kernel list
Available kernel symlink targets:
  [1]   linux-5.15.1-gentoo
(chroot) ubuntu / # eselect kernel set 1
(chroot) ubuntu / # cd /usr/src/linux
(chroot) ubuntu /usr/src/linux # make defconfig
(chroot) ubuntu /usr/src/linux # make localyesconfig

This gives us a sane baseline to what we're probably going to need.  Now all we have to do is customize!

In addition to whatever the Gentoo handbook recommends, we'll also be enabling all of the following to get luks, btrfs, efi support, and initramfs:

General setup  --->
  [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
  (/usr/src/initramfs) Initramfs source file(s)
Processor type and features  --->
    [*] EFI runtime service support 
    [*]   EFI stub support
    [ ]     EFI mixed-mode support
    [*] Built-in kernel command line
    (root=UUID=<UUID OF /dev/mapper> rd.luks.uuid=<UUID of /dev/nvme0n1p3>)
[*] Enable loadable module support
Device Drivers  --->
  Generic Driver Options --->
    [*] Maintain a devtmpfs filesystem to mount at /dev 
    [*]   Automount devtmpfs at /dev, after the kernel mounted the rootfs
  [*] Multiple devices driver support (RAID and LVM)  --->
    {*}   RAID support
    [*]     Autodetect RAID arrays during kernel boot
    <*>     Linear (append) mode
    <*>     RAID-0 (striping) mode
    {*}     RAID-1 (mirroring) mode
    {*}     RAID-10 (mirrored striping) mode
    {*}     RAID-4/RAID-5/RAID-6 mode
    <*> Device mapper support
    <*> Crypt target support
    <*> Snapshot target
    <*> Mirror target
    <*> Multipath target
    <*> Multipath target
      <*> I/O Path Selector based on the number of in-flight I/Os
      <*> I/O Path Selector based on the service time
Firmware Drivers ---> (may be in Device Drivers ---> Firmware Drivers)
  EFI (Extensible Firmware Interface) Support  --->
    <*> EFI Variable Support via sysfs
    [*] Export efi runtime maps to sysfs
File systems  --->
  <*> Btrfs filesystem support
  [*]   Btrfs POSIX Access Control Lists
  <*> FUSE (Filesystem in Userspace) support
  DOS/FAT/EXFAT/NT Filesystems  --->
    <*> MSDOS fs support
    <*> VFAT (Windows-95) fs support
    (437) Default codepage for FAT
    (iso8859-1) Default iocharset for FAT 
    [*] Enable FAT UTF-8 option by default
    <*> exFAT filesystem support
    (utf8) Default iocharset for exFAT
    # Do this if you're on 5.15!
    <*> NTFS Read-Write file system support
      [ ]   64 bits per NTFS clusters
      [*]   activate support of external compressions lzx/xpress
      [ ]   NTFS POSIX Access Control Lists
  Pseudo filesystems --->
    <*> EFI Variable filesystem # This is needed for Efibootmgr
Cryptographic API --->
    <*> XTS support
    <*> RIPEMD-160 digest algorithm
    <*> SHA224 and SHA256 digest algorithm
    <*> Whirlpool digest algorithms 
    <*> LRW support
    <*> AES cipher algorithms
    <*> AES cipher algorithms (x86_64)
    <*> Serpent cipher algorithm 
    <*> Serpent cipher algorithm (x86_64/SSE2) # If your CPU supports these
    <*> Serpent cipher algorithm (x86_64/AVX)  # If your CPU supports these
    <*> Serpent cipher algorithm (x86_64/AVX2) # If your CPU supports these
    <*> Twofish cipher algorithm
    <*> User-space interface for hash algorithms
    <*> User-space interface for symmetric key cipher algorithms 

For the kernel command line, we'll want to pass the uuid of the decrypted kernel (/dev/mapper) to the kernel and the uuid encrypted drive to the initramfs.

If using genkernel for the initramfs:

     "dobtrfs crypt_root=UUID=47e7fa0b-2472-4015-96fb-e92ff1c21df7 root=UUID=2c98ff1c-a40a-4b8d-9131-e5fbc0446091"

If using dracut for the initramfs: 

    "rd.luks.uuid=47e7fa0b-2472-4015-96fb-e92ff1c21df7 root=UUID=2c98ff1c-a40a-4b8d-9131-e5fbc0446091"

I don't recommend compiling the kernel yet, so once you get to Compiling and installing in the handbook, just hold off.

Configuring the Initramfs

The next steps change depending on if you compile your kernel with modules or not, and if you're using dracut.

With modules?

If you did have any modules compiled into your kernel (you can verify by running grep "=m" .config) then you'll have to build the kernel first.  In the below command, replace "-j41" with your corecount plus one.

(chroot) ubuntu /usr/src/linux # mkdir -p /usr/src/initramfs # so we can precompile kernel
(chroot) ubuntu /usr/src/linux # make -j41 && make modules_install
If you do not have modules, then continue on.  The plan is to generate an initramfs, but then we'll actually be compiling it into the kernel.  So first, let's install the required apps:
(chroot) ubuntu /usr/src/linux # cd
(chroot) ubuntu ~ # emerge -avq sys-fs/cryptsetup sys-fs/btrfs-progs sys-apps/busybox sys-kernel/dracut sys-boot/efibootmgr

 And add dmcrypt to boot:

(chroot) ubuntu ~ # rc-update add dmcrypt boot
 * service dmcrypt added to runlevel boot

 Create the /usr/src/initramfs symbolic link:

(chroot) ubuntu ~ # KERNVERN=$(basename $(cd -P /usr/src/linux && pwd) | cut -c 7-)
(chroot) ubuntu ~ # mount /boot
(chroot) ubuntu ~ # rm /usr/src/initramfs # this should be a symbolic link if exists
(chroot) ubuntu ~ # mkdir /usr/src/initramfs-$KERNVERN
(chroot) ubuntu ~ # ln -s /usr/src/initramfs-$KERNVERN initramfs
(chroot) ubuntu ~ # cd /usr/src/initramfs

Generate the initramfs and extract it to /usr/src/initramfs 

    If using genkernel:

(chroot) ubuntu /usr/src/initramfs # genkernel --btrfs --luks --microcode --makeopts=-j41 --no-ramdisk-modules initramfs
(chroot) ubuntu /usr/src/initramfs # xzcat /boot/initramfs-$KERNVERN.img | cpio -imdv
(chroot) ubuntu /usr/src/initramfs # rm /boot/initramfs-$KERNVERN.img

    If using dracut:

(chroot) ubuntu /usr/src/initramfs # mkdir -p /lib/modules/$KERNVERN
(chroot) ubuntu /usr/src/initramfs # dracut --hostonly --kver $KERNVERN -f /usr/src/initramfs-$KERNVERN.img (chroot) ubuntu /usr/src/initramfs # cpio -imdv < /usr/src/initramfs-$KERNVERN.img

The reason for that  intial mkdir is a workaround to a dracut bug when kernels are built without modules. The KERNVERN is just a quick way of getting the current kernel version (in my case, 5.15.1-gentoo).

Now /usr/src/initramfs should contain everything we need to build the kernel!

(chroot) ubuntu /usr/src/initramfs # cd /usr/src/linux
(chroot) ubuntu /usr/src/linux # make clean (chroot) ubuntu /usr/src/linux # make -j41 && make modules_install

(chroot) ubuntu /usr/src/linux # mkdir -p /boot/EFI/Gentoo (chroot) ubuntu /usr/src/linux # cp arch/x86_64/boot/bzImage /boot/EFI/Gentoo/bzImage-$KERNVERN.efi
(chroot) ubuntu /usr/src/linux # efibootmgr --create --part 1 --disk /dev/nvme0n1 --label "Gentoo" --loader "\EFI\Gentoo\bzImage-$KERNVERN.efi"

 Mounting /, swap

Next step is to configure dm-crypt to automatically unlock the swap partition.  Add the following to /etc/conf.d/dmcrypt:

# Definition for /dev/mapper/swap

(replace the UUID with the one for the swap partition, e.g. /dev/nvme0n1p2)

And finally, add this to /etc/fstab under the boot entry we made earlier:

# Swap partition (/dev/mapper/swap)
UUID="3303e028-df54-424a-876d-1f0c9197d965" none swap sw 0 0
# rootfs (/dev/mapper/root)
UUID="2c98ff1c-a40a-4b8d-9131-e5fbc0446091" / btrfs noatime,nodiratime,ssd,compress=zlib 0 1

At this point, it should be safe to continue with the Gentoo Handbook, either at the Kernel Modules section if you built modules, or Configuring the System.  Remember, we're not using a bootloader, so you can skip that part! (feel free to read the entry on efibootmgr though)

 If everything worked, you should get a prompt asking you to enter your password to decrypt the root filesystem.  Unfortuanetly you'll probably also see other "noise" happening (like init systems loading), not much we can do about that without installing something like Plymouth, which I may write another guide for later.

I'll also look at writing a short script for kernel upgrades, once I do, I can amend this blog post.

Sunday, September 1, 2019

Push notifications on linux with cronjobs and smartmontools

Up until very recently, I was having smartmontools (and other cron processes such as backups) email me when a hard drive was failing.  For some reason my already-finicky mail-forwarding setup ended up failing and I couldn't get it to consistently send again, so I thought to myself: What other ways can I have my server notify me in an obvious way?  The answer was Push Notifications!
Now, I wouldn't attest to knowing the details behind post-notifications; my quest was for a quick and easy solution for Android:
Pros of Simplepush:
  • Super easy to set up
  • "Just works"
  • Supports end-to-end encryption
  • Works for android
  • Costs $4
  • Doesn't seem to work for iOS
  • Isn't a self-hosted solution, if that matters to you.
I did find a free solution that would also work with iOS using Telegram, and maybe I'll do that if ever disappears:

Anyway, let's get into how I did this!

First, I used the script freely available by simplepush:

I saved this in /usr/local/bin

Next, I created a new script in /usr/local/bin/ called "" with the following content:

# Edit this part to contain your PSWD='PASSWORD' # Change PASSWORD to correct password (include single-quotes)
SALT='SALT'   # Change SALT to correct salt
SPKEY='KEY'    # Change KEY to the simplepush key generated by the app.

# No need to edit anything below.
# Set MSG, TITLE, and EVENTs based on arguments, if they exist.
if [ -z "$1" ]; then MSG='empty-msg'; else MSG="$1"; fi
if [ -z "$2" ]; then TITLE='Push-notification'; else TITLE="$2"; fi
if [ -z "$3" ]; then EVENT='misc'; else EVENT="$3"; fi

# Usage "/usr/local/bin/ 'message' 'title' 'event'
/usr/local/bin/ -k "$SPKEY" -p "$PSWD" -s "$SALT" -m "$MSG" -t "$TITLE" -e "$EVENT"
(be sure to replace PASSWORD, SALT, and KEY with your own password, salt, and key from the app!)

Now, I can call this wrapper script and it will always send a push notification to my phone like so:
ajadmin@nazz ~ $ /usr/local/bin/ "A message" "A title" "Event."

This is the result:

Now I can start rewriting my cron jobs to utilize this wrapper.  Most of these are simple.  For example, my array uses btrfs for RAID10.  Every Sunday at 9am, I run a scrub on the RAID10 in my NAS.  The result (regardless of how it completes) are emailed to me:
# Run every sunday at 9pm, email me the results
0 21 * * 0 btrfs scrub start -c3 -B /mnt/RAID10

The simplest way I thought to rewrite this would just to capture the output of this command and send it in as an argument to my wrapper.  Thankfully we have "$()" for that!
# Run every sunday at 9pm, push the results
0 21 * * 0 /usr/local/bin/ "$(btrfs scrub start -c3 -B /mnt/RAID10)" 'btrfs-scrub' 'status'

Simple, right?  Let's try something a little more complicated.

Every hour, I check to see if there's any errors being reported to btrfs with the following crontab entry:
@hourly /sbin/btrfs device stats /mnt/RAID10 | grep -vE ' 0$'

Since the command /sbin/btrfs device stats /mnt/RAID10 normally ends in " 0" if there is no error for that disk, this command will normally yield no output.  For my cronjob, it means I would not get an email.  The moment the array starts to spit out errors, I would get an email from my server on the hour, indicating which drive is failing.  Fun!

While I may be able to be just as hacky with my wrapper, I figured I'd just put it in a one-line if statement:
# Check btrfs- email me on drive failure
@hourly if /sbin/btrfs device stats /mnt/RAID10 | grep -vE ' 0$'; then /usr/local/bin/ "$(/sbin/btrfs device stats /mnt/RAID10)" 'btrfs error' 'failure'; fi

Of course, I'm going to have to modify some of my backup scripts to be more nicely formatted, instead of perhaps dumping everything under the sun to me like it used to, but you get the point.

What about smartmontools?

While it's nice that I'm getting push notifications from crontab, smartmontools has a bit of a different way of handling things.  By default, it will email you the alerts when something is going FUBAR but you can actually change this in the /etc/smartd.conf file.
Originally, I had something along the lines of this:
/dev/sdX -a -o on -S on -s (S/../.././16|L/../../6/00) -W 4,35,40 -m -M daily

I found that smartmontools actually has an argument, -M exec script that allows you to execute a specific script while passing a variable $SMARTD_MESSAGE instead of sending an email.  So I created /usr/local/bin/ with the following content:
# Relay SMARTD_MESSAGE to our simplepush wrapper.
/usr/local/bin/ "$SMARTD_MESSAGE" 'smartd' 'failure'

With that out of the way, I modified /etc/smartd.conf with the following:
/dev/sdX -a -o on -S on -s (S/../.././16|L/../../6/00) -W 4,35,40 -m root -M exec /usr/local/bin/ -M daily

A note on security

Since crontab is touching these scripts in /usr/local/bin, I would ensure that they're only editable by root, but executable by anyone:

ajadmin@nazz ~ $ sudo chown root:root -R /usr/local/bin && sudo chmod 755 -R /usr/local/bin
ajadmin@nazz ~ $ ls -lh /usr/local/bin
total 24K
-rwxr-xr-x 1 root root 4.2K Aug  5  2017
-rwxr-xr-x 1 root root 1.5K Aug 25 00:02
-rwxr-xr-x 1 root root  736 Aug 31 18:18
-rwxr-xr-x 1 root root  136 Sep  1 18:06
ajadmin@nazz ~ $ 

That should be about it!  Did I miss anything?  What do you think?  Let me know in the comments!

Tuesday, July 17, 2018

Backups with rsnapshot and SSH

This is a simple schema on how I configure backups over SSH between my various servers. The howto on this stuff can be found all over the web, I'm mostly consolidating it here for my own records, in case I want to do it again later. Big thanks to a few guides on helping me put this together, if you'd like some further reading:
All devices in question run Gentoo, but this should work on anything with rsnapshot and openssh.

Naming Conventions

For the purpose of this guide, the system we're backing things up to will be called NAS and the system we're backing up will be called DESKTOP.
Your naming conventions may very.

Step 1: Install needed software

On backup server, we need to install rsnapshot.  In Gentoo this is:
aj@NAS ~ $ sudo emerge -av rsnapshot
These are the packages that would be merged, in order:

Calculating dependencies... done!

[ebuild  N     ] dev-perl/Lchown-1.10.0-r1::gentoo  4 KiB
[ebuild  N     ] app-backup/rsnapshot-1.4.2::gentoo  477 KiB

Total: 2 packages (2 new), Size of downloads: 481 KiB

Would you like to merge these packages? [Yes/No]
On the desktop, we need to install rsync.  In Gentoo that's sudo emerge -av rsync (I already have rsync).

Step 2: Create "backup" user on NAS and DESKTOP

This is the account that will be used to ssh into both of the computers.
aj@NAS ~ $ useradd -c "Backup user" backup
aj@DESKTOP ~ $ useradd -c "Backup user" backup

Step 3: Create rsync validation and wrapper scripts.

Big props to Arun Natarajan for coming up with this one.  The purpose of this script/check is to only allow rsync as a possible command.  This way, if anyone gets hold of the "backup" account on NAS, they can't do anything (too) bad on my DESKTOP.

Authenticate as the "backup" user on DESKTOP, and create a new file called
aj@DESKTOP ~ $ # Change to the backup user
aj@DESKTOP ~ $ sudo su - backup
backup@DESKTOP ~ $ # Create an "sh" directory if it doesn't exist, we'll put it in there.
backup@DESKTOP ~ $ mkdir -p ~/sh
backup@DESKTOP ~ $ cd ~/sh
backup@DESKTOP ~/sh $ # Create script file
backup@DESKTOP ~/sh $ touch
backup@DESKTOP ~/sh $ # Edit script file
backup@DESKTOP ~/sh $ vi
Open "" with your favorite text editor and paste the following:
    echo "Rejected 1"
    echo "Rejected 2"
    echo "Rejected 3"
Once saved, we'll want to make sure it's executable and not readable by anyone else:
backup@DESKTOP ~/sh $ chmod 700
Now we do the same for the wrapper:
backup@DESKTOP ~/sh $ touch
backup@DESKTOP ~/sh $ vi
Paste the following into

logger -t backup $@
/usr/bin/sudo /usr/bin/rsync "$@";
And make it executable:
backup@DESKTOP ~/sh $ chmod 755 /home/backup/sh/

Step 4: Generate SSH keys for backup user authentication

The backup user on NAS will have to authenticate on DESKTOP. The most secure way of doing this is by creating a certificate. I choose to go with Ed25519.

aj@NAS ~ $ # Change to the backup user
aj@NAS ~ $ sudo su - backup
backup@NAS ~ $ # Generate the key
backup@NAS ~ $ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/backup/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
The key's randomart image is:
+--[ED25519 256]--+
|pretty image here|
backup@NAS ~ $ # Give the key a relevant name
backup@NAS ~ $ mv ~/.ssh/id_ed25519 ~/.ssh/id_ed25519_DESKTOP
backup@NAS ~ $ mv ~/.ssh/ ~/.ssh/

Step 5: Add the new certificate to DESKTOP

We'll need to retrieve the public key from NAS and add it to the desktop's authorized_keys file. Usually this is as simple as running ssh-copy-id, but remember, our backup system accounts don't have passwords, so we won't be able to authenticate. If you're not like me, and actually created a password for both backup accounts, then you can simply run:
aj@NAS ~ $ # Change to backup user
aj@NAS ~ $ sudo su - backup
backup@NAS ~ $ cat ~/.ssh/
ssh-ed25519 backup@NAS
Copy that whole thing, then, on DESKTOP:
aj@DESKTOP ~ $ # Once again, we should be backup user
aj@DESKTOP ~ $ sudo su - backup
backup@DESKTOP ~ $ # Edit the "authorized_keys" file
backup@DESKTOP ~ $ vi .ssh/authorized_keys
Please use your editor of choice.  On the last free line of the authorized_keys file, place:
Then, a space, and the entire contents of "".  For example, this is what it might look like:

command="/home/backup/sh/" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIF3ZY+TF+/Whv4768hHxxgoPXWDnq7/iyiadSgE7DCq backup@NAS
After saving, make the file read-only:
backup@DESKTOP ~ $ chmod 600 .ssh/authorized_keys

Step 4.5: Configure SSHD

If you haven't configured SSHD on DESKTOP yet, this is probably a good time to do it. It may be a good idea to lock it down completely, except for key-based authentication. The Gentoo Security Handbook has a pretty good writeup on securing SSH, so feel free to check that out.
Warning: If your only access to DESKTOP is through SSH, you can lock yourself out of the system by doing this.

Step 5: Configure rsnapshot on NAS

We're going to create a new configuration for rsnapshot.  I'm just going to put it in /etc/, but you can have it in the /home/backup folder if you'd like.  I haven't tested it outside /etc/:
aj@NAS ~ $ sudo vi /etc/rsnapshot-DESKTOP.conf
Here's my example config for rsnapshot; you'll want to modify ssh_argssnapshot_root, and backup:
# Be sure to use TABS to separate values, not spaces!

# So the hard disk is not polluted in case the backup filesystem is not available
no_create_root 1

# Standard settings
cmd_cp  /bin/cp
cmd_rm  /bin/rm
cmd_rsync /usr/bin/rsync
cmd_ssh  /usr/bin/ssh
link_dest 1

verbose  5
loglevel 5

rsync_long_args -evaAX --rsync-path=/home/backup/sh/
ssh_args -i /home/backup/.ssh/id_ed25519_DESKTOP

# For convenience, so that mount points can be taken as backup starting points
one_fs  1

# Store all backups in one directory per machine
# A useful alternative may be to create a separate directory for each interval
snapshot_root /backups/DESKTOP

# increments, which are kept
retain daily 7
retain weekly 4
retain monthly 12

# Exclude pattern (refer to --exclude-from from rsync man page)
exclude  /dev
exclude  /proc
exclude  /sys
exclude  /run
exclude  /var/tmp
exclude  /var/run
exclude  /tmp
exclude  /lost+found
exclude  /mnt

# backup of server
backup backup@DESKTOP:/  DESKTOP/
NOTE: If /home or other things are on seperate partitions, you'll have to add them to the bottom of the config, e.g.:

backup  backup@DESKTOP:/home             DESKTOP/home

Step 6: Configure sudo on DESKTOP

We'll want to configure sudo to allow the backup user to do what it needs to do with rsync.  To edit the sudo file I use the visudo command:
aj@DESKTOP ~ $ sudo visudo
And add this somewhere (probably near root ALL=(ALL) ALL):
## backup user
backup    ALL=(root) NOPASSWD: /usr/bin/rsync
Step 7: Cron Jobs
Before we go any further, let's see if you have access to the ionice command.  In your shell, just type:
ionice --help
If you get command not found, I recommend consulting your distribution's documentation to install it.  For more information, check out the article Why aren't you using ionice yet???
Now, onto the cron jobs!

Crontab vs cron.[time]

There are two ways to add a crontob; through crontab (usually by running the command crontab -e) or by adding the commands in /etc/cron.daily / .weekly / .monthly.  On my server, I use crontab, as I want to specify the exact times that I want my backups to run; however, there's nothing wrong with using the scripts in /etc/...  I'll cover both.


Create scripts in /etc/cron.{daily,weekly,monthly}:
aj@NAS ~ $ sudo touch /etc/cron.{daily,weekly,monthly}/
Put in /etc/cron.daily/
ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf daily || echo "Daily backup for DESKTOP failed."
Put in /etc/cron.weekly/
ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf weekly || echo "Weekly backup for DESKTOP failed."
Put in /etc/cron.monthly/
ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf monthly || echo "Monthly backup for DESKTOP failed."
Note: If you don't have ionice, remove ionice -c 3 nice -n +19 from the beginning of those commands.


Run the command:
aj@DESKTOP ~ $ sudo crontab -e
And add somewhere in the crontab file:

# Backup DESKTOP
#12am every day
0 0 * * *    ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf daily || echo "Daily backup for DESKTOP failed."
#1am every week
0 1 * * 1    ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf weekly || echo "Weekly backup for DESKTOP failed."
#4am every month
0 4 2 * *    ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf monthly || echo "Monthly backup for DESKTOP failed."
Note: If you don't have ionice, remove ionice -c 3 nice -n +19 from the beginning of those commands.

Wrapping up

If I forgot something or screwed up a command horribly, please let me know!


Wednesday, December 20, 2017

Changed domains!

This blog is no longer using a subdomain!  It is, however, still hosted by  However, the blog is now under the domain.

If you're in the Ianozi family and want an subdomain, visit for more details.

Friday, July 7, 2017

Dynamic DNS with Cloudflare, ddclient, and DNS-O-Matic (with Multiple Domains)

It's been a very long time since I've updated this blog.  For a while I've kept a separate one,, but the home server I've been using to host it failed about a year ago.  I will be getting AJfox back up and running at some point, or I'll port the posts to this site, I haven't decided yet.

Anyway, this topic is about how to configure dynamic DNS with Cloudflare, using DNS-O-Matic and ddclient.

Reason for this post

There were plenty of guides on getting Dynamic DNS up and running with cloudflare, and one or two with DNS-O-Matic, but absolutely none for multiple domains.  If you've found this guide by a google search, I'm assuming you already know at least what Dynamic DNS and Cloudflare are, if not you can check out the basics here: Replacing DynDNS with CloudFlare DDNS

I was trying to set up an easy way to set the DNS for multiple domains (e.g. and, which doesn't seem to be covered anywhere online.

What is DNS-O-Matic?

To answer this question, let me quote someone who explains it better than I:
DNS-O-Matic is a service from OpenDNS for distributing dynamic DNS updates to supported authoritative DNS services. Cloudflare is one of the supported services. Source: Using DNS-O-Matic dynamic DNS updates with Cloudflare 

And what is ddclient?

ddclient is a daemon that periodically updates your DNS information to a Dynamic DNS service.  I've seen plenty of guides where people have been trying to get it working with Cloudflare, but it apparently requires patching the standard version to get it working correctly, I wasn't able to.

Instead, I set up ddclient to sync with DNS-O-Matic, and then set DNS-O-Matic to sync with Cloudflare.

With that out of the way, let's get down to business!

Step 1: Register accounts with Cloudflare and DNS-O-Matic

If you're reading this, you probably already have a cloudflare account.  If not, check out Cloudflare's Getting Started page to see how to sign up.  DNS-O-Matic is very easy, just head over to their website and sign up.

Step 2: Configure ddclient

That depends on what linux distribution you're using.  For me, with Gentoo, I just had to run:
emerge --ask ddclient

On Debian-based systems it would be:
apt-get install ddclient

Step 3: Configure Cloudflare

This is probably the weirdest part.  The problem with DNS-O-Matic is that it will only support one A-record across all domains.  As I've mentioned, every guide I've seen recommends that you create a hostname called "dynamic" (ergo; however, DNS-O-Matic literally will not allow you to create the same hostname even for different domains (e.g. 

Here's what I did:

Generate a random string for every domain you have
Any supported random string generator will work; for simplicity's sake I used the Random Letter Sequence Generator and set the length of each sequence to 16.  One feature that I liked in this generator was the ability to generate several strings at once.  For two of my domains, I generated two strings:
  • sjnuuqvibcmvycpa
  • bghpdqbwupqqdpmp
Create an A record with the random string that points to any IP address
For each domain, you'll have to create an A record with that random string.  This record can point to any IP address (I used

Here's what it looks like in Cloudflare for
Type Name Value
A sjnuuqvibcmvycpa points to

Create two CNAMEs for root and www

You'll want to create a CNAME for your root domain pointing to the A record and a WWW CNAME pointing to the root domain like so:
Type Name Value
CNAME is an alias of
CNAME www is an alias of

Once you're done, you should have something like this:

Afterwards, do the same thing for the other domain, using the other random string.

Step 4: Configure DNS-O-Matic:

This part is pretty straightforward.

Get Cloudflare API Key
First, you'll need your Global Cloudflare API.  Log into Cloudflare and click on your name in the top-right corner, then go to Settings.  From there, scroll down to API Key and under Global API Key click on View API Key.  This is your secret API key that DNS-O-Matic needs for logging into Cloudflare.

Add Cloudflare Service in DNS-O-Matic
You'll do this for every domain that you own:
  1. Log into DNS-O-Matic and click on Your Services, then Add a service.
  2. Select Cloudflare from the drop down menu
  3. Enter your cloudflare email address under Email
  4. Enter your API Token under API Token
  5. Enter your domain's random string (from above) under Hostname
  6. Enter your domain under Domain
You can use the same API and cloudflare email, but you must use a different string for Hostname and Domain for each domain.  Here's what I have (note: not using my real cloudflare email):

Step 5: Configure ddclient

The configuration file is located at: /etc/ddclient/ddclient.conf

You'll want to put something like this (be sure to change the USERNAME and PASSWORD to your DNS-O-Matic credentials):
use=web,,    \
protocol=dyndns2,                \
login=USERNAME,                  \
password=PASSWORD                \

That's it!  You can test to see if this configuration worked by manually running ddclient from the command line:
ddclient -daemon=0 -debug -verbose -noquiet

In my case it did:

Good luck!

Tuesday, June 10, 2014

The flooding situation in Rutherford (with video/rant)

My backyard got flooded again this morning.  This is happening more and more often, so I decided to take some pictures.

There's a terrible flooding situation in Rutherford, a small neighborhood in New Castle County, outside Newark, Delaware.  Due to the Christiana Hospital (who insisted to the residents of this neighborhood that it wouldn't affect us), the pipes for Rutherford are backed up and overloaded.

Tuesday, February 21, 2012

Final Update on Dvorak

Hello everyone, this is my final update on the Dvorak Series.  On December 26, 2010, I decided to switch to the Dvorak keyboard layout.  You can go read my posts on the progression.

Well, today, I decided to take a typing test, and here are the results:
Yes, you read that right: 103 wpm.  I'm officially up to my QWERTY speed.
Here is a pretty nice log of what I've done since the switch:
Time ElapsedWords Per Minute (WPM)
1 Day5 WPM
5 Days15 WPM
2 Weeks40 WPM
9 Months85 WPM
13 Months103 WPM

You know, it took about 8 or 9 years to reach 100+ wpm with QWERTY.  It took a little over 1 year with Dvorak.  Will this limit be my ceiling?  I doubt it, but I'm pretty happy with this speed at the moment.

I highly encourage using the Dvorak layout, and I'm happy to say it was well worth learning!

That is all!