custom gentoo nas, part 6: inotify-watch, write detection

I want to minimize writes to the boot/root filesystem for a bunch of bullshit reasons:

  • It’s a flash drive, so repeated writing will eventually wear it out,
  • “security”,
  • keeping myself honest from a reproduceability perspective, in terms of modifying the in-place system vs. the virtual machine image.

Some idealized version of the system would employ a really-read-only boot drive with some sort of overlay/union filesystem to capture changes to reasonable non-volatile storage (such as the RAID array).

In any case, the following has been useful to log writes, in order to determinate what needs to be moved to spinning rust:

$# inotifywait --daemon -e modify,attrib,move,create,delete -r / --exclude='^/(dev|run|proc|sys|data|tmp|home)' --outfile=/data/system/var/log/file-modifications

Since this creates watches on the individual inodes, you will probably need to do something like this, first:

$# echo $(( 8 * (2 ** 20) )) > /proc/sys/fs/inotify/max_user_watches

While I’d already proactively moved /var/log/ and /var/lib subdirectories for most services, the following were missed as being very write heavy:

/var/lib/samba/private/msg.sock
/var/cache/samba
/var/cache/man
/var/lib/mlocate

After moving/symlinking these to /data/system/ on the RAID, there’s just a handful of regular writes that I will need to do further investigation to understand how appropriately to mitigate:

/var/lib/private/systemd/timesync/clock
/var/lib/systemd/timers/stamp-cron-hourly.timer
/var/log/{lastlog,wtmp}
/root/.bash_history

custom gentoo nas, part 4: single-user, hostname changing, nas.target

When I boot the virtual machine with the NAS system software, I usually interrupt the boot to start in single-user mode. This gives me an opportunity to:

$ echo terra > /proc/sys/kernel/hostname

so that the virtual machine comes online with a distinct network identity.

For a while, after then returning to the normal boot, once the machine was online I would quickly ssh in to terminate a handful of services (tor, plex, sonarr, &c.) as quickly as possible to prevent them from generally … interferring with the world.

After a while, I got tired of this, so I looked into creating a specific runlevel to separate multi-user from “running as a NAS”.

In the traditional runlevel system, runlevels 1 (single-user), 3 (multi-user), 5 (graphical) and 6 (reboot) are spoken for, while 2 and 4 are “free” to be used for other purposes.

systemd takes a different (imho better) tack, with a bunch of semantically-defined “targets”, to which other units can be associated.

In my case, I created a nas.target, which you can think of as runlevel 4 if it makes you feel better.

# /etc/systemd/systemd/nas.target
# 2018-04-06, jsled: create a new target!
[Unit]
Description=Network Attached Storage
# Documentation=nfs://earth/data/system/ChangeLog
Requires=multi-user.target
Conflicts=rescue.service rescue.target
After=multi-user.target rescue.service rescue.target
AllowIsolate=yes

$ rm /etc/systemd/system/default.target
$ ln -s /etc/systemd/system/nas.target /etc/systemd/system/default.target

At this point, the system will boot to nas.target by default.

After this, one needs to modify /lib/systemd/system/{whatever}.service to replace

[Install]
WantedBy=nas.target

And then re-systemctl enable whatever.service to get it linked appropriately.

(Unfortunately, it seems that adding /etc/systemd/system/whatever.service.d/override.conf will add the target to the WantedBy set, not replace the value. :/)

At this point, booting the virtual machine becomes:

# boot to single user: add "single" or "systemd.unit=rescue.target" to the boot line.
# wait for single-user...
$# echo terra > /proc/sys/kernel/hostname
$# systemctl isolate multi-user

Then the boot will continue to multi-user, without further loading all the services associated with nas.target.

And, yes, I should figure out how to configure grub / grub-mkconfig to add a boot-menu entry for single-user rather than editing the damn boot every time.

(And, it seems like there should be a kernel param for hostname, but I can’t seem to find it.)

In my case, the following services are associated with nas.target:

root@earth [~]# ls -l /etc/systemd/system/nas.target.wants/
total 0
lrwxrwxrwx 1 root root 40 Apr  6 18:25 avahi-daemon.service -> /lib/systemd/system/avahi-daemon.service
lrwxrwxrwx 1 root root 36 Apr  6 18:25 collectd.service -> /lib/systemd/system/collectd.service
lrwxrwxrwx 1 root root 35 Apr  6 18:25 grafana.service -> /lib/systemd/system/grafana.service
lrwxrwxrwx 1 root root 38 May  6 09:02 lm_sensors.service -> /lib/systemd/system/lm_sensors.service
lrwxrwxrwx 1 root root 38 Apr  6 18:25 nfs-server.service -> /lib/systemd/system/nfs-server.service
lrwxrwxrwx 1 root root 41 Apr 12 18:01 node_exporter.service -> /etc/systemd/system/node_exporter.service
lrwxrwxrwx 1 root root 38 Apr 15 09:16 nut-driver.service -> /lib/systemd/system/nut-driver.service
lrwxrwxrwx 1 root root 38 Apr 15 09:14 nut-server.service -> /lib/systemd/system/nut-server.service
lrwxrwxrwx 1 root root 45 Apr  6 18:25 plex-media-server.service -> /lib/systemd/system/plex-media-server.service
lrwxrwxrwx 1 root root 38 Apr  6 18:25 prometheus.service -> /lib/systemd/system/prometheus.service
lrwxrwxrwx 1 root root 36 Apr  6 18:26 sabnzbd@default.service -> /lib/systemd/system/sabnzbd@.service
lrwxrwxrwx 1 root root 34 May  6 09:02 smartd.service -> /lib/systemd/system/smartd.service
lrwxrwxrwx 1 root root 32 Apr  6 18:25 smbd.service -> /lib/systemd/system/smbd.service
lrwxrwxrwx 1 root root 34 Apr  6 18:27 sonarr.service -> /lib/systemd/system/sonarr.service
lrwxrwxrwx 1 root root 31 Apr  6 18:25 tor.service -> /lib/systemd/system/tor.service
lrwxrwxrwx 1 root root 47 Apr  6 18:25 transmission-daemon.service -> /lib/systemd/system/transmission-daemon.service

Basically, everything related to physical hardware (lm_sensors, nut), or proactive networking (nfs, samba, avahi, plex, sonarr/sabnzbd, tor).

custom gentoo nas, part 3: syslog-ng, systemd journal in memory

systemd, as is its nature, subsumes into itself a number of facilities that historically have separate processes and utilities.

In particular, system logging is now part of systemd, recorded in binary files, and accessible via the journalctl utility.

And while systemd has some compelling aspects, logging is somewhere I wanted to continue to use syslog-ng and logrotate to have a familiar experience and easily tailable, grepable files.

Both of these packages have support for integration with systemd and journald.

As well, with this OS image, I wanted to minimize writes to the boot/root filesystem, to prevent excess wear on the flash storage.

A key component of this is to reconfigure journald (/etc/systemd/journald.conf): for Storage=volatile. This will ensure that the journal is only retained in memory. While you’re at it, set RuntimeMaxUse to something reasonable. Note that you really are only going to need this to be large enough to spool syslogs for the period of time before syslog-ng starts up and can start consuming the logs from journald to be put into persistent storage.

Then, simply configure syslog-ng with:

    # /etc/syslog-ng/syslog-ng.conf
    source src { systemd-journal(); [...] };

…to consume logs from journald once it starts. In my case, I have it configured to spool logs to:

    # /etc/syslog-ng/syslog-ng.confg
    destination messages { file("/data/system/var/log/messages"); };

wish (x10dev) 2.1.5 ebuild

I got frustrated last week and fixed the wish-2.1.3 ebuild to install the /dev/x10 nodes into /lib/udev/devices/. It was relatively hard to find this out, but when traditional mknod-created nodes are in this directory, they’ll be copied into the otherwise-dynamic /dev tree at boot time. As such, wish finally boots cleanly. In the course of … Continue reading “wish (x10dev) 2.1.5 ebuild”

I got frustrated last week and fixed the wish-2.1.3 ebuild to install the /dev/x10 nodes into /lib/udev/devices/. It was relatively hard to find this out, but when traditional mknod-created nodes are in this directory, they’ll be copied into the otherwise-dynamic /dev tree at boot time. As such, wish finally boots cleanly.

In the course of trying to track down other system lockups, I upgraded to the 2.6.18 kernel, and wish-2.1.3 stopped building for me. A devfs header file that it used was finally deprecated out of the module-building header directory. Luckily, in the mean time, wish-2.1.5 was released without this dependence on devfs. I re-generated the patches, and updated the bug to be an ebuild for wish-2.1.5, a.k.a. x10dev.

Org-mode (vs. planner-mode)

Yesterday I switched from using planner-mode to using Org-Mode, both emacs note-tasking and todo-task management apps. I’ve been using planner-mode for the last year and a half, and rely on it pretty extensively to track upcoming todo items, in both personal and work contexts. I’ve read in the last couple of months, however, that Org-Mode … Continue reading “Org-mode (vs. planner-mode)”

Yesterday I switched from using planner-mode to using Org-Mode, both emacs note-tasking and todo-task management apps. I’ve been using planner-mode for the last year and a half, and rely on it pretty extensively to track upcoming todo items, in both personal and work contexts.

I’ve read in the last couple of months, however, that Org-Mode is already in emacs-cvs, and will be part of the standard distribution of emacs-22. I decided to try it out, was quickly impressed, and transitioned over.

Both support the same basic data model: files of mixed notes and todo-items, where the items are simply specially-formatted lines in the file. Items have an optional due date and an optional priority (ABC). For example, in planner-mode, they appear:

A _ Write up planner-/org-mode post (2006.11.24)

B X Make faux-chicken dinner for Thanksgiving (2006.11.23)

For Org-mode:

** TODO [#A] Write up planner-/org-mode post SCHEDULED: <2006-11-24> ** DONE Make faux-chicken dinner for Thanksgiving SCHEDULED: <2006-11-23> CLOSED: [2006-11-23 Thu]

Both support the same basic execution flow: when requested, the package looks through the special files to find lines that fit the pattern, and constructs another file (planner) or a transient view (Org) over past-due and presently-due items. There are operations for creating items, scheduling them on a particular date, transitioning them closed, jumping from the item to associated notes, &c. As well, both support some form of unidirectional publishing of the content to HTML, ostensibly to serve as some public project-planning status.

Org-mode, I’ve discovered, goes far beyond planner-mode in terms of features.

  • Core
  • It is based on outliner-mode, and its documents are inherently foldable, hierarchical documents.
  • It uses file extension (“.org”) for mode-selection, so planning document can live next-to or within projects.

  • Todo

  • There’s a bit more latitude where items can be placed in the notes documents.
  • It supports — separately from the scheduled to-do time for a task — a deadline, which is brought up differently in the “now” view.
  • Overdue items are indicated more clearly.
  • Items can be scheduled for a range of time.
  • Items can be tagged arbitrarily and searched/viewed by tags (“WORK”, “STORE”, “@LAPTOP”, …)
  • Supports progress logging
  • Supports hierarchical sub-tasks (yay!)

  • Linking

  • Explicit linking rather than CamelCase (though CamelCase is an option), which was the major problem I was having with planner-mode (it starts to become really slow for large documents).
  • Patterned links (“bugs:1234”, “http://bugs/show_bug.cgi?id=%s” → “[…]?id=1234”) are supported.

  • Other

  • Has a table/spreadsheet editor. The table-editing part appears similar to table.el, but the spreadsheet functionality is just pure awesome.

Update Mon 2006-11-27: after being contacted by Carten Dominik, the author of org-mode, I’ve revised the comment at the end about the table/spreadsheet editor.

creating many graph instances in cacti

cacti_db.py is a script which will clone an existing data-template and graph-template for multiple (host,rrd_file,template) instances. In the scenario it was created for, data is (externally) collected into a (directory) tree of RRD files. While there are only a handful of unique RRD file and graph types, there are a large number of instances of … Continue reading “creating many graph instances in cacti”

cacti_db.py is a script which will clone an existing data-template and graph-template for multiple (host,rrd_file,template) instances.

In the scenario it was created for, data is (externally) collected into a (directory) tree of RRD files. While there are only a handful of unique RRD file and graph types, there are a large number of instances of those templates. Good examples of this are:

  • cpu load
  • 1,5,15 minute averages
  • 1 per machine
  • disk util
  • avail, used, free
  • ~5 per machine
  • cache stats
  • (hit, miss, size) stats
  • 2 .. 20 caches/”node”, depending type.

This collected data might exist in the following tree of RRD files:

/servers/hostA/cpu.rrd /servers/hostA/disk/root.rrd /servers/hostA/disk/usr.rrd /servers/hostA/disk/data/logs.rrd /node/nodeA/cache/foo.rrd /node/nodeA/cache/bar.rrd […15 more caches…]

Cacti has the following templates manually configured:

/servers//cpu /servers//disk /node/*/_cache

The script, then, knows how to relate RRD files of the regex pattern r'''/servers/([^/]+)/cpu''' to the template(s) “/servers/*/cpu”, using the regexp group 1 as the name of the host; edit the ‘rrd_types‘ global to suit your scenario.

Instances are named as their RRD file name (yes, skipping Cacti’s own |template_formatting|).

The script will delete instances with the same name, allowing you to re-run it pretty liberally to pick up new rrd files, changes in templates, &c.

The process of cloning is as follows; see ‘execute_plan_alpha(...)‘ in the script for the gory details.

  • find the host in table ‘host‘.
  • data template instantiation
  • find the data template id (by name) in ‘data_template‘.
  • if an instance with the name already exists: delete it.
  • insert into data_local, get new (autoinc) id for the instance.
  • get the list of rrd DSes from the template (from ‘data_template_rrd‘)
  • copy each, maintaining the instance -> template id map.
  • instantiate the data_template itself (‘data_template_data‘)
  • copy the RRAs as well (‘data_template_data_rra‘)
  • graph instantiation
  • get the graph_template_id (by name) from ‘graph_templates
  • insert into ‘graph_local‘, get autoinc id.
  • instantiate graph (‘graph_template‘).
  • copy ‘graph_instance_item‘s, as this is where the graph-ds/rrd-ds mapping is stored, run the relevant ids through the map retained earlier.

For details on Cacti 0.8’s rather atrocious database schema, see Cacti Forums: Database schema… WTF? and Cacti Forums: Cacti data relationship diagram.

squirrel sql 2.2 gentoo ebuild

Attached to Gentoo Bug#133629: –– shell-script –– Copyright 2006 Joshua Sled Distributed under the terms of the GNU General Public License v2 inherit eutils java-pkg MY_PV=${PV}final DESCRIPTION=”SQL client” HOMEPAGE=”http://squirrel-sql.sourceforge.net/” SRCURI=”mirror://sourceforge/${PN}/${PN}-${MYPV}-src.zip” @fixme: log4j, commons-cli, nano-xml DEPEND=”>=virtual/jdk-1.4 dev-java/ant-core” LICENSE=”LGPL-2″ SLOT=”0″ KEYWORDS=”x86″ IUSE=”” S=”${WORKDIR}/sql12″ src_unpack() { unpack ${A} cd ${S} } srccompile() { cd ${S}/build ant jarsource compile_plugins … Continue reading “squirrel sql 2.2 gentoo ebuild”

Attached to Gentoo Bug#133629:

– shell-script –

Copyright 2006 Joshua Sled

Distributed under the terms of the GNU General Public License v2

inherit eutils java-pkg

MY_PV=${PV}final

DESCRIPTION=”SQL client” HOMEPAGE=”http://squirrel-sql.sourceforge.net/” SRCURI=”mirror://sourceforge/${PN}/${PN}-${MYPV}-src.zip”

@fixme: log4j, commons-cli, nano-xml

DEPEND=”>=virtual/jdk-1.4 dev-java/ant-core” LICENSE=”LGPL-2″ SLOT=”0″ KEYWORDS=”x86″ IUSE=””

S=”${WORKDIR}/sql12″

src_unpack() { unpack ${A} cd ${S} }

srccompile() { cd ${S}/build ant jarsource compile_plugins }

src_install() { dodir /opt/${P} dodir /opt/${P}/doc

cp -R ${WORKDIR}/squirrel-sql-dist/squirrel-sql/core/dist/* ${D}/opt/${P}/ cp ${D}/opt/${P}/squirrel-sql.sh ${T} echo ${T} sed -e “s#%INSTALLPATH#/opt/${P}#” < ${T}/squirrel-sql.sh > ${D}/opt/${P}/squirrel-sql.sh makedesktop_entry squirrel-sql.sh “SQuirreL SQL ${PV}” /opt/${P}/icons/acorn.png “Utility” /opt/${P}/ }

squirrel sql 2.0 gentoo ebuild

The following is a ghetto ebuild for the very nifty Squirrel SQL 2.0. It’s especially ghetto in that – ignoring The Gentoo Way – it doesn’t even attempt to re-build from sources. It also probably doesn’t do things in the right way. But it works… use /opt/squirrel-sql/squirrel-sql.sh to run. –– shell-script –– Copyright 2005 Joshua … Continue reading “squirrel sql 2.0 gentoo ebuild”

The following is a ghetto ebuild for the very nifty Squirrel SQL 2.0. It’s especially ghetto in that – ignoring The Gentoo Way – it doesn’t even attempt to re-build from sources. It also probably doesn’t do things in the right way. But it works… use /opt/squirrel-sql/squirrel-sql.sh to run.

– shell-script –

Copyright 2005 Joshua Sled

Distributed under the terms of the GNU General Public License v2

inherit eutils java-pkg

MY_PV=${PV}final

DESCRIPTION=”SQL client” HOMEPAGE=”http://squirrel-sql.sourceforge.net/” SRCURI=”mirror://sourceforge/${PN}/${PN}-${MYPV}-standard.tar.gz” DEPEND=”>=virtual/jdk-1.4″ LICENSE=”LGPL-2″ SLOT=”0″ KEYWORDS=”x86″ IUSE=””

S=”${WORKDIR}/SQuirreL SQL Client”

src_unpack() { unpack ${A} cd “${S}” }

src_compile() { echo “no compilation” }

src_install() { dodir /opt/${PN} cp -r “${S}”/* ${D}opt/${PN} }