=pod =head1 NAME arlogd - a remote logger =head1 SYNOPSIS arlogd-server [-h] [-s] [-t] [--version] [-c config] [start|stop|state] arlogd-client [-h] [-s] [-t] [--version] [-c config] [start|stop|state] =head1 DESCRIPTION Arlogd provides I ; a daemon on the I transfers data to a daemon on the I (loghost). =over 4 =item * Programs on a client host use remote logging by writing to a I (fifo). =item * On a client, a daemon periodicly searches directory I<$fifos> (default C) for readable fifo's. Note : a fifo is created with C =item * Anything written into such a fifo, appears immediately in a corresponding file on the loghost : client: /var/log/fifos/foo/bar loghost: /var/log/arlog/foo/bar =back =over 4 =item start Start the (loghost or client) daemon with : arlogd-server start arlogd-client start The exit status is 0 if a daemon is started (or was already running) ; 1 otherwise. =item stop Stop a daemon with : arlogd-server stop arlogd-client stop The exit status is 0 if a daemon was stopped (or wasn't running) ; 1 otherwise. =item state To query the running daemon's state, use argument C : arlogd-server state arlogd-client state The daemon reports : =over 4 =item * the current list of open logfiles (server) or watched fifos (client), =item * the current incoming and outgoing connections, =item * the service ports it is listening on. =back The exit status is 0. =item I Without an argument, C and C just shows the daemon's status. arlogd-server arlogd-client The exit status is 0 if the daemon is running ; 1 otherwise. =back The default location for a config-file is C or C. If/when I<$loghost> is unavailable, the client will : =over 4 =item * save log-entries in a file in I<$save_dir> (default C) ; =item * periodicly try to reconnect to I<$loghost> ; =item * on reconnect, send I<$save_dir>CI<$loghost>C<.log> to I<$loghost>. =back You can send commands to a running daemon ; see L and L below. See below : =over 4 =item * IMPORTANT : notes on using B in L =item * more on L =item * L =back =head2 how to use arlogd To use remote logging for some I, just replace the I by a symlink to a fifo : # remote logging for /var/log/httpd/vhost/access.log % mkdir -p /var/log/fifos/httpd/vhost/ % mkfifo /var/log/fifos/httpd/vhost/access.log % cd /var/log/apache % mv access.log TEMP % ln -s /var/log/fifos/httpd/vhost/access.log . % cat TEMP > access.log Things to check or do : =over 4 =item * Restart utilities that keep their logfile open. Some (like apache) do, others (like rsync) don't. C may tell you which processes are still using C. =item * Remove C. =item * Don't forget to B log processing (like log-rotation by logrotate(8) or other processes) for the logfile you just made I. =back =head1 OPTIONS =over 4 =item -h Show help and default config ; then exit. =item -s Be silent ; use logs instead of stdout ; used in C scripts. =item -t Just check the config. =item --version Just print the version and exit. =item -c I Use config I. =back =head1 CONFIGURATION A configuration file must be specified with B<-c config-file> or be present as ./arlogd.conf /etc/arlogd/conf In the config-file, lines starting with C<#> are skipped ; as are empty lines ; lines with leading and/or trailing white space are stripped. Each line contains a name/value pair, separated by white space. The list below shows the default value for each I. I<$name> indicates the configured value of entry I. =head2 Config entries you should configure =over 4 =item * C Clients send their data to host I<$loghost>. Specify a host.domain name, or an IP address ; eg loghost loghost.my.org =item * C I =item * C I The hosts that are allowed to connect to the I<$loghost>. By default, C is allowed to connect. The default I is I (C<''>). Configure a C-line for each allowed host. allow_host localhost allow_host webfarm.my.org allow_host monitor.my.org If a I is specified, the I<$loghost> prepends a prefix received from I with I. For instance, if a client supplies a log-line like access_log 81.189.215.252 - - [09/Dec/2014:12:49:41 ... then, by default, the line is appended to C<$log_dir/access_log>. However, if you configure : allow_host webfarm.my.org webfarm/ allow_host monitor.my.org monitor_ log_lines are written to C<$log_dir/webfarm/access_log> and C<$log_dir/monitor_access_log>. =back =head2 Config entries you may want to set =over 4 =item * C By default, the daemon on the loghost writes the logfiles with I on : the daemon flushes files after every write. =over 4 =item * Autoflush is expensive, but it will keep the contents of the remote logfiles as up-to-date as possible. =item * It is probably wise to turn off I later, when things are running smoothly. =back To turn autoflush off, configure : autoflush 0 See also commands L and L below. =item * C =over 4 =item * The specified file should contain a list of (I,I)-pairs like : mv file /path/to/dest Path I<$file> is used I to I<$log_dir> ; that is, I<$log_dir>CI<$file> should exist. =item * The I<$rename_list> is used by command L (see below). It attempts to rename $log_dir/$src $dst for each (I,I)-pair. =item * Renaming can be useful if you want to integrate I logfile processing (rotation, analysis etc) into your I process. Use renaming to move your I logfiles into your I log-tree, just before normal processing begins. If the list is long, you should probably generated it ; the list is re-read every time L runs. =back =back =head2 Config entries that are probably sane =over 4 =item * C On the loghost, the daemon can manipulate logfiles on demand. Pattern I<$log_names> is the default I for the daemon. In I, C<*> means I> ; it is always used to match as a suffix. =item * C The loghost data port ; I<$port+1> is used on I<$loghost> for commands . =item * C The server- and client-daemons log into file I<$logfile>. =item * C The server- and client-daemons keep pid-files etc in directory I<$run_dir>. =item * C The server on I<$loghost> writes the logs in directory I<$log_dir>. =item * C A client-daemon watches named pipes in directory I<$fifos>. =item * C If/when the loghost is unavailable, a client saves log-entries in I<$save_dir>CI<$loghost>C<.log>. =item * C If/when the loghost is unavailable, a client will try to reconnect to the loghost every I<$reconnect> seconds. On reconnect, the client sends the contents of I<$save_dir>CI<$loghost>C<.log> to I<$loghost>. =item * C A client searches the fifos directory, looking for new (or disappeared) fifo's, every I<$upd_fifos> seconds. =item * C A client writing data to a server, will timeout after I<$send_timeout> seconds ; when a timeout occurs the client starts writing to a local file, and will try to reconnect later. =item * C When a client connects to a server, it will timeout after I<$send_connect> seconds ; when a timeout occurs the client starts writing to a local file, and will try to reconnect later. =item * C The server closes a data-connection when nothing was received in the last I<$recv_timeout> seconds. =back =head1 arlogd-server Program arlogd-server opens two service ports (default 2207 and 2208). On port 2207 the daemon accepts data connections from hosts in L. The daemon expects lines like file-path log-line The I is appended to file I<$log_dir>CI, or I<$log_dir>CICI, if the a I is set for the allowed host. If the I starts with C or contains C<..> as a path-component, the log-line is appended to file I<$log_dir>C. On port 2208 the daemon accepts command connections from localhost. To issue a I, on I<$loghost> (for now) use I : echo command | nc -v localhost 2208 =head2 arlogd-server command-list =over 4 =item * AUTOFLUSH? =item * AUTOFLUSH =item * NO_AUTOFLUSH These commands let you get/set the value of L in the running daemon. Command C gets the value ; it prints : COMMAND AUTOFLUSH? AUTOFLUSH current_value_autoflush Commands C and C set the value of L, and print : COMMAND [NO_]AUTOFLUSH AUTOFLUSH old_value -> new_value =item * CLOSE The daemon closes all files in I<$log_dir>. Command C prints something like : COMMAND CLOSE FILES 3 FILE: /path/to/file1 FILE: /path/to/file2 FILE: /path/to/file3 If/when the daemon is using (normal) buffered IO (I<$autoflush> is I), and you want to see the actual contents of a remote logfile, then you can use I to make the daemon close (and thus flush) its open files. =item * FIND [pattern] =over 4 =item * The daemon finds files in I<$log_dir> matching I as a suffix ; symlinks are ignored. =item * The default pattern is I<$log_names> (default C<*.log>). =item * Command C prints something like : COMMAND FIND FILES 3 FILE: /path/to/file1 FILE: /path/to/file2 FILE: /path/to/file3 and possibly some lines like : ERROR some error =back Use I to test I before you use it in C. =item * PING On I, the daemon prints : PONG =item * RENAME The daemon renames files as indicated in file I<$rename_list> (default C) ; see config entry L. =over 4 =item * File I<$rename_list> must contain lines like mv file /path/to/dst where I is a relative path in I<$log_dir> : C operates on I<$log_dir>CI. Lines starting with C<#> are skipped, as are empty lines ; lines with leading and/or trailing white space are stripped. =back Because (a priori) C will move anything anywhere, we need some constraints : =over 4 =item * The owner of the file I<$rename_list> must be the same as the owner of the (running) daemon executing the C command. The daemon trusts that its owner knows what he/she is doing. =item * The I<$rename_list> must not be I- or I-writable. The daemon trusts I its owner. =item * C will I rename I if I is a directory or a symlink. =item * C will I rename I to I if I is a directory or a symlink. =item * C will not create I directories for I : I must exist. =back Please note : =over 4 =item * Before each C, I is removed ; so, file I is always lost. =item * If I is missing, C attempts to create an empty file I. A missing I is assumed to be empty. If/when you run C daily, and nothing is logged in I during the previous day, then I will not exist. =back If you want to change the I of the I<$rename_list>, you have to reconfigure and restart the daemon ; this is intentional. Command C prints something like : COMMAND RENAME FILE: $rename_list FILES 3 DONE: /path/to/src1 /path/to/dst1 DONE: /path/to/src2 /path/to/dst2 DONE: /path/to/src3 /path/to/dst3 and possibly some lines like : ERROR can't open list (why) ERROR can't rename $file (why) ERROR bad line [$line] or, if I is missing : WARN: can't find $file DONE: EMPTY /path/to/dst ERROR can't create empty /path/to/dst (why) =item * ROTATE [pattern] The daemon rotates files in I<$log_dir> matching I as a suffix. =over 4 =item * While searching for targets, symlinks are ignored. =item * The default pattern is I<$log_names> (default C<*.log>). =item * To I I means : close C and rename C as C. arlog/www.my.org/access.log -> arlog/www.my.org/access.log.rot =back A separate utility must further rotate, compress, analyse stuff. Command C prints something like : COMMAND ROTATE FILES 3 FILE: /path/to/file1 FILE: /path/to/file2 FILE: /path/to/file3 and possibly some lines like : ERROR can't rename /path/to/file =item * STATE The daemon shows it's I : the current list of open logfiles, the incoming and outgoing connections, and the service ports it is listening on. =item * STOP I The daemon checks the I, closes all files in I<$log_dir> and stops. The secret is generated by the daemon on start-up. The daemon writes the secret to I<$run_dir>C with mode C<0600>, so command C can only be used by the user who owns the daemon. Command C prints : STOPPED or BAD SECRET C is used by C to cleanly stop the running daemon. =back =head1 arlogd-client =over 4 A client-daemon watches named pipes in directory I<$fifos>. =item * A fifo must not have white space in its name, or path-components. Such fifo's are flagged as I. They are watched, but anything read from them will be discarded. =item * Program arlogd-client opens a service port (default 2209), where it accepts command connections from localhost. To issue a I, on I<$loghost> (for now) use I : echo command | nc -v localhost 2209 =back =head2 arlogd-client command-list =over 4 =item * PING On I, the daemon prints : PONG =item * STATE The daemon shows it's I : the fifos it is watching, the incoming and outgoing connections, and the service ports it is listening on. =item * STOP I The daemon checks the I, closes all fifos in I<$fifos> and stops. The secret is generated by the daemon on start-up. The daemon writes the secret to I<$run_dir>C with mode C<0600>, so command C can only be used by the user who owns the daemon. Command C prints : STOPPED or BAD SECRET C is used by C to cleanly stop the running daemon. =back =head1 INSTALL =head2 requirements Installation of B requires very little : =over 4 =item * Perl, perl core-modules C, C, C. =item * C in C, C, C or C. =item * On a loghost, you need I nc(1) if you want to use the daemons I facility. =back =head2 installation Installation is simple : =over 4 =item fetch the software subversion check-out : svn co https://svn.science.uu.nl/repos/project.arlogd/pub/trunk arlogd rsync : rsync rsync.cs.uu.nl::arlogd package : http://www.staff.science.uu.nl/~penni101/arlogd/arlogd.tar.gz =item install On a server (loghost) : make -f install-server.mk On a client : make -f install-client.mk B is added as a service, but the service is not I or started. To turn on the service use chkconfig arlogd-server on chkconfig arlogd-client on =item configure To configure, edit C. On the server (loghost), configure entry L (default C). allow_host localhost allow_host webfarm.my.org allow_host monitor.my.org On a client, configure entry L (default C). loghost loghost.my.org =item run On a server (loghost) : /etc/init.d/arlogd-server start On a client : /etc/init.d/arlogd-client start =back =head2 try it =over 4 =item watch the logs On the loghost and the client watch the log : tail -F /var/log/arlogd.log =item make a fifo On the client, make a fifo : mkfifo /var/log/fifos/test Within a minute, C will start watching this fifo ; see the log. =item write something Write something to the fifo : echo something > /var/log/fifos/test On the loghost, I should immediately appear in file /var/log/arlog/test =item a fifo in a subdirectory You can also try to create a fifo in a subdirectory ; and write to it immediately : mkdir -p /var/log/fifos/foo/bar mkfifo /var/log/fifos/foo/bar/test echo something else > /var/log/fifos/foo/bar/test Expect C to block ; be patient ; C is waiting for a reader. As soon as C finds fifo C : =over 4 =item * ... C unblocks ; =item * and on the loghost, I appears in /var/log/arlog/foo/bar/test =back =item loghost : talk to the daemon On the loghost, you can talk to the daemon if you have I C. echo PING | nc localhost 2208 This should reply C. =back =head2 production =over 4 =item * Configure C and C Make sure you have configured L (on the loghost) and L (on the clients). The defaults (C for both) are probably I sane. =item * Set environment variable C<$GREP_OPTIONS --devices=skip> By default, grep(1) reads I, and will I when it encounters a fifo ; for instance, when you do grep -r some-pattern /var/log/ Grep(1) will silently ignore devices if you set environment variable GREP_OPTIONS : $GREP_OPTIONS = --devices=skip When you I want to grep devices, use grep --devices=read ... =item * Make sure the daemons are I running. The daemons should run from (multi-user) I until I. =over 4 =item * Make sure that the arlogd service (server or client) is I. chkconfig --list arlogd-server chkconfig --list arlogd-client If not, turn I the service with (either) : chkconfig arlogd-server on chkconfig arlogd-client on =item * On a client, make sure that C is started I, because a utility that logs to a fifo will block until C is started. Inspect C and assert that C is listed before CI for all services that use remote logging. =item * Less important, check that (on shutdown) C is stopped I, because you may lose some log-lines when the daemon (with autoflush I) is stopped too early. Inspect C and assert that C is listed after CI for all services that use remote logging. =item * Note: to see the C start/stop priorities of C, look at C and C. The relevant line looks like : # chkconfig: 2345 $start_prio $stop_prio Note: configuration management of the daemon-startup-sequence varies wildly between Linux/Unix platforms and installations ; YMMV ... =back =item * Where to look for fifo's Every minute, the client deamon has to walk the I<$fifos>-tree, looking for fifo's. That's cheap if the tree is flat (has only a few flat sub-directories) ; the flatter, the better. F<> But if your client isn't very busy, or C isn't very big, you may want to configure fifos /var/log and just replace logfiles with fifo's. Use find(1) to see how long it takes to walk C, looking for fifo's : time find /var/log -type p -print =item * Where to put the fifo's Maybe having fifo's in C (or in C) isn't such a good idea. Think about C ; it will I ; although C will not (it skips devices). So, if it C a problem, the default for I<$fifos> should be something ugly like C. Or, if it C a problem, the default for I<$fifos> could be C ; that is preferable because it simplifies usage for Joe average ("just replace a logfile by a fifo"). Scanning C is cheap, unless it is I big. Try : C