[time-nuts] Best OS for small time server

Poul-Henning Kamp phk at phk.freebsd.dk
Sat Feb 23 17:48:35 EST 2008


In message <91981b3e0802231443k333aeea4vaffec57e4514bf7b at mail.gmail.com>, "Chri
s Kuethe" writes:

I have a set of utilities on the way into FreeBSD soon for solving the
logfile problem on flash media,

Rudimentary manual page looks like this:

FIFOLOG(1)		FreeBSD General Commands Manual 	    FIFOLOG(1)

NAME
     fifolog_create -- Initialize storage for fifolog
     fifolog_write -- Write data to fifolog
     fifolog_read -- Seek and extract data from fifolog

SYNOPSIS
     fifolog_create [-l record-size] [-r record-count] [-s size] file
     fifolog_reader [-t] [-b tstart] [-B Tstart] [-e tend] [-E Tend]
		    [-o ofile] [-R regexp] [-T timefmt] file
     fifolog_writer [-w write-rate] [-s sync-rate] [-z compression] file

DESCRIPTION
     Fifologs are designed to provide a round-robin circular storage for
     recording text and binary information to permanent storage in a bounded
     and predictable fashion, time and space wise.

     A fifolog can be stored either directly on a disk partition or in a regu-
     lar filesystem file.

     The input data stream is encoded and compressed and marked up with time-
     stamps before it is written to storage, such that it is possible to seek
     out a particular time interval in the stored data, without having to
     decompress all of the logfile.

     Writes happen whenever the output buffer is filled with compressed data
     or when either of two timers expire forcing the partially filled buffer
     to be written.

     The first and faster timer just forces available data to be written but
     does not flush and reset the compression dictionary.  This timer is
     intended to minimize the amount of logdata lost in RAM in case of a crash
     and by default it fires every 10 seconds after the previous write.

     The second and slower timer forces a full reset of the compression engine
     and forces the next record written to be a synchronization point with its
     own timestamp, making it possible to initiate reading from that record.
     By default this timer fires every minute.

     The fifolog_create program is used to initialize the first sector of a
     disk device or filesystem file to make it a fifolog and should be called
     only once.

     Running fifolog_create on an existing fifolog will reset it so that
     fifolog_reader and fifolog_writer will not see the previous contents.
     (The previos contents is not physically erased, and with a bit of hand-
     work, all but the first record can be easily recovered).

     If the file does not already exist fifolog_create will attempt to create
     and ftruncate(3) it to the specified size, defaulting to 86400 records if
     no -r argument is specified.

     fifolog_writer will read standard input and write it to the end of the
     fifolog according to the parameters given.

     fifolog_reader will retrieve records from the fifolog according to the
     specified parameters and write them either to stdout or the file speci-
     fied with -o.

     It is possible to specify a start and end time to limit the amount of
     data fifolog_reader will report.  The lower-case variants -b and -e take
     a time_t value, whereas the upper-case variants -B and -E take human red-
     able specifications such as "1 hour ago".

     The -t argument forces timestamps to be formatted as "YYYYMMDDhhmmss"
     instead of as time_t, and -T allows the specification of a strftime(3)
     formatting string.

     Finally, records can be filtered such that only records matching the
     (REG_BASIC) regular expression specified with -R is output.

IMPLEMENTATION NOTES
     The data stored in the fifolog consists of three layers, an outher layer
     that allows searches to synchronization points based on timestamps with-
     out having to decompress and decode the actual contents, a compression
     layer implemented with zlib(3) and an inner serialization and timestamp-
     ing layer.

     The exact encoding is described in the fifolog.h file.

     Fifolog is particularly well suited for use on Flash based media, where
     it results in much less write-wear, than a filesystem with regular log-
     files and newsyslog(8) etc.

EXAMPLES
     Create a fifolog with 1024*1024 records of 512 bytes:

     fifolog_create -s 10m /tmp/fifolog

     Write a single record to this file:

     date | fifolog_writer /tmp/fifolog

     Read it back with human readable timestamps:

     fifolog_reader -t /tmp/fifolog

     One particular useful use of fifolog_writer is with syslogd(8) using a
     line such as this in /etc/syslog.conf(5):

     *.* |fifolog_writer /var/log/syslog_fifolog

HISTORY
     The fifolog tools have been liberated from an open source SCADA applica-
     tions called "measured", which monitors and controls remote radio naviga-
     tion transmitters for the Danish Air Traffic Control system.

AUTHORS
     The fifolog tools were written by Poul-Henning Kamp

FreeBSD 8.0			  Feb 9, 2008			   FreeBSD 8.0


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



More information about the time-nuts mailing list