Some Software Tools for
Time and Frequency Data Analysis
stable-stats.pl
I've put together a perl program called stable-stats.pl which will read a file containing timetag/phase pairs and generate useful statistics such as drift, offset, and Allan Variance. It will generate a web page containing those statistics, as well as charts of accumulated phase and Allan variance. You can see what it does by going to www.febo.com/time-freq/plots and clicking on some of the files there.stable-stats.pl was developed under, and primarily for use with, Linux. It should work with any other Unix flavor for which the Grace plotting program is available. There's no reason the perl code wouldn't work on a Windows system with perl installed, but I'm not sure if there is a version of Grace for Windows that will work properly.
The program is packaged as stable-stats-0.2.tgz. It requires a perl module called Statistics-OLS. as well as a couple of support files that are included in the tar archive.
While this is labeled as a version 0.2 alpha release, it seems to work well, and at least in my environment doesn't have any known significant bugs. However, please let me know if you have any problems, suggestions, or bug fixes. In particular, please scrutinize the calculation routines as there are plenty of opportunities for off-by-one and other indexing problems, as well as who knows what other kinds of mistakes.
ntp-stats
I've also written a program called ntp-stats which uses the same Grace plotting program as stable-stats to process the NTP "peerstats" log files that provide information about the performance of each server an NTP server is talking to.You can see its output at www.febo.com/time-freq/ntp/stats.
Like stable-stats, ntp-stats is a young program but seems to work well. The main place where it's likely to go wrong is in the regular expression processing used to select hosts to include and exclude from the output. Again, please let me know if you find any problems.
Other Tools
I've hacked together some other very simple perl filters which help me manipulate phase data for plotting and analysis. These were all thrown together quickly by someone who does not do this for a living and who doesn't know the ins and outs of perl. In other words, they're ugly, inefficient, an maximally inelegant. But they seem to work, and they're easy to modify as I continue to experiment. All are designed as filters that read STDIN and write STDOUT.License terms: If you can make them work, have at it. Please leave my comments and notices intact, and if you distribute after modifying them, please note the changes you've made, and send me a copy so I can take advantage of them.
prefilter.pl reads the input (one value per line) and (a) discards any non-numeric lines (like comments), and (b) discards outlier data, replacing bad values with the last good value. The algorithm to do this is still being tested; it needs to take into account the fastest valid slew rate of the input, as well as the fact that the data is in a range from 0 to 10 and wraps around at the edges.
10us-wrap.pl detects data that's crossing the edges of the chart (e.g., from 0.5 to 9.9 is probably a decrease, not an increase) and adjusts the data by adding or subtracting increments of 10. This is probably the hardest of these filters to tune, and I still haven't gotten rid of all the spurious 10us steps in the output.
average.pl is a very simple script that averages ten input values to generate one output value.
stats.pl is a simple statistics script that reads values on STDIN and outputs the time/date, the minimum and maximum values, the mean, and the standard deviation of the sample.
plot-10us-3h is an example of a script I feed to gnuplot (a plotting program from the Free Software Foundation) to generate the charts. Note that this is full of hard-coded filenames that will need to be modified for your use.