[time-nuts] calculating stats with gaps in the data
jim77742 at gmail.com
Thu May 25 18:18:42 EDT 2017
If you do want to Fourier transform your data and you *do* have missing
data points, I can highly recommend the Lomb-Scargle periodogram. I put it
through its paces a while back. I took 3 sin waves of different periods and
amplitude of 1 and added them together. I used 10000 data points. Then I
added Gaussian noise of various standard deviations. And I also removed
various amounts of data up to 90%.
The LSP still found the periods with noise of sd=5 and 90% of the data
gone. With sd=10 it could still find signal with 50% of the points removed.
I have some nice plots of all this if anyone is interested.
The main disadvantage of LSP is speed. It performs as O(n^2). This was a
huge disadvantage back in the 70s when it was published, but with today's
computing power it's not a problem. Unless you have 1000000 data points,
then patience is required.
On 26 May 2017 at 07:12, Michael Wouters <michaeljwouters at gmail.com> wrote:
> There are 'better' ways of handling gaps when calculating ADEV and
> siblings. Patrizia Tavela has a nice method: you pad out the time series,
> tagging missing points with NaNs say, and then if a difference contains a
> missing data point, you drop it. It works very well. I expect this is in
> Stable32. I think it's implemented in allantools. It's definitely
> implemented in the Matlab functions I wrote (tftools on GitHub).
> On Fri, 26 May 2017 at 12:00 am, Tom Van Baak <tvb at leapsecond.com> wrote:
> > Only Stable32 handles data gaps seamlessly. Give it a try (read the
> > for details).
> > But also ask yourself how much gaps matter. Yes, they affect the accuracy
> > of your y-axis sigma scale and your x-axis tau scale. A few seconds every
> > 30 minutes is, what, a 0.1% error? That's like one pixel in a ADEV plot;
> > not significant.
> > What I've done when I need a perfectly seamless data set is just
> > interpolate for rare and obviously missing phase data points. That keeps
> > the timescale intact. This is especially important if you plan to Fourier
> > transform the data: under no circumstances do you want to slip a sample
> > that case.
> > /tvb
> > ----- Original Message -----
> > From: "jimlux" <jimlux at earthlink.net>
> > To: "Discussion of precise time and frequency measurement" <
> > time-nuts at febo.com>
> > Sent: Thursday, May 25, 2017 6:11 AM
> > Subject: [time-nuts] calculating stats with gaps in the data
> > > I'm looking at the python AllanTools package.. does it deal with gaps
> > > the data series (e.g. I've got a series of phase and/or frequency
> > > measurements, 1 per second, but there's gaps of a few seconds every 30
> > > minutes or so)
> > _______________________________________________
> > time-nuts mailing list -- time-nuts at febo.com
> > To unsubscribe, go to
> > https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
> > and follow the instructions there.
> time-nuts mailing list -- time-nuts at febo.com
> To unsubscribe, go to https://www.febo.com/cgi-bin/
> and follow the instructions there.
More information about the time-nuts