[time-nuts] calculating stats with gaps in the data

Jim Palfreyman jim77742 at gmail.com
Thu May 25 18:18:42 EDT 2017


If you do want to Fourier transform your data and you *do* have missing
data points, I can highly recommend the Lomb-Scargle periodogram. I put it
through its paces a while back. I took 3 sin waves of different periods and
amplitude of 1 and added them together. I used 10000 data points. Then I
added Gaussian noise of various standard deviations. And I also removed
various amounts of data up to 90%.

The LSP still found the periods with noise of sd=5 and 90% of the data
gone. With sd=10 it could still find signal with 50% of the points removed.

I have some nice plots of all this if anyone is interested.

The main disadvantage of LSP is speed. It performs as O(n^2). This was a
huge disadvantage back in the 70s when it was published, but with today's
computing power it's not a problem. Unless you have 1000000 data points,
then patience is required.

Jim Palfreyman


On 26 May 2017 at 07:12, Michael Wouters <michaeljwouters at gmail.com> wrote:

> There are 'better' ways of handling gaps when calculating ADEV and
> siblings. Patrizia Tavela has a nice method: you pad out the time series,
> tagging missing points with NaNs say, and then if a difference contains a
> missing data point, you drop it. It works very well. I expect this is in
> Stable32. I think it's implemented in allantools. It's definitely
> implemented in the Matlab functions I wrote (tftools on GitHub).
>
> Cheers
> Michael
>
> On Fri, 26 May 2017 at 12:00 am, Tom Van Baak <tvb at leapsecond.com> wrote:
>
> > Only Stable32 handles data gaps seamlessly. Give it a try (read the
> manual
> > for details).
> >
> > But also ask yourself how much gaps matter. Yes, they affect the accuracy
> > of your y-axis sigma scale and your x-axis tau scale. A few seconds every
> > 30 minutes is, what, a 0.1% error? That's like one pixel in a ADEV plot;
> > not significant.
> >
> > What I've done when I need a perfectly seamless data set is just
> > interpolate for rare and obviously missing phase data points. That keeps
> > the timescale intact. This is especially important if you plan to Fourier
> > transform the data: under no circumstances do you want to slip a sample
> in
> > that case.
> >
> > /tvb
> >
> > ----- Original Message -----
> > From: "jimlux" <jimlux at earthlink.net>
> > To: "Discussion of precise time and frequency measurement" <
> > time-nuts at febo.com>
> > Sent: Thursday, May 25, 2017 6:11 AM
> > Subject: [time-nuts] calculating stats with gaps in the data
> >
> >
> > > I'm looking at the python AllanTools package.. does it deal with gaps
> in
> > > the data series (e.g. I've got a series of phase and/or frequency
> > > measurements, 1 per second, but there's gaps of a few seconds every 30
> > > minutes or so)
> >
> > _______________________________________________
> > time-nuts mailing list -- time-nuts at febo.com
> > To unsubscribe, go to
> > https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
> > and follow the instructions there.
> >
> _______________________________________________
> time-nuts mailing list -- time-nuts at febo.com
> To unsubscribe, go to https://www.febo.com/cgi-bin/
> mailman/listinfo/time-nuts
> and follow the instructions there.
>


More information about the time-nuts mailing list