[time-nuts] Allan variance by sine-wave fitting

[Magnus Danielson]

Hi,

On 11/27/2017 07:37 PM, Attila Kinali wrote:
> Moin Ralph,
> 
> On Sun, 26 Nov 2017 21:33:03 -0800
> Ralph Devoe <rgdevoe at gmail.com> wrote:
> 
>> The issue I intended to raise, but which I'm not sure I stated clearly
>> enough, is a conjecture: Is least-square fitting as efficient as any of the
>> other direct-digital or SDR techniques?
> 
> You stated that, yes, but it's well hidden in the paper.

Least-square fitting done right is very efficient.

A good comparison would illustrate that, but it is also expected. What 
does differ is how well adapted different approaches is.

>> If the conjecture is true then the SDR
>> technique must be viewed as one several equivalent algorithms for
>> estimating phase. Note that the time deviation for a single ADC channel in
>> the Sherman and Joerdens paper in Fig. 3c is about the same as my value.
>> This suggests that the conjecture is true.
> 
> Yes, you get to similar values, if you extrapolate from the TDEV
> data in S&J Fig3c down to 40µs that you used. BUT: while S&J see
> a decrease of the TDEV consistend with white phase noise until they
> hit the flicker phase noise floor at about a tau of 1ms, your data
> does not show such a decrease (or at least I didn't see it).

There is a number of ways to do this.
There is even a number of ways that least square processing can be applied.

The trouble with least square estimators is that you do not maintain the 
improvement for longer taus, and the paper PDEV estimator does not 
either. That motivated me to develop a decimator method for phase, 
frequency and PDEV that extends in post-processing, which I presented 
last year.

>> Other criticisms seem off the mark:
>>
>> Several people raised the question of the filter factor of the least-square
>> fit.  First, if there is a filtering bias due to the fit, it would be the
>> same for signal and reference channels and should cancel. Second, even if
>> there is a bias, it would have to fluctuate from second to second to cause
>> a frequency error.
> 
> Bob answered that already, and I am pretty sure that Magnus will comment
> on it as well. Both are better suited than me to go into the details of this.

Yes, see my comment.

Least square estimator for phase and frequency applies a linear ramp 
weighing on phase samples or parabolic curve weighing on frequency 
samples. These filter, and the bandwidth of the filter depends on the 
sample count and time between samples. As sample count increases, the 
bandwidth goes way down.

>> Third, the Monte Carlo results show no bias. The output
>> of the Monte Carlo system is the difference between the fit result and the
>> known MC input. Any fitting bias would show up in the difference, but there
>> is none.
> 
> Sorry, but this is simply not the case. If I undestood your simulations
> correctly (you give very little information about them), you used additive
> Gaussian i.i.d noise on top of the signal. Of course, if you add Gaussian
> i.i.d noise with zero mean, you will get zero bias in a linear least squares
> fit. But, as Magnus and I have tried to tell you, noises we see in this area
> are not necessarily Gauss i.i.d. Only white phase noise is Gauss i.i.d.
> Most of the techniques we use in statistics implicitly assume Gauss i.i.d.

Go back to the IEEE Special issue on time and frequency from february 
1966 you find a nice set of articles. In there is among others David 
Allans article on 2-sample variance that later became Allans variance 
and now Allan variance. Another article is the short but classic write 
up of another youngster, David Leeson, which summarize a model for phase 
noise generation which we today refer to the Leeson model. To deeper 
appreciate the Leeson model, check out the phase-noise book of Enrico 
Rubiola, which gives you some insight. If you want to make designs, 
there is more to it, so several other papers needs to be read, but here 
you just need to understand that you get 3 or 4 types of noises out of 
an oscillator, and the trouble with them is that noise does not converge 
like your normal textbook on statistics would make you assume. The RMS 
estimator on your frequency estimation does not converge, in fact it 
goes astray and vary with the amount of samples. This was already a 
known problem, but the solution came with Dave Allans paper. It in fact 
includes a function we later would refer to a bias function that depends 
on the number of samples taken. This motivates the conversion from one M 
sample variance to a 2-sample variance and a N sample variance to a 
2-sample variance such that they can be compared. The bias function 
varies with the number of samples and the dominant noise-form.

The noiseforms are strange and their action on statistics is strange.
You need to understand how they interact with your measurement tool, and 
that well, in the end you need to test all noiseforms.

>> Attila says that I exaggerate the difficulty of programming an FPGA. Not
>> so. At work we give experts 1-6 months for a new FPGA design. We recently
>> ported some code from a Spartan 3 to a Spartan 6. Months of debugging
>> followed.
> 
> This argument means that either your design was very complex, or it
> used features of the Spartan3 that are not present in Spartan6 anymore.
> The argument does not say anything about the difficulty of writing
> a down-mixer and sub-sampling code (which takes less than a month,
> including all validation, if you have no prior experience in signal
> processing). Yes, it's still more complicated than calling a python
> function. But if you'd have to write that python function yourself
> (to make the comparison fair), then it would take you considerably
> longer to make sure the fitting function worked correctly.
> Using python for the curve fitting is like you would get the VHDL code
> for the whole signal processing part from someone. That's easy to handle.
> Done in an afternoon. At most.
> 
> And just to further underline my point here: I have both written
> VHDL code for FPGAs to down mix and subsample and done sine fitting
> using the very same python function you have used. I know the complexity
> of both. As I know their pitfalls.

Another sample point: One of my designs was converted to Virtex 7, we 
lost a week not because of the design was broken, but library 
dependencies where a bit old so synthesis got things incorrect. Once we 
realized that, it was trivial to fix and it works. VHDL for FPGA has 
good uses, and some will never really be competed by software. I do 
both, as they have different benefits. There is god and bad ways of 
designing things for FPGA or as software, it takes skills to design 
things in a portable way. It also takes skills and time to port a design 
to use most of a new chip or CPU.

>> FPGA's will always be faster and more computationally efficient
>> than Python, but Python is fast enough. The motivation for this experiment
>> was to use a high-level language (Python) and preexisting firmware and
>> software (Digilent) so that the device could be set up and reconfigured
>> easily, leaving more time to think about the important issues.
> 
> Sure. This is fine. But please do not bash other techniques, just because
> you are not good at handling them. Especially if you hide the complexity
> of your approach completely in a sidermark. (Yes, that ticked me off)

Indeed. This is not a good way to convince anyone. If you don't work 
well with certain tools, either don't use them or learn to understand 
how to use them. I tend to do the later so I learn.

The trick is that you want to use the benefit of both to achieve that 
extreme performance, and when you do it right, you use their strengths 
together without the actual design being very complex. That's the beauty 
of good designs.

>> Attila has about a dozen criticisms of the theory section, mostly that it
>> is not rigorous enough and there are many assumptions. But it is not
>> intended to be rigorous.
> 
> If it is not intended as such, then you should make it clear in
> the paper. Or put it in an appendix. Currently the theory is almost 4 of
> the 8 pages of your paper. So it looks like an important part.
> And you still should make sure it is correct. Which currently it isn't.
> 
>> This is primarily an experimental paper and the
>> purpose of the theory is to give a simple physical picture of the
>> surprizingly good results. It does that, and the experimental results
>> support the conjecture above.
> 
> Then you should have gone with a simple SNR based formula like S&J
> or referenced one of the many papers out there that do this kind of
> calculations and just repeated the formula with a comment that the
> derivation is in paper X.

Break it up and focus on what is important in separate papers.

>> The limitations of the theory are discussed in detail on p. 6 where it is
>> called "... a convenient approximation.." Despite this the theory agrees
>> with the Monte Carlo over most of parameter space, and where it does not is
>> discussed in the text.
> 
> Please! This is bad science! You build a theory on flawed foundations,
> use this theory as a foundation in your simulations. And when the
> simulations agree with your theory you claim the theory is correct?
> Please, do not do this!
> 
> Yes, it is ok to approximate. Yes it is ok to make assumptions.
> But please be aware what the limits of those approximations and
> assumptions are. I have tried to point out the flaws in your
> argumentation and how they affect the validity of your paper.
> If you just want to do an experimental paper, then the right
> thing to do would be to cut out all the theory and concentrate
> on the experiments.

Agree fully.

Please remember that while Attila, Bob and I may be critical, we do try 
to make you aware of relevant aspects you need to consider.

I have tried to hint that it would be useful to see how different 
estimator methods perform. The type of dominant noise for a certain tau 
is relevant, and is how we have been forced to analyze things for the 
last 50 years because of the problems we try to indicate.

> I hope it comes accross that I do not critisize your experiments
> or the results you got out of them. I critisize the analysis you
> have done and that it contains assumptions, which you are not aware
> of, that invalidate some of your results. The experiments are fine.
> The precision you get is fine. But your analysis is flawed.

We are doing the friendly pre-review. A real reviewer could easily just 
say "Not worthy to publish", as I have seen.

There is nothing wrong about attempting new approaches, or even just 
test and idea and see how it pans out. You should then compare it to a 
number of other approaches, and as you test things, you should analyze 
the same data with different methods. Prototyping that in Python is 
fine, but in order to analyze it, you need to be careful about the details.

I would consider one just doing the measurements and then try different 
post-processings and see how those vary.
Another paper then takes up on that and attempts analysis that matches 
the numbers from actual measurements.

So, we might provide tough love, but there is a bit of experience behind 
it, so it should be listened to carefully.

Cheers,
Magnus