[time-nuts] Using a frequency synthesizer replacement for motherboard oscillator

Erich Heine sophacles at gmail.com
Mon Dec 3 15:43:07 UTC 2012

On Sun, Dec 2, 2012 at 3:16 PM, Magnus Danielson <magnus at rubidium.dyndns.org
> wrote:

> Erich,
> On 12/02/2012 08:54 PM, Erich Heine wrote:
>> Jonathan,
>> My research group has had some good experiences using products from
>> Endace (
>> http://www.endace.com/) for network timing measurement at the ethernet
>> level. I don't have a pointer immediately to the work, but if there is
>> interest can ask tomorrow at work. The gist of it though was to understand
>> precise timing characteristics of network switches for better simulation.
>> Examining the time "in switch" for various packets at the microsecond
>> level
>> was needed to understand various delay curves for different network loads,
>> with an ultimate goal of proper statistical modeling reflecting reality as
>> close as possible.
> This is a bold challenge, it's a difficult task (clear speak: there is a
> reason for this to be a research field, industry never *really* got it
> under control).
It is a challenge. I'll have to re evaluate my understanding of the timing
characteristics of the card in light of my new and growing knowledge on
timing. However, this is what I understand now:

I'm sure the timing isn't perfect, but we mitigated some of the potential
clock issues by doing many runs of tests. Further a single endace DAG card
has 2 or 4 network ports on it, so timing can be measured across a network
to the same card with the same reference clock which helps simplify error
source.  (The card has a feature which allows time stamping packets on the
card from that reference clock). Also there is a mechanism by which the
endace cards can be connected directly to each other to synchronize their
clocks to each other, so packet timings across cards can well measured.

I wasn't very involved in the details of the project I'm describing, so I
can't speak to how well any one of those functionalities of the cards
performed, nor how much error we saw or tolerated, I'll ask those folks
later today (as I happen to have a meeting with them about other stuff
anyway :))

One thing I can speak to though, is that in simulation of the type they are
doing - the general granularity of the simulation results are on the order
of 10^-3, so understanding the switches at 10^-6 isn't used directly in the
simulation, but rather to build probability distribution functions that
capture the behavior of the switches well enough to see cumulative results
comparable to observed networking conditions (e.g. the result of N switches
handling serializing events under certain loads will get packet trains like
$X and other loads like $Y and so on). One thing the PI on that project
mentioned was the shape of those functions being correct is the highest
priority. I understand this to mean that the errors "average out" (not an
actual mean, but through some statistics I don't understand all the way yet
they stop being significant compared to other issues).

Perhaps I'll need to point those researchers at this list to find out ways
to better get the timings they want :)

>  I personally have also used endace products to measure packet timings for
>> research, but I didn't need so much precision for that work - however I
>> can
>> say they have a good API and decent tech support for interacting with
>> their
>> cards and products.
> Is there native support with Linux kernels?
> It would be very nice to have the support using SO_TIMESTAMP and friends.
It's been a couple of years since I got deep on this, so I don't remember
details. I do know all the work we've done with the cards was on Linux, and
it worked nicely - I had no issues that were OS related. One thing about
the paradigm for the DAG cards is they are not in the standard networking
stack. They expose themselves from the kernel in a different way via an API
(which Endace provides the source for). To do IP networking with them, you
need to provided your own network stack in userland - however this is out
of the scope of the companies goal. They are really about providing high
speed layer2 access. I believe their primary use cases are research like we
did, and extremely low-latency communications between machines in clusters
(targeting high frequency traders and the like).

> Cheers,
> Magnus
> ______________________________**_________________
> time-nuts mailing list -- time-nuts at febo.com
> To unsubscribe, go to https://www.febo.com/cgi-bin/**
> mailman/listinfo/time-nuts<https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts>
> and follow the instructions there.

More information about the time-nuts mailing list