[time-nuts] trade timestamps

Wojciech Owczarek wojciech at owczarek.co.uk
Tue Nov 10 23:17:42 EST 2015

Long time lurker here. Hefty read follows - apologies. Can't blame anyone
for TL;DR.

On 11 November 2015 at 02:07, Alexander List <alex at list.priv.at> wrote:
> On Wednesday, November 11, 2015 03:15 AM, Don Latham wrote:
> Indeed. HFT firms have been working on their own "advanced" timekeeping
> using PTP and GPS and what not, but it seems the MIFID-II directive and
> equivalents in other markets force the entire industry to up their
> systems from NTP to PTP...

It's all nice and simple when you have a GPSDO in your lab and nice, steady
10 MHz, and that's that.

So far MIFID-II sets a precedent. The tightest US requirement is currently
50 ms to UTC (time error, not timestamp granularity), but the US is bound
to follow EU at some point. PTP has been pretty much a standard for bigger
trading firms for 5+ years. I first saw it in 2007. Until now, it has only
been used by traders to gain advantage in establishing data arrival times,
and exchanges used it to maintain timestamp consistency between different
components of a trading system. Now it becomes a legal requirement
for both. They were all screaming and shouting, but the EU settled on
100 us from UTC (only for firms with low latency systems). Initially they
suggested 1 us! NTP is on the way out, but it's not the protocol's fault,
it's the implementations' fault. Long poll intervals, small amount of samples
and no hardware timestamping. PTP resolves all of those in its very
core definitions - and implementations follow. Your clock may be stable,
but if you're polling every 16 seconds and packet delay variation of the
network stack outweighs your path delays, and you have no accurate
transmission timestamp, no Allan Intercept will help. Suboptimal
measurement, and you have crap in, crap out.

Hardware can do small nanoseconds, the problem is getting the time
to the operating system where the application lives. People have to
realise that they are not driving actual crystals, just software models
describing OS clocks, that only use assumed known frequency
as a starting point. It is advanced timekeeping, considering scale,
network conditions, monitoring, holdover, mitigating GPS vulnerabilities,
complex GPS distribution into data centre bunkers, calibration, etc.
OK, it's not exactly China Mobile with 700,000 base stations
using PTP, but it's pretty advanced in my humble opinion.

With software-only PTP, with good filtering you get low microseconds,
albeit somewhat noisy. With a PTP timestamping NIC you can
get sub-150 ns between OS and NIC, and the NIC can definitely
get sub-500 ns to reference, assuming little or no delay asymmetries,
or use of PTP transparent clocks across the whole path. 100 us
is easy if you architect it right, the Finance community just has
a lot to learn. For many engineers from the "IP generation",
phase and frequency concepts are completely foreign, they mix
definitions and generally get confused. NTP you could just leave
running, syncing to something, and it gave you a number. With
PTP, suddenly every small detail affecting phase offset has to be
understood and accounted for, because your target is smaller.

It also works the other way round: explaining the problems of
time sync in enterprise networks to frequency / telecom people
(vendors) is a tough task. I have been doing this for a few years
now, and today they mostly get it. Now they have to explain "their"
stuff back to the users and everything will be fine (ha-ha).

Before the legal requirement is cut down to 10 us, hardware
timestamping will be available on every NIC (mostly is today
for new systems), and sync will be much less challenging.
Solving the "Last two inch problem" ( (c) John Eidson), that
is building computer systems that are time-aware and time-
correct by design, will be the last task on the road to ubiquitous
sync. Intel have already laid the foundations allowing to relate
the TSC counter to PTP time, and there is the PTM functionality
in PCI Express 3.x+ spec (basically PTP over PCIE). Finally,
there is Synchronous Ethernet, providing tight frequency lock
between network ports - add PTP for phase and time, and you've
got White Rabbit (picosecond accuracy).

The funniest thing about MIFID-II, or rather ESMA, the regulatory
body, is that they question the traceability of GPS time to UTC!
Perhaps because the EU has Galileo. Apparently.

[disclaimer: personal opinions only]



Wojciech Owczarek

More information about the time-nuts mailing list