Network Time Protocol server using PC gnu/linux and freebsd

Low cost GPS receivers make it tempting to convert one for NTP use. Here is documented the development of a server using an ISA serial port interface. The NTP distribution and Linux / PPSKit were used in the original implementation during June 2002. We've also used PPSKit 2.0.2 (with linux 2.4.19) in October 2002, and linux 2.6.3 with the PPSKit "lite" patch. Note that the freebsd 4.x kernel includes the Dave Mills nanokernel without patching.

Using the Gentoo Linux, I was able to install glibc 2.11.1 with linux kernel 2.6.34 and the ntp-dev 4.2.7p32@1.2144-o code for a native working PPS and nanokernel Linux system.

A sound card and external radio can also be used to provide time to ntpd via WWV broadcasts. We have had success with linux 2.2, 2.4, 2.6 and freebsd 4, 5, 7, and 8 in doing so.

Table of Contents

PC Motherboard Time Components

There are several different design components making up a typical PC motherboard's time related functions. It is possible that there are variations in the number of components and their functions due to a particular set of design constraints chosen in a particular implementation. For example, a laptop design probably has a working ACPI implementation, and a 24x7 server of 2004 or so likely also has a power consumption related feature set. Both of these will impact ntp operation if they involve a change of processor "megahertz" (cpu/system clock frequency).

Here follows a description of some of the time related functions in a PC.

The NTP application is at the mercy of the various clocks and their implementation in the particular motherboard in use. One example of various issues is where a system management function is not able to be controlled by standard BIOS settings, requiring a user to run a "unsupported" custom BIOS in order to obtain acceptable NTP operation. Looking at the AMD 8111 specs, the device requires six different clocks! It would be possible to design an ASIC (Application Specific Integrated Circuit) that could supply these clocks, and include a facility to provide an external high stability clock source as input.

The TSC was a nice hack for timestamping in the days of fixed CPU and system clocks. With all the modern cpu frequency smashing going on, it can no longer be relied upon. A while ago I'd had the idea to move the timestamping into the interrupt controller itself, so that it is no longer dependent upon software and a possibly variable CPU clock frequency, or discontinuities due to low power or "suspend" operations. Since I am not sure that Intel will redesign the i8259 for me (grin), an ISA wirewrap card could listen to the IRQ line on the bus, and with an independent clock setup, could grab a timestamp with, say, a 50MHz clock rate, for a theoretical 20 nanosecond resolution. The interrupt would trigger one latch that grabs a first timestamp. Then, the os' interrupt routine would trigger another timestamp, and then immediately take a TSC system timestamp. The delta of the interrupt timestamp and the os' triggered timestamp would allow removal of the os' interrupt latency from the TSC (system) timestamp. The PC's I've seen here typically have on the order of one to two microseconds jitter. The 20 nanosecond timestamp would allow a factor of 100 greater resolution.

There are some interesting PCI bus based FPGA hacks available these days. One of these could implement my timestamping scheme as outlined above.

Operating System Time Components

A typical linux system provides hwclock and adjtimex utilities to manipulate kernel time parameters. The man page for adjtimex states "Linux uses David L. Mills' clock adjustment algorithm (see RFC 1305). The system call adjtimex reads and optionally sets adjustment parameters for this algorithm." The linux kernel you have might set the RTC clock from system time every eleven minutes. The 2.6 kernel is making Brave Changes to this area of code. I have had trouble with 2.6.18-rc2's changes to the time code while running ntp, as the kernel and ntp seem unable to agree, and the offsets flap and spike, and the ticks swish and sway.

A freebsd 5.4 system describes the various software interfaces of its time facilities in the "clocks" man page. The RTC clock is used for profiling and statistics. The PIT is used by the scheduler to generate timeslices. It is useful to be able to set the hardware clock to what the system time is, especially if ntp has fixed up your time. So a teeny utility to set the hwclock is a very useful tool to have with your freebsd system. I let ntp stabilise, then simply run this utility and it writes the RTC with what ntp thinks the time should be.

I tried to bring up freebsd 7.1 on an old pentium133, and it seems to hang at boot after printing "Calibrating clocks..." (You only see this message if you use the "verbose boot" option.) Actually, it will eventually continue after a very lengthy wait. There is supposedly a flag to config for "CLK_CALIBRATION_LOOP" but there is a silly bug in the sys/i386/isa/clock.c file--in the startrtclock() routine, just before the "if CLK_CALIBRATION_LOOP" that is supposed to protect a call to calibrate_clocks(), there is an un-protected call "freq = calibrate_clocks()." Cough. I hacked it so that "freq = 1193182;" and moved calibrate_clocks() inside the #ifdef, this gives a "defined but not used" warning but lets me boot today instead of next week.

Linux, FreeBSD, and Solaris NTPD Configuration

An end user with no attached reference clock should utilise the NTP pool project when configuring their ntp.conf file. Here is an example suitable for Linux, FreeBSD and Solaris end users that are running the ntp version 4 software. Please adjust the drift file location per your local policy:

driftfile /var/ntp/ntp.drift

server version 4
server version 4
server version 4
server version 4

#server version 4
#server version 4

It is unfortunate, but some configuration parameters have changed their meaning both from earlier versions, and within the current version (version 4). There are thus far too many conflicting README's and HowTo's for the ntp software. The recommended practice is to only use the documentation packaged with the particular ntpd distribution when installing and configuring. Pairing the ntp version with the correct copy of the documentation may not always be done carefully by the folks that package a linux "distribution" or other OS software so let the buyer beware.

The ntp service uses port 123 and udp. NTP does not use tcp. The redhat and freebsd /etc/services file has incorrectly shown tcp for many years now.

NTP does NOT use port 37. Port 37 is the RFC868 "rdate" service.

Freebsd 7 os issue

The freebsd 7 release was installed here recently (May 2008). The performance was dismal. A kernel rebuild did not improve things at all. It turns out that the ACPI timers were getting a higher "rating" than the TSC counter. The following line in /etc/rc.local forces use of TSC:

sysctl kern.timecounter.hardware="TSC"

Vendor NTP binaries

Sometimes vendor supplied ntp software will work fine. Often, best results are obtained by installing the most recent version available. In the case of a physically attached reference clock, taking the time to install and configure the latest version should be automatic. Some vendors, it is true, make the job more difficult by for example removing any C compiler from their unix software installation.

Firewall Rules, /etc/services

Many problems will likely be related to incorrect firewall rules, or trying to configure an older ntp with the newer configuration rules. If you are trying to use the ntp pool servers, your firewall must allow udp/123 access. The unix file /etc/services should also contain a line eg. ntp udp/123 for proper operation. An indication of firewall problems is shown when the ntpq billboard shows a "0" in the reachability column for a remote ntp server.. A "0" (zero) reachability indicates that no packets have been received back from the remote ntp server. In normal operation, there should be a "377" (8 bit, octal) shown for reachability.

PC RTXO Hardware

Your pc hardware and software must be within a certain functionality range before the ntp software can work properly. The ntp.drift file should have a range +/- 100 (ppm) for decent performance. This indicates that your crystal is of minimum desired functionality. Note that the ntp.drift file is written by ntp. You cannot simply set this file to a value by hand unless you know the proper value for the temperature that the crystal has at that moment. This number reflects physical characteristics of the crystal used for timing in your pc motherboard. A quality crystal would be in the +/- 30 ppm range. Replacing the crystal with a I2C tunable +/- 2 ppm oscillator would be the highest performance pc motherboard for use with ntp. Beyond that is the realm of GPS receivers and atomic reference standards such as rubidium, cesium or hydrogen radiation.

Hardware rustlers might contemplate a homebew OCXO which has come up in some corners of the web such as this lm723 based OCXO hack.

Operating System Configuration

Your host operating system should be configured in a minimal fashion for best performance. The various components of a gnome or kde environment can interfere with services that have not been engineered for careful integration. My choice is to run a simple X based, twm system where I have an ntp installation in service. This allows good performance with ntp, although a web browser refresh cycle can impact operation of the audio driver based time recovery, for example, as the heavy CPU and memory utilization a browser makes will impact ntpd performance.

GPS Receiver and ISA Interface

The Garmin gps35 offers two serial channels and a PPS output. It was decided to purchase the version capable of +5v operation and "true" rs232 serial drivers. Doing so allows minimal changes to a 486 era ISA serial interface card modified to supply power to the Garmin and receive the data and time signals.

The ISA bus interface will be the limiting agent (2 microseconds) for a typical PC based NTP server. In fact, the algorithms used by the ntpv4 code are able to get down into the 100's of nanoseconds on the computers here in the lab.

Another unit with two serial channels and PPS is the Trimble SVeeSix-CM3, in jargon, SV6. One such unit was found surplus and featured TTL interface levels and was provided with a Micro Pulse GPS antenna. (They sold out a batch of Motorola Oncore GT's without antenna quickly.) The max232 device was used to provide rs232 signal levels that duplicate the connector wiring used on the gps35. This allows either unit to plug into the same ISA card, or use the differential cable.

The particular ISA card chosen allowed very simple PPS signal interfacing. A 74HC04 was piggy backed onto another DIP and wired up to the PPS signal. A trace cut and jumper inserted the signal into the DCD path to the UART. Only the stock length of cable attached to the gps35 was used, and soldered directly to a db25 connector. Two other wires carry the secondary serial data that provides the RTCM SC-104 standard differential GPS compatibility.

An isa all in one card has also been modified. This card was able to be rejumpered for the second two serial ports, and the second parallel port. This configuration ran well with the linux nanokernel in ppsapi mode. The parallel port shm driver also ran well on both the motherboard's built-in parallel port and this cards' parallel port. Note that a PPSAPI serial port should not share an interrupt with any other source, including another serial port. With freebsd I have been able to use three different serial ports with the PPSAPI timestamping function.

Differential GPS and NTP NMEA Mode

The gps35 can run in differential GPS mode, which was wired up and tested with the dgpsip 1.32 software. The $GPGGA sentence was configured in the gps35 for the dgpsip operation, and the ntp configuration file thus includes the mode flag to recognize this NMEA sentence and operate properly. The refclock_nmea.c source wistfully mentions differential gps and suggests that its use would result in better time.

NTP Source Code and Reference Clock Configuration

We've been using bitkeeper to maintain a code tomb here. Otherwise most sites should use the latest tarball. The Reference Clock documentation for the NMEA driver shows gps25 configuration commands e.g. enabling the PPS signal. The gps35 commands are similar, but you should check your documentation for exact details. The Trimble SV6 is operated using the generic or PARSE driver which includes support for several different tickers. These can be build by specifying the --enable-parse-clocks flag to the configure script.

Careful reading of source files and many experimental test runs resulted in simplified ntp.conf configuration lines. The order enable pps, and radio clock lines are used currently with good results.

enable auth stats monitor pps
server mode 2 version 4 prefer  # for nmea (gps35)

server mode 138 version 4 prefer   # for tsip/pps 
fudge flag2 0 # timestamp assert or 1 for clear

The "enable pps" activates the RFC 2783 linux PPSkit.

The NMEA reference clock "mode 2" allows $GPGGA code compatible with the dgpsip operation.

In ntp versions previous to 4.2, the PPS reference clock had the poll interval fixed at 16 seconds with the "minpoll 4 maxpoll 4" configuration. The maxpoll 4 should be removed for later ntp versions, or ntp will not scale out the polling interval for its other configured clocks, thus causing unnecessary ntp traffic. Even worse, those earlier versions of ntp would not scale out polling of external references if there were a local clock configured.

In a previous configuration that included a line for the pps driver (type 22), I had persistent jitter in the 6-8 microsecond range. This was due to "interrupt collision" between the "atomized" (i.e., driver uses pps internally) NMEA driver (type 20) and the pps driver (type 22). So, the revised config omits the pps driver and just uses the NMEA config. This dropped the jitter down to the 2 microsecond range, which is the typical limit for an isa bus computer. The NMEA type 20 fudge flag3 can be set to 1 which lights up the kernel pps. Dave suggests that the userland ntpd has more stages of grooming and filtering than the 3 stage kernel pps and recommends not enabling the kernel pps function.

It is possible to run two NMEA units in a single host computer. Doing this with linux, a sample run shows very good performance.[1606] ntpq -p ku
     remote           refid      st t when poll reach   delay   offset  jitter
*GPS_NMEA(0)     .GPS.            0 l    4   64  377    0.000    0.011   0.002
-GPS_NMEA(1)     .GPS.            0 l   35   64   37    0.000    0.007   0.002
+wraith.wraith.s .GPS.            1 u   31   64  377    0.246   -0.042   0.015
-smidge.wraith.s     2 u   27   64  377    0.333   -0.121   0.008
+ring.wraith.sf.     2 u   60   64  377    0.218   -0.048   0.024
+thrall.wraith.s     2 u   44   64  377    0.237   -0.054   0.028

Preliminary Integration Issue

The original development ix86 unit is a Tyan S1830 "Tsunami" with 266mhz CPU clock and 66mhz SDRAM memory clock, using the 440BX chipset. The superio chip has enabled the two built in serial ports as ttyS0 and ttyS1 under linux, with the ISA card being ttyS2 (for the NMEA and PPS interface), and ttyS3 (for the differential GPS data). The linux kernel is built with serial interrupt sharing and the PPSAPI configured into the kernel, i.e., not a module, with the low_latency flag set in the serial ports. The linux kernel can be set for this mode with the following utility:

# setserial /dev/gps0 low_latency
Currently the motherboard BIOS is set up with "plug and play aware os NO" and the two serial IRQ lines "ISA" which seems to be a solid combination for this particular chipset and BIOS. The performance was gruesome until functional BIOS settings were in place.

At first, the 440BX seemed ok, with the motherboard serial ports disabled and the ISA card set up for the first two ports. Big Trouble started when the card was addressed to the 3rd and 4th serial ports. The PPS signal was known to be present at the DCD input of the UART, verified with a logic probe. One problem was traced to a bad switch. The offending DIP switch was crushed and the pieces unsoldered and replaced with a header in a socket. The current board being used has a National Semiconductor NS16550AFN UART as the serial data receiver. A UMC UM8250B has also been used.

You should dedicate a separate irq for each PPSAPI device that you plan to use. Trying to share interrupts will not work properly with the PPSAPI. I have been able to run freebsd 4.7 with three serial ports, using PPSAPI on two of them. Each serial port had a separate serial interrupt. A linux installation can use the setserial program to route com channel interrupts thusly:

# setserial /dev/ttyS2 irq 9

The IDE interface has been a source of trouble with regards to lost interrupts and should be characterised for your environment if you are using it. The "HZ" kernel configuration for linux is generally recommended to be configured for 100 Hz. The 1000 Hz setting has been reported to impact NTP.

So far, it appears that a BIOS setting change may be needed in a particular motherboard before predictable performance is achieved using the ISA serial card. Performance with the motherboard chipset serial port may be similar. It is known that some motherboards have significant amounts of time constant (capacitance) added to the DCD signal, such that a 1 microsecond PPS signal is unable to trigger an interrupt in the chipset. The resourceful user then usually decides to "stretch" the PPS signal with a one-shot such as a 74123. This has worked here but you should recognise that accuracy can suffer.

We've tested PPSKit 2.0.2 for linux 2.4.19 running on an AMD k6-2 at 400mhz with a 100mhz bus in a FIC 503+ VIA Apollo MVP3 chipset motherboard, using the "fudge flag3 1" mode. The installation efforts are well rewarded by good performance.

The Tyan "Tomcat IV" (S1564S) Pentium/mmx motherboard does not appear to work very well with ntp. The "Tomcat III" (S1563D) did not work well here either. None of the three boards I have here seem to be stable enough to be a good choice for an ntp server. The Tyan "Tsunami" (S1830), however, seems to be a very stable board in my operation here. I've used 486 motherboards with freebsd and gotten better ntp performance that with the tomcats. Perhaps there is an issue with the clock frequency synthesizer chip in the tomcat. I have tried changing XO's and not been able to improve the tomcats problem when running ntp.

Identified Errors

The "time_pps_kcbind" error in the syslog was observed during preliminary configuration. This ended up being due to too many option flags being unfurled, in particular having the "enable kernel pps" and the "fudge flag3 1" both fluttering. First power on did produce functioning signals. Very quickly, however, a 100ms offset was apparent. The gps35 does have a 100ms pulse by default for the PPS signal. This 100ms offset was due to one too many inverters in the original hack, easily fixed by moving one wire. If you think that your ntpd needs the "use clear" flag and has some kind of fixed offset on the pps, likely you just need an extra inverter (or to remove one) somewhere in your signal chain.

There are useful items on debugging to be found in the nanokernel reference.

A plot of loopstat data shows a periodic unsynchronized interval. The gps35 was sitting in a windowsill during this testing period, and clearly lacks sufficient sky view for expected performance. The unit was running on its side with no cover during this period. Earlier testing with the kernel flag enabled did show the proper status bits in the ntptime query (PLL, PPSFREQ, PPSTIME, PPSSIGNAL, NANO). The presence of the nanokernel can be checked with the "ntpdc -c kern" command. One minor issue is that the NANO flag is not displayed by name in the version of ntpdc that is built on a non-nanokernel system. However, the flags value itself is displayed properly. The nano flag is displayed as the bit 2000. Ulrich also has posted about this problem with ntpd/ntpdc.

Received Timecode Display

The received timecode can be viewed with ntpq in a two step process. Use "ntpq -c las clockhost" to get a list of the associations configured on the clockhost. Using the association number of the reference clock of interest (for example, 666), give the "ntpq -c 'cv 666' clockhost" command. Note that under unix, the 'cv 666' should be in quotes for proper handling by your shell.

If there is only a single radio clock attached to the unit, the shorthand form "ntpq -c cv" can be used. If this form is used on a system with multiple clocks, the returned data reflects the current synchronisation source.

Access Policy Listing

A useful convention is to add an ntp "system variable" outlining the units access policy. The ntpd setvar command is used:

setvar access_policy="experimental server not suitable for production" default
An ntpq -c rv command will display the access_policy variable remotely.

SELinux ntpd shm Policy

The Fedora Core 3 iso images at CCRMA have SELinux support. I've written a policy for ntpd shm refclock support. Append the rules to your domains/misc/local.te and make reload. This allows the type 8 refclock ( to work, for instance with the "shm-splc" shared memory serial parallel linux clock program available on this site.

Differential Cable

A TTL PPS signal cannot be used directly for long (greater than stock cable length on gps35) cable runs. I solved this issue here by using a National Semiconductor DM8831 differential line driver and DM8820A balanced line receiver to drive a cable long enough to have the gps35 antenna mounted on the roof. These two circuits will add to the PPS signal offset: 20ns for the 8831 driver, and 30ns for the 8820A receiver. The 74HC04 buffer (see above) adds 20ns to the delay. There is also a propagation delay for the approximately 15 meters of cat5 cable used. For our length of cable, about 75 nanoseconds delay using 1.5 ns/foot.

The DM8820A receiver is set up as in the National Semiconductor data sheet. Here's the DM8831 data sheet. The parts used are obsolete, but were on hand and could work with the five volt power available. Looks like TI second sources both 8820A and 8830 (but not 8831) as 75182 and 75183. The DS9638 driver and DS9637A receiver are more modern parts. As a hack, I've taken a 74251 eight input mux with complementary ouputs and used it to drive the input of the 8820 receiver. I used a 330pf cap in the termination section of the receiver. A 74151 might work as well, although the higher current drive in the high state of the '251 is preferred.

After several years of service, moisture took a toll of the driver. Despite the obvious corrosion, the unit continued to function! I've rebuilt the driver using a salvaged 9638 differential driver removed from an old Micropolis ESDI hard drive controller board, and added a 26ls32 receiver on the fanout box circuit board. This time I've enclosed the driver in a plastic box on the side of the house.

We use differential signaling with the PPS signal to preserve the inherent accuracy of the GPS timing chain. The isa bus has a jitter of 2 microseconds according to Dave's usenet post which is the limit we've reached using our hacked up serial card.

Why not use RS232? Several issues come into play. One is that driving long cable runs with large voltage swings slows down the transition. Another is that many receiver designs use slew rate limiting capacitors in an effort to reduce noise coupling. These capacitors of course introduce delays, that will also drift with temperature and the voltage of the drivers. Basically you are throwing away some amount of timing accuracy when using RS232 for the PPS signal. It is slightly more convenient to use RS232 to drive the PPS signal of course.

Fanout Box

I've built an rs232 fanout box for distributing the NMEA data and PPS signals. This unit has power supplies and connectors that provide power to the GPS unit and takes the rs232/ttl signals and buffers them, along with a 74123 "one-shot" to stretch the PPS pulse, and 1488 level converters to turn TTL PPS to rs232. I've built it into an old AT pc box. This allowed me to use the PC style brackets to mount the DB9/DB25 connectors which saved a lot of mechanical metal cutting. The board is designed to supply buffered data to eight units, and also has differential outputs for the NMEA and PPS signals for connecting further fanout boxes or other uses. The gps35 (or SV6 hack) is placed outside with good sky view, and the differential cable is used to bring the signals into the fanout box, which also supplies the five volt power to the cable/gps, rather than using the pc's five volt power.

It turns out that the pc motherboards I have running here all seem to accept the PPS via DCD just fine, as compared to the TTL level I used on the hacked up ISA cards. The rs232 cables from the fanout box are short, since they only run a few feet to the servers and not all the way down from the roof. I am using MC1488 rs232 drivers with plus and minus twelve volt power. The January 1974 National Semiconductor Data Book specifies the 1488 propagation delay to the logic 1 level to be 230 nanoseconds typically, 300 nanoseconds maximum. The Maxim data sheet for the MAX232 style TTL -> RS232 converter/driver specifies 1.5 microseconds typical, 3.5 microseconds maximum propagation delay.

The fanout box has worked very well here so far. Testing with the Trimble SV6, four units have consistently been within ten to twenty microseconds offset, and frequently get down to single digit numbers. There will be cumulative offset as compared to the NIST's ultimate tick, composed of the various gate, buffer, and other circuit and timing components of the fanout box. Since all the servers are fed by the fanout box, they all see the same offsets.


How well does it work? Using both freebsd and linux servers running NTP 4.1.72, the modified host was selected as sys.peer (in ntp speak) after running for a while, without the "prefer" flag being used on those units. The systems selected the gps35 host from among several other stratum 1 servers configured out on the internet. In a facility with a LAN connecting to a single radio clock on a local server, the LAN hosts should have this server set as the "prefer" host as explained in the NTP documentation. This will act to eliminate clockhopping. The radio clock server should have the radio clock set as the preferred server.

We have Asymmetric DSL (ADSL) (1500k / 384k) connectivity here, with 17ms transit times to the closest configured remote servers. Note, however, that the asymmetric nature of this internet link introduces an offset in remote configured hosts. By observation, this offset is on the order of 2 milliseconds, when the link had a 1500k/128k (11 to 1) ratio, and seems to be around 1 millisecond with the current 1500k/256k (5 to 1) ratio. These same hosts, when using a Symmetric DSL (SDSL) line, cranked down to zero offset.

We've been running a linux PPSkit 2.0.2 server using a gps35 with the atomized NMEA driver in kernel mode (fudge flag3 1) and have had very nice results. There does not seem to be any spiking, or any temperature drift, and the offset seems to be relatively constant. This lack of spikes is due to the daily cron activity having been suppressed on this host. There are other cron jobs in weekly and monthly catagories that could affect performance in a typical linux or freebsd system that should be investigated and considered.

Dave Mills was kind enough to post some matlab code to plot the Allan Deviation from data contained in the loopstats output. I modified the code to work with Octave, and you can grab it here.

It is a challenge to identify the behaviour of a particular interface, motherboard and cpu combination. Linked are gnuplot graphs of two motherboard/cpu combinations performance. The "fuzz" or width of the plotted sample line was thought due in main part due to linux interrupt latency. We currently understand much of the fuzz would have been due to "interrupt collision" between the atomized NMEA driver and the PPS driver, given the original ntp.conf configuration shown above on this web page. The noticable spike around 40000 is due to disk activity triggered by the crontab running the "updatedb" job on the RedHat system. The overall swish and sway is likely your room temperature crystal oscillator (RTXO) reacting to ambient temperature. The somewhat high drift value (70 ppm) could result in other units selecting Internet hosts as peer, rather than this unit, during intervals of rapid ambient temperature change, as the offset wanders around.

The NIST folks have an excellent discussion of some of the issues that affect GPS used in the timing mode.

Another interesting read with regards to rubidium oscillators gives details of design tradeoffs. There is a design for a "rubidium PC" done by S L Moshier. It requires an external reference clock which is used to stabilise the 14.3181818 mhz master timing reference found in most PC designs.

The rubidium stabilised level of performance begins to require serious laboratory equipment of a kind not usually available in a typical IT shop. An ambitious amateur would have a task just to aquire eg surplus/ebay gear allowing some level of measurement to be performed. At this point I have developed this site and the hardware described here with just a soldering iron, logic probe and voltmeter.

The linux 2.6.3 kernel seems to be working well with ntpd and the shm parallel port driver. I'd been testing with the alsa sound and the ntpd audio driver to see if a low-latency linux config would make any difference with the ntpd audio driver. Good results are possible as shown further along this site.

Motherboard Issues

Some motherboard chipsets may not function properly, or have buggy os drivers, when attempting to operate a configuration that is not a pure standard PC. In my case, I have jumpered an add on ISA serial and parallel card so that the parallel port is lpt2/irq 5, and the serial port is com4/irq 9. The card works fine in that configuration under freebsd 5.4 in an old 486 PCI/ISA motherboard. I can configure ntp for the serial port and it works as expected. The same card, in a more recent K6-2 PCI/ISA motherboard, does not work properly. It seems that the irq routing has a problem somewhere. The port shows up but freebsd never gets delivered any interrupts from the card. This could be a chipset driver problem within freebsd, or it could be a BIOS/motherboard bug. I've not figured out which at this point.

This is an example of why a lab stocked with a variety of units is of value when characterising new devices with NTP operation. It also explains to me why I could not make linux work with that sio/irq combination in that motherboard.

BIOS Issues

A common entry in modern BIOS software is a selection for "spread spectrum." A typical "PC" processor must generate a master system timing reference (often called a "clock" even though it does not have anything to do with what the time, or day, is) to allow various parts of the system to work together. When a PC gets qualified to (US) FCC radiation emission standards, having that nice fixed clock in the PC makes for too much spurious radiation. The chip that generates the system clock from the crystal thus is designed with a "spread spectrum" feature that modulates the frequency generated across a range of frequencies (but still within operational requirements of the cpu et al involved). However, the ntp daemon operation wants as fixed and stable a clock as possible. You must therefore insure that any spread spectrum BIOS setting is disabled for proper ntp operation.

Some "green" or power save modes will alter the system clock value. Insure that there is a constant system clock with the current ntp code. This usually means disabling power save mode.

The ntp faq also points out that the "SMM" functions can impact your ntp performance. OEM BIOS code seems to be a culprit here, as aftermarket motherboards do not seem to have the kinds of problems seen with "name brand" computers. It is of course true that the OEM designs their unit with a certain set of engineering and system performance goals that may not have NTP requirements as a target. This web site itself is an example of the scope and difficulty of some of these goals.

Local Oscillator Issues

The particular PC motherboard has an ntp.drift value of about 70 ppm, on the high side compared to some other systems around the lab. Investigation of the motherboard clock chip suggests the possibility of replacing the existing crystal with something else, either another 14.31818Mhz crystal, or a more exotic 28.63636Mhz crystal and divider driving the clock chip input. Actually, with modern clock generation techniques, you start out with stable medium frequencies, and regenerate higher.

There is a specialty TCXO that appears to hold some promise. This unit seems to be a replacement product for an earlier version (DS4000). The SOIC 16 package seems much nicer than the previous devices' LPBGA package. Unfortunately, there is no longer a version with 14.31818 MHz output.

An interesting site at the host "finetune" in Japan has taken a TCVCXO and replaced their motherboard 14.31818 rtxo in a freebsd 4.11 setup. The 31 PPM drift of the motherboard when using the rtxo was thereby reduced by a factor of 100, judging by their plots. However, the offset reported by ntp remained in the 1-2 microsecond range. My understanding is this offset results from the latency between the hardware reporting the PPS edge, and the software being able to sample the TSC (time stamp counter).

NIST appears to have developed a remarkable "chip scale atomic clock." Such a device appears to be perhaps not immediately available to a casual developer such as this site.

One possibility is to cannibalise a cellular phone and extract the timing signal used by the phone for external use. This cell phone stability requirement is quite high and would make a significant improvement to an RTXO circuit. There is a commercial firm that does intercept the cellular transmission in this fashion to provide an NTP server's timing source. An NTP appliance is being sold that does this.

Some simple temperature voodoo should also yield good results. Much serious work has gone into studying temperature variations and other characteristics of these devices. The K2K gps clock system has a local clock interface which is inspiration for a similar crystal clock project being considered here in this facility. Such a circuit could be used to capture a timestamp of the PPS signal to measure and eliminate the linux operating system interrupt latency.

Homebrew Oven Controlled Crystal Oscillator (OCXO)

Having been inspired by Remco's ocxo project I went ahead and built my own. My project was a bit different, in that I built an external oscillator that I use to drive the input to the motherboard's clock generator. You can see the wires running from the homebrew board to the motherboard. I used the DM8831 driver again since I have them. It turns out the poor little DM8820's work fine for a PPS differential circuit, but fail at 14,318,181 Hertz. Maybe I'll try a UA9637 receiver later. But for now, the thing seems to work ok despite the foot of dangling wirewrap wire of my hack job. Whups, the motherboard's clock generator chip (w48s111) is a 3.3v job. Uuh my only saving grace is that ttl typically only pulls up to about 3.5v or so, so even though I'm hammering the chip pretty hard, it is (cough) working. I'll rebuild the oscillator circuit using CMOS devices with a 3.3v power supply rail. Looks like a T.I. sn65lvds050d is the ticket and you can get them at Jameco.

I built the oscillator circuit with an ajustable capacitor so I could trim the frequency. Here's how I trim the frequency of the oscillator. Note that ntpd should not be running if you are duplicating this hack. If you have previously run ntpd, you should reset the kernel's PLL frequency to zero using ntptime, for instance:

# ntptime -f 0.0

My technique is to run the shm_splc code and listen on a serial port to the PPS signal off a gps unit. Then, I watch the reported offset as printed by shm and trim the capacitor so that the offset does not change, or at least changes by a small amount. Once you have the offset stabilised, you can then start ntpd up and check out your pll frequency report. If you're careful and lucky, it should get down to one part per million or so. The adjustment is "precise" (cough) to say the least! as we are talking about parts per million. Results are interesting as at the moment I am using a stock debian kernel (but with ntp-dev code), so there are none of the Rodolpho or Ulrich goodies in there.

Ok we now have one whole day of experience here with the ocxo. I've managed to overcome debian and built one of their kernels, with the PLL fix (/usr/src/linux/include/linux/timex.h, #define SHIFT_PLL 2) and the linuxpps patch, using 1000 Hz tick, and it is working much better than the stock kernel (lenny). The debian lenny kernel config is here. A sample ntpq billboard looks like this. Note the pll frequency is 1.755 ppm after some fussing with the oscillator adjustment. The original crystal that was in the motherboard had a -30 ppm or so drift, if I recall. The current crystal was one I pulled out of my parts drawer, nothing special there. I have no idea what its uncompensated drift was. There is a very good Fairchild application note on building crystal oscillators with 74HC gates which I should have used (wry smile). I'll go back and tear out the original simple oscillator and build the "stable high frequency" version as described in the note.

I've had a bit of trouble with the oven breaking into oscillation. I reduced the capacitance from 47 uF to .1 and this reduced but did not halt the oscillation. Studying the '723 data sheet suggests grounding the frequency compensation capacitor. I've made this modification and so far it has at least been more stable than the original circuit.

Even without the oven, the simple ajustable oscillator has been working amazingly well. I went and disconnected the power from the oven and have been running just the homebrew oscillator circuit. Here's a sample billboard:[544] ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
*SHM(0)          .SHM.            0 l    6   16  377    0.000   -0.001   0.002
-wraith.wraith.s .PPS.            1 u    3   16  377    0.169    0.012   0.011
+thrall.wraith.s .PPS.            1 u    2   16  377    0.162    0.008   0.004
+smidge.wraith.s .GPS.            1 u    1   16  377    0.546    0.009   0.016[545] ntpq -c rv
associd=0 status=0915 leap_none, sync_telephone, 1 event, clock_sync,
version="ntpd 4.2.5p206@1.1972 Sun Aug 30 07:10:18 UTC 2009 (3)",
processor="i686", system="Linux/2.6.26-doobian", leap=00, stratum=1,
precision=-19, rootdelay=0.000, rootdisp=0.344, refid=SHM,
reftime=ce46f25d.86a40bd1  Mon, Aug 31 2009 18:07:41.525,
clock=ce46f264.f9ee85ed  Mon, Aug 31 2009 18:07:48.976, peer=16693,
tc=4, mintc=3, offset=0.001, frequency=-0.009, sys_jitter=0.005,
clk_jitter=0.002, clk_wander=0.000,
access_policy=ocxo hack machine. Not recommended.[546] ntpdc -c kern
pll offset:           9.26e-07 s
pll frequency:        -0.009 ppm
maximum error:        0.005739 s
estimated error:      1e-06 s
status:               2001  pll nano
pll time constant:    4
precision:            1e-09 s
frequency tolerance:  500 ppm
It is quite impressive to see the pll frequency of -0.009 ppm. The pll offset can drop below the ability of ntpq to properly display it, actually. And this performance is with the userland shm_splc driver, and not the kernel pps stuff. Now I do admit that this run was during very stable temperature conditions here (San Francisco bay area's fog), and I carefully hand tuned things. But it is still interesting to see that this is possible.

Trimble Sveesix Notes

I've put in service one of the Trimble Sveesix units once available as surplus. The unit was delivered with a case used by the OEM, which had a battery connection but no PPS connection. So I built it up out of the scrap box using the same cabling arrangement as for the gps35. I modified the unit as received quite minimally, merely disconnecting the battery backup connection and substituting the PPS signal, by cutting and jumpering an internal flexible cable. The rest of the mod was a max232 for serial connection, and 74hc04 buffering of the PPS signal. No "stretching" of the pps signal was done, as the UART is edge sensitive on the DCD signal. Also, I'm using the differential cabling arrangement to preserve the pps timing as much as possible. RS232 Considered Harmful (in the tradition of RFC titles).

Without any kind of initial almanac, the unit must do a "cold start" after power on. This can result in Ominous messages in your syslog (line wrapped for web):

PARSE receiver #0: FAILED TIMECODE: ">RTM0000000000000000000900000000;*6D<"
      (check receiver configuration / cableling)

Actually this is a good indication, because you can clearly see the received string, with crc, indicating your serial connection is good. Give the unit ten or fifteen minutes and you should get reachability.

Trimble TAIP protocol--at one point I had found some programming documentation for the sveesix unit, and here are some notes. The pps actually seems reasonable, the problem so far appears that the sloppy TAIP (never intended for its type of use by ntp here) goes falseticker, this makes ntp ditch the PPS. Eventually we get chime out of the sveesix again, this picks up PPS and repeat. This not so good example of stratum 1 was picked up by the MIT NTP survey.

I had a few strange problems with the TAIP/TSIP/NMEA protocols until I discovered that I needed to get the exact "toolkit" for the firmware version of my unit off the Trimble ftp server. Once I used the correct version of the toolkit, I could switch the SV6 to TSIP fine.

There is a patch for the GENERIC (type 8) refclock that adds the PPSAPI support. This patch is in the latest NTP code these days. I've used the following configuration for this refclock:

server mode 138 version 4
fudge flag2 0
My setup is the surplus/hacked SV6 with the differential cable running into the hacked ISA serial card. Anyways here's what a sample billboard looks like after a couple of hours running on my workstation.

The SV6 seems to read 60 to 80 meters as the altitude. There is also an offset of 8 to 10 microseconds in the ntpq billboard. The GPS35 matches the topo map altitude here better (about 100 meters).

Freebsd WWV Audio and PPS Notes

I've built a signal splitter for the PPS signal off the gps35 going into the linux box. It is simply db25's connected by ribbon cable, with ground and the pps signal wired in on an extra db25. This allowed me to bring up a freebsd 4.3 unit with pps sync. One little item that had me cross-eyed was getting the pps signal recognised by freebsd. Well, I forgot my bsd4.2/sunos skills for a moment--then I recalled:

# cd /dev
# ./MAKEDEV pps0

Note that the MAKEDEV will fail, if your kernel has not been properly configured for pps.

I could only get the sb awe64 to run on irq 5, which conflicts with the 2nd parallel port. Three serial ports with clocks on all three have been tested here with 4.x and work fine (with a different motherboard). As it turns out, my problem with trying to get the sound card to work with irq 9 was a motherboard problem with the ISA bus irq routing. Trying to use an ISA card serial port with irq 9 on this motherboard (FIC VA 503+) also failed. A different motherboard has no problem with an ISA serial port using irq 9 with freebsd 5.4.

Set the 16550 (8250) config to have "flags 0x02" which should disable the FIFO. This should result in a lower jitter for that port. I've built kernels both ways and both seem to work ok.

I've run a freebsd 4.7 kernel with the following option and device specs. Here is the 4.7 config fragment for the pps, parallel and newpcm audio entities. The config spec for the older audio is shown commented out.

I've switched over to freebsd 5.4 on one of my servers. I've used this freebsd 5.4 kernel config file and kernel hints file. These files are set up for three serial ports and two parallel ports, with PPS support on the parallel ports included. Note that you must disable the plip configuration if you want to use PPS on the parallel port. Sound support is also included. I am using this kernel setup with WWVB serial and PPS support on a Spectracom Netclock/2, WWV audio on a soundblaster, and a NMEA GPS device.

Linux and FreeBSD / Nanokernel Comparison

A set of plots was generated during a test run of the gps35 and the SV6 hack. The top plot is the wraith host (gps35/linux), the middle plot is the ku host (SV6/linux) and the bottom plot is the smidge host (gps35 pps/freebsd). The operational setup was the gps35 on the roof, differential cable to splitter, NMEA and pps to wraith host, pps only to smidge host, and the SV6 on deck railing, differential cable to ku, feeding TAIP and pps.

It is interesting that the ku (middle) and smidge (bottom) plots are very similar, even though the ku pps signal is the trimble, and the smidge pps signal is the gps35. The ku host is a DFI k6bv3+, k6-2/500mhz, 100mhz fsb system, and the smidge host is a FIC va503+, k6-2/396mhz, 66mhz fsb system. The wraith host is a Tyan S1830, PII/266, 66mhz fsb system. It appears that the DFI/FIC XO design is quite sensitive to ambient temperature change. Also note that the faster cpu and system bus on the DFI appears to make little difference, and that the slow speed of the PII, in relation to the K6-2's, is of no consequence in relation to the much better stability of the Tyan. The smidge va503+ has -64ppm drift, the ku k6bv3+ has -48ppm drift, and the S1830 has 70ppm drift. Given the comment on the K2K site, and the temperature excursions here in this testing room, an ideal site for the stratum 1 unit would be anyplace where the temperature is stable, such as the K2K 1000 meter basement.

Recently the freebsd unit seems to have had a jump in PLL frequency. Interesting what the plots can reveal, but so far, no further difficulties. I'd started ntp-dev code at that point, but running the old 4.1.73a again hasn't bothered the plot, so I guess it was a real change in the crystal's structure.

After a while, that system started getting spikes in the frequency plots. I took the motherboard out of service and discovered that the CPU had been jumpered improperly, resulting in a core overvoltage. It ran fine this way for five years or so. Rejumpering and placing in test service seems to have eliminated the frequency spiking.

Linux PPS

The current (1 June 2010) state of the art linux kernel, 2.6.34, was installed along with glibc 2.11.1 on a Gentoo linux distribution, and is working with a nanokernel and PPS using the ntp-dev code. There was one ntpd compile issue, where the linux system timex.h needs a #define MOD_NANO STA_NANO fix. Operation after several hours has reached the normal unit microseconds offset level of performance seen on the 2.4/PPSKit and freebsd 8 systems here.

It is useful to run ntpd with elevated priority. This is especially good when running the wwv audio driver, but is generally useful as well. With linux I use the "nice" command along with an ntpd flag as follows. The nice command spawns ntpd at a given (enhanced or reduced, negative is enhanced) priority, and the "-N" ntpd flag asks for what the kernel thinks is realtime:

# nice -n -5 /usr/local/bin/ntpd -N -c /var/ntp/ntp.conf
Using the "top" monitoring program, we can see ntpd running thusly:
Tasks:  73 total,   3 running,  70 sleeping,   0 stopped,   0 zombie
Cpu(s): 86.6%us, 13.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    381740k total,   193476k used,   188264k free,    39648k buffers
Swap:  1048944k total,     2460k used,  1046484k free,    63848k cached

 1992 root      RT  -5  4396 1228  788 S  3.8  0.3  23:51.28 ntpd    

TRAK Model 8400 IRIG Driver

The TRAK Model 8400 can sync to IRIG-B and has an external oscillator input. The outputs include IRIG-B, TTL, and RS-232. Lacking documentation, I fussed with the switches until I got the unit to produce an ASCII timecode on the serial data port at 9600 baud. Then I modified the type 11 driver to accept this timecode stream via the serial port. Just get this file refclock_irig_ascii.c and rename/replace the refclock_arbiter.c in ntpd and make. This file is unsupported and the mods are public domain.

GPSD Notes

The gpsd is a utility that provides gps data over the network. One side effect of running the gpsd utility is that it turns on several of the NMEA data strings. This turns out to impact the linuxpps somewhat--it appears that all the extra NMEA data (other than the $GPGGA string) makes the linuxpps server suffer offset and jitter.

Linux SHM Parallel Port PPS Driver

I've hacked up David Schwartz' linux shm serial port PPS driver for the PC parallel port interface. It uses the same setup as freebsd for the pps, so I was able to use the same split cable arrangement I built up a while ago for the smidge pps use. The current version of the code is a merge of the parallel and serial code.

Here's the latest code. It has worked nicely here with both linux 2.4.2x and 2.6.3 kernels.

I've hacked on this so that it can use the linux RTC driver as a clock. It's a real jitterbug but its fun to tinker with. It even runs in user space. I add the shm clock dynamically with ntpdc then fire this up.

A current ( linux kernel with HZ=1000 running the userland shm pps driver (using the serial DCD for pps) gets single digit microsecond performance. I'm using fudge time1 0.0014 with the shm driver configuration.

Other folks have modified the code and added some support scripts. Konstantin Belov found the code, added a patch, and sent it to me. This shmpps tar file has the shell scripts and modified code. Thanks to everyone that has added to the code.

With kernel modules, insert the parallel port driver as follows:

modprobe parport_pc io=0x378 irq=7
modprobe ppdev
The ntp.conf file should have:
Start up a parallel port listener on unit 1:
./shm -d/dev/parport0 -p -u 1
If your hardware is ticking, you should see the interrupt count increasing when you cat < /proc/interrupts.

The pps signal is wired to pin 10 (*ACK) of the db25, and ground to pin 18. Also, freebsd uses pin 14 (*auto-linefeed) to "echo" the pps signal back out the parallel port. Versions starting 0.4 have this feature creep.

You can have multiple shm units with the -u flag. My test unit here is wired to dcd on unit 0, and parallel on unit 1:[1158] ntpq -4 -p
     remote           refid      st t when poll reach   delay   offset  jitter
-SHM(0)          .SHM.            0 l   15   16  377    0.000   -0.053   0.016
+SHM(1)          .SHM.            0 l    5   16  377    0.000   -0.039   0.015
*wraith.wraith.s .GPS.            1 u    8   16  377    0.191    0.013   0.006
+smidge.wraith.s .PPS.            1 u   12   16  377    0.275    0.034   0.047

There seems to be a fair number of spurious irq 7 interrupts caught by the driver. The signal is the same ttl output that triggers the dcd on the serial port (I've bypassed the rs232, see the differential cable), but monitoring the "echo" pin with a logic probe shows more than just the pps pulses. Running the shm code with the -D flag shows this as well, by the appearance of "out of range" messages among the regular ticking of the pps.

Some sample performance after calibration by eyeball:

     remote           refid      st t when poll reach   delay   offset  jitter
oPPS(0)     1 l   15   16  377    0.000    0.001   0.001
+SHM(0)          SHM(0)           1 l   14   16  377    0.000    0.003   0.003
+wraith.wraith.s .GPS.            1 u    1   16  377    0.256    0.005   0.004
+smidge.wraith.s     2 u    4   16  377    0.322    0.005   0.010
-thrall.wraith.s .SHM.            1 u    5   16  377    0.254    0.043   0.004
Here the interface is a parallel port at 0x278 with irq 5. An isa card was jumpered for the second serial and parallel ports, and modified to supply power to the surplus SV6. The pps is via the linux PPSkit, and the shm is off the parallel port. Both signals are driven by the 8820 differential receiver using the split cable arrangement as used with the freebsd host and the gps35, only here, the signal merely feeds the two inputs on the same machine. The host is used as a normal workstation, and has xwindows, emacs, and mozilla running, along with the ntp daemon. The only thing a bit unusual is the motherboard has two megabytes of cache, instead of the more typical 1 megabyte of the era's K6-2 cpus.

PC Soundcard and WWV Receiver

Well worth the effort is to set up a pc sound card to receive the audio output of a radio tuned to one of the WWV frequencies and have the ntp daemon extract the timing information from the audio.

The NOAA Space Environment Center has a status page for wwv reception.

Here's an example status report showing excellent wwv radio clock performance after calibration for the various radio propagation and other delays. This billboard was from freebsd 4.7 with the hacked 4.1.73 refclock_wwv. Linux 2.6.3 works with the latest 4.2 code as well (see below).

There is an impressive plot of the audio refclock performance on Dr. Mills web pages at the University of Delaware. There is quite a bit of work being done in software defined radio the last few years and it is remarkable what performance can be obtained. I am looking at a rough sketch here of a quadrature demodulator using parts I found I had (National AH5010 analog switches). The '5010s work in a breadboard but their switching speed limits them to aroud the 160 Meter band (1.8 MHz).

Note that you should use the alsamixer (linux) or mixer (freebsd) to mute any other channels in your sound card, to provide maximum possible signal to noise ratio for the wwv input. The systems allow concurrent operation of ntpd with the mixer programs. You can run alsamixer and watch as the ntpd adjusts the record level. I used a script to set the mixer up for freebsd 4.7 use.

It takes some fiddling to get your paticular sound card and wwv receiver adjusted well. With an sb awe64, I had to crank the output of the receiver up, and used the "line" control for agc. With an es1373 chipset soundcard (an sb pci128), I needed to turn the receiver gain down quite a bit compared to the awe64. You can run the ntpd with -D2 debug flag to watch the audio_gain numbers. I also run aumix under linux, and can mute the unneeded channels and adjust the line gain until reasonable operation is acheived. Its probably best to ssh in, or use X/twm, when working on this setup. The VGA scrolling tends to put huge latency spikes into the system, which will not be good for the audio or other ntp reference clocks.

With a Soundblaster live it took some fussing to deduce that ntp could use the line input, but that I needed to set the "rec" gain by hand for it to work. I used a value of 10 with the mixer command under freebsd for the "rec" setting.

The audio wants to use a config file /etc/ntp.audio0 to set the device name used for agc. So, use the "fudge flag2 0" for mic, or "flag2 1" for line to get the sound card into record mode. I use a config file something like this:

idev    /dev/audio0
cdev    /dev/mixer0
agc     line
monitor vol

I'm using an inexpensive build-it-yourself wwv radio receiver from Ramsey. Make sure you set the fudge time values. The clockstuff/propdelay.c program from 4.1.73 works under linux using this patch. The latest propdelay has broken in a different fashion. Fix is to delete #include "ntp_stdlib.h" and then use the above patch.

Ramsey HFRC1 Modifications

You can nicely enhance the 100 Hz time code output of the Ramsey HFRC1 receiver with a simple addition to the circuit. The National Semiconductor LM389 amp used in the kit can be tailored with a capacitor and resistor for up to 6db more gain at 100hz. A 0.033 uF capacitor to '389 pin 1, to a 10k ohm resistor, to '389 pin 12, will provide this "bass boost" to the audio output. More details and a gain curve are shown in the device data for the LM386 that is available on the National web site. This modification has worked well here and is recommended if you use your receiver with ntp. The kit seems to have vanished from their web site (Nov 2006), although their catalog download shows it.

Further Audio Notes

The NOAA provides the "Geophysical Alert" text of solar/magnetic storms that affect the WWV radio reception. There is a link to the latest report and a description of their reporting scheme at that site.

One little trick I pulled was with the sb awe64 card. This card seems to support a tone control with the mixer under freebsd. Recalling that Dave mentioned receiver audio tended to roll off the 100hz time code carrier, I used the mixer to set the bass control from 50 to 75. This seems to have made the lock happen more quickly than with flat tone settings.

I unearthed the old ide drive I had used with linux in my first ntp experiments and was using it with an old FIC PA-2007 I had rescued from the dumpster. This motherboard had 1 megabyte of cache and seemed to run well with an AMD k6-233 (clocked at 200Mhz). This was running linux 2.2.18 with the ppskit (1.0.2). I added in a "mad16" sound card and managed to get the WWV audio to sync. The slow (by todays standards) CPU sometimes will hit 25% CPU usage when monitored by top while running the ntpd audio setup. I copied over aumix off my workstation to check the card settings with since I had only installed a restricted userland on the drive. It turned out I needed to adjust the channels by hand before ntp was able to sync ok. The (mad16 clone) card config I am using has this /etc/conf.modules with the 2.2.18-NANO/redhat 6.2 install. Make sure that you set the BIOS to reserve DMA 0 and 1, and IRQ 10 and 5 for "legacy ISA" (not PNP). The file uses "agc mix" and I set the record to "mix" with aumix and set igain to about the first notch.

Sometimes a PC can have trouble with DMA on the IDE drives. I have set up a Tyan S1668D (dual 200Mhz ppro) with freebsd 5.4 and an SB awe64 ISA sound card. The NTP audio config seemed to be working ok, so I decided to do a pkg_add of emacs. This caused the unit to hang with an IDE DMA timeout. It turns out that a BIOS upgrade (to 5.01) fixed this problem.

The refclock_wwv.c file has had some big changes in the 4.2 code. This code has not worked very well here, and indeed, will at times dump core. I took the refclock_wwv.c subfile out of the 4.1.73 distribution modified with the patches below, and used it in place of the version included in 4.2, and have been able to get sub 100 microsecond performance on a fairly regular basis. Note that the linked billboard shows the unit synced to WWVH in Hawaii. The radio receiver was located in the San Francisco Bay Area of California when the data was taken. Go ahead and grab this prepatched refclock_wwv.c file but beware that this is an unofficial, unsupported hack. This file is copyright as per the ntp source code, and the included modifications are public domain, without warranty of any kind. Ok I have the "real" version of this now, I'd had the wrong version up here for a while. Sorry about that.

For the adventurous, Ross Alexander ( has developed a patch that allows dual audio drivers under freebsd. It was very cool to see his setup with both wwv and chu active. Thanks Ross! Note that this patch was for the 4.1.73 code base and will probably not apply to 4.2.

I've gotten an sb awe64 working in freebsd and linux with the wwv audio. I tried an old pas16 with freebsd, and could work it with the mixer program, but ntpd wasn't able to get any audio out of it. Experimenters and bug reports welcome, get thee to and attain bliss.

I had a working setup with a particular sound card, and due to the motherboard showing an apparent oscillator glitch, I moved the disk drive and card set to another chassis/cpu/motherboard setup, of the same variety as the working system. Well... Now it seems NTP can no longer attain WWV sync through the audio setup. Lacking an oscilloscope, I can only guess that perhaps there is more power supply noise in this second chassis/motherboard setup. I have also had a PCI card fail to gain sync (with IRIG) in a freebsd setup, and work fine when put into a Linux setup. Testing with IRIG shows that the card gains sync very quickly. Perhaps my angst is merely a result of poor HF radio conditions. Checking the space weather archives show that yes, there were some minor disturbances during this time period (6 May 2005). And, indeed, the WWV audio driver grabbed sync today (7 May).

Old Audio Notes

You can use this hack for the agc and monitor functions (applies to 4.1.73 and possibly earlier tarballs). I've tested it to correctly select line or microphone on both freebsd and linux systems, and adjust the desired record level. It also mutes the right channel and uses the left channel, which is what you seem to get when you plug a mono jack (from the wwv receiver) into the stereo jack on a pc sound card. The latest ntp-dev sources should have the patch already applied.

You will need this patch (thanks again Ross!) to fix a few problems. The audio patch applies to vintage ntp-dev code (4.1.72c-r and 4.1.73-r), and is not needed for 4.2 series code. I've built it for alsa on rh 7.1 linux with kernel 2.4.8-ac7, and freebsd 4.7, and the patch works on both. The freebsd setup did get the wwv clock reachable with a refclock_wwv patch.


The type 6 reference clock (IRIG-B Audio) is working here. The reference clock was set up to use the "line in" on the PC motherboard audio interface:

assID=37493 status=0101 clk_noreply, last_clk_noreply,
device="Generic IRIG Audio Driver",
timecode="00 0  0 366 04:48:33 3025 255  0.594 20   -0.1  215.2 3313457311.931335",
poll=933, noreply=1160, badformat=0, baddata=0, fudgetime1=0.000,
stratum=0, refid=IRIG, flags=10

     remote           refid      st t when poll reach   delay   offset  jitter
*GPS_NMEA(0)     .GPS.            0 l   32   64  377    0.000    0.002   0.008
xIRIG_AUDIO(0)   .IRIG.           0 l   16   16  377    0.000   68.658   0.049
-SHM(0)          SHM(0)           4 l   35   64  377    0.000   -0.027   0.008
+wraith.wraith.s .GPS.            1 u    6   16  377    0.168    0.005   0.005
+thrall.wraith.s .GPS.            1 u    5   16  377    0.174    0.007   0.006
-smidge.wraith.s .GPS.            1 u   11   16  377    0.322   -0.001   0.006
The IRIG generator was a TrueTime 900 model 914. The offset is due to having set the time by hand and eyeball. The jitter is quite low considering the use of the audio driver for the time transfer. Some extra jitter is introduced when the system performs heavy web page rendering or other computationally intensive tasks. The TrueTime had been bent somewhat but still works.

Note that we are in day 366 (leap year, 2004) here. The TrueTime needed to be configured by the front panel for this. Feeding IRIG-B out of the 914 into a Datum 9150 and trying to sync kept failing--I finally found a jumper that let the Datum take day 366. Quite a coincidence that I was testing these units just on this day's UTC rollover to day 366!

The OS was linux-2.4.20-NANO with the ALSA audio driver. It took some shoveling to google the IRIG 200-4 Standard manual linked here. The ALSA 'echo "ntpd 2 128" > /proc/asound/card0/pcm0c/oss' configuration was used for low jitter with the IRIG audio driver. The SHM clock is the parallel port linux driver documented elsewhere on this web page.

Audio Tone Test Software

Here's a wwv tone hack that uses your sound card to play the wwv audio, slaved to your unit's time. I found this via google at some web archive. The formatting had been terribly butchered by the web software. I've unmangled it and managed to compile it on freebsd and linux. It seemed to hang under linux and would coredump after not too long under freebsd. Since I usually don't give up too easily, I was munging around with this code the other day, and changed the define for SAMPHZ to 8000. I was pleased when I fired up the code under linux and it worked! I've let it play overnight and it seems to work with no coredumps on linux 2.4.32 using the ALSA sound drivers here.

The performance of the tones program is quite decent. A plot showing the offset of about 200 microseconds of a freebsd host running ntpd with the tones input to the wwv refclock driver is impressive.

It seems that there was a bug, in the 10's of years encoding string. This was making the year show as 2066 for 2006, and prevented the ntpd from using the wwv refclock as a sync source. There was also an off-by-one bug in the day of year encoding. I've fixed these in the code linked here on this web site.

There is a test program written by Dave Mills that is now included in the ntp distribution in the util directory that can generate IRIG and WWV tones out of the sound card. I have run this under solaris 8 'x86. It seems to have the habit of buffering up a Large amount of audio. When this large buffer finally all trickles out the sound card, there ends up being significant skew, or so it seems, as the TrueTime unit I tried this program with (using IRIG) tries to track the audio. The unit would indicate a constant "pll skew" as it attempts to follow the test tones.

Here's a description of why this skew occurs. Thanks Dave!

The tool works for IRIG, too. The problem with the WWV generator is that 
there is no feedback loop correcting for the audio codec intrinsic 
frequency error. The only way to do this is looping the soundcard output 
to input and disciplining the generator to the system clock as 
maintained by NTP. Not really hard, but intricate.


The tones program here does have some support for deskewing the audio output, so it should provide a better WWV time than the ntp utility.

Spectracom Netclock/2 Notes

The type 4 reference clock works fairly well here with the PC setup:[215] ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
-WWV_AUDIO(0)    .WV20.           0 l    5   64  377    0.000   -0.154   0.059
*GPS_NMEA(0)     .GPS.            0 l    1   64  377    0.000    0.001   0.008
+SPECTRACOM(0)   .WWVB.           0 l    8   64  377    0.000   -0.072   0.105
-PPS(0)          .PPS.            0 l    8   64  377    0.000    0.143   0.008
+wraith.wraith.s .GPS.            1 u   13   64  377    0.339    0.022   0.009
+ring.wraith.sf. .GPS.            1 u   17   64  377    0.311    0.023   0.071
+thrall.wraith.s .GPS.            1 u   16   64  377    0.313    0.021   0.008
To get the fudge value, I ran with the gps35 out of the fanout box, and let the netclock stabilize. This seemed to take a good 6-8 hours with my used/repaired unit. Of course we have also recently had (09 November 2004) some "extreme" level geomagnetic storms the last few days. Residual effects of this storm might have affected the netclock/2 performance here. This billboard was taken off a freebsd 4.7 unit with the pps out of the netclock driving the parallel port on the PC. I've used two "prefer" commands, the first with the gps, and the second with the spectracom. This is bogus--but I wanted to have the spectracom have a prefer so that the ATOM driver would light up the PPS even if the gps went away. The problem will be if the spectracom goes away, signal wise, the PPS will still be lit due to the gps, but then the normal mitigation should kick in. Or so I think.

The two "prefer" statements do seem to induce some minor "clockhopping:"[218] ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
+WWV_AUDIO(0)    .WV15.           0 l   32   64  376    0.000    0.037   0.051
+GPS_NMEA(0)     .GPS.            0 l   14   64  377    0.000    0.004   0.008
+SPECTRACOM(0)   .WWVB.           0 l   29   64  377    0.000   -0.334   0.253
oPPS(0)          .PPS.            0 l   20   64  377    0.000    0.048   0.054
+wraith.wraith.s .GPS.            1 u   27   64  377    0.359    0.022   0.013
+ring.wraith.sf. .GPS.            1 u   32   64  377    0.296    0.000   0.009
+thrall.wraith.s .GPS.            1 u   38   64  377    0.299    0.011   0.008
Notice that the freebsd unit has the netclock/2 PPS as the sys.peer, rather than the gps35 clock, even though the GPS "atomized" signal is noticably superior to the WWVB derived PPS from the Netclock. I've run a similar setup recently with FreeBSD 5.4 and the PPS out of the Netclock was in the 2 microsecond jitter range, quite similar to the GPS. The 5.4 setup was using an ISA card parallel port set up for LPT1 and irq 5, however, and was not using the PC's built-in (chipset) parallel port (LPT0, irq 7).

The above freebsd setup used the "serial comm" port on the netclock/2.

I've run the "remote output" port into another system and configured the type 4 driver there, and have actually gotten better performance on the unit with this port. The "remote output" will continuously send its time data, once per second, where the "serial comm" port will only do so upon poll. It would seem that there is added latency and jitter when polling, as the netclock/2 must of course receive the poll request and then process and output the data.

There is a comment with regards to the PPS output in the code, although I don't know what model it would apply to. Here's what it says:

   PPS calibration fudge time 1: format 0 .003134, format 2 .004034
I was using "time1 0.026513" with the fixed output on the NetClock/2. With smidge connected to the "serial comm" port I'm using "time1 0.0281" fudge factor. The PPS output from the Netclock/2, when used on smidge, got a "time1 0.0176" fudge.

Linux 2.6 WWV Audio Notes

Things are looking good for linux 2.6.3 and wwv audio. I took stock linux 2.6.3 and built with low latency (and patched with ppskit lite) and used the alsa sound driver and oss emulation with a pci sound card. Here is the config file I used to compile the linux 2.6.3 kernel. I then built ntp-dev with my patched 4.1.73 era refclock_wwv. The alsa sound setup lets you configure with an echo command, so I told it to use two fragments of 128 bytes for ntpd:
echo "ntpd 2 128" > /proc/asound/card0/pcm0c/oss
This brought up wwv on the ramsey with this kind of performance:
     remote           refid      st t when poll reach   delay   offset  jitter
-WWV_AUDIO(0)    .WV20.           0 l   59   64  376    0.000    0.582   0.051
-PPS(0)     1 l    4   16  377    0.000    0.004   0.001
*wraith.wraith.s .GPS.            1 u   48  128  377    0.305    0.030   0.012
+smidge.wraith.s PPS(0)           2 u   20  256  377    0.384    0.036   0.008
-ring.wraith.sf. PPS(0)           2 u   37  128  377    0.238   -0.030   0.009
+thrall.wraith.s     2 u   41  128  377    0.284    0.038   0.018

assID=38004 status=0101 clk_noreply, last_clk_noreply,
device="WWV/H Audio Demodulator/Decoder",
timecode=" 0 2004 064 00:01:59  S -4 0 0 WV20 6 9 88.0 64", poll=420,
noreply=415, badformat=0, baddata=0, fudgetime1=1.993, stratum=0,
refid=WV20, flags=10
The new, post 4.1.73 refclock_wwv is also working with linux 2.6.3. I found it necessary to "renice -10" the ntpd process to get residual jitter to the 20-40 microsecond range.

Solaris 2.8 x86 WWV and PPS notes

The es1373 audio worked in solaris x86 with a driver downloaded off the net, but a debug run of ntpd shows no gain adjustments being done. The cde audiotool runs with the 1373, though. You have to grab the nanokernel tarball and put the timepps.h file into /usr/include/sys for solaris, in order to get ppsapi version of ntp compiled.

The pps works with solaris8 on x86, but you have to have the PC's db25 connector wired up with pins 4 and 5 connected, and pins 6 and 20 connected, or their equivalent on the DB9. You can then use the type 22 (atom) driver with the ntp 4.2 code. I tried the type 20 (NMEA) driver, but there was 600 milliseconds offset with 100 milliseconds jitter on the solaris8 host. The 4.2 code base appears to not daemonize properly. I was only able to run the code in the foreground, in debug mode. The solaris xntpd did seem to daemonize properly.

Current Lab Setup

The current lab setup as of May, 2005 has a Garmin GPS35 (via a differential PPS data cable) feeding a fanout box which drives eight channels of NMEA and PPS timing data, along with a set of differential outputs for further expansion. The NMEA data is sent to four server hosts and another cable is run to the test bench. There is a WWV 10Mhz radio with audio output to a sound card in the smidge host. A Spectracom Netclock/2 feeds smidge via the "serial comm" port which allows smidge to gather the daily reception report out of the Netclock/2. The additional data port of the Netclock/2 feeds the thrall host.

The smidge and thrall hosts peer with the other units since they have redundant time sources. The ring host is a workstation and is subject to various test indignities upon occasion, including IRIG audio testing, RTC SHM hacks, and PPS SHM drivers in the recent past.

The lab also has several IRIG devices, including a TrueTime 914, a Trak Systems 8400, and two Datum 9150's, one with MIL-STD-1553B output. An additional GPS device is a surplus Trimble SV6. The IRIG and SV6 units normally are run in the test bench area and are not used by the production systems.

GPS Modules

There are various OEM GPS modules available that look to be suitable for ntp use. One that I would like to obtain is the "Jupiter T" module, which has a 10Khz gps steered output available. This would be useful to derive 10Mhz via pll to feed some of the IRIG devices around the lab. This unit is labled "not recommended for new designs," however.

Trimble appears to have a current production "Resolution T" module. They state a PPS aligned to UTC within 15 nanoseconds.

Furuno has the GT-8031 timing module. Their GF-7706 subsystem provides 10Mhz and other outputs.

Meinberg, Germany has long been active on the usenet group comp.protocols.time.ntp and here is their NTP Server product group web page.

NTP Time Server - Galleon provide NTP Time Servers for time synchronisation of computer systems.

Using the CDMA signals of various telephone operators in the US, it is possible to derive a timing signal. The "Praecis Ce" unit does this.

The Oscilloquartz folks have a hydrogen maser. I'm not sure that my budget of the $25 surplus sv6 is going to deliver an H maser.

The US NIST folks have list of time and frequency device manufacturers.

This is only a brief list of units that caught my eye or that I had found references to. I had used this google search string to pick out these sources.

The current production modules all appear to require +3.3 volt power. This is attractive for battery operation, but does represent a minor inconvenience to 7400 style TTL interfacing such as done on the GPS35 or SV6 here in the lab. I have contemplated production of a "universal" gps module interface, that would provide the power supply and buffering facilities to interface these 3v modules to the ntp style of operation.

Data and Performance Monitoring

I have written a set of scripts that use "ntpq -c rv" probes to gather statistics from the various servers. I'm using a list-of-systems mode with auto generation of the index.html file. The first script to run is do-newntpstat to create the database. Make sure the database that is created gets moved to the place where the next script can find it. Add your host name(s) to the list-of-systems and run do-mung-newhost to create the index.html file. The script takes template files 1. index-head, 2. template, 3. index-tail and churns out the updated web page. The cron job then runs do-xntp to update the png data images. A further script do-ntp-rrdstats probes the ntp-dev servers and extracts the desired data it then uses to populate the rrd database.

Thanks to Berny Stapleton for sending these links for my site. It was nice to finally understand what the rrd's I made were doing (wry smile).

Info on creating your rrd.
Ganglia config details.
More rrd config examples.

Experimental plot of the offset, jitter, noise, stability, and frequency data obtained with the scripts for the hosts here, gathered with "ntpq -c rv" probes.

Monitoring the rtxo frequency drift is quite informative. The smidge host began to have fan harmonic oscillation (buzzing) problems and this shows up clearly in the plot as a spike.

Helpful Sites

There is an archive of an email list (time-nuts) here.


Thanks to Rodolfo Giometti for linuxpps, Ulrich Windl for PPSkits, Wolfgang Rupprecht for GPS Hacks and dgpsip, the NTP FAQ and the Josef Stefan Institute for guidance and inspiration, and Dr Dave Mills for Godfathering the Internet and the genesis of NTP.

Copyright 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 by Steven Bjork ( Content reproduction as part of the connected internet granted, such as a web proxy or browser performs.
Use and distribution encouraged in the hope of better chime for all.