wraith.sf.ca.us Citizen Weather Observer Program (CWOP)
Data Source and Analysis of CW3879

  • Temperature plots of various local devices.
  • CWOP data display at findu.com.
  • MADIS java tool
  • Map selection for SFO East Bay
  • MESOWEST regression analysis.
  • MESOWEST at utah.edu
  • Third party data analysis based on MESOWEST data.
  • Latency to MESOWEST

    I followed the MESOWEST link above and generated this image of the last 30 days latency to the MESOWEST servers. I wonder what happened at the start of the month?

    June 2005 to June 2009 -- Four Years of Service

    The National Semiconductor based sensors that I've been using to gather data delivered under CW3879 have been running for four years at this point. The firmware in the sensor microcontroller has remained the same during this time. The sensor hardware has been changed a couple of times--an ADC with 1/2 lsb rather than 1 lsb resolution was installed, and the voltage reference has had two modifications to the circuit. The unix software has changed slightly, as the nc program was leaving partly connected sockets around--an additional option flag to nc was used to time out (a protocol violation...) these sockets. And the cwop servers used have been adjusted as needed on occasion.

    The temperature plots all went to zero on 16 July 2009. I noticed this a couple days later and investigated. Power cycling the sensor unit brought back almost all the channels--except for a couple of them. Pulling the unit apart and testing various parts identified the failure as the ADC0808 chip. Replacing this device with the original ADC0809 brought back the two dead channels. I'm not sure what happened. There was no general power failure here, or other electrical glitch. There was no thunderstorm here. No excessively hot, dry air (static electricity) either.

    Well (cough) I suspect that your HF amplifier that puts out 600 watts has coupled RF into the long wires that carry the small sensor voltage to the A to D converter, and this amount of energy fried the A/D inputs of the ADC0808. Nov 2009

    There's been nominal operation with the "hardened" script and the ADC0809 input lines using .01 bypass with some having ferrite beads also. The script got a retry triggered and 2nd time around did work. The failed1 part illustrates the dropped character, good to see the evidence of it. The timeframe for the droppage was during my operation of CQ 160 SSB contest. Antenna was 530' loop with 100 watts out. Feb 29 2012

    Sensor Development

    Our data source is a National Semiconductor LM335 mounted under the house's back deck structure, about 6 feet above ground. There is mostly pine shaded yard in the back, with one sunny patch of ground that also sees reflected heat off the rear of the house.

    As an amusing test, I ran a sensor directly to the surface of the deck where, during a hot spell, the temperature has registered 138 degrees F! I plan to insulate this device and see what it will indicate during the next cold snap. Sometimes the deck can be covered in frost, although near the house under the eaves it will typically remain frost free.

    Data reduction and transmission is performed by a 89C52 microcontroller with an ADC0809 (MM74C949-1) , 8 bit, analog to digital converter. The implementation has a range of -151 C to +104 C, and the LM335 sensor is good for -40C to +100C.

    The sensor arrangement worked well inside on the test bench. The original design and software was tested with ice water, and it did read 0 degrees Centigrade. Watching the temperature as plotted with RRDtool, with the LM335 sensor next to a glass darkroom chemical thermometer sitting on the lab table, gave good agreement.

    I've noticed that the QA reports indicated an offset in my data values, so I checked out the op amp reference, and did adjust it. Later I checked the LM336 based voltage reference (ref) and adjusted it, the opamp settings were ok but the ref was off, of course once I set the ref back I had to adjust the opamp, too. Measuring things is tricky sometimes, and of course you can't wire wrap analog this way (cough). It's time to get one of the adc0808 (+/- 1/2 lsb) devices. There is an LM335 "A" version with tighter spec, as well.

    I had noticed an occasional glitch in the temperature data plots. I'm not sure what was going on, but today (June 2007) I did modify the voltage reference so that it uses two diodes, rather than four, for the 2.49 volt reference. I also replaced the resistor with a FET current source. The four diode arrangement was pretty marginal--the four diode drops add up to very near the 2.49 volt reference. The National data books show the four diode arrangement for the 5 volt version of the reference. Looks like I got a bit confused when I first built the prototype.

    It appears that RF energy may be coupling into the unit and causing a glitch. A 50 degree or more spike of positive or negative reading can happen. It does not always hit every line when it happens, although it sometimes does. I suspect the serial data line to the computer are a culprit as it is pretty long. I should build a 2nd unit so I can have one in service while I modify the other.

    I've checked the voltage reference today (October 2006) and it is 1.22 for the "-" and 3.77 for the "+" reference. Exactly correct! Note to self: Remember to place the meter's negative probe on the op amp ground pin or the measurements will be off by a significant amount (several millivolts).

    I've installed the +/- 1/2 lsb version (ADC0808) now (October 2006). And I ran a sensor into the freezer for a bit. Here in this part of California, that's usually the only place you'll find freezing. Well it is interesting to see that the 'fridge runs a defrost cycle, very visible on the freezer plot! There is a large spike in the plot where the temperature gets above freezing, to about 33 F (freezing is 32F). Then a bit later, the temperature plunges back down to the usual 5 or so degrees F, or about -15 C.
    The recent cold evenings have been freezing the birdbath water and roughly treating many local plants. Jan 2014

    The initial sensor cable was not a twisted pair and the data plots did show a noisy signal. A 40 foot long home made twisted pair with teflon #22 stranded, and an added .1 microfarad capacitor, seems to have reduced the noise in the data. There is also a home made twisted pair of number 30 solid wire (also about 40 feet long) that is looped and gathered and generally in a mess. It had worked fine without a capacitor in the original setup, however moving the cable a bit introduced quite noticable noise pickup.

    Calibration and Siting

    There is a wind that comes in off the Bay and the Pacific, that can be cool and foggy. The air can be still and quite warm from the heating of the deck and part of the yard, then a little bit of that breeze can feel quite cool! It also makes for temperature swings in the sensor.

    A further check of the LM335 placed outdoors was performed by using an Oregon Scientific wireless sensor placed in the vicinity of the prototype sensor. Placing the remote sensor about two feet below the LM335, puts the sensor about five feet above ground. The ambient temperature reads one degree warmer on the OS sensor, although there was sun shining on a part of the sensor earlier. Of course I need to adjust that and try again, even though the sun was not shining on the sensor when I compared the readings. There seems to be reasonable agreement, within the range of the ADC converter error (+/- 1 LSB) during casual inspection of the wall display and the computer data. On today's (30 June 2005) warmer day, the OS sensor is reading about four degrees low at 1417 hrs.

    I've shielded the LM335 by placing it inside a plastic bag that was taken from a cereal box. This has been working nicely for the last few years now (writing Jan 2014).

    A suggestion about rain gauge performance suggests a manual 4 inch gauge to monitor an automatic station for casual use. The professional standard is an eight inch gauge if your budget is good.

    A microcontroller would be useful to gather the time series for wind to allow 2 minute mean/10 minute gust CWOP data.

    Quality Statistics

    The April 2011 statistics have been very good. The Mesowest regression plot is very uniform compared to the plot below. Perhaps this is due to there being more (and more accurate?) data sources locally. The summary plots of all sites sending data to CWOP displayed in an "error box" format shows this site has excellent performance in the "all data" plot and the separate day and night plots. Note that the cw3879 data point is identified with a black circle around the data point. The last day's data for 29 April 2011 is show for reference.

    Systematic Offsets

    The sensor is in a location that has the sun drop behind some trees in the neighbor's yard after about 3pm local time. The trees are at the top of the hillside along a ridge, and the sensor is located about 30--40 feet below the ridgecrest. There are a few gaps that allow some sun through, but for the most part some shade covers the area in the afternoon. Especially in the long warm California summer afternoons, this shade makes a noticable difference in the temperature.

    The data QC reports tend to treat data from this site with some grumpiness, not surprising given the afternoon shade and sun going behind the hill. The MESOwest late summer 2006 CW3879 regression analysis does show the systematic offset. The last few days have been foggy/cloudy and today it was overcast almost the whole day. The Gladstone Weather Quality plot for CW3879 shows a greater agreement with the "predicted" local temperature during this overcast period.

    Data Quality Control

    Some independent data checks are performed by various folk and made available on the internet. Picking around at the CWOP site shows some cross checks of data: Meso West, NOAA/MADIS, and the CWOP qc. One weather station was "stuck" providing the same set of values and it was interesting to watch the MADIS flag or accept (!) the datasets from the malfunctioning system.

    Mostly CWOP is a "run what you brung" wild and wooly data stream, and thus data quality control checking is necessary.

    There is a cwop mailing list archive with good information and suggestions.

    At the moment the adc0809 data stream receives V routinely from MADIS, and has had a few gliches with Q (V = passed triple check, Q = failed one). Am paying close attention to the MADIS overview.

    There was a big spike (over 120F) in the data 16 July 2005. This was the lm335 getting wet when I watered the plants on the deck. The TO-92 plastic package is suggested to be enclosed in a "void filling" heat shrink according to National Semiconductor application notes. Sounds like a good opportunity for some creative shielding.

    Unix Data Handling Software

    The microcontroller is polled by the unix cron arrangement. There is a kermit script that sends a query string and gathers the response over a serial port connection. The unix system is very useful and powerful in text processing and grep and awk tools are used in the script. Our kermit data polling script (used with 8.0.211, earlier versions had bugs) is run from a bash crontab script which sends the data off to the CWOP server, and also copies the data over to the local web server for plotting.

    A more modular script has entries for the various weather data strings one can send to the CWOP servers. Remember that any program run from cron must be found in cron's executable search path, so if you add on any programs check that they are in /usr/bin or equivalent.

    Our microcontroller software is written in C in the SDCC compiler framework with the tar file name cwop-adc.tar.gz containing the source code. Formal schematics soon (we hope).

    Our observation time and location are both GPS derived. We operate an NTP stratum 1 time server directly on the unix polling host with the reference clock being a Garmin GPS35. Backup servers include a WWVB receiver and a WWV receiver operating on a secondary computer. This will provide 1 second accuracy in data update transmissions.

    Once the data has been polled and the timestamp appended, the netcat program (nc in the script) makes the TCP telnet connection over the Internet to the Automatic Position Reporting Service (APRS) servers that gather the data and hand it off to MADIS and other software.