Developing Field Program Legacy Sonde Datasets for Research Applications

1. Developing Field Program Legacy Sonde Datasets for Research Applications Paul Ciesielski and Richard Johnson Department of Atmospheric Science Colorado State University

2. Sounding arrays for major tropical and midlatitude field campaigns during the last 4 decades superimposed on TRMM 3B43 rain map • These field program were designed to study the primary convective regimes around the globe. • The component of these field program data sets that tends to have the greatest long-term value to the scientific community is the atmospheric vertical profile represented by the upper-air sounding data.

3. The importance of a high quality sounding dataset in field programs cannot be overstated. • Such sounding datasets have numerous applications: • -describe environmental conditions which provide a context for validating and understanding other observations (radar, satellite, etc) • diagnostic studies including computation of heat and moisture budgets from which the properties of convection can inferred • compute advective properties of heat and moisture to force cloud-resolving and single column models which aide in the improvement of cumulus parameterization schemes • initialize models for reanalyses and case studies • model validation • long term climate studies • Developing quality-controlled sounding datasets has been and continues to be an important priority all field programs.

4. STEPS TO CREATING A FIELD PROGRAM LEGACY DATASET (the 4 P’s) • good PROPOSAL – an important scientific question to be answered • proper PLANNING – designing an experiment to answer proposed question • PREPARATION – making sure that the necessary resources are available to carry out the plan • Careful POST FIELD-PHASE PROCESSING – quality controlling and corrections to the data

5. STEPS TO CREATING A FIELD PROGRAM LEGACY DATASET for TiMREX • good PROPOSAL – an important scientific question to be answered: better prediction of terrain-induced heavy rain events • proper PLANNING – designing an experiment to answer proposed question: better upstream obs (enhanced sonde network over southern Taiwan and ship soundings in the upstream flow) • PREPARATION – making sure that the necessary resources are available to carry out the plan (allocation of resources to each site; quality assurance) • Careful POST FIELD-PHASE PROCESSING – quality controlling the data (to achieve a level of data quality needed to answer the proposed scientific question).

6. Doing a great job in the first 3 steps (posing a problem, planning and preparing) but failing to complete the 4th step, is like running a long race only to fall short before you cross the finish line.

7. An example from TOGA COARE: why sounding QC is important 3 sonde types were used in TOGA COARE: Vaisala RS80A, Vaisala RS80H (capacitive sensors which had a dry bias) and VIZ (carbon hygristor had a moist bias). Rainfall map during COARE showing a maximum south of equator with a secondary peak around 4°N

8. To explore the impact of the humidity correction (HC) on convection, the Raymond-Blyth (1992) buoyancy sorting cloud model was used to compute the convective mass flux with and without the HC. • 4 month-mean convective massflux peak shifts from 8N in the uncorrected analysis to just south of the equator using the HC data, which agrees better with the diagnosed vertical motion and observed rainfall for this period. • Rainfall in ECMWF and NCEP reanalysis, which used the uncorrected data, placed a rainfall maximum around 7°N for the COARE period. Only with humidity-corrected data could convection be correctly simulated!

9. Proper quality control of sonde data takes time and effort exp. name dates # snds time* TOGA COARE: Nov. 1992 – Feb.19 93 ~14,000 10 yrs SCSMEX: May and June 1998 ~23,000 3+ yrs NAME: 1 July 1 – 15 August, 2004 ~3,000 4 yrs AMMA: 1 June – 30 Sept. 2006 ~6,600 ???? TiMREX: 15 May – 25 June 2008 ~2,300 2 yrs * - time to complete corrections

10. TiMREX/SoWMEX Sounding and GPS Network • 5 different sounding types • Taiwan was responsible for 13 upsonde sites including Laoag • 16 aircraft missions were flown over ocean releasing 190 dropsondes • 46 ground-based GPS sites used to compute TPW (Total-column Precipitable Water) Laoag Different sonde types have different biases

11. RS92 RS90 RS80H RS80A • GPS_PW is accurate to within ±2mm and has a small negative bias (-0.2 mm) such that it is an excellent reference for sonde PW. • Vaisala sonde types have a dry bias but the bias is smaller in newer models. • Only one GRAW and 2 Meisei sites were examined - GRAW and one Meisei site showed little bias, other Meisei site had large dry bias; Shang sondes generally have small wet bias. • Large standard deviation in some sonde types due to batch differences

12. Perform a variety of QC (quality control) checks on data and compare data to independent measurements which allows us to identify problems (such as moisture biases). Using a variety of methods (e.g, intercomparison studies, statistical techniques, etc) correction algorithms are developed and applied.  Level 3 High resolution sounding data (Level 1) is put into a common ASCII format and then run through the ASPEN (Atmospheric Sounding Processing Environment) software. Level 2 (see Ciesielski et al 2010 for further details) Create 5-hPa vertical resolution data set, apply some additional subjective QC checks to create QC flags for each data value. Visually inspect each sonde in a Skew-T format and subjectively adjust QC data flags as needed.  Level 4 Stages to developing a research-quality sonde dataset

13. Level 3 Processing: Identification of sonde biases: development and application of correction algorithms During the field experiment we noted unusually large decreases in moisture near the surface at the four sites using Vaisala RS80 sondes. This example Skew-T plot from Pingdong on 13 May at 11LT shows a well-mixed temperature profile, but a 9ºC drop in Td from the surface to first sonde data point (where the surface value is an independent obs from the sonde). There were also clouds at the time yet the data indicated no saturation. First sonde data point Obs from independent surface site

14. Evidence for sonde RH biasescan be identified by comparing sonde data to independent measurements of humidity. • Difference between sonde water vapor mixing ratio (q) at first point above the sfc to independent surface measurement of q • realistic values should be < 0.5 • uncorrected VS80 sites (top of white bars) have mean q > 1 indicates significant low-level sonde dry bias • q at Meisei, Graw and VS92 look reasonable • negative value at Liou-Guei is related to how sfc obs was taken

15. Recommendations for quality assurance (prior to experiment) • Because of the usefulness of surface obs in indentifying sonde biases the surface site should be collocated (within a few meters) of where the sonde is launched. • Surface instruments should be calibrated on a regular basis (1-2/year) to ensure their accuracy. • operators should be well trained in proper launch procedures • sondes should be well ventilated prior to launch allowing instruments to equilibrate to ambient conditions • for RS92 sondes, desiccant in ground check chamber should be kept fresh with RHs kept below 1% (GC values of RH >2% almost certainly reflect a faulty GC or a bad sonde) • unless absolutely necessary, sonde should not be launched into a strong convective storm (for safety, icing issues and poor representation of larger scale environment).

16. A second method for identifying sonde humidity biases is with GPS-derived Total-column PW (TPW). • The advantage of GPS TPW is that it is available in all weather conditions, has high temporal resolution (1-hr), and is highly accurate (bias is -0.2 mm, accuracy within 1-2 mm). • Map shows the location of GPS and sounding sites in Taiwan. • GPS TPW data will be used to identify sonde humidity biases and evaluate the bias corrections.

17. TPW analysis during TiMREX SOP • Another source of TPW is from microwave retrievals over the ocean from AMSR-E. In this region, the overpass times are at 02 and 14 LT and are available about 70% of the time. • GPS and AMSR TPW are combined using an objective interpolation scheme to create a merged analysis from which we can estimate TPW at any point in the TiMREX domain. • TPW max exists over SW Taiwan and off the NE coast with a minimum over interior. Special thanks to Jing-Shan Hong for GPS data.

18. Comparison of sonde PW to independent observations • In cases with elevation differences > 50 m, non-sonde values were adjusted by an appropriate scaling. • Dry bias on the order of 12-20% present in VS80 sondes. • Dry bias of 3-5% in Meisei sonde • Little or no bias present in VS92 and Graw sondes.

19. Further evidence for sonde RH biases • In other monsoon-type experiments (e.g. SCSMEX) the freq. of saturated layers in soundings has been observed to be on the order of 5-15% • Only the VS92 sites and S. Ship are in this range. S.Ship looks okay; Laoag appears slightly dry MS80 appear to have slight dry bias VS80 have large a dry bias VS92 have slight daytime dry bias

20. Reasons for dry bias in Vaisala sondes thermistor • Contaminants on humidity sensor prevent water vapor molecules from adhering to it which limits its capacity to sense water vapor. • Vaisala RS92 sondes go thru a pre-conditioning process which burns the impurities off the sensors. Daytime radiative heating of the sensor arm results in lower measured RHs (note: RH = e(Td)/e(T) )

21. Recognizing these problems, a series of intercomparison sonde launches were conducted. The idea here is to use the VS92 sonde as a reference sonde, since these sondes have been shown to have little or no RH bias for nighttime launches, and a well-documented daytime dry bias (related to solar heating effects) that can easily be removed. • With this in mind: • Under the direction of Dr. Po-Hsiung Lin, 12 launches were conducted at Banchio intercomparing the VS92 sondes with Meisei and GRAW sondes. 4 launches were conducted in April 2008 and 8 additional launches in October 2008. (6 daytime and 6 nighttime launches). Unfortunately these sondes were not representative of monsoon conditions and thus are less than optimal for correcting TiMREX sondes. • Under the direction of Dr. Ching-Hwang Liu, 18 launches were conducted at Pingdong intercomparing the VS92 and VS80 sondes for the period from 18 May to 9 June. The objective was to document the atmosphere as broadly as possible (day/night, wet/dry). They ended up with 10 daytime and 8 nighttime launches (6 rainy, 3 cloudy, and 9 fair).

22. Various stages of Banchiao intercomparison launch on 15 April 2008

23. How are how these intercomparison datasets used to construct a correction algorithm? • The approach taken here is referred to as Cumulative Distribution Function (CDF) matching, which attempts to adjust the statistics of questionable data to be similar to that of reliable data from a reference sonde (in this case data from a Vaisala RS92). • used to correct humidity biases in sonde dataset from other field programs such as AMMA and NAME. • This method corrects all sondes to the standard of the Vaisala RS92. A second, additional correction is needed to correct the daytime dry bias present in the Vaisala RS92.

24. CDF matching method at Pindong • Application of the CDF matching method using the 18 intercomparison launches results in the RH correction tables. • Shown below are correction tables for night (left) and daytime (right) sondes at Pingdong; nighttime correction is larger because the daytime Vaisala RS92 sondes contains a dry bias themselves. This correction table could potentially be used to correct RH at all four VS80 sites.

25. As an alternative to using the CDF matching method for correcting RH in the VS80 sondes: • The Vaisala company took a sample of 70 VS80 sondes from the four sites using this sonde type for further investigation of the problem (thanks to Dr. Lin, Po-Hisung). • They determined that the dry bias in sondes at three of the sites (Pingdong, Makung and Green Is) was indeed beyond what was typical and that its cause was an unusually high level of contaminants on the humidity sensors. • These sondes were given a heat treatment to burn off the contaminants, then recalibrated resulting a set of corrected calibration coefficients. • Using a statistical average correction for these 70 sondes, Vaisala provided corrected humidity coefficients for all the VS80 sondes used during TiMREX • Vaisala did not feel that the sondes used at Hualien were affected and thus no corrected coefficients were provided for this site. • In the end the Vaisala Corrected Coefficients (VCC) were used at Pingdong, Makung and Green Is and CDF matching was used at Hualien.

26. Corrections used at four VS80 sites • Choice of using CDF matching method using Pingdong intercomparison dataset(18 launches) or Vaisala Corrected Coefficients (VCC) • Analyses showed VCC provided slightly better validation at Pingdong than CDF, so VCC was used at all these sites but Hualien

27. Daytime solar heating correction • After application of Vaisala Corrected Coefficients (VCC), an additional correction is needed for daytime heating problem. • One of the simpler corrections, which is independent of height, has been suggested by Cady-Pereira et al. 2008. This correction is solely a function of solar zenith angle (SZA) as shown below. SF = 0.067 x exp[-0.2*SZA] Scale Factor (SF) used to multiply specific humidity for correcting RS92 sondes for daytime dry bias. X’s along curve - indicate sonde times at Pingdong.

28. Corrections used at VS92 sites • of the many corrections proposed for VS92 sondes, the most appropriate comes from Yoneyama’s et al 2009 (YON) study based on sondes taken in an oceanic, convective environment. • smooth curve is a polynomial fit to the red line (VS92-SW RH difference based on 14 intercomparison launches) • correction is function of solar zenith angle and height with little correction near the surface but increasing with height

29. Corrections used at MS80 and Graw sites • corrections used CDF matching method based on Banchio intercomparison data set (12 launches in non-monsoonal conditions comparing VS92, MS80 and Graw). Dataset was less than optimal. • Results from intercomparison showed small differences in winds and temps (0.5C at low levels to 1.5C aloft) • Slight dry bias at high RH (3-5%) with only VS92 sondes attaining saturation in clouds.

30. Summary of corrections used at each site Ultimately, the correction chosen at each site was the one that provided the best validation to independent data.

31. Impact of correction on near surface q gradient and frequency of stable layers • Near surface moisture gradients computed with corrected data now show values much closer to the range of expected values (< 0.5). • the corrected VS80 sites may still be slightly too dry but are greatly improved from their uncorrected values • frequency of saturated layers with corrected data now are in more reasonable range (5-15%), except at Hualien and Banchiao but these sites may actually be drier.

32. Impact of correction on PW • Black bars show increase in PW due to correction. • Corrected values are now close to or within uncertainty limits of independent estimates (1-2 mm). (independent - sonde)

33. Correction results in an excellent match between corrected sonde PW and nearby GPS site (r =0.95). • PWGPS - PWsonde = 0.45mm • PW differences at individual time of day are reduced from 5-8 mm to less than 1mm. # of obs at each time Validation of humidity correction at Pingdong

34. +303 +210 +27 +490 -3 -15 -36 -82 Impact off humidity corrections on CAPE and CIN • Corrections results in huge increase in CAPE and decrease in CIN at VS80 sites (e.g. 4-fold increase in CAPE • These parameters at VS80 sites now comparable to nearby sites. Convection simulated with the humidity corrected sondes will occur with much greater intensity and frequency.

35. Perform a variety of QC (quality control) checks on data and compare data to independent measurements which allows us to identify problems (such as temperature and moisture biases). Using a variety of methods (e.g, intercomparison studies, statistical techniques, etc) correction algorithms are developed and applied.  Level 3 Create 5-hPa vertical resolution data set, apply some additional subjective QC checks to create QC flags for each data value. Visually inspect each sonde in a Skew-T format and subjectively adjust QC data flags as needed.  Level 4 High resolution sounding data is put into a common ASCII format and then run through the ASPEN (Atmospheric Sounding Processing Environment) software. Level 2 Stages in developing a research-quality sonde dataset Level 4 processing deals with problems that are identifiable but not correctable

36. Example of bad winds at Dongsha Winds at 06Z on 18 May are of good quality Winds at 12Z on 18 May are extremely noisy in the vertical

37. Level 4: 5-hPa interpolated sonde dataset with QC flags marking suspicious data How is it created? • Level 3 data is linearly interpolated in log pressure to equal 5-hPa pressure levels • A variety of Quality-Control (QC) checks are applied which flag data as questionable or bad. • QC checks include gross limit, hydrostatic, and vertical consistency checks. • Vertical consistency QC involves checks for unreasonable values of vertical wind shear and temperature lapse rates (strongly adiabatic layers). • After assigning objective QC flags, a software program (xsnd) is used to visually examine each sounding to subjectively adjust these flags.

38. Subjective determination of data reliability Determination of counterfeit money requires that one become intimately familiar with the real thing in a variety of situations. In a similar way, subjectively identifying questionable sonde data requires one to have a good understanding of the environment that the sonde was launched in and to have examined numerous good sondes in this type of environment.

39. xsnd is a software tool developed at CSU to visually QC soundings. It’s written in TCL and runs in Linux environment. Example of xsnd display

40. Example of xsnd displaying a sounding from Banchiao on 10 June 2008 The strong superadiabatic layer at the top of the moist layer is example of wetbulbing - wetting on thermistor which results is excessive cooling by evaportation or sublimation once the sonde leaves the cloud.

41. Example of zoom capability of xsnd Suspicious super-adiabatic at top of cloud layer is marked as bad Selecting filter option shows sounding with suspicious points removed.

42. Example of marking suspicious winds at Dongsha Vertical winds profiles at Dongsha from soundings 6 hours apart

43. Bad and questionable data points are marked simply by clicking on points By selecting the filter option, you can see what sounding looks like with bad points removed.

44. QP - pressure QH - height QT - temperature QD - dew point QW - winds FLAG Notation 1 - good 2 - objective ques. 3 - subjective ques. 4 - objective bad 5 - subjective bad 6 - interpolated 7 - estimated 8 - unchecked 9 - missing Clicking on a point in xsnd changes the value of the QC flag in the file; the data value is unchanged. The user must decide what quality level they are willing to accept in their analysis. Example of sounding file in CSU format with QC flags

45. . } } { } Meisei RS80 RS92 Graw The errors identified in Level 3 data flagged in L4 using a software program called xsnd (eXamine SouNDing). • Questionable data (3-6 from temporal mean); bad data (> 6 from temporal mean) • Meisei system had most problems, note : these systems used Radiotheodolite Tracking. • Overall, the quality of sondes from Vaisala 92 and Graw systems quite good.

46. In analyzing GATE sounding data Vic Ooyama (1987) made this assertion, “After having explored every possible avenue to extract “facts” from the observational data, the author cannot hide his empathy with Bernand Trevisan (alchemist, 1406-1490) who uttered with his last breath his conviction, “To make gold, one must start with gold.”

47. Concluding remarks • the Taiwanese scientists are to be commended for their efforts in TIMREX to provide the scientific community with gold - a legacy dataset for the study of heavy rainfall in a monsoon environment. • the corrections and QC efforts described in this talk are an effort to refine this gold. • the corrected sonde dataset has resulted in much improved description of the humidity field during TiMREX • these improvements should lead to more accurate diagnostic studies, better initialization of models and a high fidelity dataset useful for calibration and validation of independent datasets. • for further details of corrections and quality-control procedure please refer to our recent article in JTECH.

48. Questions? Xie Xie Picture taken from ASTER dropsonde flight on 30 May 2008, Cb penetrating through low-level stratus deck is off the southwest coast of Taiwan.

49. Supplemental material

50. Example of CDF matching method • Since the bias varies in the vertical, CDFs are computed for every 20ºC temperature bin from +40ºC to • -80ºC. • CDFs for each sonde type are matched for each percentile and in each temperature interval as shown in top panel. • In this example the RS80 value of 72% at the 40% percentile is matched to the corresponding RS92 value of 81%, resulting in 9% bias correction for an RS80 data value of 72% (as shown in bottom panel). 9%