Darlene Goldstein

1. A Comparison of Microarray PlatformsNUS – IMS Workshop7 January 2004 Darlene Goldstein

2. Talk Outline • Bioinformatics Core Facility at ISREC • Purpose of study • Platform technologies and study design • Comparisons between platforms • Conclusions and study completion

3. DAF NCCR biomedical BCF microarray research NCCR biomedical DAFL & BIOINF BCF bioinf. research biostatisticsEPFL BCF: What is it ? • ISREC-based, supported by the NCCR for molecular oncology, member group of the SIB • Created by the NCCR molecular oncology to assist its DAF (which is now absorbed into the DAFL) and its microarray users in their biomedical research • A group devoted to the bioinformatics and statistical aspects of gene expression research, in particular to the analysis of data generated with microarraytechnologies

4. BCF: Main Components • Technical Support • advice inexperimental designand data analysis • production, control, development of spotted arrays • processing of microarray data, quality assessment • Education • practical trainingthrough classes / workshops • Collaboration • statistical data analysis of research projects • Research & Development • development / testing tools & methods

5. Platform Comparison Study • Purpose • to assess accuracy and reproducibility of different gene expression platforms • to compare features of different measurement types • to understand the system (important for normalization and downstream analysis) • Impact • practical advice to DAF(L) and to NCCR microarray users • benefit to wider scientific community, especially if possible to somehow combine results across array types

6. Platforms and Study Design • Platforms • Affymetrix GeneChips, high-density short oligo arrays • Agilent long oligo arrays • in-house spotted cDNA arrays • MPSS (massively parallel signature sequencing, a digital gene expression technology patented by Lynx); in collaboration with the Ludwig Institute for Cancer Research; originally intended as ‘gold standard’ • Basic Design • 3 replicate measurements for two mRNAs (human placenta and testis) • dye swap for two-color systems (Agilent, cDNA) • 2 to 3 million tags sequenced for MPSS

7. Methods • Experimental Method (as recommended by ‘specialists’): • Affymetrix: Biozentrum Basel • Agilent: Institut Goustav Roussy, Paris • Spotted cDNA arrays: Otto Hagenbuechle's group (DAF, now DAFL) • MPSS: Lynx (California), Victor Jongeneel's group (LICR) • qRT-PCR followup (~ 250 genes), Robert Lyle, Patrick Descombes (UniGE) • Expression Quantification as recommended by ‘specialists’ (above), but : RMA for Affymetrix

8. Spotted cDNA arrays Human 10k Array 8x4 subarrays

9. Affymetrix GeneChips Image of hybridized array

10. MPSS • Uses microbeads with ~100k identical DNA molecules attached • Captures and identifies transcript sequences of expressed genes by counting the number of individual mRNA molecules representing each gene • Individual mRNAs are identified through generated 17-to 20-base signature sequence • Can use without organism sequence information • ‘MPSS can accurately quantify transcripts as low as 5 transcripts per million (tpm) to above 50,000 tpm’ (information from Lynx web site)

11. Other comparison studies (I) • Yuen et al. 2002; Nuc. Acids Res. 30(10):e48 • Affy MGU-74A, cDNA; cell lines; qRT-PCR 47 genes • both arrays sensitive (TP) and specific (TN) at identifying regulated transcripts • found comparable rank-order of gene regulation, but only modest correlation in fold-change • both array types biased downwards (FC under-estimated compared to qRT-PCR) • Evans et al.2002; Eur. J. Neuroscience 16:409-413 • Affy RG-U34A, SAGE to detect brain transcripts; 43 rat hippocampi; evaluation based on 1000 transcripts • ~55% low, ~90% high abundance transcripts detected

12. Other comparison studies (II) • Li et al.2002; Toxicological Sciences 69:383-390 • Affy HuGene FL, HGU-95Av2, IncyteGenomics UniGemV 2.0 (‘long cDNA’); drug-treated cell lines at 8h and 24h; qRT-PCR 9 genes • cross-hyb contributed to platform discrepancies • found Affy ‘more reliable’ (sensitive) • Kuo et al. 2002; Bioinformatics 18:405-412 • Affy HU6800, cDNA, publicly available data on NCI 60; 2895 genes • found low correlationbetween measurements (but no control over lab procedures – different groups had performed the original studies)

13. Other comparison studies (III) • Barczak et al.2003; Genome Res. 13:1775-1785 • 2 versions of spotted long oligo (Operon), Affy HGU-95Av2; cell lines; 7344 genes • this large-scale analysis found strong correlations between relative expression measurements • similar results for amplified and unamplified targets • Tan et al. 2003; Nuc. Acids Res. 31:5676-5684 • Agilent Human 1, Affy HGU-95Av2, Amersham Codelink UniSet Human I (30-mers); cell lines in serum-rich medium and 24h after serum removal; 2009 genes • modest correlations • little overlap in genes called DE • best agreement on DE calls (varying criteria) only 21% • comparison studies by other groups world-wide are also in progress

14. Comparison Principle • Cross-platform gene matching done through the trome database of transcripts (constructed with the Transcriptome Analyzer program tromer) • Use only those genes we classify as ‘reliably mapped’ between platforms (~2500 genes); we have not (yet) looked at probe(set)s that could not be well-mapped to known transcripts • ‘Peak technical performance’ : this is a case study, not a systematic study; does not take into account normal user variation, other mRNAs, etc. • Comparison based on M (log ratio) and A (average log intensity) • Unfortunately, accuracy cannot be properly assessed, as true M values are not known

15. cDNA array Performance

16. MA plots (examples) Affy U133A range background NCCR h10kd Agilent

17. |M| (putative effect) densities

18. (Difference in M) vs. A: reproducibility Affy U133A y = difference in M x = average A Agilent h1A NCCR h10k

19. D (error) densities

20. Affy Agilent 5977 5099 5797 3365 435 549 2514 NCCR Gene Matching Probe(sets) / genes 18325 Agilent h1A 15688 24808 Affy U133A 14876 7812 NCCR h10k 6853

21. Affy Agilent - - 2494 - - - NCCR Gene matching also with MPSS 2494 Tromer clusters 4060 Affy probesets 2869 Agilent probes 2685 NCCR clones

22. Concordance in M density plots (I) Agilent Affy NCCR

23. Concordance in M density plots (II) Agilent Affy NCCR

24. Difficulty in comparing to MPSS ratios

25. MPSS difficulties, another illustration

26. Correlations first quartile (25% least frequent RNAs) fourthquartile (25% most frequent RNAs)

27. Affy Agilent 30 40 48 96 34 26 44 NCCR Agreement: top up 200 (placenta) M range Affy: 1.66 - 7.94 Agil: 1.48 - 6.17 NCCR: 1.83 - 7.12

28. Affy Agilent 41 38 46 87 34 26 53 NCCR Agreement: top down 200 (testis) M range Affy: -8.27 - -1.65 Agil: -6.07 - -1.47 NCCR: -6.18 - -1.79

29. Comparison with MPSS, 99% CI (up)

30. Comparison with MPSS, 99% CI (Down)

31. MPSS CI Overlap Overlap with the 99% CI for MPSS Overlap with the 99.9% CI for MPSS

32. Overlap with MPSS 38 MPSS 74 (similar numbers also for Affy and Agilent); 56 of the 88 are in common to all 4 88 112 NCCR missing or classified as unreliably mapped (tag to gene not unique)

33. Conclusions (I) • The three microarray platforms compared performed very similarly in terms of which genes are detected as differentially expressed, distributions of M values, variability between replicate measurements ... ... so similarly that it seems hard to find real differences • Most disagreement for low-expressed genes • RMA M values (Affy) are better variance-stabilized, but reproducibility is good for all platforms except for weak signals in Agilent (likely due to bg treatment) • RMA M values are more strongly compressed towards zero at low intensity; reduces false positive calls but might make DE at low intensity undetectable (but is it detectable at all?)

34. Conclusions (II) Microarrays vs MPSS M values, quantitative comparison: the disagreement is large ... ... so large that it is hard to reconcile the values, making it impossible to use MPSS as the ‘gold standard’ M values, qualitative comparison: there is a good degree of agreement - approximately the same to all three microarray platforms

35. Conclusions (III) • MPSS predicts many more low-abundance genes to be (strongly) differentially expressed • The hybridization methods lose signal of low-abundance genes (due to the background fluorescence estimation?) • microarrays miss detection of most of the differential expression of low abundance transcripts, but it is also possible that MPSS is biased for many genes or less precise than this approach suggests • approach with confidence intervals for MPSS (currently approximate CI that takes into consideration the sampling error on the counts, we have no replicated measurements for MPSS)

36. Completion of Study Choose genes for qRT-PCR for which the platforms and MPSS disagree and (attempt to) address the questions: • which platform is more accurate? • how does accuracy depend on the signal intensity? • do the microarrays miss DE frequently....? • ....and especially at weak signal intensity ? • which platform best detects low abundance RNAs? • does MPSS agree with QT-PCR? • Suggestions are welcome !! 

37. Acknowledgements • Ludwig Institute for Cancer Research Victor Jongeneel, Christian Iseli, Brian Stephenson • DAF/DAFL Otto Hagenbuechle, Josiane Wyniger • UniGE Robert Lyle, Patrick Descombes • BCF Mauro Delorenzi, Eugenia Migliavacca • and everyone I inadvertently left out!