1 / 31

A Longitudinal Analysis of the ads.txt Standard

Learn about ads.txt Standard, combatting domain spoofing fraud in Real Time Bidding auctions, transparency in advertising ecosystem, adoption trends, compliance, and data collection methodology with insights for privacy researchers.

gfrisch
Download Presentation

A Longitudinal Analysis of the ads.txt Standard

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Longitudinal Analysis of the ads.txt Standard Muhammad Ahmad Bashir Sajjad Arshad Engin KirdaWilliam Robertson Christo Wilson October 22, 2019

  2. What is ads.txt Standard? Introduced by Interactive Advertising Bureau (IAB)to combat ‘domain spoofing’ fraud in Real Time Bidding (RTB) auctions

  3. Online Advertising Website / Publisher Advertising & Analytics (A&A) Companies Users’ Profile Users’ Profile

  4. Shift Towards Real Time Bidding (RTB) Observable via browser Not observable Website / Publisher Advertising & Analytics (A&A) Companies Advertisers Demand Side Platform (DSPs) Ad Exchange RTB Auction $0.05 $0.10 $0.20 RTB currently holds 42% share of US Digital Advertising.

  5. An Example of RTB Auction Observable via browser Not observable RTB Auction UserX Publisher Ad Exchange Demand Side Platforms (DSPs) Bid for User 123 ? UserX=123 $0.15 GET, CNN’s Cookie UserX=xo$ Bid for User 123 ? $0.40 GET, DoubleClick’s Cookie = 123 DSPs don’t receive the request directly from the browser. UserX=ABC GET, RightMedia’s Cookie = ABC

  6. Potential for Fraud in RTB Auctions Observable via browser Publishers Ad Exchanges DSPs (Bidders) Not observable DSPs will think the user is on CNN, rather than FNN

  7. Ad Fraud = Loss in $$$

  8. Programmatic Buying & Selling via ads.txt Standard Authorized Digital Sellers (ads.txt) Introduced by Interactive Advertising Bureau (IAB) in May 2017 Enables publishers to list authorized exchanges At the root of the top-level domain https://cnn.com/ads.txt google.com, pub−7439281311086140, DIRECT rubiconproject.com, 11078, DIRECT c.amazon−adsystem.com, 3159, DIRECT openx.com, 183753,RESELLER • File Format: • <FIELD #1>, <FIELD #2>, <FIELD #3> • #1: Exchange domain * • #2: Publisher’s account ID • #3: Type of account/relationship * Not always an exchange

  9. Potential for Fraud in RTB Auctions https://cnn.com/ads.txt google.com, pub−7439281311086140, DIRECT rubiconproject.com, 11078, DIRECT c.amazon−adsystem.com, 3159, DIRECT openx.com, 183753,RESELLER Observable via browser Publishers Ad Exchanges DSPs (Bidders) Not observable

  10. Why Study ads.txt standard? Potential to combat domain spoofing fraud Check for any shortcomings. Lessons can be learnt for upcoming standards. • Might be useful for privacy researchers • Means of bringing transparency in the advertising ecosystem • Publishers are disclosing their relationships with ad exchanges

  11. Goals / Research Questions Are publishers adopting the ads.txt standard? Are major A&A companies embracing the standard? Which A&A companies have adopted? Do these companies appear to be complying with the standard? Can this data be useful for transparency of the ad ecosystem?

  12. Talk Overview RTB background & motivation for the ads.txt standard Data collection Adoption trends Compliance of the ads.txt standard Means of providing transparency

  13. Data Collection Methodology https://cnn.com/ads.txt Example Inclusions pub/ index.html google.com, pub−7439281311086140, DIRECT rubiconproject.com, 11078, DIRECT c.amazon−adsystem.com, 3159, DIRECT openx.com, 183753,RESELLER ads/ script.js logo.jpg Using Python’s requests moduleFetched /ads.txt URL Alexa top-100Kpublishers Stored outputas base64 ads/ frame.html ads/ img_a.jpg adnet/ data.js Crawled 15 URLs perpublisher using Chrome Debugging Stored Inclusion Chains • Collected 26 snapshots between January 2018 - April 2019

  14. Authorized Exchanges Over Time Increase: 860 —> 1400Total unique exchanges: 2381 All Exchanges Valid Exchange Mistakes in ads.txt files Number of Exchanges A. Syntax Errors google.compub−7439281311086140, DIRECT openx.com , pub−7439281311086140 B. Semantic Errors Misspelled rubicnprject.com, 11078, DIRECT Number of authorized exchange domains over time doubleclick.net, 3159, DIRECT Should be google.com

  15. Fixing Semantic Errors Misspelled rubicnprject.com, 11078, DIRECT All Exchanges Valid Exchange Should be google.com doubleclick.net, 3159, DIRECT Increase: 860 —> 1400Total unique exchanges: 2381 Number of Exchanges Fetch WHOIS record Exists Not Found 1035 domains 1346 domains Perform DNS lookup Number of authorized exchange domains over time No Resolution

  16. Talk Overview RTB background & motivation for the ads.txt standard Data collection Adoption trends Compliance of the ads.txt standard Means of providing transparency

  17. How Have Publishers Adopted? Adoption has been modest Increase: 12% —> 20% Publishers that run RTB ads Adoption is pretty decent. ~64% ads.txt adoption by Alexa Top-100K publishers over time

  18. Authorized Exchanges Discovered All Exchanges Valid Exchange Number of Exchanges Number of authorized exchange domains over time Number of unique publishers per authorized exchange domain 1035 valid domains Out of 1035 domains, 20% appear only on a single ads.txt file

  19. Adoption Trends are Encouraging 64% of the publishers from Alexa top-100K who run RTB ads. Major A&A domains support ads.txt

  20. Talk Overview RTB background & motivation for the ads.txt standard Data collection Adoption trends Compliance of the ads.txt standard Means of providing transparency

  21. A Simple Check for Compliance Observable via browser Publisher Ad Exchange DSP (Bidder) Not observable https://cnn.com/ads.txt ✔ google.com, pub−787, DIRECT rubiconproject.com, 11078, DIRECT openx.com, 183753,RESELLER https://cnn.com/ads.txt ✗ google.com, pub−787, DIRECT rubiconproject.com, 11078, DIRECT

  22. Checking Compliance at Scale Example Inclusions pub/ index.html Advertisements (Detected Using EasyList) ads/ script.js logo.jpg ads/ frame.html ads/ img_a.jpg adnet/ data.js

  23. Checking Compliance at Scale Example Inclusions pub/ index.html Advertisements (Detected Using EasyList) ads/ script.js logo.jpg ads/ frame.html ads/ img_a.jpg adnet/ data.js Chain must have at least 3 elements

  24. Extracting Tuples For Compliance Analysis Associated Tuples Advertisements (Detected Using EasyList) Chain must have at least 3 elements

  25. Compliance Per Tuple ads.txt ads.txt Compliance has increased over the year 75% publishers have 100% compliant inventory Percentage of non-compliant Exchange-DSP tuples per publisher Compliance over time

  26. Top Non-Compliant Tuples • Sellers • Some sellers are systematically non-compliant (e.g. NativeRoll, JustPremium). • Top exchanges do not appear as non-compliant. • Buyers • Most are actually top exchanges (sellers). • 8/10 top sellers are buying non-compliant inventory.

  27. Compliance of ads.txt Standard Top exchanges and DSPs appear to be honoring the standard More than 75% of publishers that run RTB have 100% compliance Major ad exchanges purchasing non-compliant inventory

  28. Talk Overview RTB background & motivation for the ads.txt standard Data collection Adoption trends Compliance of the ads.txt standard Means of providing transparency

  29. Identification of A&A Domains Efforts to identify Advertising and Analytics related domains. Filter lists (e.g. EasyList) are manually curated Machine learning based approaches https://cnn.com/ads.txt https://washingtonpost.com/ads.txt indexexchange.com, 197545, DIRECT adtech.com, 232547, DIRECT openx.com, 203799,RESELLER google.com, pub−7439281311086140, DIRECT rubiconproject.com, 11078, DIRECT c.amazon−adsystem.com, 3159, DIRECT openx.com, 183753,RESELLER RESELLER Top Exchanges like Google, openX have adopted 1035 valid domains

  30. New Standards Coming As Well Have the potential to make the ecosystem even more transparent App-ads.txt (For mobile apps) Sellers.json (To reveal relationships of sellers in the ecosystem)

  31. Conclusions / Takeaways A 15-month study to study the ads.txt standard We observed heathy adoption (64% publishers have adopted) Major exchanges & DSPs are part of it, and honoring the standard A step towards making the ecosystem transparent Data needs to be cleaned Datasethttps://personalization.ccs.neu.edu/Projects/Adstxt Questions? ahmad@ccs.neu.edu

More Related