270 likes | 435 Views
(In)security in URL shortening and redirection services. Anthony Bettini McAfee Labs 6 May 2011 – CARO 2011 Workshop. Table of Contents Analysis of a URL shortening and redirection provider. What is a URL shortening and redirection provider? (Shortening vs. Redirection)
E N D
(In)security in URL shortening and redirection services Anthony Bettini McAfee Labs 6 May 2011 – CARO 2011 Workshop
Table of Contents Analysis of a URL shortening and redirection provider • What is a URL shortening and redirection provider? • (Shortening vs. Redirection) • Who are the popular providers? • Who uses a shortening provider and why? • Hypotheses • Case study: Deep-dive of a popular shortening provider • Hardening the Net • For malware researchers, why does all this (app-level security) matter? • Future research areas • Conclusions • Questions 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
What are we talking about? URL shortening and redirection what? • Technically, they are two separate services, although generally all providers combine them into one seamless service via APIs • Shortening concatenates a string • Redirection does an HTTP 3xx redirect • Shortening example: • We take this string: • http://www.gadgetguy.com.au/small-kitchen-appliances-toaster-kettle-coffee-machine-blender-juicer-channel7-sunrise-australia-42.html • We are returned this: • http://bit.ly/2Ngz9Q 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
What are we talking about? Redirection as compared to shortening • Redirection example: • We make an HTTP request for this, via an HTTP connection to bit.ly:80: • GET /2Ngz9Q HTTP/1.1 • (with other appropriate HTTP headers) • Our client gets an HTTP 3xx code and makes a connection to: • www.gadgetguy.com.au:80 • While making a seamless request for: • /small-kitchen-appliances-toaster-kettle-coffee-machine-blender-juicer-channel7-sunrise-australia-42.html • Throughout the remainder of the presentation, we’ll just say shortening, when we really mean shortening & redirection (we do this to save space!) 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Who are the players in the game? Crowded space with big names • Most widely used providers: • bit.ly (bitly – startup, widely used) • goo.gl (Google) • fb.me (Facebook) • t.co (Twitter) • There are many other less popular ones… 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Who is shortening URLs and why? Only consumers, right? • Users of various Twitter clients are having their URLs auto-shortened to stay within the 140 character limit • Facebook and other pages are doing similar things, for similar reasons • Corporate users are increasingly using shortening providers to convert long internal URLs (such as direct links to docs buried on an internal Sharepoint) to to ease sending a link to a coworker • Evidence of SPAM and malware authors leveraging shortening providers to mask URLs from being immediately read as suspect by users (making programmatic URL checking/filtering more important) 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Hypotheses These came before the testing, really • Complexity of the systems would increase over time as competition in this space increased • Developers building URL shortening and redirection applications had limited exposure to real world security attacks; same for QA • This limited understanding would result in mistakes, from a defense, safety, and security standpoint • Testing for a wide variety of imperfections, across all providers, would have surprising security findings • Summary: It would be easy to find INFOSEC problems 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service Overview • Examined one of the major URL shortening and redirection providers for security implementation limitations (weaknesses) • Many provide APIs that require API keys • Reading the API documentation gives a good understanding of the system • API keys exist to: • Track behavior • Rate limit requests • Restrict functionality • We skipped the API key and wrote our own HTTP client • Problem solved 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service Some of the tests we ran, at a high-level • Example tests we ran (basic unit testing, primarily for security): • Shortening of invalid URLs • Space efficiency when shortening strings with meaningless repetitions • Sequence predictability for keys in the dictionary • Shortening of self-referential URLs • Quick note on dictionary sizing • CAPTCHA requirements • Updating the source URL • Attempting to obtain competitive intelligence from providers on marketing campaigns 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • TEST NAME: Shorten invalid URL (invalid based on TLD) • EXPECTATION: GOOD_URL should be properly shortened, BAD_URL should result in an INVALID_REQUEST • RESULT: • Result for GOOD_URL: www.bing.com -> http://svc/9kkMVF • Result for BAD_URL: www.microsoft.ny -> http://svc/a0ewwA • #fail 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • TEST NAME: Shorten what obviously isn’t a URL • EXPECTATION: GOOD_URL should be properly shortened, BAD_URL should result in an INVALID_REQUEST • RESULT: • Result for GOOD_URL: www.bing.com -> http://svc/9kkMVF • Result for BAD_URL: junk -> INVALID_REQUEST • #success 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • TEST NAME: The same URLs should have the same result • EXPECTATION: GOOD_URL should be properly shortened, SAME_URLS should receive the same result • RESULT (additional related findings on next slide): • Result for GOOD_URL[0]: www.bing.com -> http://svc/9kkMVF • Result for SAME_URLS[0][0]: wWw.bing.com -> http://svc/b43j5I • #fail • Result for SAME_URLS[0][1]: http://www.bing.com -> http://svc/9kkMVF • #success 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • RESULT (continued): • Result for GOOD_URL[1]: www.google.com/advanced_search -> http://svc/aRxMmM • Result for SAME_URLS[1][0]: www.google.com////advanced_search -> INVALID_REQUEST • #fail • Result for SAME_URLS[1][1]: www.google.com///advanced_search -> http://svc/alc49z • #fail • Result for SAME_URLS[1][2]: www.google.com//advanced_search -> http://svc/dpTc57 • #fail 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • TEST NAME: Shorten the shortener’s URL (self-reference) • EXPECTATION: GOOD_URL should be properly shortened, BAD_URL should result in an INVALID_REQUEST • RESULT: • Result for GOOD_URL: www.bing.com -> http://svc/9kkMVF • Result for BAD_URL: http://svc/9kkMVF -> INVALID_REQUEST • #success 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • Quick note on dictionary/key space sizing, which we’ll discuss later • Provider A uses a default keyspacesize, excludingcustomURLs (we’ll get tothis later), of: 62^6 ([A-Z][a-z][0-9]^[fixednumber of characters]) [56 billion] • Provider B uses a default keyspacesize of: 36^6 [2 billion] 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • TEST NAME: Quick check on predictability of keys in the dictionary • EXPECTATION: Keys should at least appear pseudo-random at first glance • RESULT: • Resultfor www.d9204faa-d91b-4b85-8503-6d83a6b49842.ny -> http://svc/bd9Huo • Resultfor www.806b675b-da10-4791-b801-e7c78577c9b9.ny -> http://svc/cMuTnB • Resultfor www.559b37d2-505d-4155-8153-c84507decfde.ny -> http://svc/d8k7Zx • Resultfor www.7b21b10c-9380-497c-b933-a51ad56b66e2.ny -> http://svc/bbzXBM • Resultfor www.e0c42320-16be-4b3d-a627-b5cc7a2eae05.ny -> http://svc/cG2EpA • Resultfor www.0df5718c-7738-45bc-a1ac-a8c67b0433f4.ny -> http://svc/aGp6P8 • Resultfor www.be2cb653-072b-44d6-a280-2f060a2e7e02.ny -> http://svc/dbIyo2 • #success 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • TEST NAME: Determine if programmatic scraping is occurring • EXPECTATION: People are brute force scanning URL shortening providers for URLs • RESULT: • Resultfor www.d9204faa-d91b-4b85-8503-6d83a6b49842.ny -> http://svc/bd9Huo • Resultfor www.806b675b-da10-4791-b801-e7c78577c9b9.ny -> http://svc/cMuTnB • Resultfor www.559b37d2-505d-4155-8153-c84507decfde.ny -> http://svc/d8k7Zx • Didanalytics checks on the results; 0 clicks • #success • However, if the scraping was forhttp://svc/123456+, the inverse lookupscrapingwouldn’tbedetectable in thismanner (otherthanby the provider) 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • TEST NAME: Account registration valid email and CAPTCHA • EXPECTATION: We estimated that registered users of the system have “more rights” and that CAPTCHA was a requirement upon registration (to make it harder to programmatically generate accounts) • RESULT: • Confirmedifyou register, you have additional privileges • #success • No registration information is validated; fake email addresspermitted • #fail • No CAPTCHA requiredtocreatean account • #fail 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • TEST NAME: Source URL replacement (i.e. the rickrolling test) • EXPECTATION: We estimated that registered users of the system may have some means to replace a source URL • RESULT: • Result for GOOD_URL: link.confusion.com/assumed_clean-> http://svc/fB3xbu • (Requested to “Edit” the link) • http://svc/fB3xbu now points to link.confusion.com/assumed_dirty • #fail • An attacker could get Pointer A to become popular, where Pointer A redirects to Site A; then the attacker edit Pointer A to redirect to Site B, while keeping the integrity of Pointer A, thus compromising user’s who viewed the page after the change 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • TEST NAME: Two providers friends that won’t play nicely together • EXPECTATION: We estimated that registered users of the system may have some means to replace a source URL • EXAMPLE TEST: • Shorten URL http://blah.invalid.url.com/blah on Provider A and get http://svc/dAf32X • Shorten URL http://svc/dAf32X on Provider B and get http://svc2/Adf392z • Edit http://svc/dAf32X on Provider A and redirect it to http://svc2/Adf392z • What happens? • Left as a reader exercise… 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Case study: Analysis of a major shortening service The findings… • HACK IS WACK • “Upload your 2-min or less anti-cybercrime rap video” • Really clever and interesting marketing campaign http://www.hackiswack.com/ • Went to bitly and did an inverse lookup (shortened an already shortened URL) • Was provided with: http://bit.ly/bOJg8r • Made an HTTP request for: http://bit.ly/bOJg8r+ • Was handed all the analytics on the campaign’s “success” 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Future research areas Why dictionary size matters • Have established that creating a bunch of keys in the dictionary: • Can be programmatically done and doesn’t require valid URLs • There are a lot of way to hand the provider an invalid URL and get a key • Doesn’t require a valid email • Doesn’t require CAPTCHA • Seems a lot like the traditional buffer overflow problem • When the dictionary fills up, what happens? • Do old links get expired? If so, can they be tracked and then re-used? • Do links get overwritten? If so, can we first determine the most popular links? Can we predict which will get overwritten first? • How fast can a botnet fill up a keyed dictionary with between 2-56 billion keys? • Answer: TBD 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Hardening the Net Making the interweb a safer place • As we learned yesterday in Igor’s keynote, we’re not going to find problems where we’re not looking • The same way we proactively scrape such providers as IRC, Usenet, and web sites for malware, we should be scraping the shortening providers • While web security scraping systems and web site reputation systems have had to deal with site redirection, the mass-use of shortening and redirection services presents new challenges to these systems 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Why does application security matter? URL shortening security design’s applicability for malware researchers • New systems to scrape for malware • New user experience (at a consumer level), where URLs are intentionally obfuscated and redirected, leading to a higher importance level for web site warning systems (ex. SiteAdvisor) • Their increasing usage, presents new opportunities as well, to scan URLs prior to user’s even browsing to the page • New challenges as well, such as the dynamic nature of shortened URLs pointing to new locations 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Conclusions As with any new and widely used technology, exercise caution • New technology brings new risks • Game theory can be used to present a nice backdrop to INFOSEC • As new systems become prevalent, which were built limited awareness of attacks, those systems will become attacked • The security of URL shortening and redirection providers will improve with time • But whether they improve their security, faster than their complexity increases, is TBD! • Programmatic scraping of shortening providers is likely in the best interest of anti-malware vendors, for the purpose of finding suspect behavior (the same way older systems, like IRC and Usenet are scraped) • In short, exercise caution 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague
Q&A Questions? • New technologies bring new challenges; if you need assistance, don’t hesitate to ask for help • For additional questions and follow up, I can be reached at: • Anthony_Bettini@McAfee.com 6 May 2011 (In)security in URL shortening and redirection providers CARO 2011 Workshop: Prague