1 / 17

Open Source Intelligence:

Open Source Intelligence:. Access All Intelligence, in All Languages, All the Time. Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC.

charles-salas
Download Presentation

Open Source Intelligence:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Open Source Intelligence: Access All Intelligence, in All Languages, All the Time Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP ’06 Sheraton Premier, Tysons Corner, Virginia January 16-20

  2. About Deep Web Technologies (DWT) DWT is a New Mexico based company focused on providing state-of-the-art software solutions which search, retrieve, aggregate, and analyze content. • Deployed first “federated search” portal in the Federal Government, 1999 • Major clients include: • DOE Office of Scientific & Technical Information • Defense Technical Information Center • Science.gov Alliance • DOE Office of Science • National Agricultural Library

  3. Open Source Intelligence The Problem: • Collecting and analyzing enormous quantities of information in any language, in myriad formats, located anywhere, accessible through a large variety of means, with a majority not accessible through the Internet

  4. Shared Challenge: OSINT and Knowledge Discovery/Diffusion OSINT Challenges Knowledge Discovery/ Diffusion Challenges DWT for the past six years has been the lead technical organization addressing these challenges in collaboration with DOE Office of Scientific & Technical Information

  5. The DWT Proposition To apply DWT’s technology, expertise and ongoing innovations* to address the challenges of OSINT *Developed in partnership with DOE/OSTI

  6. Challenges in Working with Thousands of Data Sources Locate Reliable Sources Categorize Sources by Content Configure Sources for Searching Maintain Sources

  7. Challenges in Searching Thousands of Sources Automatically Select Sources to Search Perform Many Searches in Parallel Translate, Analyze and Organize Results Relevance Rank Extract Key Information Cluster/ Visualize

  8. ResearchAssistantTM DWT’s State-of-the-art Federated Search Engine • Scalable, grid-computing based federated search engine • Sophisticated Search Conductor • Supports custom connectors • Multi-tier relevance ranking • Framework accepts integration of advanced linguistic, analyses, and visualization modules

  9. Grid Computing: Distributing the Workload

  10. Perform search Enough good results? Deliver results to user Can I get more results from “good” sources? Search Conductor Select sources to search YES NO YES NO

  11. Multi-tier Relevance Ranking • QuickRankTM – Ranks results based on occurrence of search terms in title and snippet • MetaRankTM – Ranks results utilizing custom algorithms applied to metadata • DeepRankTM – Downloads and indexes full-text documents

  12. Science.gov Alliance Consortium of 12 Federal Government Agencies Dept of Agriculture Dept of Commerce Dept of Defense Dept of Education Dept of Energy Dept of Health/Human Services Dept of Interior Environmental Protection Agency NASA National Science Foundation US Government Printing Office National Archives & Records Administration Sponsoring Science.gov Portal (Access to most of Federal Government R&D

  13. Science.gov Advanced Search Page

  14. Science.gov Results Page

  15. A Science.gov Document

  16. Next Steps Identify Sponsors and development partners that can collaborate on the development of a pilot that integrates best-of-breed technologies of value to OSINT. This pilot will result in a portal that aggregates content of different types, generating actionable intelligence.

  17. Contact Us Abe Lederman 122 Longview Drive Los Alamos, NM 87544 abe@deepwebtech.com www.deepwebtech.com http://www.deepwebtech.com/talks/IOP.ppt

More Related