SharePoint 2010 Search First Migration Strategy & Implementation Tips

1. SharePoint 2010 Search First Migration Strategy & Implementation Tips Guy Mounier, Technical Services Sang Shin, Business Development

2. Presentation to Harvard University Agenda – 12pm to 1:30pm • 15 min: Introductions & Grab Lunch • 20 min: What’s New in SharePoint 2010 Search What’s New in MS Enterprise Search 2010 offering for Users, Admins, Developers. Differences between FAST Search 2010 for SharePoint and SharePoint 2010 Search out-of-the-box. • 20 min: Search First Migration Strategy How to take advantage of SharePoint 2010 Search first in a phased migration strategy from MOSS. • 20 min: Best Practices for SharePoint 2010 Search Implementations Capacity Planning and Architecture Design Recommendations from the Trenches. • 10 min: About BA Insight Integrated Search Technologies for SharePoint & FAST Quick Overview & Demo • 5 min: Next Steps

3. Introductions Tell us a bit more about your environment: • Current use of MOSS? • Current plans to upgrade to SharePoint 2010? • Any plans to leverage FAST for SPS? • Any plans to integrate external applications?

4. Introductions Exclusively Supporting Microsoft Enterprise Search Since Day 1 • Microsoft Managed Partner ISV focused on Unified Enterprise Search Since 2004 • Headquartered in NYC, with Offices & 2m+ users worldwide • Google Compete & Autonomy Compete Team Members • TAP Member for FAST, SharePoint, Office365, and Azure • MTC Global Alliance Partner • MCS Certified Sub-Contractor • Technical Team Led by Jeff Fried, former Senior Product Manager for Microsoft Enterprise Search products • Key Competitive Wins include ExxonMobil, Accenture, Siemens, eBay, ADP, DIA, US Army, US Navy

5. A Track Record of Success

6. BA Insight for Education Sector • Harvard JFK School of Government • 3,000 User Licenses of Longitude Search for SharePoint • Deployed in 2010 • Contact: Don Oppenheimer

7. What’s New in SharePoint 2010

8. Microsoft Enterprise Search SKUs

9. Flavors of FAST Bringing high-end search to the mainstream market

10. FAST Search Extends SharePoint Superset of capabilities; common base functionality and platform Best High-End Content Search based on FAST ESP Advanced Content Processing Content SSA FAST Search for SharePoint Common query and federation APIs Common connectors and content APIs Content search Query SSA SharePoint Server 2010 People search End Users Content Unique Social Search from SharePoint Server 2010 + Profiles Common IT Pro and developer experience

11. SharePoint 2010 and FAST search General Productivity search Light customization Social, Conversational Connectivity, Security, Relevance, Performance, Administration, Scale Customized productivity search and search driven applications Social, Conversational++, Visual, Contextual Connectivity+, Security, Relevance+, Performance, Administration+, Scale++ SharePoint Standard CAL + SharePoint server license SharePoint Enterprise CAL + FAST Search for SharePoint server license

12. SharePoint 2010 & FAST End User Perspective

13. FAST Search Interface Sorting on any property Query completion Relatedsearches & people Scrolling PPT Document thumbnails Read in Office Web Apps Federated results

14. Sorting and Deep Refinement Enables precise control of results • Enables conversational experience across all of the results • Better findability and exploration • Discover non-obvious relationships across the entire result set • Exact counts shows relative weight • Provides analytic view of your results • Indicates priority and importance • The right lever to slice and dice your content • Sort on any field • Rearrange the result set to meet specific criteria • Alphabetical, numeric, and date Out of the Box Sorting Options Exact Counts

15. SharePoint 2010 & FAST IT Pro Perspective

16. Introducing the Processing Pipeline A systematic approach to interpreting your content • Sequential stages perform specific tasks while ingesting content • Breaks down content to the smallest addressable chunks to build meaning • Understands file encoding, data formats, and written languages • Supports 400+ file formats, 80+ languages • Process your content to make it searchable • Normalizes content so that a consistent relevancy model can be applied • Identifies structured and unstructured metadata in your content • Maps document metadata to SharePoint Crawled Properties

17. Extending Pipeline capabilities Straightforward way to add custom text analysis functionality Configure Optional Processing Steps • XML Properties mapper • Offensive Content Filter • Verbatim (whole word) extractor • Use a dictionary for custom extraction • Pipeline Extensibility • Calls external applications for custom item processing • Field Collapsing • Entity Extraction Add Custom Processing Pipeline Extensibility is a specially defined stage that takes a set of crawled properties, as flat text as input and maps output to another crawled property Sandboxed execution Executable arguments and temporary files are automatically handled with timeouts. Runs just before the Crawled Property Mapper, providing accessibility within SharePoint

18. Tune relevancy Improve accuracy and control with Rank Profiles Rank Profiles are made by combining multiple ranking elements Create custom ranking algorithms to combine multiple ranking properties Rank Profiles are exposed by modifying the sorting web part. Rank Profiles created in PowerShell

19. SharePoint 2010 & FAST Developers Perspective

20. Customizing the Search Experience IP Portfolio Mgt. Intel/Surveillance Drug Discovery …. Intranet Search People Search Site Search Research Portal Case Management Save Results to Excel …..

21. Configuring Web Parts to extend UI Example: Add new Refiner to Results page • Expose custom collections (verbatim extractors) in refinement panel • Modify shared refinement web part on results page • Add new Managed Properties to existing XML file Custom Collections

22. Query Language Expressiveness Use FAST Query Language (FQL) for precise query development • FQL provides a robust and expressive query language • Wildcard support - *, ? • Numeric Data types (Integer, Float, Decimal, Datetime) • Operators • Direct field access (e.g., title:othello, author:shakespeare) • Numeric (COUNT, RANGE, <, <=, >, >=) • Boolean (AND, OR, ANY, NOT) • Rank (RANK, XRANK) • Proximity (NEAR, ONEAR) • String (operator support for strings) • Boundary (starts-with, ends-with, equals) • Filter

23. User Interface Extensibility Extend the interface by modifying web parts or creating new ones • Create custom interactive Web Parts • Bar Charts, Pie Charts, Maps, Tag Clouds • All Web Parts communicate through common interface • Create new Web Parts that can interact with others • All Web Parts are public • Extend Web Parts to change default behavior • Expand Queries and Results with SharePoint data • Use Federation Object Model to query: • SharePoint Search • FAST Search for SharePoint • Open Search (both synchronous and asynchronous) • Federation Object Model • Extend/Add Web Parts on the search page • Connect to new data sources (e.g.. Exchange) • Combine results from multiple locations.

24. Search First Migration

25. Search First Migration Project • Facilitate SP2010 migration and adoption: «Search First» • Form • Documentation and guidance (on TechNet) • Migration scripts and Web Parts (free and supported on partner site) • BAinsight.com/Resources –Free Downloads

26. Migration Scenarios covered 1) New search instance with search center in SP 2010 2) New search instance with sites/search box in MOSS 2007 3) Existing search in MOSS 2007, migrating to search in SP2010

27. Required Steps • Implement a new SP2010 Farm • Crawl SP2007 content • Set up search center • Migrate settings • Scopes • Best bets • Synonyms • Modify existing SP2007 Farm • Redirect query requests to SP2010

28. Search First Migration SP 2007 SP 2010 Web Application A Site Collection A-1 Search Center Search Center A Site Collection A-N Search Center Web Application B Site Collection B-1 Search Center Search Center B Site Collection B-N Search Center

29. Tools and Web Part provided • Migration tools for • Scopes • Best Bets • Site Collection Search Settings • Updating Web Parts for redirection to SP2010 • Profile Replication • SP 2007 Search Box Control • Displays Query Completion • Supports SP 2010 Scopes

30. Master page webparts Specific page webparts

31. Need to replicatescopes in SP2010

34. Need to update all links to advancedsearch

35. This Site and This List This Site and This List searches search from top of site

36. SharePoint Search 2010Deployment Best Practices

37. SP2010 Search Architecture • Multiple Query Servers • Partition index across multiple query servers • Mirror index for high availability • 100 mio. items with sub-second query latency • Multiple Crawlers (aka Index Servers) • Better crawl performance • Stateless, enabling failover • No copy of index on server, propagates indexed data to query servers • Multiple Property Databases • Minimize DB bottleneck known from SP2007

38. SP2010 Search Architecture Partioned and Mirrored Index with 4xQuery + 2xProp. DB A A B B 2x2 Distributed Stateless Crawlers + 2xCrawl DB Admin Component + Admin DB

39. Search Application Boundaries • Max. 20 index partitions • Max. 10 million items in each index partition • Max. 100 million items across all partitions • Max. 16 crawl components • Max. 10 crawl databases • Max. 10 property databases • Max. 64 query components • Max. 50 content sources • Max. 100 crawl rules

40. Scale-Out Decision Points

41. Scale-Out Decision Points

42. Storage Architecture Search Database File Spindle Priority

43. Hardware Requirements

44. Search System Testing • Create a test plan • Create a test environment from the architecture you have designed • Migrate your SharePoint content to the test environment for realistic testing of indexing performance. Also, setup indexing of external content if applicable • Use load test tools to simulate expected demand, e.g. Visual Studio 2010 Ultimate • Analyze results, identify bottlenecks and optimize the architecture. Retest if needed

45. Search System Monitoring • Monitor the live search system to • Identify and address performance bottlenecks • Address changes in demand like increasing size of search index and increasing query latency • Use the crawl log to verify that all content is successfully added to the index • Use performance counters to monitor CPU, Memory, Network and Disk usage. • Use search administration reports to monitor crawl rate and query latency

46. Monitoring the Crawl Rate

47. FAST Search 2010 for SPSDeployment Best Practices

48. FAST Search – Main Components SharePoint Crawler Capacity: ~30 mill items per crawler node, SQL server needs to be scaled for high IO Web Analyzer CPU/disk footprint can vary by a factor of 10 depending on the content: - number of links - length of links - internal cross link ratio Average capacity: ~30 mill items per web analyzer node Can be deployed with the Indexer in normal scenarios Indexer/search node Two supported models: - Normal mode: ~15 mill items per node ~25 QPS - High Density Mode: ~ 40 mill. items per node ~ 7 QPS

49. FAST Search HW – Best Practices CPU: 2 x 2GHz+ (Quad/six core) Memory: 24-48 GB Disk: 2 x 300 GB, SAS, 10K RPM (RAID 1) CPU: 2 x 2GHz+ (Quad/six core) Memory: 24-48 GB Disk alternatives: 1.0 TB: 8 x 300 GB, SAS, 10K RPM (RAID10) 1.8 TB: 8 x 300 GB, SAS, 10K RPM (RAID 5) 3.6 TB: 16 x 300 GB, SAS, 10K RPM (RAID 5+0) New: 7.2 TB: 16 x 600 GB, SAS, 10K RPM (RAID 5+0) SAN: Configured for “database performance”

50. FAST Search – Server Calculation Assumptions • Query rate: < 25 QPS • Intranet search with 70% items and 30% documents, source doc between 10-100Kb. • Standard deployment and configuration Disclaimer: The numbers might not be representative for the customer environment and data. Please use caution when using these numbers for sizing.