Extending the SharePoint Search Infrastructure

1. Extending the SharePoint Search Infrastructure

2. Outline SharePoint 2010 Search Architecture The Connector Framework Enhancing the Search User Interface Creating Custom Ranking Models

3. SharePoint 2010 Search Product Line FAST and SharePoint Search form a unified productivity search line GOAL : To get the message across that FS2010 is integral part of SharePoint FS2010 = FAST search for SharePoint SS2010 = SharePoint search Enterprise search = entire array of search offering from Microsoft. Includes both SS2010 and FS2010. FS2010 is part of the SharePoint family of products Most of platform components shared Common query layer/api's/front end's Common indexing connector layer With FS2010 you get enhanced features and functionality both OOB and as a platform. GOAL : To get the message across that FS2010 is integral part of SharePoint FS2010 = FAST search for SharePoint SS2010 = SharePoint search Enterprise search = entire array of search offering from Microsoft. Includes both SS2010 and FS2010. FS2010 is part of the SharePoint family of products Most of platform components shared Common query layer/api's/front end's Common indexing connector layer With FS2010 you get enhanced features and functionality both OOB and as a platform.

4. SharePoint Search Architecture Index servers reads config information from the crawl DB Index server crawls the content and it goes through the indexing pipeline Send index + metadata to Query sever + metadata database WFE sends query to QP QP sends to QS + MD QP gets restuts back and sends to WFEIndex servers reads config information from the crawl DB Index server crawls the content and it goes through the indexing pipeline Send index + metadata to Query sever + metadata database WFE sends query to QP QP sends to QS + MD QP gets restuts back and sends to WFE

5. FAST Search for SharePointIntegration Architecture This slide shows the integration architecture of FS2010 as part of sharepoint There are two ways in which data can be indexed in FS2010 1. By using the index servers of FS2010 2. By using crawler of SS2010, which then uses the shared sharepoint conenctor to send data to fast using content API. The data does not go through any processing on SS2010 side.. All processsing is done at the FS2010 side Both sharepoint and fast shares the APIs to access the query capabilties Both sharepoint and fast uses the same interface to communicate between the WFE and QP. Make a point that new verticals can go against sharepoint and fast and OOB we use people to go against sharepoint This slide shows the integration architecture of FS2010 as part of sharepoint There are two ways in which data can be indexed in FS2010 1. By using the index servers of FS2010 2. By using crawler of SS2010, which then uses the shared sharepoint conenctor to send data to fast using content API. The data does not go through any processing on SS2010 side.. All processsing is done at the FS2010 side Both sharepoint and fast shares the APIs to access the query capabilties Both sharepoint and fast uses the same interface to communicate between the WFE and QP. Make a point that new verticals can go against sharepoint and fast and OOB we use people to go against sharepoint

6. Enterprise Search PlatformConnecting to Repositories Now we get into the details of the platform bulding blocks in ES 2010. As I mentioned before most of the platform elements are shared between SS2010 and FS2010. We got a lot of feedback about inflixibility of the search platform in o12 and specially around the ability to programmatically change the end user experience. Based on the feedback we are making all our webparts public, which means you can inherit from it. There are query and resutls hook provided at page level that cn be used to modify the query and resutls There is anew OM that can be used to send multiple queries. Public refinement om that can be used to provide refinement optinons over a resutl set. Enhanced query om and web service for fast There is a common indexing layer using the shared shaerpoint conector,. Now we get into the details of the platform bulding blocks in ES 2010. As I mentioned before most of the platform elements are shared between SS2010 and FS2010. We got a lot of feedback about inflixibility of the search platform in o12 and specially around the ability to programmatically change the end user experience. Based on the feedback we are making all our webparts public, which means you can inherit from it. There are query and resutls hook provided at page level that cn be used to modify the query and resutls There is anew OM that can be used to send multiple queries. Public refinement om that can be used to provide refinement optinons over a resutl set. Enhanced query om and web service for fast There is a common indexing layer using the shared shaerpoint conector,.

7. Enterprise Search PlatformManaging Content April 27-30, 2009 MICROSOFT CONFIDENTIAL FS2010 differentiators.. FS2010 differentiators..

8. Enterprise Search PlatformEnhancing End User Experience April 27-30, 2009 MICROSOFT CONFIDENTIAL FS2010 differentiators.. FS2010 differentiators..

9. Enterprise Search PlatformQuery Capabilities April 27-30, 2009 MICROSOFT CONFIDENTIAL FS2010 differentiators.. FS2010 differentiators..

10. Outline SharePoint 2010 Search Architecture The Connector Framework Enhancing the Search User Interface Creating Custom Ranking Models

11. The Connector Framework MOSS 2007 Indexing Protocol Handlers Business Data Catalog IDEnumerators O2010 Connectors Exchange Notes Documentum File Net Open Text Custom .NET Connector MOSS 2007 Indexing The first method called �Protocol Handler� method has existed for nearly a decade (present since old ages of 13 Site Server following into SPS2003) and provided an unmanaged COM interface that 3rd party 2010 partners/developers could write custom code into. This interface was directly plugged into the search 15 indexer making it a highly performant way to index data from custom repositories. Also, the model was low 16 level enough that any type of custom repository could be expressed to the Search System. This method is 17 known for its complexity which made it unusable by many customers. Hence Search Team has decided to 18 dramatically simply what we offer to our customers. The second method got introduced as recent as Office12. Search took advantage of a new product in Office 21 called �Business Data Catalog� (BDC) as it specialized in interacting with data in complex LOB repositories. 22 Search made use of BDC to index data from databases and web services backed repositories. There was a 23 significant payoff as the new technology enabled crawls of back-end systems by just expressing connections 24 to repositories declaratively (XML) without having to write any code. Search Team will continue to leverage 25 BDC as a way to connect to LOB data since that�s the strategy prescribed in Office Team as BDC becomes 26 front and center for interacting w/business repositories. MOSS 2007 Indexing The first method called �Protocol Handler� method has existed for nearly a decade (present since old ages of 13 Site Server following into SPS2003) and provided an unmanaged COM interface that 3rd party 2010 partners/developers could write custom code into. This interface was directly plugged into the search 15 indexer making it a highly performant way to index data from custom repositories. Also, the model was low 16 level enough that any type of custom repository could be expressed to the Search System. This method is 17 known for its complexity which made it unusable by many customers. Hence Search Team has decided to 18 dramatically simply what we offer to our customers. The second method got introduced as recent as Office12. Search took advantage of a new product in Office 21 called �Business Data Catalog� (BDC) as it specialized in interacting with data in complex LOB repositories. 22 Search made use of BDC to index data from databases and web services backed repositories. There was a 23 significant payoff as the new technology enabled crawls of back-end systems by just expressing connections 24 to repositories declaratively (XML) without having to write any code. Search Team will continue to leverage 25 BDC as a way to connect to LOB data since that�s the strategy prescribed in Office Team as BDC becomes 26 front and center for interacting w/business repositories.

12. The Connector Framework This discussion will draw on the information presented earlier in the BCS module. Will explain how entities can be searched and how this is the new model for searching external sources.This discussion will draw on the information presented earlier in the BCS module. Will explain how entities can be searched and how this is the new model for searching external sources.

13. BDC Based Connector New Features for Search in Model Files Support for attachments Item-level security Crawl across entity associations Control the resulting display URL April 27-30, 2009

14. Outline SharePoint 2010 Search Architecture The Connector Framework Enhancing the Search User Interface Creating Custom Ranking Models

15. Creating a .NET Connector demo This demo will be a walkthrough of the VS10 shim creation process. I may need Boris to do this since my VS10 doesn�t have the projectThis demo will be a walkthrough of the VS10 shim creation process. I may need Boris to do this since my VS10 doesn�t have the project

16. Outline SharePoint 2010 Search Architecture The Connector Framework Enhancing the Search User Interface Creating Custom Ranking Models

17. Customizing User Interface Extend OOB Web parts Extend any Web Part to change default behavior Add new Web Part that interacts with existing Web Parts on the search page All Web Parts communicate through public interface (SharedQueryManager object) Ability to modify query before it leaves the end user page Override the GetXPathNavigator method of core results Web Part Ability to modify the search results before the results are displayed to the users Override AggregateResults method of LocationList class

18. Search Parts demo

19. Outline SharePoint 2010 Search Architecture The Connector Framework Enhancing the Search User Interface Creating Custom Ranking Models

20. Ranking Models Admins use XML files to describe new Ranking Models Custom ranking models rank using a weighted-average of the features in the XML descriptor BM25F (Query Dependent Feature) Static features Click Features (static) Manage Ranking Models using PowerShell Use Custom Ranking Model in End user using Web Part Property Multiple OOB Ranking Models Different for Core Results and People Cannot be modified In MOSS 2010 search, there will be an out of the box default ranking model � a two-layer neural net model � and 3 possibly several other ranking models for people search and �Sort by Date� etc. We won�t provide a way to 4 customize these models directly. Instead we will provide a template ranking model that can be edited in order to 5 create a new model. In addition the following relevance features will only be supported in the default models � Edit 6 distance, minspan, document length. They will not be included in the ranking model schema that we will publish as 7 part of the Relevance OM documentation on MSDN. 8 There will be 2 options for creating ranking models: 1) Starting with our �default� list of static and dynamic 9 ingredients in the linear model by getting the template xml model and then editing and uploading the modified 10 ranking model xml file 2) Creating a new model in xml which conforms to our ranking model schema. Custom ranking 11 models will not be versioned. Any updates/changes to a model override the existing model. 12 We expect that creating a new model based on the �default� and customizing it will satisfy requirements for 13 changing weights on our OOB relevance features, similar to the O12 support for altering weights on managed 2010 properties. However, admins will need to do the extra step of associating the model to the search results Web parts 15 on Web sites where they want the new ranking model. Creation of a new xml model based on our schema will satisfy 16 requirement for specialized search applications to create a unique model targeted to their business requirements. 17 For each new release of MOSS/ MSS a new version of the ranking model schema may be created. In order to 18 maintain compatibility of custom ranking models created using previous versions of the ranking model schema, the 19 version number of the schema will be indicated using the schema version attribute. New versions of the schema will 20 retain all previous elements and attributes, although new elements and attributes may be added. In MOSS 2010 search, there will be an out of the box default ranking model � a two-layer neural net model � and 3 possibly several other ranking models for people search and �Sort by Date� etc. We won�t provide a way to 4 customize these models directly. Instead we will provide a template ranking model that can be edited in order to 5 create a new model. In addition the following relevance features will only be supported in the default models � Edit 6 distance, minspan, document length. They will not be included in the ranking model schema that we will publish as 7 part of the Relevance OM documentation on MSDN. 8 There will be 2 options for creating ranking models: 1) Starting with our �default� list of static and dynamic 9 ingredients in the linear model by getting the template xml model and then editing and uploading the modified 10 ranking model xml file 2) Creating a new model in xml which conforms to our ranking model schema. Custom ranking 11 models will not be versioned. Any updates/changes to a model override the existing model. 12 We expect that creating a new model based on the �default� and customizing it will satisfy requirements for 13 changing weights on our OOB relevance features, similar to the O12 support for altering weights on managed 2010 properties. However, admins will need to do the extra step of associating the model to the search results Web parts 15 on Web sites where they want the new ranking model. Creation of a new xml model based on our schema will satisfy 16 requirement for specialized search applications to create a unique model targeted to their business requirements. 17 For each new release of MOSS/ MSS a new version of the ranking model schema may be created. In order to 18 maintain compatibility of custom ranking models created using previous versions of the ranking model schema, the 19 version number of the schema will be indicated using the schema version attribute. New versions of the schema will 20 retain all previous elements and attributes, although new elements and attributes may be added.

21. Custom Ranking Model demo This demo will b. This will include PowerShell commandletse a custom ranking model based on the linear templateThis demo will b. This will include PowerShell commandletse a custom ranking model based on the linear template

22. Summary SharePoint 2010 Search Architecture The Connector Framework Enhancing the Search User Interface Creating Custom Ranking Models