Toronto SharePoint User Group

1. Toronto SharePoint User Group Bill BrockbankNavantis Inc.billb@navantis.com

2. Architecture and Topology • Logical Architecture - Components • Crawl • Content Processing • Analytics Processing • Index • Administration • Query Processing • Search Process • Topologies - Examples

3. Logical Architecture

4. Crawling the Content • The crawl role is responsible for crawling content sources. It delivers crawled items – both the actual content as well as their associated metadata – to the content processing component • Invokes connectors or protocol handlers to content sources to retrieve data • Does not do any document parsing (Content Processing Component does that) • Information about content sources, schedules, etc. are synchronized to the registry on crawl role servers from the search admin database • The Crawl Database is used by the crawl component to store information about crawled items and to track crawl history • Holds information such as the last crawl time, the last crawl ID and the type of update during the last craw.

5. Content Processing • Content Processing Component • Processes crawled items and feeds these items to the index component • Document Parsing happens through Format Handlers • iFilter is supported through a Generic iFilter format handler • iFilters are still the extensibility platform for SharePoint 15 • Heart of the indexing process: transforms crawled items into artifacts that can be included in the search index by carrying out operations such as document parsing and property mapping • Performs linguistic processing at index time (e.g. language detection and entity extraction) • Writes information about links and URLs to the Link Database directly • Generates phonetic name variations for people search

6. Analytics Processing Component • Search Analytics analyzes crawled items and how users interact with search results. • Usage analytics analyzes usage events e.g. like views from the event store • When an user does an action (e.g. view a page) the event is collected in usage files on the WFE’s and regularly pushed to event store where they are stored until processed • Results are then returned to the Content Processing Component to be included in the search index • You can add write code to add custom events • Analytics Processing Component supports scaling out: • Add more APC roles to have analysis complete faster • Adding more Link databases to increase capacity for links and search clicks, and potentially speed up the DB extraction • Adding more reporting databases to store more reports as well as improving SQL throughtput in retrieveing reports

7. List of Sub Analyses • Search Analytics • Link and Anchor text analysis • Click Distance • Search Clicks • Deep Links • Social Tags • Social Distance • Search Reports • Usage Analytics • Recommendations • Usage Counts • Acivity Ranking

8. Index Component • Used in both feeding and query processes: • Feeding: receives processed items from the content processing component and writes those items to index files • Query: receives queries from the query processing component and provides results sets in return • It also physically moves around indexed content when the index architecture is changed by the Search Administration Component

9. Query Processing Component • Performs linguistic processing at query time: • Word breaking, stemming, query spellchecking, thesaurus • Analyzes and processes search queries and results. • When the component receives a query from the search front-end, it analyzes and processes the query to attempt to optimize precision, recall and relevancy. The processed query is then submitted to the index component(s). • As part of this it also decides which query rules are applicable, which index to send the query to, and whether to do any pre- or post-processing of the query • The index component returns a result set based on the processed query back to the query processing component, which in turn processes that result set before sending it back to the search front-end.

10. Search Administration • Search Admin Component • Runs number of system processes required for search • Is responsible for search provisioning and topology changes • Coordinates search components – Content Processing, Query Processing, Analytics, and Indexing. • Search Admin DB • Stores search configuration data: • Topology • Crawl rules • Query rules • Managed property mappings • Content sources • Crawl schedules • Stores Analytics settings • Does not store ACLs anymore

11. Search Processes • Host Controller • A Windows Service that supervises NodeRunner process(es) on a given box • It restarts failed NodeRunner processes: if a NodeRunnerfails, the HostController will detect this and restart it • NodeRunner.exe • Is the process that hosts the search components • There might be several instances of this process on a single box • MSSearch.exe • Is the Windows Service who hosts the Crawl Component

12. Search Host Process NodeRunner.exe NodeRunner.exe • Multiple NodeRunner instances can run on the same server • Each NodeRunner instance hosts one search component • E.g. If you have two index replicas on one server you will have two NodeRunner instances – one for each replica • On a default single server install there will be 5 instances of the NodeRunner.exe process Crawl Component Content Processing Component NodeRunner.exe NodeRunner.exe Index Component Query Processing Component NodeRunner.exe Analytic Processing Component

13. Enterprise Search – Query Configuration

14. Query Configuration • Ranking • Query Spell Correction • Query Rules

15. Ranking • Ranking model describes which criteria are included in sorting, how much they contribute to the rank score and how they relate to one another • Custom ranking models are managed through PowerShell and the public OM using XML files as it was in SharePoint 2010 • Ranking model for a specific query can be selected at query time by setting the RankingModelId of the query; otherwise default is used • NOTE: This property is currently not available in the OOB search results web part

16. Working with Ranking Models • A search service administrator can do the following basic operations on the rank models: • List ranking models • Specify a default ranking model • Change an existing custom ranking model • Delete an existing custom ranking model • Create a new ranking model (either based on an existing one or from scratch) • Import and export a ranking model to XML • Specify how much weight the various properties should have in the model • These operations are all administered through PowerShell • There may be a CodePlex GUI-based solution after RTM

17. Query Spelling Correction • Customizations to Query Spelling Correction is now managed in the term store – both inclusions and exclusions • We also still have a dynamic dictionary that is created based on content in the index, or you can switch to the static OOB dictionary • Here’s an example: • Provides customizable • “did you mean” functionality:

18. Query Rules • Query Rules are a new feature in SharePoint 15 that help act upon the “intent” of a query • Query Rules are composed of three top level elements: • Query Conditions (i.e. matching rules) • Query Actions (i.e. what do you do when you find a match) • Publishing Options (i.e. when should this rule be active) • Rules are created at the site collection level, but you will probably also be able to create them at the SSA level through PowerShell (TBD by RTM) • Several Query Rules are provided out of the box and can be found in the Site Settings for each site collection

19. Query Rules Implementation • Query Rules allow you to have search requests from a user trigger multiple queries and multiple result sets. • This differs from SharePoint 2010 where there were only simple queries – one query has one set of results. • All matching Query Rules can generate results; the query orchestration engine retrieves all the different sets of search results and organizes them for rendering to the user • The results themselves are then rendered using a new feature called Query Results display templates, which is discussed later

20. Query Rules - Conditions • Conditions allow you to define the circumstances under which your query rule will apply • You can choose to match on these use cases: • Query contains a specific word or words • Query contains a word in a specific dictionary • Query contains an action word that matches a specific phrase or term set • Query is common in a different source (like Videos Result Source) • Results include a common result type (like file type) • Advanced rule – match across a set of terms, dictionary, regular expression, etc. • You can also match if a query is against a specific Result Source or category (like a Topic Page category) • Each Query Rule can have multiple conditions

21. Query Rule Actions • There are three types of actions: • Assign a “promoted result” to the query • Promoted result is similar to Best Bets with SharePoint 2010 or Visual Best Bets with FAST Search 2010, but much more flexible • Create and display a Result Block • Result blocks specify an additional query to run and how to display results • The feature includes a full query designer so you can build and test queries before finalizing them • You can include the results above those returned by core results, or interleaved by ranking • You can also choose custom display templates instead of the default for the result • Change the query that returns the core results • Change the original query by changing the query terms, adding additional terms, apply an XRANK formula to the query, etc. • You can add multiple best bets or results blocks for each query rule

22. Query Rules Publishing Options • Publishing options for a Query Rule determine when it will be used • It works just like rules for publishing a page in a publishing site template • You decide whether the rule is active • You decide when to start and when to finish using the rule, or don’t set dates at all and it’s always used • You can configure a review date, which is basically an email reminder for you to look at the rule again • Publishing a rule is particularly useful in commerce scenarios. For example, queries that should return items you want to percolate to the top and sell during a particular period of time

23. Enterprise Search – New Search Results Display Options

24. Search UI Configuration • Result Types • Display Templates • Search Navigation • Search Refinement • Query Suggestions • Thumbnail Previews • Site Level Search Admin Summary

25. Result Types • SharePoint 15 includes a new framework for presenting search results to end users called result types • Result types have several elements that enable each type to be rendered differently:

26. Result Type Rules • You can create a variety of different rules on which to match: • Equality(= or !=), comparison (< or >), or logical (AND or ORor NOT) • You can have multiple rules as a condition • You can use managed properties in a rule • In B1R there will be a new UI for creating these rules, so what you see in the B1 bits will change quite a bit Example

27. Result Type Management • Result types are managed as follows: • Result Types can apply to an entire search service application or to a specific site or web • OOB Result Types are immutable but can be overridden with a copy • There is an inheritance hierarchy to result types in a site: the current web, plus the site collection root, plus the Search Service Application. • So if you adda new result type to some random web, the sub webs under it won’t inherit that type unless you are working in the site collection root web

28. Display Templates • Display templates contain the following characteristics: • They define the visual layout of a Result Type • A template is really just HTML • You can edit it with ANY HTML designer you want – DreamWeaver, SharePoint Designer, Visual Studio, Notepad – doesn’t matter • You add placeholders to your HTML file where managed properties should be emitted • It gives you a real WYSIWIG experience when designing templates • A set of display templates is included OOB; you can find them in the Master Pages/Display Templates/Search folder • Site collection admins can upload new display templates.

29. Result Type with Display Template Example • Here’s an example from one of the Hands On Labs that demonstrates building a custom result type and display template for .TXT file search results:

30. Result Types demo Steve Peschka Sr. Principal Consultant Microsoft Corporation

31. Search Refinement • There are two different modes for the refiner web part: search results and faceted navigation • With search results the refinement data works essentially the same as SharePoint 2010 • With faceted navigation it uses a term from the term store to filter what kind of data should be displayed (explained on next slide) • Refinement is different with SharePoint 15 in that you can define display templates to use for rendering different kinds of refinements • In SharePoint 2010 you had to write your own custom refiner

32. Faceted Navigation with Search Refiners • With Faceted Navigation, it is used in conjunction with term sets that are used for navigation • With each term you select which managed properties should be used as refiners with that term • Within the managed property you need to configure it as “Refinable” • Example: • You have term store terms Camera and Laptop • You have managed properties Megapixel Count, Colorand Manufacturer • For Cameraterm, you add refiners for Megapixel Countand Manufacturer • For Laptopterm you add refiners for Colorand Manufacturer • Another Example: • Department is the term store term, Customers and Projects could be refiners

33. Query Suggestions • Query suggestions take a big leap forward in usability. It improves on the experience in SharePoint 2010 as follows: • Your personal SharePoint activity factors into the query suggestions, i.e. you have a personal query log • It includes weighting based on sites that you have previously visited • It uses the most frequent queries across all users that “match” the search terms • The behavior of the query suggestions turns into more of a “browse and find” kind of experience • You can also add inclusion and exclusion lists for suggestions via the SSA admin pages

34. Query Suggestion Types • There are two types of query suggestions – what you see when you are entering a query, and what you see when you get results • When entering a query you will see two types of suggestions: • A list of items you have clicked on before from your personal query log • A list of items that others are typing for their queries • When you get query results back, you will get another set of suggestions • They are a list of links that you have clicked through at least twice before and match your search criteria

35. Query Suggestion Examples • Here are some examples of how query suggestions might look: • Before the Query Suggestion • Above are items based on what other users have queried and found • Bottom 3 items are ones you clicked before • After the Query Suggestion • Top 3 items are ones you clicked before

36. Thumbnail Previews • The new Office Web Apps is the engine for thumbnail previews in SharePoint 15 • The BIG WIN HERE – you can now browse through the entire document in the preview • See all pages, see animations, zoom in, scroll through the entire document • The point of this is to allow users to find the exact item they’re looking for right in search results – no more clicking a result, hitting the back button, and on and on until they find the one they’re looking for • It also addresses the two major shortcomings of thumbnails in SharePoint 2010: • It could only be used with FAST Search • It did NOT work with claims authentication  • Since there’s only one search engine in SharePoint 15 you get document previews out of the box • In a different twist, previews only work with claims authentication – it will not work with classic Windows authentication

37. Summary: New Site Search Admin Tasks • Just to summarize, here are things site admins can do to manage search themselves:

38. Online Content links: • IT Pro training for SharePoint 2013 http://technet.microsoft.com/en-us/sharepoint/fp123606 • SharePoint 2013 enterprise search Module:http://technet.microsoft.com/en-us/sharepoint/fp123606