Support/Kitsune/features/search-ranking

Purpose

• Improve CTR to about 80%
• Increase the quality of results (based on heuristics)
• Increase helpfulness rating for articles when compared with users sent from Google vs SUMO Search (vs clicking an article on the home page)
• Decrease the rank of searches people click (e.g. the first, 3rd, 8th)
• Decrease the ratio of forum questions per thousand visitors

Outcome

• Reports that give us insight into what to do to get better search results
• Tools that let us manipulate the display of search results.

Open issues/risks

• Can we get the data to produce the reports we need?
• Do we need all of this, if we are already doing good?
  • First order of business: Figuring out, what our status regarding CTR and accuracy are.

Definition

Feature overview

Reports

• Aggregated top searches
• A report of the top 200 searches per week with their CTR, what the top 5 articles were for those searches (including the share of clicks per result)
  • Should also allow us to manually enter what the accuracy was for each search and calculate the overall accuracy (accuracy is estimated by a human looking at the results)
• Report on the bail rate (people who click on the "Ask a Question" button) and the searches with the highest bail rate.
  • A best bet should be regarded as a Click through
• Visualization for searches that have the lowest CTR (frequent searches with a low CTR)
• Best bet/tailored search audit report: How often was a certain best bet triggered and what was the CTR.
  • Clicking on a best bet should show you what search terms were used to trigger it.
• A report that lists words the dictionary didn't recognize by frequency (to add those non-words as synonyms)

Features

• Capture the path someone has taken before they used the search or any facets a user has selected prior to running a search.
• Add a field that is indexed by third party search. This field is NOT displayed when viewing the article. This will allow the KB authors to add phrases to the articles that will improve third party search without sacrificing the quality of the article.

We need two types of boosting related features:

1. Type A: A way to boost a whole range of articles based on certain criteria
2. Type B: A way to manipulate search results for a user/admin defined query or set of queries

Type A

• Boosting results based on article source (KB, forum, marketing?)
• Boosting results by field. (e.g. title vs. content vs. keywords ...)
• Boosting based on tags.
• Boosting trending articles/threads.
• Boosting articles/threads based on helpfulness rating.

Type B

• An interface to build query logic for tailored search: [synonym] 'stem match' "string match" AND OR NOT ()- Active and Passive
  • [synonym]: Match on the word or any synonyms including stemming
  • 'stem match': Match only the word in and stems. Do NOT match synonyms or stem matches of synonyms.
  • "string match": should match exact string. No synonym OR stem matches
  • AND OR NOT: Logical operators
  • (): To separate queries that are connected by the operators
  • Active and Passive: Active queries manipulate the search results, passive queries are only for metrics tracking.
• Topic restrictions: only show tailored searches when certain topics are active (may come from path taken or from facet selected)
• Time constraints: The option to set start and end dates/times for tailored searches.
• The tool should also let us define the results to return when each tailored search is invoked. Needs to let us define
  • only a few articles to show on top of otherwise regular results
  • a complete list of results to be shown (with nothing added after that)
• A way to change the display name or description for ranked searches
• Best bets: a special box that shows separately from the search results list with user/admin defined content. ie, links to articles, informational text, external links,etc
• A process wizard builder.

Users & use cases

Dependencies

Requirements

Non-Goals

Design

Functional Spec

What contributors should do

Things that will need to be figured out

• Which metric sources do we already have access to, and which ones do we still need?

User experience design

Implementation plan