Searchkick [alpha]

:rocket: Search made easy

Searchkick provides sensible search defaults out of the box. It handles:

• stemming - tomatoes matches tomato
• special characters - jalapenos matches jalapeños
• extra whitespace - dishwasher matches dish washer
• misspellings - zuchini matches zucchini
• custom synonyms - qtip matches cotton swab

Runs on Elasticsearch

Battle-tested at Instacart

Get Started

Install Elasticsearch.

brew install elasticsearch

Add this line to your application’s Gemfile:

gem "searchkick"

Add searchkick to models you want to search.

class Product < ActiveRecord::Base
  searchkick
end

Add data to the search index.

Product.reindex

And to query, use:

Product.search "2% Milk"

Query Like SQL

Search specific fields

Product.search "Butter", fields: [:name, :brand]

Filter queries

Product.search "2% Milk", where: {in_stock: true}, limit: 10, offset: 50

Where

where: {
  expires_at: {gt: Time.now}, # lt, gte, lte also available
  orders_count: 1..10,        # equivalent to {gte: 1, lte: 10}
  aisle_id: [25, 30],         # in
  store_id: {not: 2},         # not
  aisle_id: {not: [25, 30]},  # not in
  or: [
    [{in_stock: true}, {backordered: true}]
  ]
}

Order

order: {_score: :desc} # most relevant first - default

Explain

explain: true

Facets

Product.search "2% Milk", facets: [:store_id, :aisle_id]

Advanced

Product.search "2% Milk", facets: {store_id: {where: {in_stock: true}}}

Synonyms

class Product < ActiveRecord::Base
  searchkick synonyms: [["scallion", "green onion"], ["qtip", "cotton swab"]]
end

You must call Product.reindex after changing synonyms.

Make Searches Better Over Time

Improve results with analytics on conversions and give popular documents a little boost.

First, you must keep track of search conversions. The database works well for low volume, but feel free to use redis or another datastore.

class Search < ActiveRecord::Base
  belongs_to :product
  # fields: id, query, searched_at, converted_at, product_id
end

Add the conversions to the index.

class Product < ActiveRecord::Base
  has_many :searches

  searchkick conversions: true

  def to_indexed_json
    {
      name: name,
      conversions: searches.group("query").count.map{|query, count| {query: query, count: count} }, # TODO fix
      _boost: Math.log(orders_count) # boost more popular products a bit
    }
  end
end

After the reindex is complete (to prevent errors), tell the search method to use conversions.

Product.search "Fat Free Milk", conversions: true

Zero Downtime Changes

Product.reindex

Behind the scenes, this creates a new index products_20130714181054 and points the products alias to the new index when complete - an atomic operation :)

Searchkick uses find_in_batches to import documents. To filter documents or eagar load associations, use the searchkick_import scope.

class Product < ActiveRecord::Base
  scope :searchkick_import, where(active: true).includes(:searches)
end

There is also a rake task.

rake searchkick:reindex CLASS=Product

Thanks to Jaroslav Kalistsuk for the original implementation.

Reference

Reindex one item

product = Product.find(1)
product.update_index

Partial matches (needs better name)

Item.search "fresh honey", partial: true # matches organic honey

Migrating from Tire

1. Change search methods to tire.search and add index name in existing search calls

  Product.search "fruit"

should be replaced with

  Product.tire.search "fruit", index: "products"

1. Replace tire mapping w/ searchkick method

  searchkick index_name: "products_v2"

1. Deploy and reindex

  rake searchkick:reindex CLASS=Product # or Product.reindex in the console

1. Once it finishes, replace search calls w/ searchkick calls

Elasticsearch Gotchas

Inconsistent Scores

Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, set the search type to dfs_query_and_fetch. Alternatively, you can just use one shard with settings: {number_of_shards: 1}.

TODO

• Autocomplete
• Option to turn off fuzzy matching (should this be default?)
• Option to disable callbacks
• Exact phrase matches (in order)
• Focus on results format (load: true?)
• Test helpers - everyone should test their own search
• Built-in synonyms from WordNet
• Dashboard w/ real-time analytics?
• Suggest API "Did you mean?"
• Allow for "exact search" with quotes
• Make updates to old and new index while reindexing possibly with an another alias

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request