Merged master

Andrew Kane
2 parents 6281297d 3a08814c
Showing 19 changed files with 851 additions and 402 deletions Show diff stats
CHANGELOG.md
Gemfile
README.md
lib/searchkick.rb
lib/searchkick/model.rb
lib/searchkick/query.rb
lib/searchkick/reindex.rb
lib/searchkick/results.rb
lib/searchkick/search.rb
lib/searchkick/version.rb
searchkick.gemspec
test/autocomplete_test.rb
test/boost_test.rb
test/index_test.rb
test/inheritance_test.rb
test/match_test.rb
test/query_test.rb
test/sql_test.rb
test/test_helper.rb
+## 0.5.3
+
+- Fixed bug w/ word_* queries
+
+## 0.5.2
+
+- Use after_commit hook for ActiveRecord to prevent data inconsistencies
+
+## 0.5.1
+
+- Replaced stop words with common terms query
+- Added language option
+- Fixed bug with empty array in where clause
+- Fixed bug with MongoDB integer _id
+- Fixed reindex bug when callbacks disabled
+
+## 0.5.0
+
+- Better control over partial matches
+- Added merge_mappings option
+- Added batch_size option
+- Fixed bug with nil where clauses
+
 ## 0.4.2
 - Added `should_index?` method to control which records are indexed
@@ -3,5 +3,7 @@ source &#39;https://rubygems.org&#39;
 # Specify your gem's dependencies in searchkick.gemspec
 gemspec
+gem "sqlite3"
+gem "activerecord"
 # gem "activerecord", "~> 3.2.0"
 # gem "activerecord", "~> 3.1.0"
@@ -21,12 +21,14 @@ Plus:
 - “Did you mean” suggestions
 - works with ActiveRecord and Mongoid
-:zap: Even better with [Searchjoy](http://ankane.github.io/searchjoy/)
-
 :tangerine: Battle-tested at [Instacart](https://www.instacart.com)
 [![Build Status](https://travis-ci.org/ankane/searchkick.png?branch=master)](https://travis-ci.org/ankane/searchkick)
+We highly recommend tracking queries and conversions
+
+:zap: [Searchjoy](http://ankane.github.io/searchjoy/) makes it easy
+
 ## Get Started
 [Install Elasticsearch](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html). For Homebrew, use:
@@ -148,6 +150,44 @@ To change this, use:
 Product.search "fresh honey", partial: true # fresh OR honey
 ```
+By default, results must match the entire word - `back` will not match `backpack`. You can change this behavior with:
+
+```ruby
+class Product < ActiveRecord::Base
+  searchkick word_start: [:name]
+end
+```
+
+And to search:
+
+```ruby
+Product.search "back", fields: [{name: :word_start}]
+```
+
+Available options are:
+
+```ruby
+:word # default
+:word_start
+:word_middle
+:word_end
+:text_start
+:text_middle
+:text_end
+```
+
+### Language
+
+Searchkick defaults to English for stemming.  To change this, use:
+
+```ruby
+class Product < ActiveRecord::Base
+  searchkick language: "German"
+end
+```
+
+[See the list of languages](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-snowball-tokenfilter.html)
+
 ### Synonyms
 ```ruby
@@ -193,7 +233,7 @@ Searchkick uses `find_in_batches` to import documents.  To eager load associatio
 ```ruby
 class Product < ActiveRecord::Base
-  scope :search_import, includes(:searches)
+  scope :search_import, -> { includes(:searches) }
 end
 ```
@@ -293,14 +333,14 @@ First, specify which fields use this feature.  This is necessary since autocompl
 ```ruby
 class City < ActiveRecord::Base
-  searchkick autocomplete: ["name"]
+  searchkick text_start: [:name]
 end
 ```
 Reindex and search with:
 ```ruby
-City.search "san fr", autocomplete: true
+City.search "san fr", fields: [{name: :text_start}]
 ```
 Typically, you want to use a Javascript library like [typeahead.js](http://twitter.github.io/typeahead.js/) or [jQuery UI](http://jqueryui.com/autocomplete/).
@@ -314,7 +354,7 @@ First, add a controller action.
 class CitiesController < ApplicationController
   def autocomplete
-    render json: City.search(params[:query], autocomplete: true, limit: 10).map(&:name)
+    render json: City.search(params[:query], fields: [{name: :text_start}], limit: 10).map(&:name)
   end
 end
@@ -454,8 +494,9 @@ Dog.reindex # equivalent
 And to search, use:
 ```ruby
-Animal.search "*" # all animals
-Dog.search "*"    # just dogs
+Animal.search "*"                   # all animals
+Dog.search "*"                      # just dogs
+Animal.search "*", type: [Dog, Cat] # just cats and dogs [master]
 ```
 **Note:** The `suggest` option retrieves suggestions from the parent at the moment.
@@ -530,6 +571,24 @@ And use the `query` option to search:
 Product.search query: {match: {name: "milk"}}
 ```
+To keep the mappings and settings generated by Searchkick, use:
+
+```ruby
+class Product < ActiveRecord::Base
+  searchkick merge_mappings: true, mappings: {...}
+end
+```
+
+## Experimental [master]
+
+Modify the query generated by Searchkick.
+
+```ruby
+query = Product.search "2% Milk", execute: false
+query.body[:query] = {match_all: {}}
+products = query.execute
+```
+
 ## Reference
 Searchkick requires Elasticsearch `0.90.0` or higher.
@@ -574,9 +633,9 @@ end
 or temporarily
 ```ruby
-Product.disable_search_callbacks # use Searchkick.disable_callbacks for all models
+Product.disable_search_callbacks # or use Searchkick.disable_callbacks for all models
 ExpensiveProductsTask.execute
-Product.enable_search_callbacks
+Product.enable_search_callbacks # or use Searchkick.enable_callbacks for all models
 Product.reindex
 ```
@@ -601,6 +660,50 @@ class Product &lt; ActiveRecord::Base
 end
 ```
+Change import batch size
+
+```ruby
+class Product < ActiveRecord::Base
+  searchkick batch_size: 200 # defaults to 1000
+end
+```
+
+Asynchronous reindexing
+
+```ruby
+class Product < ActiveRecord::Base
+  searchkick callbacks: false
+
+  # add the callbacks manually
+
+  # ActiveRecord - one callback
+  after_commit :reindex_async
+
+  # Mongoid - two callbacks
+  after_save :reindex_async
+  after_destroy :reindex_async
+
+  def reindex_async
+    # delayed job
+    delay.reindex
+  end
+end
+```
+
+Reindex conditionally
+
+**Note:** With ActiveRecord, use this feature with caution - [transaction rollbacks can cause data inconstencies](https://github.com/elasticsearch/elasticsearch-rails/blob/master/elasticsearch-model/README.md#custom-callbacks)
+
+```ruby
+class Product < ActiveRecord::Base
+  searchkick callbacks: false
+
+  # add the callbacks manually
+  after_save :reindex, if: proc{|model| model.name_changed? } # use your own condition
+  after_destroy :reindex
+end
+```
+
 Reindex all models (Rails only)
 ```sh
@@ -655,11 +758,14 @@ For convenience, this is set by default in the test environment.
 Thanks to Karel Minarik for [Tire](https://github.com/karmi/tire), Jaroslav Kalistsuk for [zero downtime reindexing](https://gist.github.com/jarosan/3124884), and Alex Leschenko for [Elasticsearch autocomplete](https://github.com/leschenko/elasticsearch_autocomplete).
-## TODO
+## Roadmap
-- Generate autocomplete predictions from past search queries
-- Automatic failover
-- Make Searchkick work with any language
+- Search multiple fields for different terms
+- Search across models
+- Search nested objects
+- Add section on testing
+- Much finer customization
+- More transparency into generated queries (for advanced use)
 ## History
@@ -673,3 +779,11 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
 - Fix bugs and [submit pull requests](https://github.com/ankane/searchkick/pulls)
 - Write, clarify, or fix documentation
 - Suggest or add new features
+
+To get started with development and testing:
+
+1. Clone the repo
+2. Install PostgreSQL and create a database called `searchkick_test` (`psql -d postgres -c "create database searchkick_test"`)
+3. Install Elasticsearch
+4. `bundle`
+5. `rake test`
@@ -3,6 +3,7 @@ require &quot;searchkick/version&quot;
 require "searchkick/index"
 require "searchkick/reindex"
 require "searchkick/results"
+require "searchkick/query"
 require "searchkick/search"
 require "searchkick/similar"
 require "searchkick/model"
@@ -19,8 +19,12 @@ module Searchkick
         extend Searchkick::Reindex
         include Searchkick::Similar
-        after_save :reindex
-        after_destroy :reindex
+        if respond_to?(:after_commit)
+          after_commit :reindex, if: proc{ self.class.search_callbacks? }
+        else
+          after_save :reindex, if: proc{ self.class.search_callbacks? }
+          after_destroy :reindex, if: proc{ self.class.search_callbacks? }
+        end
         def self.enable_search_callbacks
           class_variable_set :@@searchkick_callbacks, true
@@ -39,13 +43,11 @@ module Searchkick
         end
         def reindex
-          if self.class.search_callbacks?
-            index = self.class.searchkick_index
-            if destroyed? or !should_index?
-              index.remove self
-            else
-              index.store self
-            end
+          index = self.class.searchkick_index
+          if destroyed? or !should_index?
+            index.remove self
+          else
+            index.store self
           end
         end
@@ -60,7 +62,9 @@ module Searchkick
           source = source.inject({}){|memo,(k,v)| memo[k.to_s] = v; memo}
           # Mongoid 4 hack
-          source["_id"] = source["_id"].to_s if source["_id"]
+          if defined?(BSON::ObjectId) and source["_id"].is_a?(BSON::ObjectId)
+            source["_id"] = source["_id"].to_s
+          end
           options = self.class.searchkick_options
@@ -0,0 +1,429 @@
+module Searchkick
+  class Query
+    attr_reader :klass, :term, :options
+    attr_accessor :body
+
+    def initialize(klass, term, options = {})
+      if term.is_a?(Hash)
+        options = term
+        term = nil
+      else
+        term = term.to_s
+      end
+
+      @klass = klass
+      @term = term
+      @options = options
+
+      fields =
+        if options[:fields]
+          if options[:autocomplete]
+            options[:fields].map{|f| "#{f}.autocomplete" }
+          else
+            options[:fields].map do |value|
+              k, v = value.is_a?(Hash) ? value.to_a.first : [value, :word]
+              "#{k}.#{v == :word ? "analyzed" : v}"
+            end
+          end
+        else
+          if options[:autocomplete]
+            (searchkick_options[:autocomplete] || []).map{|f| "#{f}.autocomplete" }
+          else
+            ["_all"]
+          end
+        end
+
+      operator = options[:operator] || (options[:partial] ? "or" : "and")
+
+      # model and eagar loading
+      load = options[:load].nil? ? true : options[:load]
+      load = (options[:include] ? {include: options[:include]} : true) if load
+
+      # pagination
+      page = [options[:page].to_i, 1].max
+      per_page = (options[:limit] || options[:per_page] || 100000).to_i
+      offset = options[:offset] || (page - 1) * per_page
+      index_name = options[:index_name] || searchkick_index.name
+
+      conversions_field = searchkick_options[:conversions]
+      personalize_field = searchkick_options[:personalize]
+
+      all = term == "*"
+
+      if options[:query]
+        payload = options[:query]
+      elsif options[:similar]
+        payload = {
+          more_like_this: {
+            fields: fields,
+            like_text: term,
+            min_doc_freq: 1,
+            min_term_freq: 1,
+            analyzer: "searchkick_search2"
+          }
+        }
+      elsif all
+        payload = {
+          match_all: {}
+        }
+      else
+        if options[:autocomplete]
+          payload = {
+            multi_match: {
+              fields: fields,
+              query: term,
+              analyzer: "searchkick_autocomplete_search"
+            }
+          }
+        else
+          queries = []
+          fields.each do |field|
+            if field == "_all" or field.end_with?(".analyzed")
+              shared_options = {
+                fields: [field],
+                query: term,
+                use_dis_max: false,
+                operator: operator,
+                cutoff_frequency: 0.001
+              }
+              queries.concat [
+                {multi_match: shared_options.merge(boost: 10, analyzer: "searchkick_search")},
+                {multi_match: shared_options.merge(boost: 10, analyzer: "searchkick_search2")}
+              ]
+              if options[:misspellings] != false
+                distance = (options[:misspellings].is_a?(Hash) && options[:misspellings][:distance]) || 1
+                queries.concat [
+                  {multi_match: shared_options.merge(fuzziness: distance, max_expansions: 3, analyzer: "searchkick_search")},
+                  {multi_match: shared_options.merge(fuzziness: distance, max_expansions: 3, analyzer: "searchkick_search2")}
+                ]
+              end
+            else
+              analyzer = field.match(/\.word_(start|middle|end)\z/) ? "searchkick_word_search" : "searchkick_autocomplete_search"
+              queries << {
+                multi_match: {
+                  fields: [field],
+                  query: term,
+                  analyzer: analyzer
+                }
+              }
+            end
+          end
+
+          payload = {
+            dis_max: {
+              queries: queries
+            }
+          }
+        end
+
+        if conversions_field and options[:conversions] != false
+          # wrap payload in a bool query
+          payload = {
+            bool: {
+              must: payload,
+              should: {
+                nested: {
+                  path: conversions_field,
+                  score_mode: "total",
+                  query: {
+                    custom_score: {
+                      query: {
+                        match: {
+                          query: term
+                        }
+                      },
+                      script: "doc['count'].value"
+                    }
+                  }
+                }
+              }
+            }
+          }
+        end
+      end
+
+      custom_filters = []
+
+      if options[:boost]
+        custom_filters << {
+          filter: {
+            exists: {
+              field: options[:boost]
+            }
+          },
+          script: "log(doc['#{options[:boost]}'].value + 2.718281828)"
+        }
+      end
+
+      if options[:user_id] and personalize_field
+        custom_filters << {
+          filter: {
+            term: {
+              personalize_field => options[:user_id]
+            }
+          },
+          boost: 100
+        }
+      end
+
+      if options[:personalize]
+        custom_filters << {
+          filter: {
+            term: options[:personalize]
+          },
+          boost: 100
+        }
+      end
+
+      if custom_filters.any?
+        payload = {
+          custom_filters_score: {
+            query: payload,
+            filters: custom_filters,
+            score_mode: "total"
+          }
+        }
+      end
+
+      payload = {
+        query: payload,
+        size: per_page,
+        from: offset
+      }
+      payload[:explain] = options[:explain] if options[:explain]
+
+      # order
+      if options[:order]
+        order = options[:order].is_a?(Enumerable) ? options[:order] : {options[:order] => :asc}
+        payload[:sort] = Hash[ order.map{|k, v| [k.to_s == "id" ? :_id : k, v] } ]
+      end
+
+      # filters
+      filters = where_filters(options[:where])
+      if filters.any?
+        payload[:filter] = {
+          and: filters
+        }
+      end
+
+      # facets
+      facet_limits = {}
+      if options[:facets]
+        facets = options[:facets] || {}
+        if facets.is_a?(Array) # convert to more advanced syntax
+          facets = Hash[ facets.map{|f| [f, {}] } ]
+        end
+
+        payload[:facets] = {}
+        facets.each do |field, facet_options|
+          # ask for extra facets due to
+          # https://github.com/elasticsearch/elasticsearch/issues/1305
+
+          if facet_options[:ranges]
+            payload[:facets][field] = {
+              range: {
+                field.to_sym => facet_options[:ranges]
+              }
+            }
+          else
+            payload[:facets][field] = {
+              terms: {
+                field: field,
+                size: facet_options[:limit] ? facet_options[:limit] + 150 : 100000
+              }
+            }
+          end
+
+          facet_limits[field] = facet_options[:limit] if facet_options[:limit]
+
+          # offset is not possible
+          # http://elasticsearch-users.115913.n3.nabble.com/Is-pagination-possible-in-termsStatsFacet-td3422943.html
+
+          facet_filters = where_filters(facet_options[:where])
+          if facet_filters.any?
+            payload[:facets][field][:facet_filter] = {
+              and: {
+                filters: facet_filters
+              }
+            }
+          end
+        end
+      end
+
+      # suggestions
+      if options[:suggest]
+        suggest_fields = (searchkick_options[:suggest] || []).map(&:to_s)
+        # intersection
+        suggest_fields = suggest_fields & options[:fields].map(&:to_s) if options[:fields]
+        if suggest_fields.any?
+          payload[:suggest] = {text: term}
+          suggest_fields.each do |field|
+            payload[:suggest][field] = {
+              phrase: {
+                field: "#{field}.suggest"
+              }
+            }
+          end
+        end
+      end
+
+      # highlight
+      if options[:highlight]
+        payload[:highlight] = {
+          fields: Hash[ fields.map{|f| [f, {}] } ]
+        }
+        if options[:highlight].is_a?(Hash) and tag = options[:highlight][:tag]
+          payload[:highlight][:pre_tags] = [tag]
+          payload[:highlight][:post_tags] = [tag.to_s.gsub(/\A</, "</")]
+        end
+      end
+
+      # An empty array will cause only the _id and _type for each hit to be returned
+      # http://www.elasticsearch.org/guide/reference/api/search/fields/
+      payload[:fields] = [] if load
+
+      tire_options = {load: load, size: per_page, from: offset}
+      if options[:type] or klass != searchkick_klass
+        tire_options[:type] = [options[:type] || klass].flatten.map(&:document_type)
+      end
+
+      @body = payload
+      @facet_limits = facet_limits
+    end
+
+    def searchkick_index
+      klass.searchkick_index
+    end
+
+    def searchkick_options
+      klass.searchkick_options
+    end
+
+    def searchkick_klass
+      klass.searchkick_klass
+    end
+
+    def document_type
+      klass.document_type
+    end
+
+    def execute
+      begin
+        response = Searchkick.client.search(index: searchkick_index.name, body: body)
+      rescue => e # TODO rescue type
+        status_code = e.message[0..3].to_i
+        if status_code == 404
+          raise "Index missing - run #{searchkick_klass.name}.reindex"
+        elsif status_code == 500 and (e.message.include?("IllegalArgumentException[minimumSimilarity >= 1]") or e.message.include?("No query registered for [multi_match]"))
+          raise "Upgrade Elasticsearch to 0.90.0 or greater"
+        else
+          raise e
+        end
+      end
+
+      # apply facet limit in client due to
+      # https://github.com/elasticsearch/elasticsearch/issues/1305
+      @facet_limits.each do |field, limit|
+        field = field.to_s
+        facet = response["facets"][field]
+        response["facets"][field]["terms"] = facet["terms"].first(limit)
+        response["facets"][field]["other"] = facet["total"] - facet["terms"].sum{|term| term["count"] }
+      end
+
+      Searchkick::Results.new(response, @search.options.merge(term: term, model_name: searchkick_klass.model_name))
+    end
+
+    private
+
+    def where_filters(where)
+      filters = []
+      (where || {}).each do |field, value|
+        field = :_id if field.to_s == "id"
+
+        if field == :or
+          value.each do |or_clause|
+            filters << {or: or_clause.map{|or_statement| {and: where_filters(or_statement)} }}
+          end
+        else
+          # expand ranges
+          if value.is_a?(Range)
+            value = {gte: value.first, (value.exclude_end? ? :lt : :lte) => value.last}
+          end
+
+          if value.is_a?(Array)
+            value = {in: value}
+          end
+
+          if value.is_a?(Hash)
+            value.each do |op, op_value|
+              case op
+              when :within, :bottom_right
+                # do nothing
+              when :near
+                filters << {
+                  geo_distance: {
+                    field => op_value.map(&:to_f).reverse,
+                    distance: value[:within] || "50mi"
+                  }
+                }
+              when :top_left
+                filters << {
+                  geo_bounding_box: {
+                    field => {
+                      top_left: op_value.map(&:to_f).reverse,
+                      bottom_right: value[:bottom_right].map(&:to_f).reverse
+                    }
+                  }
+                }
+              when :not # not equal
+                filters << {not: term_filters(field, op_value)}
+              when :all
+                filters << {terms: {field => op_value, execution: "and"}}
+              when :in
+                filters << term_filters(field, op_value)
+              else
+                range_query =
+                  case op
+                  when :gt
+                    {from: op_value, include_lower: false}
+                  when :gte
+                    {from: op_value, include_lower: true}
+                  when :lt
+                    {to: op_value, include_upper: false}
+                  when :lte
+                    {to: op_value, include_upper: true}
+                  else
+                    raise "Unknown where operator"
+                  end
+                # issue 132
+                if existing = filters.find{ |f| f[:range] && f[:range][field] }
+                  existing[:range][field].merge!(range_query)
+                else
+                  filters << {range: {field => range_query}}
+                end
+              end
+            end
+          else
+            filters << term_filters(field, value)
+          end
+        end
+      end
+      filters
+    end
+
+    def term_filters(field, value)
+      if value.is_a?(Array) # in query
+        if value.any?
+          {or: value.map{|v| term_filters(field, v) }}
+        else
+          {terms: {field => value}} # match nothing
+        end
+      elsif value.nil?
+        {missing: {"field" => field, existence: true, null_value: true}}
+      else
+        {term: {field => value}}
+      end
+    end
+
+  end
+end
 \ No newline at end of file
@@ -44,17 +44,20 @@ module Searchkick
     end
     def self.extended(klass)
-      (@descendents ||= []) << klass
+      @descendents ||= []
+      @descendents << klass unless @descendents.include?(klass)
     end
     private
     def searchkick_import(index)
+      batch_size = searchkick_options[:batch_size] || 1000
+
       # use scope for import
       scope = searchkick_klass
       scope = scope.search_import if scope.respond_to?(:search_import)
       if scope.respond_to?(:find_in_batches)
-        scope.find_in_batches do |batch|
+        scope.find_in_batches batch_size: batch_size do |batch|
           index.import batch.select{|item| item.should_index? }
         end
       else
@@ -63,7 +66,7 @@ module Searchkick
         items = []
         scope.all.each do |item|
           items << item if item.should_index?
-          if items.length % 1000 == 0
+          if items.length % batch_size == 0
             index.import items
             items = []
           end
@@ -75,7 +78,7 @@ module Searchkick
     def searchkick_index_options
       options = searchkick_options
-      if options[:mappings]
+      if options[:mappings] and !options[:merge_mappings]
         settings = options[:settings] || {}
         mappings = options[:mappings]
       else
@@ -85,24 +88,24 @@ module Searchkick
               searchkick_keyword: {
                 type: "custom",
                 tokenizer: "keyword",
-                filter: ["lowercase", "snowball"]
+                filter: ["lowercase", "searchkick_stemmer"]
               },
               default_index: {
                 type: "custom",
                 tokenizer: "standard",
                 # synonym should come last, after stemming and shingle
-                # shingle must come before snowball
-                filter: ["standard", "lowercase", "asciifolding", "stop", "searchkick_index_shingle", "snowball"]
+                # shingle must come before searchkick_stemmer
+                filter: ["standard", "lowercase", "asciifolding", "searchkick_index_shingle", "searchkick_stemmer"]
               },
               searchkick_search: {
                 type: "custom",
                 tokenizer: "standard",
-                filter: ["standard", "lowercase", "asciifolding", "stop", "searchkick_search_shingle", "snowball"]
+                filter: ["standard", "lowercase", "asciifolding", "searchkick_search_shingle", "searchkick_stemmer"]
               },
               searchkick_search2: {
                 type: "custom",
                 tokenizer: "standard",
-                filter: ["standard", "lowercase", "asciifolding", "stop", "snowball"]
+                filter: ["standard", "lowercase", "asciifolding", "searchkick_stemmer"]
               },
               # https://github.com/leschenko/elasticsearch_autocomplete/blob/master/lib/elasticsearch_autocomplete/analyzers.rb
               searchkick_autocomplete_index: {
@@ -115,10 +118,50 @@ module Searchkick
                 tokenizer: "keyword",
                 filter: ["lowercase", "asciifolding"]
               },
+              searchkick_word_search: {
+                type: "custom",
+                tokenizer: "standard",
+                filter: ["lowercase", "asciifolding"]
+              },
               searchkick_suggest_index: {
                 type: "custom",
                 tokenizer: "standard",
                 filter: ["lowercase", "asciifolding", "searchkick_suggest_shingle"]
+              },
+              searchkick_suggest_index: {
+                type: "custom",
+                tokenizer: "standard",
+                filter: ["lowercase", "asciifolding", "searchkick_suggest_shingle"]
+              },
+              searchkick_text_start_index: {
+                type: "custom",
+                tokenizer: "keyword",
+                filter: ["lowercase", "asciifolding", "searchkick_edge_ngram"]
+              },
+              searchkick_text_middle_index: {
+                type: "custom",
+                tokenizer: "keyword",
+                filter: ["lowercase", "asciifolding", "searchkick_ngram"]
+              },
+              searchkick_text_end_index: {
+                type: "custom",
+                tokenizer: "keyword",
+                filter: ["lowercase", "asciifolding", "reverse", "searchkick_edge_ngram", "reverse"]
+              },
+              searchkick_word_start_index: {
+                type: "custom",
+                tokenizer: "standard",
+                filter: ["lowercase", "asciifolding", "searchkick_edge_ngram"]
+              },
+              searchkick_word_middle_index: {
+                type: "custom",
+                tokenizer: "standard",
+                filter: ["lowercase", "asciifolding", "searchkick_ngram"]
+              },
+              searchkick_word_end_index: {
+                type: "custom",
+                tokenizer: "standard",
+                filter: ["lowercase", "asciifolding", "reverse", "searchkick_edge_ngram", "reverse"]
               }
             },
             filter: {
@@ -136,6 +179,20 @@ module Searchkick
               searchkick_suggest_shingle: {
                 type: "shingle",
                 max_shingle_size: 5
+              },
+              searchkick_edge_ngram: {
+                type: "edgeNGram",
+                min_gram: 1,
+                max_gram: 50
+              },
+              searchkick_ngram: {
+                type: "nGram",
+                min_gram: 1,
+                max_gram: 50
+              },
+              searchkick_stemmer: {
+                type: "snowball",
+                language: options[:language] || "English"
               }
             },
             tokenizer: {
@@ -152,7 +209,7 @@ module Searchkick
           settings.merge!(number_of_shards: 1, number_of_replicas: 0)
         end
-        settings.merge!(options[:settings] || {})
+        settings.deep_merge!(options[:settings] || {})
         # synonyms
         synonyms = options[:synonyms] || []
@@ -193,10 +250,12 @@ module Searchkick
           }
         end
-        # autocomplete and suggest
-        autocomplete = (options[:autocomplete] || []).map(&:to_s)
-        suggest = (options[:suggest] || []).map(&:to_s)
-        (autocomplete + suggest).uniq.each do |field|
+        mapping_options = Hash[
+          [:autocomplete, :suggest, :text_start, :text_middle, :text_end, :word_start, :word_middle, :word_end]
+            .map{|type| [type, (options[type] || []).map(&:to_s)] }
+        ]
+
+        mapping_options.values.flatten.uniq.each do |field|
           field_mapping = {
             type: "multi_field",
             fields: {
@@ -206,12 +265,13 @@ module Searchkick
               # http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-highlighting.html#_fast_vector_highlighter
             }
           }
-          if autocomplete.include?(field)
-            field_mapping[:fields]["autocomplete"] = {type: "string", index: "analyzed", analyzer: "searchkick_autocomplete_index"}
-          end
-          if suggest.include?(field)
-            field_mapping[:fields]["suggest"] = {type: "string", index: "analyzed", analyzer: "searchkick_suggest_index"}
+
+          mapping_options.each do |type, fields|
+            if fields.include?(field)
+              field_mapping[:fields][type] = {type: "string", index: "analyzed", analyzer: "searchkick_#{type}_index"}
+            end
           end
+
           mapping[field] = field_mapping
         end
@@ -246,7 +306,7 @@ module Searchkick
               }
             ]
           }
-        }
+        }.deep_merge(options[:mappings] || {})
       end
       {
@@ -23,5 +23,10 @@ module Searchkick
     def __find_records_by_ids(klass, ids)
       @options[:load] === true ? klass.find(ids) : klass.includes(@options[:load][:include]).find(ids)
     end
+
+    def model_name
+      @options[:model_name]
+    end
+
   end
 end
@@ -2,355 +2,12 @@ module Searchkick
   module Search
     def search(term, options = {})
-      if term.is_a?(Hash)
-        options = term
-        term = nil
+      query = Searchkick::Query.new(self, term, options)
+      if options[:execute] == false
+        query
       else
-        term = term.to_s
+        query.execute
       end
-
-      fields =
-        if options[:fields]
-          if options[:autocomplete]
-            options[:fields].map{|f| "#{f}.autocomplete" }
-          else
-            options[:fields].map{|f| "#{f}.analyzed" }
-          end
-        else
-          if options[:autocomplete]
-            (searchkick_options[:autocomplete] || []).map{|f| "#{f}.autocomplete" }
-          else
-            ["_all"]
-          end
-        end
-
-      operator = options[:partial] ? "or" : "and"
-
-      # model and eagar loading
-      load = options[:load].nil? ? true : options[:load]
-      load = (options[:include] ? {include: options[:include]} : true) if load
-
-      # pagination
-      page = [options[:page].to_i, 1].max
-      per_page = (options[:limit] || options[:per_page] || 100000).to_i
-      offset = options[:offset] || (page - 1) * per_page
-      index_name = options[:index_name] || searchkick_index.name
-
-      conversions_field = searchkick_options[:conversions]
-      personalize_field = searchkick_options[:personalize]
-
-      all = term == "*"
-
-      if options[:query]
-        payload = options[:query]
-      elsif options[:similar]
-        payload = {
-          more_like_this: {
-            fields: fields,
-            like_text: term,
-            min_doc_freq: 1,
-            min_term_freq: 1,
-            analyzer: "searchkick_search2"
-          }
-        }
-      elsif all
-        payload = {
-          match_all: {}
-        }
-      else
-        if options[:autocomplete]
-          payload = {
-            multi_match: {
-              fields: fields,
-              query: term,
-              analyzer: "searchkick_autocomplete_search"
-            }
-          }
-        else
-          shared_options = {
-            fields: fields,
-            query: term,
-            use_dis_max: false,
-            operator: operator
-          }
-          queries = [
-            {multi_match: shared_options.merge(boost: 10, analyzer: "searchkick_search")},
-            {multi_match: shared_options.merge(boost: 10, analyzer: "searchkick_search2")}
-          ]
-          if options[:misspellings] != false
-            distance = (options[:misspellings].is_a?(Hash) && options[:misspellings][:distance]) || 1
-            queries.concat [
-              {multi_match: shared_options.merge(fuzziness: distance, max_expansions: 3, analyzer: "searchkick_search")},
-              {multi_match: shared_options.merge(fuzziness: distance, max_expansions: 3, analyzer: "searchkick_search2")}
-            ]
-          end
-          payload = {
-            dis_max: {
-              queries: queries
-            }
-          }
-        end
-
-        if conversions_field and options[:conversions] != false
-          # wrap payload in a bool query
-          payload = {
-            bool: {
-              must: payload,
-              should: {
-                nested: {
-                  path: conversions_field,
-                  score_mode: "total",
-                  query: {
-                    custom_score: {
-                      query: {
-                        match: {
-                          query: term
-                        }
-                      },
-                      script: "doc['count'].value"
-                    }
-                  }
-                }
-              }
-            }
-          }
-        end
-      end
-
-      custom_filters = []
-
-      if options[:boost]
-        custom_filters << {
-          filter: {
-            exists: {
-              field: options[:boost]
-            }
-          },
-          script: "log(doc['#{options[:boost]}'].value + 2.718281828)"
-        }
-      end
-
-      if options[:user_id] and personalize_field
-        custom_filters << {
-          filter: {
-            term: {
-              personalize_field => options[:user_id]
-            }
-          },
-          boost: 100
-        }
-      end
-
-      if custom_filters.any?
-        payload = {
-          custom_filters_score: {
-            query: payload,
-            filters: custom_filters,
-            score_mode: "total"
-          }
-        }
-      end
-
-      payload = {
-        query: payload,
-        size: per_page,
-        from: offset
-      }
-      payload[:explain] = options[:explain] if options[:explain]
-
-      # order
-      if options[:order]
-        order = options[:order].is_a?(Enumerable) ? options[:order] : {options[:order] => :asc}
-        payload[:sort] = order
-      end
-
-      # where
-      where_filters =
-        proc do |where|
-          filters = []
-          (where || {}).each do |field, value|
-            field = :_id if field.to_s == "id"
-
-            if field == :or
-              value.each do |or_clause|
-                filters << {or: or_clause.map{|or_statement| {and: where_filters.call(or_statement)} }}
-              end
-            else
-              # expand ranges
-              if value.is_a?(Range)
-                value = {gte: value.first, (value.exclude_end? ? :lt : :lte) => value.last}
-              end
-
-              if value.is_a?(Array) # in query
-                filters << {terms: {field => value}}
-              elsif value.is_a?(Hash)
-                if value[:near]
-                  filters << {
-                    geo_distance: {
-                      field => value.delete(:near).map(&:to_f).reverse,
-                      distance: value.delete(:within) || "50mi"
-                    }
-                  }
-                end
-
-                if value[:top_left]
-                  filters << {
-                    geo_bounding_box: {
-                      field => {
-                        top_left: value.delete(:top_left).map(&:to_f).reverse,
-                        bottom_right: value.delete(:bottom_right).map(&:to_f).reverse
-                      }
-                    }
-                  }
-                end
-
-                value.each do |op, op_value|
-                  if op == :not # not equal
-                    if op_value.is_a?(Array)
-                      filters << {not: {terms: {field => op_value}}}
-                    else
-                      filters << {not: {term: {field => op_value}}}
-                    end
-                  elsif op == :all
-                    filters << {terms: {field => op_value, execution: "and"}}
-                  else
-                    range_query =
-                      case op
-                      when :gt
-                        {from: op_value, include_lower: false}
-                      when :gte
-                        {from: op_value, include_lower: true}
-                      when :lt
-                        {to: op_value, include_upper: false}
-                      when :lte
-                        {to: op_value, include_upper: true}
-                      else
-                        raise "Unknown where operator"
-                      end
-                    filters << {range: {field => range_query}}
-                  end
-                end
-              else
-                if value.nil?
-                  filters << {missing: {"field" => field, existence: true, null_value: true}}
-                else
-                  filters << {term: {field => value}}
-                end
-              end
-            end
-          end
-          filters
-        end
-
-      # filters
-      filters = where_filters.call(options[:where])
-      if filters.any?
-        payload[:filter] = {
-          and: filters
-        }
-      end
-
-      # facets
-      facet_limits = {}
-      if options[:facets]
-        facets = options[:facets] || {}
-        if facets.is_a?(Array) # convert to more advanced syntax
-          facets = Hash[ facets.map{|f| [f, {}] } ]
-        end
-
-        payload[:facets] = {}
-        facets.each do |field, facet_options|
-          # ask for extra facets due to
-          # https://github.com/elasticsearch/elasticsearch/issues/1305
-
-          if facet_options[:ranges]
-            payload[:facets][field] = {
-              range: {
-                field.to_sym => facet_options[:ranges]
-              }
-            }
-          else
-            payload[:facets][field] = {
-              terms: {
-                field: field,
-                size: facet_options[:limit] ? facet_options[:limit] + 150 : 100000
-              }
-            }
-          end
-
-          facet_limits[field] = facet_options[:limit] if facet_options[:limit]
-
-          # offset is not possible
-          # http://elasticsearch-users.115913.n3.nabble.com/Is-pagination-possible-in-termsStatsFacet-td3422943.html
-
-          facet_filters = where_filters.call(facet_options[:where])
-          if facet_filters.any?
-            payload[:facets][field][:facet_filter] = {
-              and: {
-                filters: facet_filters
-              }
-            }
-          end
-        end
-      end
-
-      # suggestions
-      if options[:suggest]
-        suggest_fields = (searchkick_options[:suggest] || []).map(&:to_s)
-        # intersection
-        suggest_fields = suggest_fields & options[:fields].map(&:to_s) if options[:fields]
-        if suggest_fields.any?
-          payload[:suggest] = {text: term}
-          suggest_fields.each do |field|
-            payload[:suggest][field] = {
-              phrase: {
-                field: "#{field}.suggest"
-              }
-            }
-          end
-        end
-      end
-
-      # highlight
-      if options[:highlight]
-        payload[:highlight] = {
-          fields: Hash[ fields.map{|f| [f, {}] } ]
-        }
-        if options[:highlight].is_a?(Hash) and tag = options[:highlight][:tag]
-          payload[:highlight][:pre_tags] = [tag]
-          payload[:highlight][:post_tags] = [tag.to_s.gsub(/\A</, "</")]
-        end
-      end
-
-      # An empty array will cause only the _id and _type for each hit to be returned
-      # http://www.elasticsearch.org/guide/reference/api/search/fields/
-      payload[:fields] = [] if load
-
-      tire_options = {load: load, payload: payload, size: per_page, from: offset}
-      tire_options[:type] = document_type if self != searchkick_klass
-      begin
-        response = Searchkick.client.search index: index_name, body: payload
-      rescue => e
-        status_code = e.message[0..3].to_i
-        if status_code == 404
-          raise "Index missing - run #{searchkick_klass.name}.reindex"
-        elsif status_code == 500 and (e.message.include?("IllegalArgumentException[minimumSimilarity >= 1]") or e.message.include?("No query registered for [multi_match]"))
-          raise "Upgrade Elasticsearch to 0.90.0 or greater"
-        else
-          raise e
-        end
-      end
-
-      # apply facet limit in client due to
-      # https://github.com/elasticsearch/elasticsearch/issues/1305
-      facet_limits.each do |field, limit|
-        field = field.to_s
-        facet = response["facets"][field]
-        response["facets"][field]["terms"] = facet["terms"].first(limit)
-        response["facets"][field]["other"] = facet["total"] - facet["terms"].sum{|term| term["count"] }
-      end
-
-      Searchkick::Results.new(response, search.options.merge(term: term))
     end
   end
 module Searchkick
-  VERSION = "0.4.2"
+  VERSION = "0.5.3"
 end
@@ -23,6 +23,4 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "bundler", "~> 1.3"
   spec.add_development_dependency "rake"
   spec.add_development_dependency "minitest", "~> 4.7"
-  spec.add_development_dependency "activerecord"
-  spec.add_development_dependency "pg"
 end
@@ -17,4 +17,39 @@ class TestAutocomplete &lt; Minitest::Unit::TestCase
     assert_search "hum", ["Hummus"], autocomplete: true, fields: [:name]
   end
+  def test_text_start
+    store_names ["Where in the World is Carmen San Diego"]
+    assert_search "where in the world is", ["Where in the World is Carmen San Diego"], fields: [{name: :text_start}]
+    assert_search "in the world", [], fields: [{name: :text_start}]
+  end
+
+  def test_text_middle
+    store_names ["Where in the World is Carmen San Diego"]
+    assert_search "where in the world is", ["Where in the World is Carmen San Diego"], fields: [{name: :text_middle}]
+    assert_search "n the wor", ["Where in the World is Carmen San Diego"], fields: [{name: :text_middle}]
+    assert_search "men san diego", ["Where in the World is Carmen San Diego"], fields: [{name: :text_middle}]
+    assert_search "world carmen", [], fields: [{name: :text_middle}]
+  end
+
+  def test_text_end
+    store_names ["Where in the World is Carmen San Diego"]
+    assert_search "men san diego", ["Where in the World is Carmen San Diego"], fields: [{name: :text_end}]
+    assert_search "carmen san", [], fields: [{name: :text_end}]
+  end
+
+  def test_word_start
+    store_names ["Where in the World is Carmen San Diego"]
+    assert_search "car san wor", ["Where in the World is Carmen San Diego"], fields: [{name: :word_start}]
+  end
+
+  def test_word_middle
+    store_names ["Where in the World is Carmen San Diego"]
+    assert_search "orl", ["Where in the World is Carmen San Diego"], fields: [{name: :word_middle}]
+  end
+
+  def test_word_end
+    store_names ["Where in the World is Carmen San Diego"]
+    assert_search "rld men ego", ["Where in the World is Carmen San Diego"], fields: [{name: :word_end}]
+  end
+
 end
@@ -57,4 +57,14 @@ class TestBoost &lt; Minitest::Unit::TestCase
     assert_first "tomato", "Tomato B", user_id: 2
   end
+  def test_personalize
+    store [
+      {name: "Tomato A"},
+      {name: "Tomato B", user_ids: [1, 2, 3]},
+      {name: "Tomato C"},
+      {name: "Tomato D"}
+    ]
+    assert_first "tomato", "Tomato B", personalize: {user_ids: 2}
+  end
+
 end
@@ -32,4 +32,17 @@ class TestIndex &lt; Minitest::Unit::TestCase
     assert_equal ["Dollar Tree"], Store.search(query: {match: {name: "Dollar Tree"}}).map(&:name)
   end
+  if defined?(ActiveRecord)
+
+    def test_transaction
+      Product.transaction do
+        store_names ["Product A"]
+        raise ActiveRecord::Rollback
+      end
+
+      assert_search "product", []
+    end
+
+  end
+
 end
@@ -25,6 +25,19 @@ class TestInheritance &lt; Minitest::Unit::TestCase
     assert_equal 2, Animal.search("bear").size
   end
+  def test_force_one_type
+    store_names ["Green Bear"], Dog
+    store_names ["Blue Bear"], Cat
+    assert_equal ["Blue Bear"], Animal.search("bear", type: [Cat]).map(&:name)
+  end
+
+  def test_force_multiple_types
+    store_names ["Green Bear"], Dog
+    store_names ["Blue Bear"], Cat
+    store_names ["Red Bear"], Animal
+    assert_equal ["Green Bear", "Blue Bear"], Animal.search("bear", type: [Dog, Cat]).map(&:name)
+  end
+
   def test_child_autocomplete
     store_names ["Max"], Cat
     store_names ["Mark"], Dog
@@ -139,4 +139,9 @@ class TestMatch &lt; Minitest::Unit::TestCase
     assert_search "*", ["Product A", "Product B"]
   end
+  def test_to_be_or_not_to_be
+    store_names ["to be or not to be"]
+    assert_search "to be", ["to be or not to be"]
+  end
+
 end
@@ -0,0 +1,14 @@
+require_relative "test_helper"
+
+class TestQuery < Minitest::Unit::TestCase
+
+  def test_basic
+    store_names ["Milk", "Apple"]
+    query = Product.search("milk", execute: false)
+    # query.body = {query: {match_all: {}}}
+    # query.body = {query: {match: {name: "Apple"}}}
+    query.body[:query] = {match_all: {}}
+    assert_equal ["Apple", "Milk"], query.execute.map(&:name).sort
+  end
+
+end
@@ -40,7 +40,7 @@ class TestSql &lt; Minitest::Unit::TestCase
     store [
       {name: "Product A", store_id: 1, in_stock: true, backordered: true, created_at: now, orders_count: 4, user_ids: [1, 2, 3]},
       {name: "Product B", store_id: 2, in_stock: true, backordered: false, created_at: now - 1, orders_count: 3, user_ids: [1]},
-      {name: "Product C", store_id: 3, in_stock: false, backordered: true, created_at: now - 2, orders_count: 2},
+      {name: "Product C", store_id: 3, in_stock: false, backordered: true, created_at: now - 2, orders_count: 2, user_ids: [1, 3]},
       {name: "Product D", store_id: 4, in_stock: false, backordered: false, created_at: now - 3, orders_count: 1},
     ]
     assert_search "product", ["Product A", "Product B"], where: {in_stock: true}
@@ -60,13 +60,21 @@ class TestSql &lt; Minitest::Unit::TestCase
     assert_search "product", ["Product A", "Product B"], where: {store_id: [1, 2]}
     assert_search "product", ["Product B", "Product C", "Product D"], where: {store_id: {not: 1}}
     assert_search "product", ["Product C", "Product D"], where: {store_id: {not: [1, 2]}}
+    assert_search "product", ["Product A"], where: {user_ids: {lte: 2, gte: 2}}
     # or
     assert_search "product", ["Product A", "Product B", "Product C"], where: {or: [[{in_stock: true}, {store_id: 3}]]}
     assert_search "product", ["Product A", "Product B", "Product C"], where: {or: [[{orders_count: [2, 4]}, {store_id: [1, 2]}]]}
     assert_search "product", ["Product A", "Product D"], where: {or: [[{orders_count: 1}, {created_at: {gte: now - 1}, backordered: true}]]}
     # all
-    assert_search "product", ["Product A"], where: {user_ids: {all: [1, 3]}}
+    assert_search "product", ["Product A", "Product C"], where: {user_ids: {all: [1, 3]}}
     assert_search "product", [], where: {user_ids: {all: [1, 2, 3, 4]}}
+    # any / nested terms
+    assert_search "product", ["Product B", "Product C"], where: {user_ids: {not: [2], in: [1,3]}}
+    # not / exists
+    assert_search "product", ["Product D"], where: {user_ids: nil}
+    assert_search "product", ["Product A", "Product B", "Product C"], where: {user_ids: {not: nil}}
+    assert_search "product", ["Product A", "Product C", "Product D"], where: {user_ids: [3, nil]}
+    assert_search "product", ["Product B"], where: {user_ids: {not: [3, nil]}}
   end
   def test_where_string
@@ -90,6 +98,35 @@ class TestSql &lt; Minitest::Unit::TestCase
     assert_search "product", ["Product A"], where: {id: product.id.to_s}
   end
+  def test_where_empty
+    store_names ["Product A"]
+    assert_search "product", ["Product A"], where: {}
+  end
+
+  def test_where_empty_array
+    store_names ["Product A"]
+    assert_search "product", [], where: {store_id: []}
+  end
+
+  # http://elasticsearch-users.115913.n3.nabble.com/Numeric-range-quey-or-filter-in-an-array-field-possible-or-not-td4042967.html
+  # https://gist.github.com/jprante/7099463
+  def test_where_range_array
+    store [
+      {name: "Product A", user_ids: [11, 23, 13, 16, 17, 23.6]},
+      {name: "Product B", user_ids: [1, 2, 3, 4, 5, 6, 7, 8, 8.9, 9.1, 9.4]},
+      {name: "Product C", user_ids: [101, 230, 150, 200]}
+    ]
+    assert_search "product", ["Product A"], where: {user_ids: {gt: 10, lt: 23.9}}
+  end
+
+  def test_where_range_array_again
+    store [
+      {name: "Product A", user_ids: [19, 32, 42]},
+      {name: "Product B", user_ids: [13, 40, 52]}
+    ]
+    assert_search "product", ["Product A"], where: {user_ids: {gt: 26, lt: 36}}
+  end
+
   def test_near
     store [
       {name: "San Francisco", latitude: 37.7833, longitude: -122.4167},
@@ -133,12 +170,34 @@ class TestSql &lt; Minitest::Unit::TestCase
     assert_order "product", ["Product A", "Product B", "Product C", "Product D"], order: "name"
   end
+  def test_order_id
+    store_names ["Product A", "Product B"]
+    product_a = Product.where(name: "Product A").first
+    product_b = Product.where(name: "Product B").first
+    assert_order "product", [product_a, product_b].sort_by(&:id).map(&:name), order: {id: :asc}
+  end
+
+  def test_order_multiple
+    store [
+      {name: "Product A", color: "blue", store_id: 1},
+      {name: "Product B", color: "red", store_id: 3},
+      {name: "Product C", color: "red", store_id: 2}
+    ]
+    assert_order "product", ["Product A", "Product B", "Product C"], order: {color: :asc, store_id: :desc}
+  end
+
   def test_partial
     store_names ["Honey"]
     assert_search "fresh honey", []
     assert_search "fresh honey", ["Honey"], partial: true
   end
+  def test_operator
+    store_names ["Honey"]
+    assert_search "fresh honey", []
+    assert_search "fresh honey", ["Honey"], operator: "or"
+  end
+
   def test_misspellings
     store_names ["abc", "abd", "aee"]
     assert_search "abc", ["abc"], misspellings: false
@@ -56,9 +56,9 @@ else
   ActiveRecord::Base.time_zone_aware_attributes = true
   # migrations
-  ActiveRecord::Base.establish_connection :adapter => "postgresql", :database => "searchkick_test"
+  ActiveRecord::Base.establish_connection adapter: "sqlite3", database: ":memory:"
-  ActiveRecord::Migration.create_table :products, :force => true do |t|
+  ActiveRecord::Migration.create_table :products do |t|
     t.string :name
     t.integer :store_id
     t.boolean :in_stock
@@ -71,11 +71,11 @@ else
     t.timestamps
   end
-  ActiveRecord::Migration.create_table :stores, :force => true do |t|
+  ActiveRecord::Migration.create_table :stores do |t|
     t.string :name
   end
-  ActiveRecord::Migration.create_table :animals, :force => true do |t|
+  ActiveRecord::Migration.create_table :animals do |t|
     t.string :name
     t.string :type
   end
@@ -112,12 +112,18 @@ class Product
     suggest: [:name, :color],
     conversions: "conversions",
     personalize: "user_ids",
-    locations: ["location", "multiple_locations"]
+    locations: ["location", "multiple_locations"],
+    text_start: [:name],
+    text_middle: [:name],
+    text_end: [:name],
+    word_start: [:name],
+    word_middle: [:name],
+    word_end: [:name]
   attr_accessor :conversions, :user_ids
   def search_data
-    serializable_hash.merge conversions: conversions, user_ids: user_ids, location: [latitude, longitude], multiple_locations: [[latitude, longitude], [0, 0]]
+    serializable_hash.except("id").merge conversions: conversions, user_ids: user_ids, location: [latitude, longitude], multiple_locations: [[latitude, longitude], [0, 0]]
   end
   def should_index?
@@ -150,6 +156,7 @@ class Minitest::Unit::TestCase
   def setup
     Product.destroy_all
+    Store.destroy_all
     Animal.destroy_all
   end
		1	+## 0.5.3
		2	+
		3	+- Fixed bug w/ word_* queries
		4	+
		5	+## 0.5.2
		6	+
		7	+- Use after_commit hook for ActiveRecord to prevent data inconsistencies
		8	+
		9	+## 0.5.1
		10	+
		11	+- Replaced stop words with common terms query
		12	+- Added language option
		13	+- Fixed bug with empty array in where clause
		14	+- Fixed bug with MongoDB integer _id
		15	+- Fixed reindex bug when callbacks disabled
		16	+
		17	+## 0.5.0
		18	+
		19	+- Better control over partial matches
		20	+- Added merge_mappings option
		21	+- Added batch_size option
		22	+- Fixed bug with nil where clauses
		23	+
1	## 0.4.2	24	## 0.4.2
2		25
3	- Added `should_index?` method to control which records are indexed	26	- Added `should_index?` method to control which records are indexed
	@@ -3,5 +3,7 @@ source 'https://rubygems.org'		@@ -3,5 +3,7 @@ source 'https://rubygems.org'
3	# Specify your gem's dependencies in searchkick.gemspec	3	# Specify your gem's dependencies in searchkick.gemspec
4	gemspec	4	gemspec
5		5
		6	+gem "sqlite3"
		7	+gem "activerecord"
6	# gem "activerecord", "~> 3.2.0"	8	# gem "activerecord", "~> 3.2.0"
7	# gem "activerecord", "~> 3.1.0"	9	# gem "activerecord", "~> 3.1.0"
	@@ -21,12 +21,14 @@ Plus:		@@ -21,12 +21,14 @@ Plus:
21	- “Did you mean” suggestions	21	- “Did you mean” suggestions
22	- works with ActiveRecord and Mongoid	22	- works with ActiveRecord and Mongoid
23		23
24	-:zap: Even better with [Searchjoy](http://ankane.github.io/searchjoy/)
25	-
26	:tangerine: Battle-tested at [Instacart](https://www.instacart.com)	24	:tangerine: Battle-tested at [Instacart](https://www.instacart.com)
27		25
28	[![Build Status](https://travis-ci.org/ankane/searchkick.png?branch=master)](https://travis-ci.org/ankane/searchkick)	26	[![Build Status](https://travis-ci.org/ankane/searchkick.png?branch=master)](https://travis-ci.org/ankane/searchkick)
29		27
		28	+We highly recommend tracking queries and conversions
		29	+
		30	+:zap: [Searchjoy](http://ankane.github.io/searchjoy/) makes it easy
		31	+
30	## Get Started	32	## Get Started
31		33
32	[Install Elasticsearch](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html). For Homebrew, use:	34	[Install Elasticsearch](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html). For Homebrew, use:
	@@ -148,6 +150,44 @@ To change this, use:		@@ -148,6 +150,44 @@ To change this, use:
148	Product.search "fresh honey", partial: true # fresh OR honey	150	Product.search "fresh honey", partial: true # fresh OR honey
149	```	151	```
150		152
		153	+By default, results must match the entire word - `back` will not match `backpack`. You can change this behavior with:
		154	+
		155	+```ruby
		156	+class Product < ActiveRecord::Base
		157	+ searchkick word_start: [:name]
		158	+end
		159	+```
		160	+
		161	+And to search:
		162	+
		163	+```ruby
		164	+Product.search "back", fields: [{name: :word_start}]
		165	+```
		166	+
		167	+Available options are:
		168	+
		169	+```ruby
		170	+:word # default
		171	+:word_start
		172	+:word_middle
		173	+:word_end
		174	+:text_start
		175	+:text_middle
		176	+:text_end
		177	+```
		178	+
		179	+### Language
		180	+
		181	+Searchkick defaults to English for stemming. To change this, use:
		182	+
		183	+```ruby
		184	+class Product < ActiveRecord::Base
		185	+ searchkick language: "German"
		186	+end
		187	+```
		188	+
		189	+[See the list of languages](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-snowball-tokenfilter.html)
		190	+
151	### Synonyms	191	### Synonyms
152		192
153	```ruby	193	```ruby
	@@ -193,7 +233,7 @@ Searchkick uses `find_in_batches` to import documents. To eager load associatio		@@ -193,7 +233,7 @@ Searchkick uses `find_in_batches` to import documents. To eager load associatio
193		233
194	```ruby	234	```ruby
195	class Product < ActiveRecord::Base	235	class Product < ActiveRecord::Base
196	- scope :search_import, includes(:searches)	236	+ scope :search_import, -> { includes(:searches) }
197	end	237	end
198	```	238	```
199		239
	@@ -293,14 +333,14 @@ First, specify which fields use this feature. This is necessary since autocompl		@@ -293,14 +333,14 @@ First, specify which fields use this feature. This is necessary since autocompl
293		333
294	```ruby	334	```ruby
295	class City < ActiveRecord::Base	335	class City < ActiveRecord::Base
296	- searchkick autocomplete: ["name"]	336	+ searchkick text_start: [:name]
297	end	337	end
298	```	338	```
299		339
300	Reindex and search with:	340	Reindex and search with:
301		341
302	```ruby	342	```ruby
303	-City.search "san fr", autocomplete: true	343	+City.search "san fr", fields: [{name: :text_start}]
304	```	344	```
305		345
306	Typically, you want to use a Javascript library like [typeahead.js](http://twitter.github.io/typeahead.js/) or [jQuery UI](http://jqueryui.com/autocomplete/).	346	Typically, you want to use a Javascript library like [typeahead.js](http://twitter.github.io/typeahead.js/) or [jQuery UI](http://jqueryui.com/autocomplete/).
	@@ -314,7 +354,7 @@ First, add a controller action.		@@ -314,7 +354,7 @@ First, add a controller action.
314	class CitiesController < ApplicationController	354	class CitiesController < ApplicationController
315		355
316	def autocomplete	356	def autocomplete
317	- render json: City.search(params[:query], autocomplete: true, limit: 10).map(&:name)	357	+ render json: City.search(params[:query], fields: [{name: :text_start}], limit: 10).map(&:name)
318	end	358	end
319		359
320	end	360	end
	@@ -454,8 +494,9 @@ Dog.reindex # equivalent		@@ -454,8 +494,9 @@ Dog.reindex # equivalent
454	And to search, use:	494	And to search, use:
455		495
456	```ruby	496	```ruby
457	-Animal.search "*" # all animals
458	-Dog.search "*" # just dogs	497	+Animal.search "*" # all animals
		498	+Dog.search "*" # just dogs
		499	+Animal.search "*", type: [Dog, Cat] # just cats and dogs [master]
459	```	500	```
460		501
461	Note: The `suggest` option retrieves suggestions from the parent at the moment.	502	Note: The `suggest` option retrieves suggestions from the parent at the moment.
	@@ -530,6 +571,24 @@ And use the `query` option to search:		@@ -530,6 +571,24 @@ And use the `query` option to search:
530	Product.search query: {match: {name: "milk"}}	571	Product.search query: {match: {name: "milk"}}
531	```	572	```
532		573
		574	+To keep the mappings and settings generated by Searchkick, use:
		575	+
		576	+```ruby
		577	+class Product < ActiveRecord::Base
		578	+ searchkick merge_mappings: true, mappings: {...}
		579	+end
		580	+```
		581	+
		582	+## Experimental [master]
		583	+
		584	+Modify the query generated by Searchkick.
		585	+
		586	+```ruby
		587	+query = Product.search "2% Milk", execute: false
		588	+query.body[:query] = {match_all: {}}
		589	+products = query.execute
		590	+```
		591	+
533	## Reference	592	## Reference
534		593
535	Searchkick requires Elasticsearch `0.90.0` or higher.	594	Searchkick requires Elasticsearch `0.90.0` or higher.
	@@ -574,9 +633,9 @@ end		@@ -574,9 +633,9 @@ end
574	or temporarily	633	or temporarily
575		634
576	```ruby	635	```ruby
577	-Product.disable_search_callbacks # use Searchkick.disable_callbacks for all models	636	+Product.disable_search_callbacks # or use Searchkick.disable_callbacks for all models
578	ExpensiveProductsTask.execute	637	ExpensiveProductsTask.execute
579	-Product.enable_search_callbacks	638	+Product.enable_search_callbacks # or use Searchkick.enable_callbacks for all models
580	Product.reindex	639	Product.reindex
581	```	640	```
582		641
	@@ -601,6 +660,50 @@ class Product < ActiveRecord::Base		@@ -601,6 +660,50 @@ class Product < ActiveRecord::Base
601	end	660	end
602	```	661	```
603		662
		663	+Change import batch size
		664	+
		665	+```ruby
		666	+class Product < ActiveRecord::Base
		667	+ searchkick batch_size: 200 # defaults to 1000
		668	+end
		669	+```
		670	+
		671	+Asynchronous reindexing
		672	+
		673	+```ruby
		674	+class Product < ActiveRecord::Base
		675	+ searchkick callbacks: false
		676	+
		677	+ # add the callbacks manually
		678	+
		679	+ # ActiveRecord - one callback
		680	+ after_commit :reindex_async
		681	+
		682	+ # Mongoid - two callbacks
		683	+ after_save :reindex_async
		684	+ after_destroy :reindex_async
		685	+
		686	+ def reindex_async
		687	+ # delayed job
		688	+ delay.reindex
		689	+ end
		690	+end
		691	+```
		692	+
		693	+Reindex conditionally
		694	+
		695	+Note: With ActiveRecord, use this feature with caution - [transaction rollbacks can cause data inconstencies](https://github.com/elasticsearch/elasticsearch-rails/blob/master/elasticsearch-model/README.md#custom-callbacks)
		696	+
		697	+```ruby
		698	+class Product < ActiveRecord::Base
		699	+ searchkick callbacks: false
		700	+
		701	+ # add the callbacks manually
		702	+ after_save :reindex, if: proc{\|model\| model.name_changed? } # use your own condition
		703	+ after_destroy :reindex
		704	+end
		705	+```
		706	+
604	Reindex all models (Rails only)	707	Reindex all models (Rails only)
605		708
606	```sh	709	```sh
	@@ -655,11 +758,14 @@ For convenience, this is set by default in the test environment.		@@ -655,11 +758,14 @@ For convenience, this is set by default in the test environment.
655		758
656	Thanks to Karel Minarik for [Tire](https://github.com/karmi/tire), Jaroslav Kalistsuk for [zero downtime reindexing](https://gist.github.com/jarosan/3124884), and Alex Leschenko for [Elasticsearch autocomplete](https://github.com/leschenko/elasticsearch_autocomplete).	759	Thanks to Karel Minarik for [Tire](https://github.com/karmi/tire), Jaroslav Kalistsuk for [zero downtime reindexing](https://gist.github.com/jarosan/3124884), and Alex Leschenko for [Elasticsearch autocomplete](https://github.com/leschenko/elasticsearch_autocomplete).
657		760
658	-## TODO	761	+## Roadmap
659		762
660	-- Generate autocomplete predictions from past search queries
661	-- Automatic failover
662	-- Make Searchkick work with any language	763	+- Search multiple fields for different terms
		764	+- Search across models
		765	+- Search nested objects
		766	+- Add section on testing
		767	+- Much finer customization
		768	+- More transparency into generated queries (for advanced use)
663		769
664	## History	770	## History
665		771
	@@ -673,3 +779,11 @@ Everyone is encouraged to help improve this project. Here are a few ways you can		@@ -673,3 +779,11 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
673	- Fix bugs and [submit pull requests](https://github.com/ankane/searchkick/pulls)	779	- Fix bugs and [submit pull requests](https://github.com/ankane/searchkick/pulls)
674	- Write, clarify, or fix documentation	780	- Write, clarify, or fix documentation
675	- Suggest or add new features	781	- Suggest or add new features
		782	+
		783	+To get started with development and testing:
		784	+
		785	+1. Clone the repo
		786	+2. Install PostgreSQL and create a database called `searchkick_test` (`psql -d postgres -c "create database searchkick_test"`)
		787	+3. Install Elasticsearch
		788	+4. `bundle`
		789	+5. `rake test`
	@@ -0,0 +1,429 @@		@@ -0,0 +1,429 @@
		1	+module Searchkick
		2	+ class Query
		3	+ attr_reader :klass, :term, :options
		4	+ attr_accessor :body
		5	+
		6	+ def initialize(klass, term, options = {})
		7	+ if term.is_a?(Hash)
		8	+ options = term
		9	+ term = nil
		10	+ else
		11	+ term = term.to_s
		12	+ end
		13	+
		14	+ @klass = klass
		15	+ @term = term
		16	+ @options = options
		17	+
		18	+ fields =
		19	+ if options[:fields]
		20	+ if options[:autocomplete]
		21	+ options[:fields].map{\|f\| "#{f}.autocomplete" }
		22	+ else
		23	+ options[:fields].map do \|value\|
		24	+ k, v = value.is_a?(Hash) ? value.to_a.first : [value, :word]
		25	+ "#{k}.#{v == :word ? "analyzed" : v}"
		26	+ end
		27	+ end
		28	+ else
		29	+ if options[:autocomplete]
		30	+ (searchkick_options[:autocomplete] \|\| []).map{\|f\| "#{f}.autocomplete" }
		31	+ else
		32	+ ["_all"]
		33	+ end
		34	+ end
		35	+
		36	+ operator = options[:operator] \|\| (options[:partial] ? "or" : "and")
		37	+
		38	+ # model and eagar loading
		39	+ load = options[:load].nil? ? true : options[:load]
		40	+ load = (options[:include] ? {include: options[:include]} : true) if load
		41	+
		42	+ # pagination
		43	+ page = [options[:page].to_i, 1].max
		44	+ per_page = (options[:limit] \|\| options[:per_page] \|\| 100000).to_i
		45	+ offset = options[:offset] \|\| (page - 1) * per_page
		46	+ index_name = options[:index_name] \|\| searchkick_index.name
		47	+
		48	+ conversions_field = searchkick_options[:conversions]
		49	+ personalize_field = searchkick_options[:personalize]
		50	+
		51	+ all = term == "*"
		52	+
		53	+ if options[:query]
		54	+ payload = options[:query]
		55	+ elsif options[:similar]
		56	+ payload = {
		57	+ more_like_this: {
		58	+ fields: fields,
		59	+ like_text: term,
		60	+ min_doc_freq: 1,
		61	+ min_term_freq: 1,
		62	+ analyzer: "searchkick_search2"
		63	+ }
		64	+ }
		65	+ elsif all
		66	+ payload = {
		67	+ match_all: {}
		68	+ }
		69	+ else
		70	+ if options[:autocomplete]
		71	+ payload = {
		72	+ multi_match: {
		73	+ fields: fields,
		74	+ query: term,
		75	+ analyzer: "searchkick_autocomplete_search"
		76	+ }
		77	+ }
		78	+ else
		79	+ queries = []
		80	+ fields.each do \|field\|
		81	+ if field == "_all" or field.end_with?(".analyzed")
		82	+ shared_options = {
		83	+ fields: [field],
		84	+ query: term,
		85	+ use_dis_max: false,
		86	+ operator: operator,
		87	+ cutoff_frequency: 0.001
		88	+ }
		89	+ queries.concat [
		90	+ {multi_match: shared_options.merge(boost: 10, analyzer: "searchkick_search")},
		91	+ {multi_match: shared_options.merge(boost: 10, analyzer: "searchkick_search2")}
		92	+ ]
		93	+ if options[:misspellings] != false
		94	+ distance = (options[:misspellings].is_a?(Hash) && options[:misspellings][:distance]) \|\| 1
		95	+ queries.concat [
		96	+ {multi_match: shared_options.merge(fuzziness: distance, max_expansions: 3, analyzer: "searchkick_search")},
		97	+ {multi_match: shared_options.merge(fuzziness: distance, max_expansions: 3, analyzer: "searchkick_search2")}
		98	+ ]
		99	+ end
		100	+ else
		101	+ analyzer = field.match(/\.word_(start\|middle\|end)\z/) ? "searchkick_word_search" : "searchkick_autocomplete_search"
		102	+ queries << {
		103	+ multi_match: {
		104	+ fields: [field],
		105	+ query: term,
		106	+ analyzer: analyzer
		107	+ }
		108	+ }
		109	+ end
		110	+ end
		111	+
		112	+ payload = {
		113	+ dis_max: {
		114	+ queries: queries
		115	+ }
		116	+ }
		117	+ end
		118	+
		119	+ if conversions_field and options[:conversions] != false
		120	+ # wrap payload in a bool query
		121	+ payload = {
		122	+ bool: {
		123	+ must: payload,
		124	+ should: {
		125	+ nested: {
		126	+ path: conversions_field,
		127	+ score_mode: "total",
		128	+ query: {
		129	+ custom_score: {
		130	+ query: {
		131	+ match: {
		132	+ query: term
		133	+ }
		134	+ },
		135	+ script: "doc['count'].value"
		136	+ }
		137	+ }
		138	+ }
		139	+ }
		140	+ }
		141	+ }
		142	+ end
		143	+ end
		144	+
		145	+ custom_filters = []
		146	+
		147	+ if options[:boost]
		148	+ custom_filters << {
		149	+ filter: {
		150	+ exists: {
		151	+ field: options[:boost]
		152	+ }
		153	+ },
		154	+ script: "log(doc['#{options[:boost]}'].value + 2.718281828)"
		155	+ }
		156	+ end
		157	+
		158	+ if options[:user_id] and personalize_field
		159	+ custom_filters << {
		160	+ filter: {
		161	+ term: {
		162	+ personalize_field => options[:user_id]
		163	+ }
		164	+ },
		165	+ boost: 100
		166	+ }
		167	+ end
		168	+
		169	+ if options[:personalize]
		170	+ custom_filters << {
		171	+ filter: {
		172	+ term: options[:personalize]
		173	+ },
		174	+ boost: 100
		175	+ }
		176	+ end
		177	+
		178	+ if custom_filters.any?
		179	+ payload = {
		180	+ custom_filters_score: {
		181	+ query: payload,
		182	+ filters: custom_filters,
		183	+ score_mode: "total"
		184	+ }
		185	+ }
		186	+ end
		187	+
		188	+ payload = {
		189	+ query: payload,
		190	+ size: per_page,
		191	+ from: offset
		192	+ }
		193	+ payload[:explain] = options[:explain] if options[:explain]
		194	+
		195	+ # order
		196	+ if options[:order]
		197	+ order = options[:order].is_a?(Enumerable) ? options[:order] : {options[:order] => :asc}
		198	+ payload[:sort] = Hash[ order.map{\|k, v\| [k.to_s == "id" ? :_id : k, v] } ]
		199	+ end
		200	+
		201	+ # filters
		202	+ filters = where_filters(options[:where])
		203	+ if filters.any?
		204	+ payload[:filter] = {
		205	+ and: filters
		206	+ }
		207	+ end
		208	+
		209	+ # facets
		210	+ facet_limits = {}
		211	+ if options[:facets]
		212	+ facets = options[:facets] \|\| {}
		213	+ if facets.is_a?(Array) # convert to more advanced syntax
		214	+ facets = Hash[ facets.map{\|f\| [f, {}] } ]
		215	+ end
		216	+
		217	+ payload[:facets] = {}
		218	+ facets.each do \|field, facet_options\|
		219	+ # ask for extra facets due to
		220	+ # https://github.com/elasticsearch/elasticsearch/issues/1305
		221	+
		222	+ if facet_options[:ranges]
		223	+ payload[:facets][field] = {
		224	+ range: {
		225	+ field.to_sym => facet_options[:ranges]
		226	+ }
		227	+ }
		228	+ else
		229	+ payload[:facets][field] = {
		230	+ terms: {
		231	+ field: field,
		232	+ size: facet_options[:limit] ? facet_options[:limit] + 150 : 100000
		233	+ }
		234	+ }
		235	+ end
		236	+
		237	+ facet_limits[field] = facet_options[:limit] if facet_options[:limit]
		238	+
		239	+ # offset is not possible
		240	+ # http://elasticsearch-users.115913.n3.nabble.com/Is-pagination-possible-in-termsStatsFacet-td3422943.html
		241	+
		242	+ facet_filters = where_filters(facet_options[:where])
		243	+ if facet_filters.any?
		244	+ payload[:facets][field][:facet_filter] = {
		245	+ and: {
		246	+ filters: facet_filters
		247	+ }
		248	+ }
		249	+ end
		250	+ end
		251	+ end
		252	+
		253	+ # suggestions
		254	+ if options[:suggest]
		255	+ suggest_fields = (searchkick_options[:suggest] \|\| []).map(&:to_s)
		256	+ # intersection
		257	+ suggest_fields = suggest_fields & options[:fields].map(&:to_s) if options[:fields]
		258	+ if suggest_fields.any?
		259	+ payload[:suggest] = {text: term}
		260	+ suggest_fields.each do \|field\|
		261	+ payload[:suggest][field] = {
		262	+ phrase: {
		263	+ field: "#{field}.suggest"
		264	+ }
		265	+ }
		266	+ end
		267	+ end
		268	+ end
		269	+
		270	+ # highlight
		271	+ if options[:highlight]
		272	+ payload[:highlight] = {
		273	+ fields: Hash[ fields.map{\|f\| [f, {}] } ]
		274	+ }
		275	+ if options[:highlight].is_a?(Hash) and tag = options[:highlight][:tag]
		276	+ payload[:highlight][:pre_tags] = [tag]
		277	+ payload[:highlight][:post_tags] = [tag.to_s.gsub(/\A</, "</")]
		278	+ end
		279	+ end
		280	+
		281	+ # An empty array will cause only the _id and _type for each hit to be returned
		282	+ # http://www.elasticsearch.org/guide/reference/api/search/fields/
		283	+ payload[:fields] = [] if load
		284	+
		285	+ tire_options = {load: load, size: per_page, from: offset}
		286	+ if options[:type] or klass != searchkick_klass
		287	+ tire_options[:type] = [options[:type] \|\| klass].flatten.map(&:document_type)
		288	+ end
		289	+
		290	+ @body = payload
		291	+ @facet_limits = facet_limits
		292	+ end
		293	+
		294	+ def searchkick_index
		295	+ klass.searchkick_index
		296	+ end
		297	+
		298	+ def searchkick_options
		299	+ klass.searchkick_options
		300	+ end
		301	+
		302	+ def searchkick_klass
		303	+ klass.searchkick_klass
		304	+ end
		305	+
		306	+ def document_type
		307	+ klass.document_type
		308	+ end
		309	+
		310	+ def execute
		311	+ begin
		312	+ response = Searchkick.client.search(index: searchkick_index.name, body: body)
		313	+ rescue => e # TODO rescue type
		314	+ status_code = e.message[0..3].to_i
		315	+ if status_code == 404
		316	+ raise "Index missing - run #{searchkick_klass.name}.reindex"
		317	+ elsif status_code == 500 and (e.message.include?("IllegalArgumentException[minimumSimilarity >= 1]") or e.message.include?("No query registered for [multi_match]"))
		318	+ raise "Upgrade Elasticsearch to 0.90.0 or greater"
		319	+ else
		320	+ raise e
		321	+ end
		322	+ end
		323	+
		324	+ # apply facet limit in client due to
		325	+ # https://github.com/elasticsearch/elasticsearch/issues/1305
		326	+ @facet_limits.each do \|field, limit\|
		327	+ field = field.to_s
		328	+ facet = response["facets"][field]
		329	+ response["facets"][field]["terms"] = facet["terms"].first(limit)
		330	+ response["facets"][field]["other"] = facet["total"] - facet["terms"].sum{\|term\| term["count"] }
		331	+ end
		332	+
		333	+ Searchkick::Results.new(response, @search.options.merge(term: term, model_name: searchkick_klass.model_name))
		334	+ end
		335	+
		336	+ private
		337	+
		338	+ def where_filters(where)
		339	+ filters = []
		340	+ (where \|\| {}).each do \|field, value\|
		341	+ field = :_id if field.to_s == "id"
		342	+
		343	+ if field == :or
		344	+ value.each do \|or_clause\|
		345	+ filters << {or: or_clause.map{\|or_statement\| {and: where_filters(or_statement)} }}
		346	+ end
		347	+ else
		348	+ # expand ranges
		349	+ if value.is_a?(Range)
		350	+ value = {gte: value.first, (value.exclude_end? ? :lt : :lte) => value.last}
		351	+ end
		352	+
		353	+ if value.is_a?(Array)
		354	+ value = {in: value}
		355	+ end
		356	+
		357	+ if value.is_a?(Hash)
		358	+ value.each do \|op, op_value\|
		359	+ case op
		360	+ when :within, :bottom_right
		361	+ # do nothing
		362	+ when :near
		363	+ filters << {
		364	+ geo_distance: {
		365	+ field => op_value.map(&:to_f).reverse,
		366	+ distance: value[:within] \|\| "50mi"
		367	+ }
		368	+ }
		369	+ when :top_left
		370	+ filters << {
		371	+ geo_bounding_box: {
		372	+ field => {
		373	+ top_left: op_value.map(&:to_f).reverse,
		374	+ bottom_right: value[:bottom_right].map(&:to_f).reverse
		375	+ }
		376	+ }
		377	+ }
		378	+ when :not # not equal
		379	+ filters << {not: term_filters(field, op_value)}
		380	+ when :all
		381	+ filters << {terms: {field => op_value, execution: "and"}}
		382	+ when :in
		383	+ filters << term_filters(field, op_value)
		384	+ else
		385	+ range_query =
		386	+ case op
		387	+ when :gt
		388	+ {from: op_value, include_lower: false}
		389	+ when :gte
		390	+ {from: op_value, include_lower: true}
		391	+ when :lt
		392	+ {to: op_value, include_upper: false}
		393	+ when :lte
		394	+ {to: op_value, include_upper: true}
		395	+ else
		396	+ raise "Unknown where operator"
		397	+ end
		398	+ # issue 132
		399	+ if existing = filters.find{ \|f\| f[:range] && f[:range][field] }
		400	+ existing[:range][field].merge!(range_query)
		401	+ else
		402	+ filters << {range: {field => range_query}}
		403	+ end
		404	+ end
		405	+ end
		406	+ else
		407	+ filters << term_filters(field, value)
		408	+ end
		409	+ end
		410	+ end
		411	+ filters
		412	+ end
		413	+
		414	+ def term_filters(field, value)
		415	+ if value.is_a?(Array) # in query
		416	+ if value.any?
		417	+ {or: value.map{\|v\| term_filters(field, v) }}
		418	+ else
		419	+ {terms: {field => value}} # match nothing
		420	+ end
		421	+ elsif value.nil?
		422	+ {missing: {"field" => field, existence: true, null_value: true}}
		423	+ else
		424	+ {term: {field => value}}
		425	+ end
		426	+ end
		427	+
		428	+ end
		429	+end
0	\ No newline at end of file	430	\ No newline at end of file
	@@ -23,5 +23,10 @@ module Searchkick		@@ -23,5 +23,10 @@ module Searchkick
23	def __find_records_by_ids(klass, ids)	23	def __find_records_by_ids(klass, ids)
24	@options[:load] === true ? klass.find(ids) : klass.includes(@options[:load][:include]).find(ids)	24	@options[:load] === true ? klass.find(ids) : klass.includes(@options[:load][:include]).find(ids)
25	end	25	end
		26	+
		27	+ def model_name
		28	+ @options[:model_name]
		29	+ end
		30	+
26	end	31	end
27	end	32	end
	@@ -2,355 +2,12 @@ module Searchkick		@@ -2,355 +2,12 @@ module Searchkick
2	module Search	2	module Search
3		3
4	def search(term, options = {})	4	def search(term, options = {})
5	- if term.is_a?(Hash)
6	- options = term
7	- term = nil	5	+ query = Searchkick::Query.new(self, term, options)
		6	+ if options[:execute] == false
		7	+ query
8	else	8	else
9	- term = term.to_s	9	+ query.execute
10	end	10	end
11	-
12	- fields =
13	- if options[:fields]
14	- if options[:autocomplete]
15	- options[:fields].map{\|f\| "#{f}.autocomplete" }
16	- else
17	- options[:fields].map{\|f\| "#{f}.analyzed" }
18	- end
19	- else
20	- if options[:autocomplete]
21	- (searchkick_options[:autocomplete] \|\| []).map{\|f\| "#{f}.autocomplete" }
22	- else
23	- ["_all"]
24	- end
25	- end
26	-
27	- operator = options[:partial] ? "or" : "and"
28	-
29	- # model and eagar loading
30	- load = options[:load].nil? ? true : options[:load]
31	- load = (options[:include] ? {include: options[:include]} : true) if load
32	-
33	- # pagination
34	- page = [options[:page].to_i, 1].max
35	- per_page = (options[:limit] \|\| options[:per_page] \|\| 100000).to_i
36	- offset = options[:offset] \|\| (page - 1) * per_page
37	- index_name = options[:index_name] \|\| searchkick_index.name
38	-
39	- conversions_field = searchkick_options[:conversions]
40	- personalize_field = searchkick_options[:personalize]
41	-
42	- all = term == "*"
43	-
44	- if options[:query]
45	- payload = options[:query]
46	- elsif options[:similar]
47	- payload = {
48	- more_like_this: {
49	- fields: fields,
50	- like_text: term,
51	- min_doc_freq: 1,
52	- min_term_freq: 1,
53	- analyzer: "searchkick_search2"
54	- }
55	- }
56	- elsif all
57	- payload = {
58	- match_all: {}
59	- }
60	- else
61	- if options[:autocomplete]
62	- payload = {
63	- multi_match: {
64	- fields: fields,
65	- query: term,
66	- analyzer: "searchkick_autocomplete_search"
67	- }
68	- }
69	- else
70	- shared_options = {
71	- fields: fields,
72	- query: term,
73	- use_dis_max: false,
74	- operator: operator
75	- }
76	- queries = [
77	- {multi_match: shared_options.merge(boost: 10, analyzer: "searchkick_search")},
78	- {multi_match: shared_options.merge(boost: 10, analyzer: "searchkick_search2")}
79	- ]
80	- if options[:misspellings] != false
81	- distance = (options[:misspellings].is_a?(Hash) && options[:misspellings][:distance]) \|\| 1
82	- queries.concat [
83	- {multi_match: shared_options.merge(fuzziness: distance, max_expansions: 3, analyzer: "searchkick_search")},
84	- {multi_match: shared_options.merge(fuzziness: distance, max_expansions: 3, analyzer: "searchkick_search2")}
85	- ]
86	- end
87	- payload = {
88	- dis_max: {
89	- queries: queries
90	- }
91	- }
92	- end
93	-
94	- if conversions_field and options[:conversions] != false
95	- # wrap payload in a bool query
96	- payload = {
97	- bool: {
98	- must: payload,
99	- should: {
100	- nested: {
101	- path: conversions_field,
102	- score_mode: "total",
103	- query: {
104	- custom_score: {
105	- query: {
106	- match: {
107	- query: term
108	- }
109	- },
110	- script: "doc['count'].value"
111	- }
112	- }
113	- }
114	- }
115	- }
116	- }
117	- end
118	- end
119	-
120	- custom_filters = []
121	-
122	- if options[:boost]
123	- custom_filters << {
124	- filter: {
125	- exists: {
126	- field: options[:boost]
127	- }
128	- },
129	- script: "log(doc['#{options[:boost]}'].value + 2.718281828)"
130	- }
131	- end
132	-
133	- if options[:user_id] and personalize_field
134	- custom_filters << {
135	- filter: {
136	- term: {
137	- personalize_field => options[:user_id]
138	- }
139	- },
140	- boost: 100
141	- }
142	- end
143	-
144	- if custom_filters.any?
145	- payload = {
146	- custom_filters_score: {
147	- query: payload,
148	- filters: custom_filters,
149	- score_mode: "total"
150	- }
151	- }
152	- end
153	-
154	- payload = {
155	- query: payload,
156	- size: per_page,
157	- from: offset
158	- }
159	- payload[:explain] = options[:explain] if options[:explain]
160	-
161	- # order
162	- if options[:order]
163	- order = options[:order].is_a?(Enumerable) ? options[:order] : {options[:order] => :asc}
164	- payload[:sort] = order
165	- end
166	-
167	- # where
168	- where_filters =
169	- proc do \|where\|
170	- filters = []
171	- (where \|\| {}).each do \|field, value\|
172	- field = :_id if field.to_s == "id"
173	-
174	- if field == :or
175	- value.each do \|or_clause\|
176	- filters << {or: or_clause.map{\|or_statement\| {and: where_filters.call(or_statement)} }}
177	- end
178	- else
179	- # expand ranges
180	- if value.is_a?(Range)
181	- value = {gte: value.first, (value.exclude_end? ? :lt : :lte) => value.last}
182	- end
183	-
184	- if value.is_a?(Array) # in query
185	- filters << {terms: {field => value}}
186	- elsif value.is_a?(Hash)
187	- if value[:near]
188	- filters << {
189	- geo_distance: {
190	- field => value.delete(:near).map(&:to_f).reverse,
191	- distance: value.delete(:within) \|\| "50mi"
192	- }
193	- }
194	- end
195	-
196	- if value[:top_left]
197	- filters << {
198	- geo_bounding_box: {
199	- field => {
200	- top_left: value.delete(:top_left).map(&:to_f).reverse,
201	- bottom_right: value.delete(:bottom_right).map(&:to_f).reverse
202	- }
203	- }
204	- }
205	- end
206	-
207	- value.each do \|op, op_value\|
208	- if op == :not # not equal
209	- if op_value.is_a?(Array)
210	- filters << {not: {terms: {field => op_value}}}
211	- else
212	- filters << {not: {term: {field => op_value}}}
213	- end
214	- elsif op == :all
215	- filters << {terms: {field => op_value, execution: "and"}}
216	- else
217	- range_query =
218	- case op
219	- when :gt
220	- {from: op_value, include_lower: false}
221	- when :gte
222	- {from: op_value, include_lower: true}
223	- when :lt
224	- {to: op_value, include_upper: false}
225	- when :lte
226	- {to: op_value, include_upper: true}
227	- else
228	- raise "Unknown where operator"
229	- end
230	- filters << {range: {field => range_query}}
231	- end
232	- end
233	- else
234	- if value.nil?
235	- filters << {missing: {"field" => field, existence: true, null_value: true}}
236	- else
237	- filters << {term: {field => value}}
238	- end
239	- end
240	- end
241	- end
242	- filters
243	- end
244	-
245	- # filters
246	- filters = where_filters.call(options[:where])
247	- if filters.any?
248	- payload[:filter] = {
249	- and: filters
250	- }
251	- end
252	-
253	- # facets
254	- facet_limits = {}
255	- if options[:facets]
256	- facets = options[:facets] \|\| {}
257	- if facets.is_a?(Array) # convert to more advanced syntax
258	- facets = Hash[ facets.map{\|f\| [f, {}] } ]
259	- end
260	-
261	- payload[:facets] = {}
262	- facets.each do \|field, facet_options\|
263	- # ask for extra facets due to
264	- # https://github.com/elasticsearch/elasticsearch/issues/1305
265	-
266	- if facet_options[:ranges]
267	- payload[:facets][field] = {
268	- range: {
269	- field.to_sym => facet_options[:ranges]
270	- }
271	- }
272	- else
273	- payload[:facets][field] = {
274	- terms: {
275	- field: field,
276	- size: facet_options[:limit] ? facet_options[:limit] + 150 : 100000
277	- }
278	- }
279	- end
280	-
281	- facet_limits[field] = facet_options[:limit] if facet_options[:limit]
282	-
283	- # offset is not possible
284	- # http://elasticsearch-users.115913.n3.nabble.com/Is-pagination-possible-in-termsStatsFacet-td3422943.html
285	-
286	- facet_filters = where_filters.call(facet_options[:where])
287	- if facet_filters.any?
288	- payload[:facets][field][:facet_filter] = {
289	- and: {
290	- filters: facet_filters
291	- }
292	- }
293	- end
294	- end
295	- end
296	-
297	- # suggestions
298	- if options[:suggest]
299	- suggest_fields = (searchkick_options[:suggest] \|\| []).map(&:to_s)
300	- # intersection
301	- suggest_fields = suggest_fields & options[:fields].map(&:to_s) if options[:fields]
302	- if suggest_fields.any?
303	- payload[:suggest] = {text: term}
304	- suggest_fields.each do \|field\|
305	- payload[:suggest][field] = {
306	- phrase: {
307	- field: "#{field}.suggest"
308	- }
309	- }
310	- end
311	- end
312	- end
313	-
314	- # highlight
315	- if options[:highlight]
316	- payload[:highlight] = {
317	- fields: Hash[ fields.map{\|f\| [f, {}] } ]
318	- }
319	- if options[:highlight].is_a?(Hash) and tag = options[:highlight][:tag]
320	- payload[:highlight][:pre_tags] = [tag]
321	- payload[:highlight][:post_tags] = [tag.to_s.gsub(/\A</, "</")]
322	- end
323	- end
324	-
325	- # An empty array will cause only the _id and _type for each hit to be returned
326	- # http://www.elasticsearch.org/guide/reference/api/search/fields/
327	- payload[:fields] = [] if load
328	-
329	- tire_options = {load: load, payload: payload, size: per_page, from: offset}
330	- tire_options[:type] = document_type if self != searchkick_klass
331	- begin
332	- response = Searchkick.client.search index: index_name, body: payload
333	- rescue => e
334	- status_code = e.message[0..3].to_i
335	- if status_code == 404
336	- raise "Index missing - run #{searchkick_klass.name}.reindex"
337	- elsif status_code == 500 and (e.message.include?("IllegalArgumentException[minimumSimilarity >= 1]") or e.message.include?("No query registered for [multi_match]"))
338	- raise "Upgrade Elasticsearch to 0.90.0 or greater"
339	- else
340	- raise e
341	- end
342	- end
343	-
344	- # apply facet limit in client due to
345	- # https://github.com/elasticsearch/elasticsearch/issues/1305
346	- facet_limits.each do \|field, limit\|
347	- field = field.to_s
348	- facet = response["facets"][field]
349	- response["facets"][field]["terms"] = facet["terms"].first(limit)
350	- response["facets"][field]["other"] = facet["total"] - facet["terms"].sum{\|term\| term["count"] }
351	- end
352	-
353	- Searchkick::Results.new(response, search.options.merge(term: term))
354	end	11	end
355		12
356	end	13	end
1	module Searchkick	1	module Searchkick
2	- VERSION = "0.4.2"	2	+ VERSION = "0.5.3"
3	end	3	end
	@@ -23,6 +23,4 @@ Gem::Specification.new do \|spec\|		@@ -23,6 +23,4 @@ Gem::Specification.new do \|spec\|
23	spec.add_development_dependency "bundler", "~> 1.3"	23	spec.add_development_dependency "bundler", "~> 1.3"
24	spec.add_development_dependency "rake"	24	spec.add_development_dependency "rake"
25	spec.add_development_dependency "minitest", "~> 4.7"	25	spec.add_development_dependency "minitest", "~> 4.7"
26	- spec.add_development_dependency "activerecord"
27	- spec.add_development_dependency "pg"
28	end	26	end
	@@ -17,4 +17,39 @@ class TestAutocomplete < Minitest::Unit::TestCase		@@ -17,4 +17,39 @@ class TestAutocomplete < Minitest::Unit::TestCase
17	assert_search "hum", ["Hummus"], autocomplete: true, fields: [:name]	17	assert_search "hum", ["Hummus"], autocomplete: true, fields: [:name]
18	end	18	end
19		19
		20	+ def test_text_start
		21	+ store_names ["Where in the World is Carmen San Diego"]
		22	+ assert_search "where in the world is", ["Where in the World is Carmen San Diego"], fields: [{name: :text_start}]
		23	+ assert_search "in the world", [], fields: [{name: :text_start}]
		24	+ end
		25	+
		26	+ def test_text_middle
		27	+ store_names ["Where in the World is Carmen San Diego"]
		28	+ assert_search "where in the world is", ["Where in the World is Carmen San Diego"], fields: [{name: :text_middle}]
		29	+ assert_search "n the wor", ["Where in the World is Carmen San Diego"], fields: [{name: :text_middle}]
		30	+ assert_search "men san diego", ["Where in the World is Carmen San Diego"], fields: [{name: :text_middle}]
		31	+ assert_search "world carmen", [], fields: [{name: :text_middle}]
		32	+ end
		33	+
		34	+ def test_text_end
		35	+ store_names ["Where in the World is Carmen San Diego"]
		36	+ assert_search "men san diego", ["Where in the World is Carmen San Diego"], fields: [{name: :text_end}]
		37	+ assert_search "carmen san", [], fields: [{name: :text_end}]
		38	+ end
		39	+
		40	+ def test_word_start
		41	+ store_names ["Where in the World is Carmen San Diego"]
		42	+ assert_search "car san wor", ["Where in the World is Carmen San Diego"], fields: [{name: :word_start}]
		43	+ end
		44	+
		45	+ def test_word_middle
		46	+ store_names ["Where in the World is Carmen San Diego"]
		47	+ assert_search "orl", ["Where in the World is Carmen San Diego"], fields: [{name: :word_middle}]
		48	+ end
		49	+
		50	+ def test_word_end
		51	+ store_names ["Where in the World is Carmen San Diego"]
		52	+ assert_search "rld men ego", ["Where in the World is Carmen San Diego"], fields: [{name: :word_end}]
		53	+ end
		54	+
20	end	55	end
	@@ -57,4 +57,14 @@ class TestBoost < Minitest::Unit::TestCase		@@ -57,4 +57,14 @@ class TestBoost < Minitest::Unit::TestCase
57	assert_first "tomato", "Tomato B", user_id: 2	57	assert_first "tomato", "Tomato B", user_id: 2
58	end	58	end
59		59
		60	+ def test_personalize
		61	+ store [
		62	+ {name: "Tomato A"},
		63	+ {name: "Tomato B", user_ids: [1, 2, 3]},
		64	+ {name: "Tomato C"},
		65	+ {name: "Tomato D"}
		66	+ ]
		67	+ assert_first "tomato", "Tomato B", personalize: {user_ids: 2}
		68	+ end
		69	+
60	end	70	end