github / searchkick

Download as
Browse Code ยป

Commit 684eecf572162fa45942f301fb39e3d5fad5e6e7

Authored by Andrew Kane
2 parents d6abba8d 07c41eea
Exists in master and in 2 other branches allow_missing, conversions_v2

Merge branch 'edge'

Showing 68 changed files with 1728 additions and 1467 deletions   Show diff stats
.github/ISSUE_TEMPLATE/bug_report.md
  Diff comments   View file @684eecf
... ... @@ -26,10 +26,13 @@ gemfile do
26 26 gem "activejob", require: "active_job"
27 27 gem "sqlite3"
28 28 gem "searchkick", git: "https://github.com/ankane/searchkick.git"
  29 + # uncomment one
  30 + # gem "elasticsearch"
  31 + # gem "opensearch-ruby"
29 32 end
30 33  
31 34 puts "Searchkick version: #{Searchkick::VERSION}"
32   -puts "Elasticsearch version: #{Searchkick.server_version}"
  35 +puts "Server version: #{Searchkick.server_version}"
33 36  
34 37 ActiveRecord::Base.establish_connection adapter: "sqlite3", database: ":memory:"
35 38 ActiveJob::Base.queue_adapter = :inline
... ...
.github/workflows/build.yml
  Diff comments   View file @684eecf
... ... @@ -21,19 +21,13 @@ jobs:
21 21 - ruby: 2.6
22 22 gemfile: gemfiles/activerecord52.gemfile
23 23 elasticsearch: 7.0.0
24   - - ruby: 2.5
25   - gemfile: gemfiles/activerecord51.gemfile
26   - elasticsearch: 6.8.23
27   - - ruby: 2.4
28   - gemfile: gemfiles/activerecord50.gemfile
29   - elasticsearch: 6.0.0
30 24 - ruby: 2.7
31 25 gemfile: gemfiles/mongoid7.gemfile
32 26 elasticsearch: 7
33 27 mongodb: true
34 28 - ruby: 2.6
35 29 gemfile: gemfiles/mongoid6.gemfile
36   - elasticsearch: 6
  30 + elasticsearch: 7
37 31 mongodb: true
38 32 runs-on: ubuntu-latest
39 33 env:
... ...
CHANGELOG.md
  Diff comments   View file @684eecf
  1 +## 5.0.0 (unreleased)
  2 +
  3 +- Searches now use lazy loading (similar to Active Record)
  4 +- Added `unscope` option to better support working with default scopes
  5 +- Added support for `:async` and `:queue` modes for `reindex` on relation
  6 +- Added basic protection from unfiltered parameters to `where` option
  7 +- Added `models` option to `similar` method
  8 +- Changed async full reindex to fetch ids instead of using ranges for numeric primary keys with Active Record
  9 +- Changed `searchkick_index_options` to return symbol keys (instead of mix of strings and symbols)
  10 +- Changed non-anchored regular expressions to match expected results (previously warned)
  11 +- Changed record reindex to return `true` to match model and relation reindex
  12 +- Updated async reindex job to call `search_import` for nested associations
  13 +- Fixed removing records when `should_index?` is `false` when `reindex` called on relation
  14 +- Fixed issue with `merge_mappings` for fields that use `searchkick` options
  15 +- Raise error when `search` called on relations
  16 +- Raise `ArgumentError` (instead of warning) for invalid regular expression modifiers
  17 +- Raise `ArgumentError` instead of `RuntimeError` for unknown operators
  18 +- Removed mapping of `id` to `_id` with `order` option
  19 +- Removed `wordnet` option (no longer worked)
  20 +- Removed dependency on `elasticsearch` gem (can use `elasticsearch` or `opensearch-ruby`)
  21 +- Dropped support for Elasticsearch 6
  22 +- Dropped support for Ruby < 2.6 and Active Record < 5.2
  23 +- Dropped support for NoBrainer and Cequel
  24 +- Dropped support for `faraday_middleware-aws-signers-v4` (use `faraday_middleware-aws-sigv4` instead)
  25 +
1 26 ## 4.6.3 (2021-11-19)
2 27  
3 28 - Added support for reloadable synonyms for OpenSearch
4   -- Added experimental support for `opensearch` gem
  29 +- Added experimental support for `opensearch-ruby` gem
5 30 - Removed `elasticsearch-xpack` dependency for reloadable synonyms
6 31  
7 32 ## 4.6.2 (2021-11-15)
... ...
Gemfile
  Diff comments   View file @684eecf
... ... @@ -7,6 +7,7 @@ gem &quot;minitest&quot;, &quot;&gt;= 5&quot;
7 7 gem "activerecord", "~> 7.0.0"
8 8 gem "activejob", "~> 7.0.0", require: "active_job"
9 9 gem "actionpack", "~> 7.0.0"
  10 +gem "elasticsearch"
10 11 gem "sqlite3"
11 12 gem "gemoji-parser"
12 13 gem "typhoeus"
... ...
README.md
  Diff comments   View file @684eecf
... ... @@ -20,7 +20,7 @@ Plus:
20 20 - autocomplete
21 21 - โ€œDid you meanโ€ suggestions
22 22 - supports many languages
23   -- works with Active Record, Mongoid, and NoBrainer
  23 +- works with Active Record and Mongoid
24 24  
25 25 Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Autosuggest](https://github.com/ankane/autosuggest) for query suggestions
26 26  
... ... @@ -43,22 +43,30 @@ Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Au
43 43 - [Reference](#reference)
44 44 - [Contributing](#contributing)
45 45  
  46 +Searchkick 5.0 was recently released! See [how to upgrade](#upgrading)
  47 +
46 48 ## Getting Started
47 49  
48 50 Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) or [OpenSearch](https://opensearch.org/downloads.html). For Homebrew, use:
49 51  
50 52 ```sh
51   -brew install elasticsearch # or opensearch
52   -brew services start elasticsearch # or opensearch
  53 +brew install elasticsearch
  54 +brew services start elasticsearch
  55 +# or
  56 +brew install opensearch
  57 +brew services start opensearch
53 58 ```
54 59  
55   -Add this line to your applicationโ€™s Gemfile:
  60 +Add these lines to your applicationโ€™s Gemfile:
56 61  
57 62 ```ruby
58 63 gem "searchkick"
  64 +
  65 +gem "elasticsearch" # select one
  66 +gem "opensearch-ruby" # select one
59 67 ```
60 68  
61   -The latest version works with Elasticsearch 6 and 7 and OpenSearch 1. For Elasticsearch 5, use version 3.1.3 and [this readme](https://github.com/ankane/searchkick/blob/v3.1.3/README.md).
  69 +The latest version works with Elasticsearch 7 and 8 and OpenSearch 1. For Elasticsearch 6, use version 4.6.3 and [this readme](https://github.com/ankane/searchkick/blob/v4.6.3/README.md).
62 70  
63 71 Add searchkick to models you want to search.
64 72  
... ... @@ -83,7 +91,7 @@ products.each do |product|
83 91 end
84 92 ```
85 93  
86   -Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html). As your search becomes more advanced, we recommend you use the [Elasticsearch DSL](#advanced) for maximum flexibility.
  94 +Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and [OpenSearch Search API](https://opensearch.org/docs/latest/opensearch/rest-api/search/). As your search becomes more advanced, we recommend you use the [search server DSL](#advanced) for maximum flexibility.
87 95  
88 96 ## Querying
89 97  
... ... @@ -144,7 +152,7 @@ select: [:name]
144 152  
145 153 ### Results
146 154  
147   -Searches return a `Searchkick::Results` object. This responds like an array to most methods.
  155 +Searches return a `Searchkick::Relation` object. This responds like an array to most methods.
148 156  
149 157 ```ruby
150 158 results = Product.search("milk")
... ... @@ -177,7 +185,7 @@ Get the full response from the search server
177 185 results.response
178 186 ```
179 187  
180   -**Note:** By default, Elasticsearch and OpenSearch [limit paging](#deep-paging) to the first 10,000 results for performance. With Elasticsearch 7 and OpenSearch, this applies to the total count as well.
  188 +**Note:** By default, Elasticsearch and OpenSearch [limit paging](#deep-paging) to the first 10,000 results for performance. This applies to the total count as well.
181 189  
182 190 ### Boosting
183 191  
... ... @@ -616,6 +624,28 @@ class Image &lt; ApplicationRecord
616 624 end
617 625 ```
618 626  
  627 +### Default Scopes
  628 +
  629 +If you have a default scope that filters records, use the `should_index?` method to exclude them from indexing:
  630 +
  631 +```ruby
  632 +class Product < ApplicationRecord
  633 + default_scope { where(deleted_at: nil) }
  634 +
  635 + def should_index?
  636 + deleted_at.nil?
  637 + end
  638 +end
  639 +```
  640 +
  641 +If you want to index and search filtered records, set:
  642 +
  643 +```ruby
  644 +class Product < ApplicationRecord
  645 + searchkick unscope: true
  646 +end
  647 +```
  648 +
619 649 ## Intelligent Search
620 650  
621 651 The best starting point to improve your search **by far** is to track searches and conversions. [Searchjoy](https://github.com/ankane/searchjoy) makes it easy.
... ... @@ -1221,7 +1251,7 @@ And [setup-opensearch](https://github.com/ankane/setup-opensearch) for an easy w
1221 1251  
1222 1252 ## Deployment
1223 1253  
1224   -Searchkick uses `ENV["ELASTICSEARCH_URL"]` for the search server. This defaults to `http://localhost:9200`.
  1254 +For the search server, Searchkick uses `ENV["ELASTICSEARCH_URL"]` for Elasticsearch and `ENV["OPENSEARCH_URL"]` for OpenSearch. This defaults to `http://localhost:9200`.
1225 1255  
1226 1256 - [Elastic Cloud](#elastic-cloud)
1227 1257 - [Heroku](#heroku)
... ... @@ -1246,13 +1276,20 @@ rake searchkick:reindex:all
1246 1276  
1247 1277 Choose an add-on: [Bonsai](https://elements.heroku.com/addons/bonsai), [SearchBox](https://elements.heroku.com/addons/searchbox), or [Elastic Cloud](https://elements.heroku.com/addons/foundelasticsearch).
1248 1278  
1249   -For Bonsai:
  1279 +For Elasticsearch on Bonsai:
1250 1280  
1251 1281 ```sh
1252   -heroku addons:create bonsai # use --engine=opensearch for OpenSearch
  1282 +heroku addons:create bonsai
1253 1283 heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`
1254 1284 ```
1255 1285  
  1286 +For OpenSearch on Bonsai:
  1287 +
  1288 +```sh
  1289 +heroku addons:create bonsai --engine=opensearch
  1290 +heroku config:set OPENSEARCH_URL=`heroku config:get BONSAI_URL`
  1291 +```
  1292 +
1256 1293 For SearchBox:
1257 1294  
1258 1295 ```sh
... ... @@ -1287,10 +1324,10 @@ heroku run rake searchkick:reindex:all
1287 1324  
1288 1325 ### Amazon OpenSearch Service
1289 1326  
1290   -Create an initializer `config/initializers/elasticsearch.rb` with:
  1327 +Create an initializer `config/initializers/opensearch.rb` with:
1291 1328  
1292 1329 ```ruby
1293   -ENV["ELASTICSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"
  1330 +ENV["OPENSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"
1294 1331 ```
1295 1332  
1296 1333 To use signed requests, include in your Gemfile:
... ... @@ -1317,10 +1354,12 @@ rake searchkick:reindex:all
1317 1354  
1318 1355 ### Self-Hosted and Other
1319 1356  
1320   -Create an initializer `config/initializers/elasticsearch.rb` with:
  1357 +Create an initializer with:
1321 1358  
1322 1359 ```ruby
1323 1360 ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
  1361 +# or
  1362 +ENV["OPENSEARCH_URL"] = "https://user:password@host:port"
1324 1363 ```
1325 1364  
1326 1365 Then deploy and reindex:
... ... @@ -1616,7 +1655,7 @@ ReindexConversionsJob.perform_later(&quot;Product&quot;)
1616 1655  
1617 1656 ## Advanced
1618 1657  
1619   -Searchkick makes it easy to use the Elasticsearch DSL on its own.
  1658 +Searchkick makes it easy to use the Elasticsearch or OpenSearch DSL on its own.
1620 1659  
1621 1660 ### Advanced Mapping
1622 1661  
... ... @@ -1683,8 +1722,8 @@ Searchkick.client
1683 1722 To batch search requests for performance, use:
1684 1723  
1685 1724 ```ruby
1686   -products = Product.search("snacks", execute: false)
1687   -coupons = Coupon.search("snacks", execute: false)
  1725 +products = Product.search("snacks")
  1726 +coupons = Coupon.search("snacks")
1688 1727 Searchkick.multi_search([products, coupons])
1689 1728 ```
1690 1729  
... ... @@ -1743,7 +1782,7 @@ class Product &lt; ApplicationRecord
1743 1782 end
1744 1783 ```
1745 1784  
1746   -If you just need an accurate total count with Elasticsearch 7 and OpenSearch, you can instead use:
  1785 +If you just need an accurate total count, you can instead use:
1747 1786  
1748 1787 ```ruby
1749 1788 Product.search("pears", body_options: {track_total_hits: true})
... ... @@ -1958,13 +1997,6 @@ class Product &lt; ApplicationRecord
1958 1997 end
1959 1998 ```
1960 1999  
1961   -Lazy searching
1962   -
1963   -```ruby
1964   -products = Product.search("carrots", execute: false)
1965   -products.each { ... } # search not executed until here
1966   -```
1967   -
1968 2000 Add [request parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#search-search-api-query-params) like `search_type`
1969 2001  
1970 2002 ```ruby
... ... @@ -2008,11 +2040,6 @@ Product.search &quot;api&quot;, misspellings: {prefix_length: 2} # api, apt, no ahi
2008 2040 Product.search "ah", misspellings: {prefix_length: 2} # ah, no aha
2009 2041 ```
2010 2042  
2011   -## Elasticsearch 6 to 7 Upgrade
2012   -
2013   -1. Install Searchkick 4
2014   -2. Upgrade your Elasticsearch cluster
2015   -
2016 2043 ## Gotchas
2017 2044  
2018 2045 ### Consistency
... ... @@ -2036,6 +2063,34 @@ end
2036 2063  
2037 2064 For convenience, this is set by default in the test environment.
2038 2065  
  2066 +## Upgrading
  2067 +
  2068 +### 5.0
  2069 +
  2070 +Searchkick 5 supports both the `elasticsearch` and `opensearch-ruby` gems. Add the one you want to use to your Gemfile:
  2071 +
  2072 +```ruby
  2073 +gem "elasticsearch"
  2074 +# or
  2075 +gem "opensearch-ruby"
  2076 +```
  2077 +
  2078 +If using the deprecated `faraday_middleware-aws-signers-v4` gem, switch to `faraday_middleware-aws-sigv4`.
  2079 +
  2080 +Also, searches now use lazy loading:
  2081 +
  2082 +```ruby
  2083 +# search not executed
  2084 +Product.search("milk")
  2085 +
  2086 +# search executed
  2087 +Product.search("milk").to_a
  2088 +```
  2089 +
  2090 +And thereโ€™s a [new option](#default-scopes) for models with default scopes.
  2091 +
  2092 +Check out the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md) for the full list of changes.
  2093 +
2039 2094 ## History
2040 2095  
2041 2096 View the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md).
... ...
benchmark/search.rb
  Diff comments   View file @684eecf
... ... @@ -45,7 +45,7 @@ if ENV[&quot;SETUP&quot;]
45 45 puts "Reindexed"
46 46 end
47 47  
48   -query = Product.search("product", fields: [:name], where: {color: "red", store_id: 5}, limit: 10000, load: false, execute: false)
  48 +query = Product.search("product", fields: [:name], where: {color: "red", store_id: 5}, limit: 10000, load: false)
49 49  
50 50 require "pp"
51 51 pp query.body.as_json
... ...
gemfiles/activerecord50.gemfile
View file @d6abba8
... ... @@ -1,15 +0,0 @@
1   -source "https://rubygems.org"
2   -
3   -gemspec path: ".."
4   -
5   -gem "rake"
6   -gem "minitest", ">= 5"
7   -gem "sqlite3", "~> 1.3.0"
8   -gem "activerecord", "~> 5.0.0"
9   -gem "actionpack", "~> 5.0.0"
10   -gem "activejob", "~> 5.0.0", require: "active_job"
11   -gem "elasticsearch", "~> 6"
12   -gem "redis"
13   -gem "connection_pool"
14   -gem "kaminari"
15   -gem "gemoji-parser"
gemfiles/activerecord51.gemfile
View file @d6abba8
... ... @@ -1,15 +0,0 @@
1   -source "https://rubygems.org"
2   -
3   -gemspec path: ".."
4   -
5   -gem "rake"
6   -gem "minitest", ">= 5"
7   -gem "sqlite3"
8   -gem "activerecord", "~> 5.1.0"
9   -gem "actionpack", "~> 5.1.0"
10   -gem "activejob", "~> 5.1.0", require: "active_job"
11   -gem "elasticsearch", "~> 6"
12   -gem "redis"
13   -gem "connection_pool"
14   -gem "kaminari"
15   -gem "gemoji-parser"
gemfiles/cequel.gemfile
View file @d6abba8
... ... @@ -1,9 +0,0 @@
1   -source "https://rubygems.org"
2   -
3   -gemspec path: ".."
4   -
5   -gem "rake"
6   -gem "minitest", ">= 5"
7   -gem "cequel"
8   -gem "activejob"
9   -gem "redis"
gemfiles/nobrainer.gemfile
View file @d6abba8
... ... @@ -1,9 +0,0 @@
1   -source "https://rubygems.org"
2   -
3   -gemspec path: ".."
4   -
5   -gem "rake"
6   -gem "minitest", ">= 5"
7   -gem "nobrainer", ">= 0.21.0"
8   -gem "activejob"
9   -gem "redis"
gemfiles/opensearch.gemfile
  Diff comments   View file @684eecf
... ... @@ -13,5 +13,5 @@ gem &quot;typhoeus&quot;
13 13 gem "redis"
14 14 gem "connection_pool"
15 15 gem "kaminari"
16   -gem "opensearch-ruby", github: "opensearch-project/opensearch-ruby"
  16 +gem "opensearch-ruby"
17 17 gem "parallel_tests"
... ...
lib/searchkick.rb
  Diff comments   View file @684eecf
1 1 # dependencies
2 2 require "active_support"
3 3 require "active_support/core_ext/hash/deep_merge"
4   -require "elasticsearch"
  4 +require "active_support/core_ext/module/attr_internal"
  5 +require "active_support/core_ext/module/delegation"
  6 +require "active_support/notifications"
5 7 require "hashie"
6 8  
  9 +# stdlib
  10 +require "forwardable"
  11 +
7 12 # modules
8   -require "searchkick/bulk_indexer"
  13 +require "searchkick/controller_runtime"
9 14 require "searchkick/index"
  15 +require "searchkick/index_cache"
  16 +require "searchkick/index_options"
10 17 require "searchkick/indexer"
11 18 require "searchkick/hash_wrapper"
12   -require "searchkick/middleware"
  19 +require "searchkick/log_subscriber"
13 20 require "searchkick/model"
14 21 require "searchkick/multi_search"
15 22 require "searchkick/query"
16 23 require "searchkick/reindex_queue"
17 24 require "searchkick/record_data"
18 25 require "searchkick/record_indexer"
  26 +require "searchkick/relation"
  27 +require "searchkick/relation_indexer"
19 28 require "searchkick/results"
20 29 require "searchkick/version"
21 30  
22 31 # integrations
23 32 require "searchkick/railtie" if defined?(Rails)
24   -require "searchkick/logging" if defined?(ActiveSupport::Notifications)
25 33  
26 34 module Searchkick
  35 + # requires faraday
  36 + autoload :Middleware, "searchkick/middleware"
  37 +
27 38 # background jobs
28 39 autoload :BulkReindexJob, "searchkick/bulk_reindex_job"
29 40 autoload :ProcessBatchJob, "searchkick/process_batch_job"
... ... @@ -33,19 +44,21 @@ module Searchkick
33 44 # errors
34 45 class Error < StandardError; end
35 46 class MissingIndexError < Error; end
36   - class UnsupportedVersionError < Error; end
37   - # TODO switch to Error
38   - class InvalidQueryError < Elasticsearch::Transport::Transport::Errors::BadRequest; end
  47 + class UnsupportedVersionError < Error
  48 + def message
  49 + "This version of Searchkick requires Elasticsearch 7+ or OpenSearch 1+"
  50 + end
  51 + end
  52 + class InvalidQueryError < Error; end
39 53 class DangerousOperation < Error; end
40 54 class ImportError < Error; end
41 55  
42 56 class << self
43   - attr_accessor :search_method_name, :wordnet_path, :timeout, :models, :client_options, :redis, :index_prefix, :index_suffix, :queue_name, :model_options
  57 + attr_accessor :search_method_name, :timeout, :models, :client_options, :redis, :index_prefix, :index_suffix, :queue_name, :model_options, :client_type
44 58 attr_writer :client, :env, :search_timeout
45 59 attr_reader :aws_credentials
46 60 end
47 61 self.search_method_name = :search
48   - self.wordnet_path = "/var/lib/wn_s.pl"
49 62 self.timeout = 10
50 63 self.models = []
51 64 self.client_options = {}
... ... @@ -54,15 +67,45 @@ module Searchkick
54 67  
55 68 def self.client
56 69 @client ||= begin
57   - require "typhoeus/adapters/faraday" if defined?(Typhoeus) && Gem::Version.new(Faraday::VERSION) < Gem::Version.new("0.14.0")
58   -
59   - Elasticsearch::Client.new({
60   - url: ENV["ELASTICSEARCH_URL"] || ENV["OPENSEARCH_URL"],
61   - transport_options: {request: {timeout: timeout}, headers: {content_type: "application/json"}},
62   - retry_on_failure: 2
63   - }.deep_merge(client_options)) do |f|
64   - f.use Searchkick::Middleware
65   - f.request signer_middleware_key, signer_middleware_aws_params if aws_credentials
  70 + client_type =
  71 + if self.client_type
  72 + self.client_type
  73 + elsif defined?(OpenSearch::Client) && defined?(Elasticsearch::Client)
  74 + raise Error, "Multiple clients found - set Searchkick.client_type = :elasticsearch or :opensearch"
  75 + elsif defined?(OpenSearch::Client)
  76 + :opensearch
  77 + elsif defined?(Elasticsearch::Client)
  78 + :elasticsearch
  79 + else
  80 + raise Error, "No client found - install the `elasticsearch` or `opensearch-ruby` gem"
  81 + end
  82 +
  83 + # check after client to ensure faraday is installed
  84 + # TODO remove in Searchkick 6
  85 + if defined?(Typhoeus) && Gem::Version.new(Faraday::VERSION) < Gem::Version.new("0.14.0")
  86 + require "typhoeus/adapters/faraday"
  87 + end
  88 +
  89 + if client_type == :opensearch
  90 + OpenSearch::Client.new({
  91 + url: ENV["OPENSEARCH_URL"],
  92 + transport_options: {request: {timeout: timeout}, headers: {content_type: "application/json"}},
  93 + retry_on_failure: 2
  94 + }.deep_merge(client_options)) do |f|
  95 + f.use Searchkick::Middleware
  96 + f.request :aws_sigv4, signer_middleware_aws_params if aws_credentials
  97 + end
  98 + else
  99 + raise Error, "The `elasticsearch` gem must be 7+" if Elasticsearch::VERSION.to_i < 7
  100 +
  101 + Elasticsearch::Client.new({
  102 + url: ENV["ELASTICSEARCH_URL"],
  103 + transport_options: {request: {timeout: timeout}, headers: {content_type: "application/json"}},
  104 + retry_on_failure: 2
  105 + }.deep_merge(client_options)) do |f|
  106 + f.use Searchkick::Middleware
  107 + f.request :aws_sigv4, signer_middleware_aws_params if aws_credentials
  108 + end
66 109 end
67 110 end
68 111 end
... ... @@ -96,14 +139,6 @@ module Searchkick
96 139 Gem::Version.new(server_version.split("-")[0]) < Gem::Version.new(version.split("-")[0])
97 140 end
98 141  
99   - # memoize for performance
100   - def self.server_below7?
101   - unless defined?(@server_below7)
102   - @server_below7 = server_below?("7.0.0")
103   - end
104   - @server_below7
105   - end
106   -
107 142 def self.search(term = "*", model: nil, **options, &block)
108 143 options = options.dup
109 144 klass = model
... ... @@ -129,17 +164,27 @@ module Searchkick
129 164 end
130 165 end
131 166  
132   - options = options.merge(block: block) if block
133   - query = Searchkick::Query.new(klass, term, **options)
  167 + # TODO remove in Searchkick 6
134 168 if options[:execute] == false
135   - query
136   - else
137   - query.execute
  169 + Searchkick.warn("The execute option is no longer needed")
  170 + options.delete(:execute)
138 171 end
  172 +
  173 + options = options.merge(block: block) if block
  174 + Searchkick::Relation.new(klass, term, **options)
139 175 end
140 176  
141 177 def self.multi_search(queries)
142   - Searchkick::MultiSearch.new(queries).perform
  178 + return if queries.empty?
  179 +
  180 + queries = queries.map { |q| q.send(:query) }
  181 + event = {
  182 + name: "Multi Search",
  183 + body: queries.flat_map { |q| [q.params.except(:body).to_json, q.body.to_json] }.map { |v| "#{v}\n" }.join,
  184 + }
  185 + ActiveSupport::Notifications.instrument("multi_search.searchkick", event) do
  186 + Searchkick::MultiSearch.new(queries).perform
  187 + end
143 188 end
144 189  
145 190 # callbacks
... ... @@ -160,13 +205,25 @@ module Searchkick
160 205 end
161 206 end
162 207  
163   - def self.callbacks(value = nil)
  208 + # message is private
  209 + def self.callbacks(value = nil, message: nil)
164 210 if block_given?
165 211 previous_value = callbacks_value
166 212 begin
167 213 self.callbacks_value = value
168 214 result = yield
169   - indexer.perform if callbacks_value == :bulk
  215 + if callbacks_value == :bulk && indexer.queued_items.any?
  216 + event = {}
  217 + if message
  218 + message.call(event)
  219 + else
  220 + event[:name] = "Bulk"
  221 + event[:count] = indexer.queued_items.size
  222 + end
  223 + ActiveSupport::Notifications.instrument("request.searchkick", event) do
  224 + indexer.perform
  225 + end
  226 + end
170 227 result
171 228 ensure
172 229 self.callbacks_value = previous_value
... ... @@ -177,12 +234,8 @@ module Searchkick
177 234 end
178 235  
179 236 def self.aws_credentials=(creds)
180   - begin
181   - # TODO remove in Searchkick 5 (just use aws_sigv4)
182   - require "faraday_middleware/aws_signers_v4"
183   - rescue LoadError
184   - require "faraday_middleware/aws_sigv4"
185   - end
  237 + require "faraday_middleware/aws_sigv4"
  238 +
186 239 @aws_credentials = creds
187 240 @client = nil # reset client
188 241 end
... ... @@ -197,7 +250,6 @@ module Searchkick
197 250 }
198 251 end
199 252  
200   - # TODO use ConnectionPool::Wrapper when redis is set so this is no longer needed
201 253 def self.with_redis
202 254 if redis
203 255 if redis.respond_to?(:with)
... ... @@ -215,29 +267,41 @@ module Searchkick
215 267 end
216 268  
217 269 # private
218   - def self.load_records(records, ids)
219   - records =
220   - if records.respond_to?(:primary_key)
221   - # ActiveRecord
222   - records.where(records.primary_key => ids) if records.primary_key
223   - elsif records.respond_to?(:queryable)
224   - # Mongoid 3+
225   - records.queryable.for_ids(ids)
226   - elsif records.respond_to?(:unscoped) && :id.respond_to?(:in)
227   - # Nobrainer
228   - records.unscoped.where(:id.in => ids)
229   - elsif records.respond_to?(:key_column_names)
230   - records.where(records.key_column_names.first => ids)
  270 + def self.load_records(relation, ids)
  271 + relation =
  272 + if relation.respond_to?(:primary_key)
  273 + primary_key = relation.primary_key
  274 + raise Error, "Need primary key to load records" if !primary_key
  275 +
  276 + relation.where(primary_key => ids)
  277 + elsif relation.respond_to?(:queryable)
  278 + relation.queryable.for_ids(ids)
231 279 end
232 280  
233   - raise Searchkick::Error, "Not sure how to load records" if !records
  281 + raise Error, "Not sure how to load records" if !relation
  282 +
  283 + relation
  284 + end
234 285  
235   - records
  286 + # private
  287 + def self.load_model(class_name, allow_child: false)
  288 + model = class_name.safe_constantize
  289 + raise Error, "Could not find class: #{class_name}" unless model
  290 + if allow_child
  291 + unless model.respond_to?(:searchkick_klass)
  292 + raise Error, "#{class_name} is not a searchkick model"
  293 + end
  294 + else
  295 + unless Searchkick.models.include?(model)
  296 + raise Error, "#{class_name} is not a searchkick model"
  297 + end
  298 + end
  299 + model
236 300 end
237 301  
238 302 # private
239 303 def self.indexer
240   - Thread.current[:searchkick_indexer] ||= Searchkick::Indexer.new
  304 + Thread.current[:searchkick_indexer] ||= Indexer.new
241 305 end
242 306  
243 307 # private
... ... @@ -251,21 +315,8 @@ module Searchkick
251 315 end
252 316  
253 317 # private
254   - def self.signer_middleware_key
255   - defined?(FaradayMiddleware::AwsSignersV4) ? :aws_signers_v4 : :aws_sigv4
256   - end
257   -
258   - # private
259 318 def self.signer_middleware_aws_params
260   - if signer_middleware_key == :aws_sigv4
261   - {service: "es", region: "us-east-1"}.merge(aws_credentials)
262   - else
263   - {
264   - credentials: aws_credentials[:credentials] || Aws::Credentials.new(aws_credentials[:access_key_id], aws_credentials[:secret_access_key]),
265   - service_name: "es",
266   - region: aws_credentials[:region] || "us-east-1"
267   - }
268   - end
  319 + {service: "es", region: "us-east-1"}.merge(aws_credentials)
269 320 end
270 321  
271 322 # private
... ... @@ -275,31 +326,55 @@ module Searchkick
275 326 def self.relation?(klass)
276 327 if klass.respond_to?(:current_scope)
277 328 !klass.current_scope.nil?
278   - elsif defined?(Mongoid::Threaded)
279   - !Mongoid::Threaded.current_scope(klass).nil?
  329 + else
  330 + klass.is_a?(Mongoid::Criteria) || !Mongoid::Threaded.current_scope(klass).nil?
  331 + end
  332 + end
  333 +
  334 + # private
  335 + def self.scope(model)
  336 + # safety check to make sure used properly in code
  337 + raise Error, "Cannot scope relation" if relation?(model)
  338 +
  339 + if model.searchkick_options[:unscope]
  340 + model.unscoped
  341 + else
  342 + model
280 343 end
281 344 end
282 345  
283 346 # private
284 347 def self.not_found_error?(e)
285   - (defined?(Elasticsearch) && e.is_a?(Elasticsearch::Transport::Transport::Errors::NotFound)) ||
  348 + (defined?(Elastic::Transport) && e.is_a?(Elastic::Transport::Transport::Errors::NotFound)) ||
  349 + (defined?(Elasticsearch::Transport) && e.is_a?(Elasticsearch::Transport::Transport::Errors::NotFound)) ||
286 350 (defined?(OpenSearch) && e.is_a?(OpenSearch::Transport::Transport::Errors::NotFound))
287 351 end
288 352  
289 353 # private
290 354 def self.transport_error?(e)
291   - (defined?(Elasticsearch) && e.is_a?(Elasticsearch::Transport::Transport::Error)) ||
  355 + (defined?(Elastic::Transport) && e.is_a?(Elastic::Transport::Transport::Error)) ||
  356 + (defined?(Elasticsearch::Transport) && e.is_a?(Elasticsearch::Transport::Transport::Error)) ||
292 357 (defined?(OpenSearch) && e.is_a?(OpenSearch::Transport::Transport::Error))
293 358 end
294   -end
295 359  
296   -require "active_model/callbacks"
297   -ActiveModel::Callbacks.include(Searchkick::Model)
298   -# TODO use
299   -# ActiveSupport.on_load(:mongoid) do
300   -# Mongoid::Document::ClassMethods.include Searchkick::Model
301   -# end
  360 + # private
  361 + def self.not_allowed_error?(e)
  362 + (defined?(Elastic::Transport) && e.is_a?(Elastic::Transport::Transport::Errors::MethodNotAllowed)) ||
  363 + (defined?(Elasticsearch::Transport) && e.is_a?(Elasticsearch::Transport::Transport::Errors::MethodNotAllowed)) ||
  364 + (defined?(OpenSearch) && e.is_a?(OpenSearch::Transport::Transport::Errors::MethodNotAllowed))
  365 + end
  366 +end
302 367  
303 368 ActiveSupport.on_load(:active_record) do
304 369 extend Searchkick::Model
305 370 end
  371 +
  372 +ActiveSupport.on_load(:mongoid) do
  373 + Mongoid::Document::ClassMethods.include Searchkick::Model
  374 +end
  375 +
  376 +ActiveSupport.on_load(:action_controller) do
  377 + include Searchkick::ControllerRuntime
  378 +end
  379 +
  380 +Searchkick::LogSubscriber.attach_to :searchkick
... ...
lib/searchkick/bulk_indexer.rb
View file @d6abba8
... ... @@ -1,173 +0,0 @@
1   -module Searchkick
2   - class BulkIndexer
3   - attr_reader :index
4   -
5   - def initialize(index)
6   - @index = index
7   - end
8   -
9   - def import_scope(relation, resume: false, method_name: nil, async: false, batch: false, batch_id: nil, full: false, scope: nil)
10   - if scope
11   - relation = relation.send(scope)
12   - elsif relation.respond_to?(:search_import)
13   - relation = relation.search_import
14   - end
15   -
16   - if batch
17   - import_or_update relation.to_a, method_name, async
18   - Searchkick.with_redis { |r| r.srem(batches_key, batch_id) } if batch_id
19   - elsif full && async
20   - full_reindex_async(relation)
21   - elsif relation.respond_to?(:find_in_batches)
22   - if resume
23   - # use total docs instead of max id since there's not a great way
24   - # to get the max _id without scripting since it's a string
25   -
26   - # TODO use primary key and prefix with table name
27   - relation = relation.where("id > ?", index.total_docs)
28   - end
29   -
30   - relation = relation.select("id").except(:includes, :preload) if async
31   -
32   - relation.find_in_batches batch_size: batch_size do |items|
33   - import_or_update items, method_name, async
34   - end
35   - else
36   - each_batch(relation) do |items|
37   - import_or_update items, method_name, async
38   - end
39   - end
40   - end
41   -
42   - def bulk_index(records)
43   - Searchkick.indexer.queue(records.map { |r| RecordData.new(index, r).index_data })
44   - end
45   -
46   - def bulk_delete(records)
47   - Searchkick.indexer.queue(records.reject { |r| r.id.blank? }.map { |r| RecordData.new(index, r).delete_data })
48   - end
49   -
50   - def bulk_update(records, method_name)
51   - Searchkick.indexer.queue(records.map { |r| RecordData.new(index, r).update_data(method_name) })
52   - end
53   -
54   - def batches_left
55   - Searchkick.with_redis { |r| r.scard(batches_key) }
56   - end
57   -
58   - private
59   -
60   - def import_or_update(records, method_name, async)
61   - if records.any?
62   - if async
63   - Searchkick::BulkReindexJob.perform_later(
64   - class_name: records.first.class.searchkick_options[:class_name],
65   - record_ids: records.map(&:id),
66   - index_name: index.name,
67   - method_name: method_name ? method_name.to_s : nil
68   - )
69   - else
70   - records = records.select(&:should_index?)
71   - if records.any?
72   - with_retries do
73   - # call out to index for ActiveSupport notifications
74   - if method_name
75   - index.bulk_update(records, method_name)
76   - else
77   - index.bulk_index(records)
78   - end
79   - end
80   - end
81   - end
82   - end
83   - end
84   -
85   - def full_reindex_async(scope)
86   - if scope.respond_to?(:primary_key)
87   - # TODO expire Redis key
88   - primary_key = scope.primary_key
89   -
90   - scope = scope.select(primary_key).except(:includes, :preload)
91   -
92   - starting_id =
93   - begin
94   - scope.minimum(primary_key)
95   - rescue ActiveRecord::StatementInvalid
96   - false
97   - end
98   -
99   - if starting_id.nil?
100   - # no records, do nothing
101   - elsif starting_id.is_a?(Numeric)
102   - max_id = scope.maximum(primary_key)
103   - batches_count = ((max_id - starting_id + 1) / batch_size.to_f).ceil
104   -
105   - batches_count.times do |i|
106   - batch_id = i + 1
107   - min_id = starting_id + (i * batch_size)
108   - bulk_reindex_job scope, batch_id, min_id: min_id, max_id: min_id + batch_size - 1
109   - end
110   - else
111   - scope.find_in_batches(batch_size: batch_size).each_with_index do |batch, i|
112   - batch_id = i + 1
113   -
114   - bulk_reindex_job scope, batch_id, record_ids: batch.map { |record| record.id.to_s }
115   - end
116   - end
117   - else
118   - batch_id = 1
119   - # TODO remove any eager loading
120   - scope = scope.only(:_id) if scope.respond_to?(:only)
121   - each_batch(scope) do |items|
122   - bulk_reindex_job scope, batch_id, record_ids: items.map { |i| i.id.to_s }
123   - batch_id += 1
124   - end
125   - end
126   - end
127   -
128   - def each_batch(scope)
129   - # https://github.com/karmi/tire/blob/master/lib/tire/model/import.rb
130   - # use cursor for Mongoid
131   - items = []
132   - scope.all.each do |item|
133   - items << item
134   - if items.length == batch_size
135   - yield items
136   - items = []
137   - end
138   - end
139   - yield items if items.any?
140   - end
141   -
142   - def bulk_reindex_job(scope, batch_id, options)
143   - Searchkick.with_redis { |r| r.sadd(batches_key, batch_id) }
144   - Searchkick::BulkReindexJob.perform_later(**{
145   - class_name: scope.searchkick_options[:class_name],
146   - index_name: index.name,
147   - batch_id: batch_id
148   - }.merge(options))
149   - end
150   -
151   - def with_retries
152   - retries = 0
153   -
154   - begin
155   - yield
156   - rescue Faraday::ClientError => e
157   - if retries < 1
158   - retries += 1
159   - retry
160   - end
161   - raise e
162   - end
163   - end
164   -
165   - def batches_key
166   - "searchkick:reindex:#{index.name}:batches"
167   - end
168   -
169   - def batch_size
170   - @batch_size ||= index.options[:batch_size] || 1000
171   - end
172   - end
173   -end
lib/searchkick/bulk_reindex_job.rb
  Diff comments   View file @684eecf
... ... @@ -2,16 +2,20 @@ module Searchkick
2 2 class BulkReindexJob < ActiveJob::Base
3 3 queue_as { Searchkick.queue_name }
4 4  
  5 + # TODO remove min_id and max_id in Searchkick 6
5 6 def perform(class_name:, record_ids: nil, index_name: nil, method_name: nil, batch_id: nil, min_id: nil, max_id: nil)
6   - klass = class_name.constantize
7   - index = index_name ? Searchkick::Index.new(index_name, **klass.searchkick_options) : klass.searchkick_index
  7 + model = Searchkick.load_model(class_name)
  8 + index = model.searchkick_index(name: index_name)
  9 +
  10 + # legacy
8 11 record_ids ||= min_id..max_id
9   - index.import_scope(
10   - Searchkick.load_records(klass, record_ids),
11   - method_name: method_name,
12   - batch: true,
13   - batch_id: batch_id
14   - )
  12 +
  13 + relation = Searchkick.scope(model)
  14 + relation = Searchkick.load_records(relation, record_ids)
  15 + relation = relation.search_import if relation.respond_to?(:search_import)
  16 +
  17 + RecordIndexer.new(index).reindex(relation, mode: :inline, method_name: method_name, full: false)
  18 + RelationIndexer.new(index).batch_completed(batch_id) if batch_id
15 19 end
16 20 end
17 21 end
... ...
lib/searchkick/controller_runtime.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,40 @@
  1 +# based on https://gist.github.com/mnutt/566725
  2 +module Searchkick
  3 + module ControllerRuntime
  4 + extend ActiveSupport::Concern
  5 +
  6 + protected
  7 +
  8 + attr_internal :searchkick_runtime
  9 +
  10 + def process_action(action, *args)
  11 + # We also need to reset the runtime before each action
  12 + # because of queries in middleware or in cases we are streaming
  13 + # and it won't be cleaned up by the method below.
  14 + Searchkick::LogSubscriber.reset_runtime
  15 + super
  16 + end
  17 +
  18 + def cleanup_view_runtime
  19 + searchkick_rt_before_render = Searchkick::LogSubscriber.reset_runtime
  20 + runtime = super
  21 + searchkick_rt_after_render = Searchkick::LogSubscriber.reset_runtime
  22 + self.searchkick_runtime = searchkick_rt_before_render + searchkick_rt_after_render
  23 + runtime - searchkick_rt_after_render
  24 + end
  25 +
  26 + def append_info_to_payload(payload)
  27 + super
  28 + payload[:searchkick_runtime] = (searchkick_runtime || 0) + Searchkick::LogSubscriber.reset_runtime
  29 + end
  30 +
  31 + module ClassMethods
  32 + def log_process_action(payload)
  33 + messages = super
  34 + runtime = payload[:searchkick_runtime]
  35 + messages << ("Searchkick: %.1fms" % runtime.to_f) if runtime.to_f > 0
  36 + messages
  37 + end
  38 + end
  39 + end
  40 +end
... ...
lib/searchkick/index.rb
  Diff comments   View file @684eecf
1   -require "searchkick/index_options"
2   -
3 1 module Searchkick
4 2 class Index
5 3 attr_reader :name, :options
... ... @@ -40,12 +38,15 @@ module Searchkick
40 38 client.indices.exists_alias name: name
41 39 end
42 40  
  41 + # call to_h for consistent results between elasticsearch gem 7 and 8
  42 + # could do for all API calls, but just do for ones where return value is focus for now
43 43 def mapping
44   - client.indices.get_mapping index: name
  44 + client.indices.get_mapping(index: name).to_h
45 45 end
46 46  
  47 + # call to_h for consistent results between elasticsearch gem 7 and 8
47 48 def settings
48   - client.indices.get_settings index: name
  49 + client.indices.get_settings(index: name).to_h
49 50 end
50 51  
51 52 def refresh_interval
... ... @@ -97,7 +98,7 @@ module Searchkick
97 98 record_data = RecordData.new(self, record).record_data
98 99  
99 100 # remove underscore
100   - get_options = Hash[record_data.map { |k, v| [k.to_s.sub(/\A_/, "").to_sym, v] }]
  101 + get_options = record_data.to_h { |k, v| [k.to_s.sub(/\A_/, "").to_sym, v] }
101 102  
102 103 client.get(get_options)["_source"]
103 104 end
... ... @@ -127,32 +128,47 @@ module Searchkick
127 128 indices
128 129 end
129 130  
130   - # record based
131   - # use helpers for notifications
132   -
133 131 def store(record)
134   - bulk_indexer.bulk_index([record])
  132 + notify(record, "Store") do
  133 + queue_index([record])
  134 + end
135 135 end
136 136  
137 137 def remove(record)
138   - bulk_indexer.bulk_delete([record])
  138 + notify(record, "Remove") do
  139 + queue_delete([record])
  140 + end
139 141 end
140 142  
141 143 def update_record(record, method_name)
142   - bulk_indexer.bulk_update([record], method_name)
  144 + notify(record, "Update") do
  145 + queue_update([record], method_name)
  146 + end
143 147 end
144 148  
145 149 def bulk_delete(records)
146   - bulk_indexer.bulk_delete(records)
  150 + return if records.empty?
  151 +
  152 + notify_bulk(records, "Delete") do
  153 + queue_delete(records)
  154 + end
147 155 end
148 156  
149 157 def bulk_index(records)
150   - bulk_indexer.bulk_index(records)
  158 + return if records.empty?
  159 +
  160 + notify_bulk(records, "Import") do
  161 + queue_index(records)
  162 + end
151 163 end
152 164 alias_method :import, :bulk_index
153 165  
154 166 def bulk_update(records, method_name)
155   - bulk_indexer.bulk_update(records, method_name)
  167 + return if records.empty?
  168 +
  169 + notify_bulk(records, "Update") do
  170 + queue_update(records, method_name)
  171 + end
156 172 end
157 173  
158 174 def search_id(record)
... ... @@ -163,20 +179,12 @@ module Searchkick
163 179 RecordData.new(self, record).document_type
164 180 end
165 181  
166   - # TODO use like: [{_index: ..., _id: ...}] in Searchkick 5
167 182 def similar_record(record, **options)
168   - like_text = retrieve(record).to_hash
169   - .keep_if { |k, _| !options[:fields] || options[:fields].map(&:to_s).include?(k) }
170   - .values.compact.join(" ")
171   -
172   - options[:where] ||= {}
173   - options[:where][:_id] ||= {}
174   - options[:where][:_id][:not] = Array(options[:where][:_id][:not]) + [record.id.to_s]
175 183 options[:per_page] ||= 10
176   - options[:similar] = true
  184 + options[:similar] = [RecordData.new(self, record).record_data]
  185 + options[:models] ||= [record.class] unless options.key?(:model)
177 186  
178   - # TODO use index class instead of record class
179   - Searchkick.search(like_text, model: record.class, **options)
  187 + Searchkick.search("*", **options)
180 188 end
181 189  
182 190 def reload_synonyms
... ... @@ -186,8 +194,9 @@ module Searchkick
186 194 raise Error, "Requires Elasticsearch 7.3+" if Searchkick.server_below?("7.3.0")
187 195 begin
188 196 client.transport.perform_request("GET", "#{CGI.escape(name)}/_reload_search_analyzers")
189   - rescue Elasticsearch::Transport::Transport::Errors::MethodNotAllowed
190   - raise Error, "Requires non-OSS version of Elasticsearch"
  197 + rescue => e
  198 + raise Error, "Requires non-OSS version of Elasticsearch" if Searchkick.not_allowed_error?(e)
  199 + raise e
191 200 end
192 201 end
193 202 end
... ... @@ -200,29 +209,35 @@ module Searchkick
200 209  
201 210 # reindex
202 211  
203   - def reindex(relation, method_name, scoped:, full: false, scope: nil, **options)
  212 + # note: this is designed to be used internally
  213 + # so it does not check object matches index class
  214 + def reindex(object, method_name: nil, full: false, **options)
  215 + if object.is_a?(Array)
  216 + # note: purposefully skip full
  217 + return reindex_records(object, method_name: method_name, **options)
  218 + end
  219 +
  220 + if !object.respond_to?(:searchkick_klass)
  221 + raise Error, "Cannot reindex object"
  222 + end
  223 +
  224 + scoped = Searchkick.relation?(object)
  225 + # call searchkick_klass for inheritance
  226 + relation = scoped ? object.all : Searchkick.scope(object.searchkick_klass).all
  227 +
204 228 refresh = options.fetch(:refresh, !scoped)
205 229 options.delete(:refresh)
206 230  
207   - if method_name
208   - # TODO throw ArgumentError
209   - Searchkick.warn("unsupported keywords: #{options.keys.map(&:inspect).join(", ")}") if options.any?
  231 + if method_name || (scoped && !full)
  232 + mode = options.delete(:mode) || :inline
  233 + raise ArgumentError, "unsupported keywords: #{options.keys.map(&:inspect).join(", ")}" if options.any?
210 234  
211   - # update
212   - import_scope(relation, method_name: method_name, scope: scope)
213   - self.refresh if refresh
214   - true
215   - elsif scoped && !full
216   - # TODO throw ArgumentError
217   - Searchkick.warn("unsupported keywords: #{options.keys.map(&:inspect).join(", ")}") if options.any?
218   -
219   - # reindex association
220   - import_scope(relation, scope: scope)
  235 + # import only
  236 + import_scope(relation, method_name: method_name, mode: mode)
221 237 self.refresh if refresh
222 238 true
223 239 else
224   - # full reindex
225   - reindex_scope(relation, scope: scope, **options)
  240 + full_reindex(relation, **options)
226 241 end
227 242 end
228 243  
... ... @@ -234,15 +249,14 @@ module Searchkick
234 249 end
235 250  
236 251 def import_scope(relation, **options)
237   - bulk_indexer.import_scope(relation, **options)
  252 + relation_indexer.reindex(relation, **options)
238 253 end
239 254  
240 255 def batches_left
241   - bulk_indexer.batches_left
  256 + relation_indexer.batches_left
242 257 end
243 258  
244   - # other
245   -
  259 + # private
246 260 def klass_document_type(klass, ignore_type = false)
247 261 @klass_document_type[[klass, ignore_type]] ||= begin
248 262 if !ignore_type && klass.searchkick_klass.searchkick_options[:_type]
... ... @@ -255,7 +269,7 @@ module Searchkick
255 269 end
256 270 end
257 271  
258   - # should not be public
  272 + # private
259 273 def conversions_fields
260 274 @conversions_fields ||= begin
261 275 conversions = Array(options[:conversions])
... ... @@ -263,10 +277,12 @@ module Searchkick
263 277 end
264 278 end
265 279  
  280 + # private
266 281 def suggest_fields
267 282 @suggest_fields ||= Array(options[:suggest]).map(&:to_s)
268 283 end
269 284  
  285 + # private
270 286 def locations_fields
271 287 @locations_fields ||= begin
272 288 locations = Array(options[:locations])
... ... @@ -285,8 +301,20 @@ module Searchkick
285 301 Searchkick.client
286 302 end
287 303  
288   - def bulk_indexer
289   - @bulk_indexer ||= BulkIndexer.new(self)
  304 + def queue_index(records)
  305 + Searchkick.indexer.queue(records.map { |r| RecordData.new(self, r).index_data })
  306 + end
  307 +
  308 + def queue_delete(records)
  309 + Searchkick.indexer.queue(records.reject { |r| r.id.blank? }.map { |r| RecordData.new(self, r).delete_data })
  310 + end
  311 +
  312 + def queue_update(records, method_name)
  313 + Searchkick.indexer.queue(records.map { |r| RecordData.new(self, r).update_data(method_name) })
  314 + end
  315 +
  316 + def relation_indexer
  317 + @relation_indexer ||= RelationIndexer.new(self)
290 318 end
291 319  
292 320 def index_settings
... ... @@ -297,9 +325,19 @@ module Searchkick
297 325 index.import_scope(relation, **import_options)
298 326 end
299 327  
  328 + def reindex_records(object, mode: nil, refresh: false, **options)
  329 + mode ||= Searchkick.callbacks_value || @options[:callbacks] || true
  330 + mode = :inline if mode == :bulk
  331 +
  332 + result = RecordIndexer.new(self).reindex(object, mode: mode, full: false, **options)
  333 + self.refresh if refresh
  334 + result
  335 + end
  336 +
300 337 # https://gist.github.com/jarosan/3124884
301 338 # http://www.elasticsearch.org/blog/changing-mapping-with-zero-downtime/
302   - def reindex_scope(relation, import: true, resume: false, retain: false, async: false, refresh_interval: nil, scope: nil)
  339 + # TODO deprecate async in favor of mode: :async, wait: true/false
  340 + def full_reindex(relation, import: true, resume: false, retain: false, async: false, refresh_interval: nil, scope: nil)
303 341 if resume
304 342 index_name = all_indices.sort.last
305 343 raise Searchkick::Error, "No index to resume" unless index_name
... ... @@ -313,9 +351,9 @@ module Searchkick
313 351 end
314 352  
315 353 import_options = {
316   - resume: resume,
317   - async: async,
  354 + mode: (async ? :async : :inline),
318 355 full: true,
  356 + resume: resume,
319 357 scope: scope
320 358 }
321 359  
... ... @@ -367,7 +405,7 @@ module Searchkick
367 405 end
368 406 rescue => e
369 407 if Searchkick.transport_error?(e) && e.message.include?("No handler for type [text]")
370   - raise UnsupportedVersionError, "This version of Searchkick requires Elasticsearch 6 or greater"
  408 + raise UnsupportedVersionError
371 409 end
372 410  
373 411 raise e
... ... @@ -382,5 +420,34 @@ module Searchkick
382 420 raise Searchkick::Error, "Safety check failed - only run one Model.reindex per model at a time"
383 421 end
384 422 end
  423 +
  424 + def notify(record, name)
  425 + if Searchkick.callbacks_value == :bulk
  426 + yield
  427 + else
  428 + name = "#{record.class.searchkick_klass.name} #{name}" if record && record.class.searchkick_klass
  429 + event = {
  430 + name: name,
  431 + id: search_id(record)
  432 + }
  433 + ActiveSupport::Notifications.instrument("request.searchkick", event) do
  434 + yield
  435 + end
  436 + end
  437 + end
  438 +
  439 + def notify_bulk(records, name)
  440 + if Searchkick.callbacks_value == :bulk
  441 + yield
  442 + else
  443 + event = {
  444 + name: "#{records.first.class.searchkick_klass.name} #{name}",
  445 + count: records.size
  446 + }
  447 + ActiveSupport::Notifications.instrument("request.searchkick", event) do
  448 + yield
  449 + end
  450 + end
  451 + end
385 452 end
386 453 end
... ...
lib/searchkick/index_cache.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,30 @@
  1 +module Searchkick
  2 + class IndexCache
  3 + def initialize(max_size: 20)
  4 + @data = {}
  5 + @mutex = Mutex.new
  6 + @max_size = max_size
  7 + end
  8 +
  9 + # probably a better pattern for this
  10 + # but keep it simple
  11 + def fetch(name)
  12 + # thread-safe in MRI without mutex
  13 + # due to how context switching works
  14 + @mutex.synchronize do
  15 + if @data.key?(name)
  16 + @data[name]
  17 + else
  18 + @data.clear if @data.size >= @max_size
  19 + @data[name] = yield
  20 + end
  21 + end
  22 + end
  23 +
  24 + def clear
  25 + @mutex.synchronize do
  26 + @data.clear
  27 + end
  28 + end
  29 + end
  30 +end
... ...
lib/searchkick/index_options.rb
  Diff comments   View file @684eecf
... ... @@ -7,18 +7,16 @@ module Searchkick
7 7 end
8 8  
9 9 def index_options
10   - custom_mapping = options[:mappings] || {}
11   - if below70? && custom_mapping.keys.map(&:to_sym).include?(:properties)
12   - # add type
13   - custom_mapping = {index_type => custom_mapping}
14   - end
  10 + # mortal symbols are garbage collected in Ruby 2.2+
  11 + custom_settings = (options[:settings] || {}).deep_symbolize_keys
  12 + custom_mappings = (options[:mappings] || {}).deep_symbolize_keys
15 13  
16 14 if options[:mappings] && !options[:merge_mappings]
17   - settings = options[:settings] || {}
18   - mappings = custom_mapping
  15 + settings = custom_settings
  16 + mappings = custom_mappings
19 17 else
20   - settings = generate_settings
21   - mappings = generate_mappings.symbolize_keys.deep_merge(custom_mapping.symbolize_keys)
  18 + settings = generate_settings.deep_symbolize_keys.deep_merge(custom_settings)
  19 + mappings = generate_mappings.deep_symbolize_keys.deep_merge(custom_mappings)
22 20 end
23 21  
24 22 set_deep_paging(settings) if options[:deep_paging]
... ... @@ -162,17 +160,14 @@ module Searchkick
162 160 settings[:number_of_replicas] = 0
163 161 end
164 162  
165   - # TODO remove in Searchkick 5 (classic no longer supported)
166 163 if options[:similarity]
167 164 settings[:similarity] = {default: {type: options[:similarity]}}
168 165 end
169 166  
170   - unless below62?
171   - settings[:index] = {
172   - max_ngram_diff: 49,
173   - max_shingle_diff: 4
174   - }
175   - end
  167 + settings[:index] = {
  168 + max_ngram_diff: 49,
  169 + max_shingle_diff: 4
  170 + }
176 171  
177 172 if options[:case_sensitive]
178 173 settings[:analysis][:analyzer].each do |_, analyzer|
... ... @@ -180,13 +175,8 @@ module Searchkick
180 175 end
181 176 end
182 177  
183   - # TODO do this last in Searchkick 5
184   - settings = settings.symbolize_keys.deep_merge((options[:settings] || {}).symbolize_keys)
185   -
186 178 add_synonyms(settings)
187 179 add_search_synonyms(settings)
188   - # TODO remove in Searchkick 5
189   - add_wordnet(settings) if options[:wordnet]
190 180  
191 181 if options[:special_characters] == false
192 182 settings[:analysis][:analyzer].each_value do |analyzer_settings|
... ... @@ -223,19 +213,7 @@ module Searchkick
223 213 type: "smartcn"
224 214 }
225 215 )
226   - when "japanese"
227   - settings[:analysis][:analyzer].merge!(
228   - default_analyzer => {
229   - type: "kuromoji"
230   - },
231   - searchkick_search: {
232   - type: "kuromoji"
233   - },
234   - searchkick_search2: {
235   - type: "kuromoji"
236   - }
237   - )
238   - when "japanese2"
  216 + when "japanese", "japanese2"
239 217 analyzer = {
240 218 type: "custom",
241 219 tokenizer: "kuromoji_tokenizer",
... ... @@ -379,16 +357,15 @@ module Searchkick
379 357 }
380 358 end
381 359  
382   - mapping_options = Hash[
  360 + mapping_options =
383 361 [:suggest, :word, :text_start, :text_middle, :text_end, :word_start, :word_middle, :word_end, :highlight, :searchable, :filterable]
384   - .map { |type| [type, (options[type] || []).map(&:to_s)] }
385   - ]
  362 + .to_h { |type| [type, (options[type] || []).map(&:to_s)] }
386 363  
387 364 word = options[:word] != false && (!options[:match] || options[:match] == :word)
388 365  
389 366 mapping_options[:searchable].delete("_all")
390 367  
391   - analyzed_field_options = {type: default_type, index: true, analyzer: default_analyzer}
  368 + analyzed_field_options = {type: default_type, index: true, analyzer: default_analyzer.to_s}
392 369  
393 370 mapping_options.values.flatten.uniq.each do |field|
394 371 fields = {}
... ... @@ -481,10 +458,6 @@ module Searchkick
481 458 ]
482 459 }
483 460  
484   - if below70?
485   - mappings = {index_type => mappings}
486   - end
487   -
488 461 mappings
489 462 end
490 463  
... ... @@ -533,7 +506,7 @@ module Searchkick
533 506 end
534 507 settings[:analysis][:filter][:searchkick_synonym_graph] = synonym_graph
535 508  
536   - if options[:language] == "japanese2"
  509 + if ["japanese", "japanese2"].include?(options[:language])
537 510 [:searchkick_search, :searchkick_search2].each do |analyzer|
538 511 settings[:analysis][:analyzer][analyzer][:filter].insert(4, "searchkick_synonym_graph")
539 512 end
... ... @@ -549,21 +522,6 @@ module Searchkick
549 522 end
550 523 end
551 524  
552   - def add_wordnet(settings)
553   - settings[:analysis][:filter][:searchkick_wordnet] = {
554   - type: "synonym",
555   - format: "wordnet",
556   - synonyms_path: Searchkick.wordnet_path
557   - }
558   -
559   - settings[:analysis][:analyzer][default_analyzer][:filter].insert(4, "searchkick_wordnet")
560   - settings[:analysis][:analyzer][default_analyzer][:filter] << "searchkick_wordnet"
561   -
562   - %w(word_start word_middle word_end).each do |type|
563   - settings[:analysis][:analyzer]["searchkick_#{type}_index".to_sym][:filter].insert(2, "searchkick_wordnet")
564   - end
565   - end
566   -
567 525 def set_deep_paging(settings)
568 526 if !settings.dig(:index, :max_result_window) && !settings[:"index.max_result_window"]
569 527 settings[:index] ||= {}
... ... @@ -587,14 +545,6 @@ module Searchkick
587 545 :searchkick_index
588 546 end
589 547  
590   - def below62?
591   - Searchkick.server_below?("6.2.0")
592   - end
593   -
594   - def below70?
595   - Searchkick.server_below?("7.0.0")
596   - end
597   -
598 548 def below73?
599 549 Searchkick.server_below?("7.3.0")
600 550 end
... ...
lib/searchkick/indexer.rb
  Diff comments   View file @684eecf
  1 +# thread-local (technically fiber-local) indexer
  2 +# used to aggregate bulk callbacks across models
1 3 module Searchkick
2 4 class Indexer
3 5 attr_reader :queued_items
... ... @@ -14,15 +16,20 @@ module Searchkick
14 16 def perform
15 17 items = @queued_items
16 18 @queued_items = []
17   - if items.any?
18   - response = Searchkick.client.bulk(body: items)
19   - if response["errors"]
20   - first_with_error = response["items"].map do |item|
21   - (item["index"] || item["delete"] || item["update"])
22   - end.find { |item| item["error"] }
23   - raise Searchkick::ImportError, "#{first_with_error["error"]} on item with id '#{first_with_error["_id"]}'"
24   - end
  19 +
  20 + return if items.empty?
  21 +
  22 + response = Searchkick.client.bulk(body: items)
  23 + if response["errors"]
  24 + # note: delete does not set error when item not found
  25 + first_with_error = response["items"].map do |item|
  26 + (item["index"] || item["delete"] || item["update"])
  27 + end.find { |item| item["error"] }
  28 + raise ImportError, "#{first_with_error["error"]} on item with id '#{first_with_error["_id"]}'"
25 29 end
  30 +
  31 + # maybe return response in future
  32 + nil
26 33 end
27 34 end
28 35 end
... ...
lib/searchkick/log_subscriber.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,57 @@
  1 +# based on https://gist.github.com/mnutt/566725
  2 +module Searchkick
  3 + class LogSubscriber < ActiveSupport::LogSubscriber
  4 + def self.runtime=(value)
  5 + Thread.current[:searchkick_runtime] = value
  6 + end
  7 +
  8 + def self.runtime
  9 + Thread.current[:searchkick_runtime] ||= 0
  10 + end
  11 +
  12 + def self.reset_runtime
  13 + rt = runtime
  14 + self.runtime = 0
  15 + rt
  16 + end
  17 +
  18 + def search(event)
  19 + self.class.runtime += event.duration
  20 + return unless logger.debug?
  21 +
  22 + payload = event.payload
  23 + name = "#{payload[:name]} (#{event.duration.round(1)}ms)"
  24 +
  25 + index = payload[:query][:index].is_a?(Array) ? payload[:query][:index].join(",") : payload[:query][:index]
  26 + type = payload[:query][:type]
  27 + request_params = payload[:query].except(:index, :type, :body)
  28 +
  29 + params = []
  30 + request_params.each do |k, v|
  31 + params << "#{CGI.escape(k.to_s)}=#{CGI.escape(v.to_s)}"
  32 + end
  33 +
  34 + debug " #{color(name, YELLOW, true)} #{index}#{type ? "/#{type.join(',')}" : ''}/_search#{params.any? ? '?' + params.join('&') : nil} #{payload[:query][:body].to_json}"
  35 + end
  36 +
  37 + def request(event)
  38 + self.class.runtime += event.duration
  39 + return unless logger.debug?
  40 +
  41 + payload = event.payload
  42 + name = "#{payload[:name]} (#{event.duration.round(1)}ms)"
  43 +
  44 + debug " #{color(name, YELLOW, true)} #{payload.except(:name).to_json}"
  45 + end
  46 +
  47 + def multi_search(event)
  48 + self.class.runtime += event.duration
  49 + return unless logger.debug?
  50 +
  51 + payload = event.payload
  52 + name = "#{payload[:name]} (#{event.duration.round(1)}ms)"
  53 +
  54 + debug " #{color(name, YELLOW, true)} _msearch #{payload[:body]}"
  55 + end
  56 + end
  57 +end
... ...
lib/searchkick/logging.rb
View file @d6abba8
... ... @@ -1,246 +0,0 @@
1   -# based on https://gist.github.com/mnutt/566725
2   -require "active_support/core_ext/module/attr_internal"
3   -
4   -module Searchkick
5   - module QueryWithInstrumentation
6   - def execute_search
7   - name = searchkick_klass ? "#{searchkick_klass.name} Search" : "Search"
8   - event = {
9   - name: name,
10   - query: params
11   - }
12   - ActiveSupport::Notifications.instrument("search.searchkick", event) do
13   - super
14   - end
15   - end
16   - end
17   -
18   - module IndexWithInstrumentation
19   - def store(record)
20   - event = {
21   - name: "#{record.searchkick_klass.name} Store",
22   - id: search_id(record)
23   - }
24   - if Searchkick.callbacks_value == :bulk
25   - super
26   - else
27   - ActiveSupport::Notifications.instrument("request.searchkick", event) do
28   - super
29   - end
30   - end
31   - end
32   -
33   - def remove(record)
34   - name = record && record.searchkick_klass ? "#{record.searchkick_klass.name} Remove" : "Remove"
35   - event = {
36   - name: name,
37   - id: search_id(record)
38   - }
39   - if Searchkick.callbacks_value == :bulk
40   - super
41   - else
42   - ActiveSupport::Notifications.instrument("request.searchkick", event) do
43   - super
44   - end
45   - end
46   - end
47   -
48   - def update_record(record, method_name)
49   - event = {
50   - name: "#{record.searchkick_klass.name} Update",
51   - id: search_id(record)
52   - }
53   - if Searchkick.callbacks_value == :bulk
54   - super
55   - else
56   - ActiveSupport::Notifications.instrument("request.searchkick", event) do
57   - super
58   - end
59   - end
60   - end
61   -
62   - def bulk_index(records)
63   - if records.any?
64   - event = {
65   - name: "#{records.first.searchkick_klass.name} Import",
66   - count: records.size
67   - }
68   - event[:id] = search_id(records.first) if records.size == 1
69   - if Searchkick.callbacks_value == :bulk
70   - super
71   - else
72   - ActiveSupport::Notifications.instrument("request.searchkick", event) do
73   - super
74   - end
75   - end
76   - end
77   - end
78   - alias_method :import, :bulk_index
79   -
80   - def bulk_update(records, *args)
81   - if records.any?
82   - event = {
83   - name: "#{records.first.searchkick_klass.name} Update",
84   - count: records.size
85   - }
86   - event[:id] = search_id(records.first) if records.size == 1
87   - if Searchkick.callbacks_value == :bulk
88   - super
89   - else
90   - ActiveSupport::Notifications.instrument("request.searchkick", event) do
91   - super
92   - end
93   - end
94   - end
95   - end
96   -
97   - def bulk_delete(records)
98   - if records.any?
99   - event = {
100   - name: "#{records.first.searchkick_klass.name} Delete",
101   - count: records.size
102   - }
103   - event[:id] = search_id(records.first) if records.size == 1
104   - if Searchkick.callbacks_value == :bulk
105   - super
106   - else
107   - ActiveSupport::Notifications.instrument("request.searchkick", event) do
108   - super
109   - end
110   - end
111   - end
112   - end
113   - end
114   -
115   - module IndexerWithInstrumentation
116   - def perform
117   - if Searchkick.callbacks_value == :bulk
118   - event = {
119   - name: "Bulk",
120   - count: queued_items.size
121   - }
122   - ActiveSupport::Notifications.instrument("request.searchkick", event) do
123   - super
124   - end
125   - else
126   - super
127   - end
128   - end
129   - end
130   -
131   - module SearchkickWithInstrumentation
132   - def multi_search(searches)
133   - event = {
134   - name: "Multi Search",
135   - body: searches.flat_map { |q| [q.params.except(:body).to_json, q.body.to_json] }.map { |v| "#{v}\n" }.join,
136   - }
137   - ActiveSupport::Notifications.instrument("multi_search.searchkick", event) do
138   - super
139   - end
140   - end
141   - end
142   -
143   - # https://github.com/rails/rails/blob/master/activerecord/lib/active_record/log_subscriber.rb
144   - class LogSubscriber < ActiveSupport::LogSubscriber
145   - def self.runtime=(value)
146   - Thread.current[:searchkick_runtime] = value
147   - end
148   -
149   - def self.runtime
150   - Thread.current[:searchkick_runtime] ||= 0
151   - end
152   -
153   - def self.reset_runtime
154   - rt = runtime
155   - self.runtime = 0
156   - rt
157   - end
158   -
159   - def search(event)
160   - self.class.runtime += event.duration
161   - return unless logger.debug?
162   -
163   - payload = event.payload
164   - name = "#{payload[:name]} (#{event.duration.round(1)}ms)"
165   -
166   - index = payload[:query][:index].is_a?(Array) ? payload[:query][:index].join(",") : payload[:query][:index]
167   - type = payload[:query][:type]
168   - request_params = payload[:query].except(:index, :type, :body)
169   -
170   - params = []
171   - request_params.each do |k, v|
172   - params << "#{CGI.escape(k.to_s)}=#{CGI.escape(v.to_s)}"
173   - end
174   -
175   - debug " #{color(name, YELLOW, true)} #{index}#{type ? "/#{type.join(',')}" : ''}/_search#{params.any? ? '?' + params.join('&') : nil} #{payload[:query][:body].to_json}"
176   - end
177   -
178   - def request(event)
179   - self.class.runtime += event.duration
180   - return unless logger.debug?
181   -
182   - payload = event.payload
183   - name = "#{payload[:name]} (#{event.duration.round(1)}ms)"
184   -
185   - debug " #{color(name, YELLOW, true)} #{payload.except(:name).to_json}"
186   - end
187   -
188   - def multi_search(event)
189   - self.class.runtime += event.duration
190   - return unless logger.debug?
191   -
192   - payload = event.payload
193   - name = "#{payload[:name]} (#{event.duration.round(1)}ms)"
194   -
195   - debug " #{color(name, YELLOW, true)} _msearch #{payload[:body]}"
196   - end
197   - end
198   -
199   - # https://github.com/rails/rails/blob/master/activerecord/lib/active_record/railties/controller_runtime.rb
200   - module ControllerRuntime
201   - extend ActiveSupport::Concern
202   -
203   - protected
204   -
205   - attr_internal :searchkick_runtime
206   -
207   - def process_action(action, *args)
208   - # We also need to reset the runtime before each action
209   - # because of queries in middleware or in cases we are streaming
210   - # and it won't be cleaned up by the method below.
211   - Searchkick::LogSubscriber.reset_runtime
212   - super
213   - end
214   -
215   - def cleanup_view_runtime
216   - searchkick_rt_before_render = Searchkick::LogSubscriber.reset_runtime
217   - runtime = super
218   - searchkick_rt_after_render = Searchkick::LogSubscriber.reset_runtime
219   - self.searchkick_runtime = searchkick_rt_before_render + searchkick_rt_after_render
220   - runtime - searchkick_rt_after_render
221   - end
222   -
223   - def append_info_to_payload(payload)
224   - super
225   - payload[:searchkick_runtime] = (searchkick_runtime || 0) + Searchkick::LogSubscriber.reset_runtime
226   - end
227   -
228   - module ClassMethods
229   - def log_process_action(payload)
230   - messages = super
231   - runtime = payload[:searchkick_runtime]
232   - messages << ("Searchkick: %.1fms" % runtime.to_f) if runtime.to_f > 0
233   - messages
234   - end
235   - end
236   - end
237   -end
238   -
239   -Searchkick::Query.prepend(Searchkick::QueryWithInstrumentation)
240   -Searchkick::Index.prepend(Searchkick::IndexWithInstrumentation)
241   -Searchkick::Indexer.prepend(Searchkick::IndexerWithInstrumentation)
242   -Searchkick.singleton_class.prepend(Searchkick::SearchkickWithInstrumentation)
243   -Searchkick::LogSubscriber.attach_to :searchkick
244   -ActiveSupport.on_load(:action_controller) do
245   - include Searchkick::ControllerRuntime
246   -end
lib/searchkick/middleware.rb
  Diff comments   View file @684eecf
1   -require "faraday/middleware"
  1 +require "faraday"
2 2  
3 3 module Searchkick
4 4 class Middleware < Faraday::Middleware
... ...
lib/searchkick/model.rb
  Diff comments   View file @684eecf
... ... @@ -7,7 +7,7 @@ module Searchkick
7 7 :filterable, :geo_shape, :highlight, :ignore_above, :index_name, :index_prefix, :inheritance, :language,
8 8 :locations, :mappings, :match, :merge_mappings, :routing, :searchable, :search_synonyms, :settings, :similarity,
9 9 :special_characters, :stem, :stemmer, :stem_conversions, :stem_exclusion, :stemmer_override, :suggest, :synonyms, :text_end,
10   - :text_middle, :text_start, :word, :wordnet, :word_end, :word_middle, :word_start]
  10 + :text_middle, :text_start, :unscope, :word, :word_end, :word_middle, :word_start]
11 11 raise ArgumentError, "unknown keywords: #{unknown_keywords.join(", ")}" if unknown_keywords.any?
12 12  
13 13 raise "Only call searchkick once per model" if respond_to?(:searchkick_index)
... ... @@ -22,52 +22,76 @@ module Searchkick
22 22 raise ArgumentError, "Invalid value for callbacks"
23 23 end
24 24  
25   - index_name =
26   - if options[:index_name]
27   - options[:index_name]
28   - elsif options[:index_prefix].respond_to?(:call)
29   - -> { [options[:index_prefix].call, model_name.plural, Searchkick.env, Searchkick.index_suffix].compact.join("_") }
30   - else
31   - [options.key?(:index_prefix) ? options[:index_prefix] : Searchkick.index_prefix, model_name.plural, Searchkick.env, Searchkick.index_suffix].compact.join("_")
  25 + mod = Module.new
  26 + include(mod)
  27 + mod.module_eval do
  28 + def reindex(method_name = nil, mode: nil, refresh: false)
  29 + self.class.searchkick_index.reindex([self], method_name: method_name, mode: mode, refresh: refresh, single: true)
32 30 end
33 31  
  32 + def similar(**options)
  33 + self.class.searchkick_index.similar_record(self, **options)
  34 + end
  35 +
  36 + def search_data
  37 + data = respond_to?(:to_hash) ? to_hash : serializable_hash
  38 + data.delete("id")
  39 + data.delete("_id")
  40 + data.delete("_type")
  41 + data
  42 + end
  43 +
  44 + def should_index?
  45 + true
  46 + end
  47 + end
  48 +
34 49 class_eval do
35   - cattr_reader :searchkick_options, :searchkick_klass
  50 + cattr_reader :searchkick_options, :searchkick_klass, instance_reader: false
36 51  
37 52 class_variable_set :@@searchkick_options, options.dup
38 53 class_variable_set :@@searchkick_klass, self
39   - class_variable_set :@@searchkick_index, index_name
40   - class_variable_set :@@searchkick_index_cache, {}
  54 + class_variable_set :@@searchkick_index_cache, Searchkick::IndexCache.new
41 55  
42 56 class << self
43 57 def searchkick_search(term = "*", **options, &block)
44   - # TODO throw error in next major version
45   - Searchkick.warn("calling search on a relation is deprecated") if Searchkick.relation?(self)
  58 + if Searchkick.relation?(self)
  59 + raise Searchkick::Error, "search must be called on model, not relation"
  60 + end
46 61  
47 62 Searchkick.search(term, model: self, **options, &block)
48 63 end
49 64 alias_method Searchkick.search_method_name, :searchkick_search if Searchkick.search_method_name
50 65  
51 66 def searchkick_index(name: nil)
52   - index = name || class_variable_get(:@@searchkick_index)
  67 + index = name || searchkick_index_name
53 68 index = index.call if index.respond_to?(:call)
54 69 index_cache = class_variable_get(:@@searchkick_index_cache)
55   - index_cache[index] ||= Searchkick::Index.new(index, searchkick_options)
  70 + index_cache.fetch(index) { Searchkick::Index.new(index, searchkick_options) }
56 71 end
57 72 alias_method :search_index, :searchkick_index unless method_defined?(:search_index)
58 73  
59 74 def searchkick_reindex(method_name = nil, **options)
60   - # TODO relation = Searchkick.relation?(self)
61   - relation = (respond_to?(:current_scope) && respond_to?(:default_scoped) && current_scope && current_scope.to_sql != default_scoped.to_sql) ||
62   - (respond_to?(:queryable) && queryable != unscoped.with_default_scope)
63   -
64   - searchkick_index.reindex(searchkick_klass, method_name, scoped: relation, **options)
  75 + searchkick_index.reindex(self, method_name: method_name, **options)
65 76 end
66 77 alias_method :reindex, :searchkick_reindex unless method_defined?(:reindex)
67 78  
68 79 def searchkick_index_options
69 80 searchkick_index.index_options
70 81 end
  82 +
  83 + def searchkick_index_name
  84 + @searchkick_index_name ||= begin
  85 + options = class_variable_get(:@@searchkick_options)
  86 + if options[:index_name]
  87 + options[:index_name]
  88 + elsif options[:index_prefix].respond_to?(:call)
  89 + -> { [options[:index_prefix].call, model_name.plural, Searchkick.env, Searchkick.index_suffix].compact.join("_") }
  90 + else
  91 + [options.key?(:index_prefix) ? options[:index_prefix] : Searchkick.index_prefix, model_name.plural, Searchkick.env, Searchkick.index_suffix].compact.join("_")
  92 + end
  93 + end
  94 + end
71 95 end
72 96  
73 97 # always add callbacks, even when callbacks is false
... ... @@ -78,33 +102,6 @@ module Searchkick
78 102 after_save :reindex, if: -> { Searchkick.callbacks?(default: callbacks) }
79 103 after_destroy :reindex, if: -> { Searchkick.callbacks?(default: callbacks) }
80 104 end
81   -
82   - def reindex(method_name = nil, **options)
83   - RecordIndexer.new(self).reindex(method_name, **options)
84   - end unless method_defined?(:reindex)
85   -
86   - # TODO switch to keyword arguments
87   - def similar(options = {})
88   - self.class.searchkick_index.similar_record(self, **options)
89   - end unless method_defined?(:similar)
90   -
91   - def search_data
92   - data = respond_to?(:to_hash) ? to_hash : serializable_hash
93   - data.delete("id")
94   - data.delete("_id")
95   - data.delete("_type")
96   - data
97   - end unless method_defined?(:search_data)
98   -
99   - def should_index?
100   - true
101   - end unless method_defined?(:should_index?)
102   -
103   - if defined?(Cequel) && self < Cequel::Record && !method_defined?(:destroyed?)
104   - def destroyed?
105   - transient?
106   - end
107   - end
108 105 end
109 106 end
110 107 end
... ...
lib/searchkick/process_batch_job.rb
  Diff comments   View file @684eecf
... ... @@ -3,34 +3,18 @@ module Searchkick
3 3 queue_as { Searchkick.queue_name }
4 4  
5 5 def perform(class_name:, record_ids:, index_name: nil)
6   - # separate routing from id
7   - routing = Hash[record_ids.map { |r| r.split(/(?<!\|)\|(?!\|)/, 2).map { |v| v.gsub("||", "|") } }]
8   - record_ids = routing.keys
  6 + model = Searchkick.load_model(class_name)
  7 + index = model.searchkick_index(name: index_name)
9 8  
10   - klass = class_name.constantize
11   - scope = Searchkick.load_records(klass, record_ids)
12   - scope = scope.search_import if scope.respond_to?(:search_import)
13   - records = scope.select(&:should_index?)
14   -
15   - # determine which records to delete
16   - delete_ids = record_ids - records.map { |r| r.id.to_s }
17   - delete_records = delete_ids.map do |id|
18   - m = klass.new
19   - m.id = id
20   - if routing[id]
21   - m.define_singleton_method(:search_routing) do
22   - routing[id]
23   - end
  9 + items =
  10 + record_ids.map do |r|
  11 + parts = r.split(/(?<!\|)\|(?!\|)/, 2)
  12 + .map { |v| v.gsub("||", "|") }
  13 + {id: parts[0], routing: parts[1]}
24 14 end
25   - m
26   - end
27 15  
28   - # bulk reindex
29   - index = klass.searchkick_index(name: index_name)
30   - Searchkick.callbacks(:bulk) do
31   - index.bulk_index(records) if records.any?
32   - index.bulk_delete(delete_records) if delete_records.any?
33   - end
  16 + relation = Searchkick.scope(model)
  17 + RecordIndexer.new(index).reindex_items(relation, items, method_name: nil)
34 18 end
35 19 end
36 20 end
... ...
lib/searchkick/process_queue_job.rb
  Diff comments   View file @684eecf
... ... @@ -3,11 +3,12 @@ module Searchkick
3 3 queue_as { Searchkick.queue_name }
4 4  
5 5 def perform(class_name:, index_name: nil, inline: false)
6   - model = class_name.constantize
  6 + model = Searchkick.load_model(class_name)
  7 + index = model.searchkick_index(name: index_name)
7 8 limit = model.searchkick_options[:batch_size] || 1000
8 9  
9 10 loop do
10   - record_ids = model.searchkick_index(name: index_name).reindex_queue.reserve(limit: limit)
  11 + record_ids = index.reindex_queue.reserve(limit: limit)
11 12 if record_ids.any?
12 13 batch_options = {
13 14 class_name: class_name,
... ...
lib/searchkick/query.rb
  Diff comments   View file @684eecf
... ... @@ -18,7 +18,7 @@ module Searchkick
18 18  
19 19 def initialize(klass, term = "*", **options)
20 20 unknown_keywords = options.keys - [:aggs, :block, :body, :body_options, :boost,
21   - :boost_by, :boost_by_distance, :boost_by_recency, :boost_where, :conversions, :conversions_term, :debug, :emoji, :exclude, :execute, :explain,
  21 + :boost_by, :boost_by_distance, :boost_by_recency, :boost_where, :conversions, :conversions_term, :debug, :emoji, :exclude, :explain,
22 22 :fields, :highlight, :includes, :index_name, :indices_boost, :limit, :load,
23 23 :match, :misspellings, :models, :model_includes, :offset, :operator, :order, :padding, :page, :per_page, :profile,
24 24 :request_params, :routing, :scope_results, :scroll, :select, :similar, :smart_aggs, :suggest, :total_entries, :track, :type, :where]
... ... @@ -148,9 +148,6 @@ module Searchkick
148 148 }
149 149  
150 150 if options[:debug]
151   - # can remove when minimum Ruby version is 2.5
152   - require "pp"
153   -
154 151 puts "Searchkick Version: #{Searchkick::VERSION}"
155 152 puts "Elasticsearch Version: #{Searchkick.server_version}"
156 153 puts
... ... @@ -210,14 +207,14 @@ module Searchkick
210 207 e.message.include?("No query registered for [function_score]")
211 208 )
212 209  
213   - raise UnsupportedVersionError, "This version of Searchkick requires Elasticsearch 5 or greater"
  210 + raise UnsupportedVersionError
214 211 elsif status_code == 400
215 212 if (
216 213 e.message.include?("bool query does not support [filter]") ||
217 214 e.message.include?("[bool] filter does not support [filter]")
218 215 )
219 216  
220   - raise UnsupportedVersionError, "This version of Searchkick requires Elasticsearch 5 or greater"
  217 + raise UnsupportedVersionError
221 218 elsif e.message =~ /analyzer \[searchkick_.+\] not found/
222 219 raise InvalidQueryError, "Bad mapping - run #{reindex_command}"
223 220 else
... ... @@ -233,7 +230,14 @@ module Searchkick
233 230 end
234 231  
235 232 def execute_search
236   - Searchkick.client.search(params)
  233 + name = searchkick_klass ? "#{searchkick_klass.name} Search" : "Search"
  234 + event = {
  235 + name: name,
  236 + query: params
  237 + }
  238 + ActiveSupport::Notifications.instrument("search.searchkick", event) do
  239 + Searchkick.client.search(params)
  240 + end
237 241 end
238 242  
239 243 def prepare
... ... @@ -268,9 +272,10 @@ module Searchkick
268 272 should = []
269 273  
270 274 if options[:similar]
  275 + like = options[:similar] == true ? term : options[:similar]
271 276 query = {
272 277 more_like_this: {
273   - like: term,
  278 + like: like,
274 279 min_doc_freq: 1,
275 280 min_term_freq: 1,
276 281 analyzer: "searchkick_search2"
... ... @@ -383,11 +388,6 @@ module Searchkick
383 388  
384 389 if field.start_with?("*.")
385 390 q2 = qs.map { |q| {multi_match: q.merge(fields: [field], type: match_type == :match_phrase ? "phrase" : "best_fields")} }
386   - if below61?
387   - q2.each do |q|
388   - q[:multi_match].delete(:fuzzy_transpositions)
389   - end
390   - end
391 391 else
392 392 q2 = qs.map { |q| {match_type => {field => q}} }
393 393 end
... ... @@ -439,7 +439,7 @@ module Searchkick
439 439 payload = {}
440 440  
441 441 # type when inheritance
442   - where = (options[:where] || {}).dup
  442 + where = ensure_permitted(options[:where] || {}).dup
443 443 if searchkick_options[:inheritance] && (options[:type] || (klass != searchkick_klass && searchkick_index))
444 444 where[:type] = [options[:type] || klass].flatten.map { |v| searchkick_index.klass_document_type(v, true) }
445 445 end
... ... @@ -696,9 +696,9 @@ module Searchkick
696 696 def set_boost_by(multiply_filters, custom_filters)
697 697 boost_by = options[:boost_by] || {}
698 698 if boost_by.is_a?(Array)
699   - boost_by = Hash[boost_by.map { |f| [f, {factor: 1}] }]
  699 + boost_by = boost_by.to_h { |f| [f, {factor: 1}] }
700 700 elsif boost_by.is_a?(Hash)
701   - multiply_by, boost_by = boost_by.partition { |_, v| v.delete(:boost_mode) == "multiply" }.map { |i| Hash[i] }
  701 + multiply_by, boost_by = boost_by.partition { |_, v| v.delete(:boost_mode) == "multiply" }.map(&:to_h)
702 702 end
703 703 boost_by[options[:boost]] = {factor: 1} if options[:boost]
704 704  
... ... @@ -763,7 +763,7 @@ module Searchkick
763 763  
764 764 def set_highlights(payload, fields)
765 765 payload[:highlight] = {
766   - fields: Hash[fields.map { |f| [f, {}] }],
  766 + fields: fields.to_h { |f| [f, {}] },
767 767 fragment_size: 0
768 768 }
769 769  
... ... @@ -797,7 +797,7 @@ module Searchkick
797 797 aggs = options[:aggs]
798 798 payload[:aggs] = {}
799 799  
800   - aggs = Hash[aggs.map { |f| [f, {}] }] if aggs.is_a?(Array) # convert to more advanced syntax
  800 + aggs = aggs.to_h { |f| [f, {}] } if aggs.is_a?(Array) # convert to more advanced syntax
801 801 aggs.each do |field, agg_options|
802 802 size = agg_options[:limit] ? agg_options[:limit] : 1_000
803 803 shared_agg_options = agg_options.except(:limit, :field, :ranges, :date_ranges, :where)
... ... @@ -836,8 +836,9 @@ module Searchkick
836 836 end
837 837  
838 838 where = {}
839   - where = (options[:where] || {}).reject { |k| k == field } unless options[:smart_aggs] == false
840   - agg_filters = where_filters(where.merge(agg_options[:where] || {}))
  839 + where = ensure_permitted(options[:where] || {}).reject { |k| k == field } unless options[:smart_aggs] == false
  840 + agg_where = ensure_permitted(agg_options[:where] || {})
  841 + agg_filters = where_filters(where.merge(agg_where))
841 842  
842 843 # only do one level comparison for simplicity
843 844 filters.select! do |filter|
... ... @@ -873,19 +874,16 @@ module Searchkick
873 874 end
874 875  
875 876 def set_order(payload)
876   - order = options[:order].is_a?(Enumerable) ? options[:order] : {options[:order] => :asc}
877   - id_field = :_id
878   - # TODO no longer map id to _id in Searchkick 5
879   - # since sorting on _id is deprecated in Elasticsearch
880   - payload[:sort] = order.is_a?(Array) ? order : Hash[order.map { |k, v| [k.to_s == "id" ? id_field : k, v] }]
  877 + payload[:sort] = options[:order].is_a?(Enumerable) ? options[:order] : {options[:order] => :asc}
881 878 end
882 879  
883   - def where_filters(where)
884   - # if where.respond_to?(:permitted?) && !where.permitted?
885   - # # TODO check in more places
886   - # Searchkick.warn("Passing unpermitted parameters will raise an exception in Searchkick 5")
887   - # end
  880 + # provides *very* basic protection from unfiltered parameters
  881 + # this is not meant to be comprehensive and may be expanded in the future
  882 + def ensure_permitted(obj)
  883 + obj.to_h
  884 + end
888 885  
  886 + def where_filters(where)
889 887 filters = []
890 888 (where || {}).each do |field, value|
891 889 field = :_id if field.to_s == "id"
... ... @@ -1007,7 +1005,7 @@ module Searchkick
1007 1005 when :lte
1008 1006 {to: op_value, include_upper: true}
1009 1007 else
1010   - raise "Unknown where operator: #{op.inspect}"
  1008 + raise ArgumentError, "Unknown where operator: #{op.inspect}"
1011 1009 end
1012 1010 # issue 132
1013 1011 if (existing = filters.find { |f| f[:range] && f[:range][field] })
... ... @@ -1036,29 +1034,25 @@ module Searchkick
1036 1034 {bool: {must_not: {exists: {field: field}}}}
1037 1035 elsif value.is_a?(Regexp)
1038 1036 source = value.source
1039   - unless source.start_with?("\\A") && source.end_with?("\\z")
1040   - # https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html
1041   - Searchkick.warn("Regular expressions are always anchored in Elasticsearch")
1042   - end
  1037 +
  1038 + # TODO handle other regexp options
1043 1039  
1044 1040 # TODO handle other anchor characters, like ^, $, \Z
1045 1041 if source.start_with?("\\A")
1046 1042 source = source[2..-1]
1047 1043 else
1048   - # TODO uncomment in Searchkick 5
1049   - # source = ".*#{source}"
  1044 + source = ".*#{source}"
1050 1045 end
1051 1046  
1052 1047 if source.end_with?("\\z")
1053 1048 source = source[0..-3]
1054 1049 else
1055   - # TODO uncomment in Searchkick 5
1056   - # source = "#{source}.*"
  1050 + source = "#{source}.*"
1057 1051 end
1058 1052  
1059 1053 if below710?
1060 1054 if value.casefold?
1061   - Searchkick.warn("Case-insensitive flag does not work with Elasticsearch < 7.10")
  1055 + raise ArgumentError, "Case-insensitive flag does not work with Elasticsearch < 7.10"
1062 1056 end
1063 1057 {regexp: {field => {value: source, flags: "NONE"}}}
1064 1058 else
... ... @@ -1069,9 +1063,7 @@ module Searchkick
1069 1063 if value.as_json.is_a?(Enumerable)
1070 1064 # query will fail, but this is better
1071 1065 # same message as Active Record
1072   - # TODO make TypeError
1073   - # raise InvalidQueryError for backward compatibility
1074   - raise Searchkick::InvalidQueryError, "can't cast #{value.class.name}"
  1066 + raise TypeError, "can't cast #{value.class.name}"
1075 1067 end
1076 1068  
1077 1069 {term: {field => {value: value}}}
... ... @@ -1150,21 +1142,13 @@ module Searchkick
1150 1142 end
1151 1143  
1152 1144 def track_total_hits?
1153   - (searchkick_options[:deep_paging] && !below70?) || body_options[:track_total_hits]
  1145 + searchkick_options[:deep_paging] || body_options[:track_total_hits]
1154 1146 end
1155 1147  
1156 1148 def body_options
1157 1149 options[:body_options] || {}
1158 1150 end
1159 1151  
1160   - def below61?
1161   - Searchkick.server_below?("6.1.0")
1162   - end
1163   -
1164   - def below70?
1165   - Searchkick.server_below?("7.0.0")
1166   - end
1167   -
1168 1152 def below73?
1169 1153 Searchkick.server_below?("7.3.0")
1170 1154 end
... ...
lib/searchkick/record_data.rb
  Diff comments   View file @684eecf
... ... @@ -39,7 +39,6 @@ module Searchkick
39 39 _index: index.name,
40 40 _id: search_id
41 41 }
42   - data[:_type] = document_type if Searchkick.server_below7?
43 42 data[:routing] = record.search_routing if record.respond_to?(:search_routing)
44 43 data
45 44 end
... ...
lib/searchkick/record_indexer.rb
  Diff comments   View file @684eecf
1 1 module Searchkick
2 2 class RecordIndexer
3   - attr_reader :record, :index
  3 + attr_reader :index
4 4  
5   - def initialize(record)
6   - @record = record
7   - @index = record.class.searchkick_index
  5 + def initialize(index)
  6 + @index = index
8 7 end
9 8  
10   - def reindex(method_name = nil, refresh: false, mode: nil)
11   - unless [:inline, true, nil, :async, :queue].include?(mode)
12   - raise ArgumentError, "Invalid value for mode"
13   - end
14   -
15   - mode ||= Searchkick.callbacks_value || index.options[:callbacks] || true
  9 + def reindex(records, mode:, method_name:, full: false, single: false)
  10 + # prevents exists? check if records is a relation
  11 + records = records.to_a
  12 + return if records.empty?
16 13  
17 14 case mode
  15 + when :async
  16 + unless defined?(ActiveJob)
  17 + raise Searchkick::Error, "Active Job not found"
  18 + end
  19 +
  20 + # we could likely combine ReindexV2Job, BulkReindexJob, and ProcessBatchJob
  21 + # but keep them separate for now
  22 + if single
  23 + record = records.first
  24 +
  25 + # always pass routing in case record is deleted
  26 + # before the async job runs
  27 + if record.respond_to?(:search_routing)
  28 + routing = record.search_routing
  29 + end
  30 +
  31 + Searchkick::ReindexV2Job.perform_later(
  32 + record.class.name,
  33 + record.id.to_s,
  34 + method_name ? method_name.to_s : nil,
  35 + routing: routing,
  36 + index_name: index.name
  37 + )
  38 + else
  39 + Searchkick::BulkReindexJob.perform_later(
  40 + class_name: records.first.class.searchkick_options[:class_name],
  41 + record_ids: records.map { |r| r.id.to_s },
  42 + index_name: index.name,
  43 + method_name: method_name ? method_name.to_s : nil
  44 + )
  45 + end
18 46 when :queue
19 47 if method_name
20 48 raise Searchkick::Error, "Partial reindex not supported with queue option"
21 49 end
22 50  
23   - # always pass routing in case record is deleted
24   - # before the queue job runs
25   - if record.respond_to?(:search_routing)
26   - routing = record.search_routing
27   - end
  51 + index.reindex_queue.push_records(records)
  52 + when true, :inline
  53 + index_records, other_records = records.partition { |r| index_record?(r) }
  54 + import_inline(index_records, !full ? other_records : [], method_name: method_name, single: single)
  55 + else
  56 + raise ArgumentError, "Invalid value for mode"
  57 + end
28 58  
29   - # escape pipe with double pipe
30   - value = queue_escape(record.id.to_s)
31   - value = "#{value}|#{queue_escape(routing)}" if routing
32   - index.reindex_queue.push(value)
33   - when :async
34   - unless defined?(ActiveJob)
35   - raise Searchkick::Error, "Active Job not found"
36   - end
  59 + # return true like model and relation reindex for now
  60 + true
  61 + end
37 62  
38   - # always pass routing in case record is deleted
39   - # before the async job runs
40   - if record.respond_to?(:search_routing)
41   - routing = record.search_routing
42   - end
  63 + def reindex_items(klass, items, method_name:, single: false)
  64 + routing = items.to_h { |r| [r[:id], r[:routing]] }
  65 + record_ids = routing.keys
43 66  
44   - Searchkick::ReindexV2Job.perform_later(
45   - record.class.name,
46   - record.id.to_s,
47   - method_name ? method_name.to_s : nil,
48   - routing: routing
49   - )
50   - else # bulk, inline/true/nil
51   - reindex_record(method_name)
  67 + relation = Searchkick.load_records(klass, record_ids)
  68 + # call search_import even for single records for nested associations
  69 + relation = relation.search_import if relation.respond_to?(:search_import)
  70 + records = relation.select(&:should_index?)
52 71  
53   - index.refresh if refresh
54   - end
  72 + # determine which records to delete
  73 + delete_ids = record_ids - records.map { |r| r.id.to_s }
  74 + delete_records =
  75 + delete_ids.map do |id|
  76 + construct_record(klass, id, routing[id])
  77 + end
  78 +
  79 + import_inline(records, delete_records, method_name: method_name, single: single)
55 80 end
56 81  
57 82 private
58 83  
59   - def queue_escape(value)
60   - value.gsub("|", "||")
  84 + def index_record?(record)
  85 + record.persisted? && !record.destroyed? && record.should_index?
61 86 end
62 87  
63   - def reindex_record(method_name)
64   - if record.destroyed? || !record.persisted? || !record.should_index?
65   - begin
66   - index.remove(record)
67   - rescue => e
68   - raise e unless Searchkick.not_found_error?(e)
69   - # do nothing if not found
  88 + # import in single request with retries
  89 + def import_inline(index_records, delete_records, method_name:, single:)
  90 + return if index_records.empty? && delete_records.empty?
  91 +
  92 + maybe_bulk(index_records, delete_records, method_name, single) do
  93 + if index_records.any?
  94 + if method_name
  95 + index.bulk_update(index_records, method_name)
  96 + else
  97 + index.bulk_index(index_records)
  98 + end
  99 + end
  100 +
  101 + if delete_records.any?
  102 + index.bulk_delete(delete_records)
70 103 end
  104 + end
  105 + end
  106 +
  107 + def maybe_bulk(index_records, delete_records, method_name, single)
  108 + if Searchkick.callbacks_value == :bulk
  109 + yield
71 110 else
72   - if method_name
73   - index.update_record(record, method_name)
74   - else
75   - index.store(record)
  111 + # set action and data
  112 + action =
  113 + if single && index_records.empty?
  114 + "Remove"
  115 + elsif method_name
  116 + "Update"
  117 + else
  118 + single ? "Store" : "Import"
  119 + end
  120 + record = index_records.first || delete_records.first
  121 + name = record.class.searchkick_klass.name
  122 + message = lambda do |event|
  123 + event[:name] = "#{name} #{action}"
  124 + if single
  125 + event[:id] = index.search_id(record)
  126 + else
  127 + event[:count] = index_records.size + delete_records.size
  128 + end
  129 + end
  130 +
  131 + with_retries do
  132 + Searchkick.callbacks(:bulk, message: message) do
  133 + yield
  134 + end
  135 + end
  136 + end
  137 + end
  138 +
  139 + def construct_record(klass, id, routing)
  140 + record = klass.new
  141 + record.id = id
  142 + if routing
  143 + record.define_singleton_method(:search_routing) do
  144 + routing
  145 + end
  146 + end
  147 + record
  148 + end
  149 +
  150 + def with_retries
  151 + retries = 0
  152 +
  153 + begin
  154 + yield
  155 + rescue Faraday::ClientError => e
  156 + if retries < 1
  157 + retries += 1
  158 + retry
76 159 end
  160 + raise e
77 161 end
78 162 end
79 163 end
... ...
lib/searchkick/reindex_queue.rb
  Diff comments   View file @684eecf
... ... @@ -8,8 +8,27 @@ module Searchkick
8 8 raise Searchkick::Error, "Searchkick.redis not set" unless Searchkick.redis
9 9 end
10 10  
11   - def push(record_id)
12   - Searchkick.with_redis { |r| r.lpush(redis_key, record_id) }
  11 + # supports single and multiple ids
  12 + def push(record_ids)
  13 + Searchkick.with_redis { |r| r.lpush(redis_key, record_ids) }
  14 + end
  15 +
  16 + def push_records(records)
  17 + record_ids =
  18 + records.map do |record|
  19 + # always pass routing in case record is deleted
  20 + # before the queue job runs
  21 + if record.respond_to?(:search_routing)
  22 + routing = record.search_routing
  23 + end
  24 +
  25 + # escape pipe with double pipe
  26 + value = escape(record.id.to_s)
  27 + value = "#{value}|#{escape(routing)}" if routing
  28 + value
  29 + end
  30 +
  31 + push(record_ids)
13 32 end
14 33  
15 34 # TODO use reliable queuing
... ... @@ -48,5 +67,9 @@ module Searchkick
48 67 def redis_version
49 68 @redis_version ||= Searchkick.with_redis { |r| Gem::Version.new(r.info["redis_version"]) }
50 69 end
  70 +
  71 + def escape(value)
  72 + value.gsub("|", "||")
  73 + end
51 74 end
52 75 end
... ...
lib/searchkick/reindex_v2_job.rb
  Diff comments   View file @684eecf
1 1 module Searchkick
2 2 class ReindexV2Job < ActiveJob::Base
3   - RECORD_NOT_FOUND_CLASSES = [
4   - "ActiveRecord::RecordNotFound",
5   - "Mongoid::Errors::DocumentNotFound",
6   - "NoBrainer::Error::DocumentNotFound",
7   - "Cequel::Record::RecordNotFound"
8   - ]
9   -
10 3 queue_as { Searchkick.queue_name }
11 4  
12   - def perform(klass, id, method_name = nil, routing: nil)
13   - model = klass.constantize
14   - record =
15   - begin
16   - if model.respond_to?(:unscoped)
17   - model.unscoped.find(id)
18   - else
19   - model.find(id)
20   - end
21   - rescue => e
22   - # check by name rather than rescue directly so we don't need
23   - # to determine which classes are defined
24   - raise e unless RECORD_NOT_FOUND_CLASSES.include?(e.class.name)
25   - nil
26   - end
27   -
28   - unless record
29   - record = model.new
30   - record.id = id
31   - if routing
32   - record.define_singleton_method(:search_routing) do
33   - routing
34   - end
35   - end
36   - end
37   -
38   - RecordIndexer.new(record).reindex(method_name, mode: :inline)
  5 + def perform(class_name, id, method_name = nil, routing: nil, index_name: nil)
  6 + model = Searchkick.load_model(class_name, allow_child: true)
  7 + index = model.searchkick_index(name: index_name)
  8 + # use should_index? to decide whether to index (not default scope)
  9 + # just like saving inline
  10 + # could use Searchkick.scope() in future
  11 + # but keep for now for backwards compatibility
  12 + model = model.unscoped if model.respond_to?(:unscoped)
  13 + items = [{id: id, routing: routing}]
  14 + RecordIndexer.new(index).reindex_items(model, items, method_name: method_name, single: true)
39 15 end
40 16 end
41 17 end
... ...
lib/searchkick/relation.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,36 @@
  1 +module Searchkick
  2 + class Relation
  3 + # note: modifying body directly is not supported
  4 + # and has no impact on query after being executed
  5 + # TODO freeze body object?
  6 + delegate :body, :params, to: :@query
  7 + delegate_missing_to :private_execute
  8 +
  9 + def initialize(model, term = "*", **options)
  10 + @query = Query.new(model, term, **options)
  11 + end
  12 +
  13 + # same as Active Record
  14 + def inspect
  15 + entries = results.first(11).map!(&:inspect)
  16 + entries[10] = "..." if entries.size == 11
  17 + "#<#{self.class.name} [#{entries.join(', ')}]>"
  18 + end
  19 +
  20 + def execute
  21 + Searchkick.warn("The execute method is no longer needed")
  22 + private_execute
  23 + self
  24 + end
  25 +
  26 + private
  27 +
  28 + def private_execute
  29 + @execute ||= @query.execute
  30 + end
  31 +
  32 + def query
  33 + @query
  34 + end
  35 + end
  36 +end
... ...
lib/searchkick/relation_indexer.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,150 @@
  1 +module Searchkick
  2 + class RelationIndexer
  3 + attr_reader :index
  4 +
  5 + def initialize(index)
  6 + @index = index
  7 + end
  8 +
  9 + def reindex(relation, mode:, method_name: nil, full: false, resume: false, scope: nil)
  10 + # apply scopes
  11 + if scope
  12 + relation = relation.send(scope)
  13 + elsif relation.respond_to?(:search_import)
  14 + relation = relation.search_import
  15 + end
  16 +
  17 + # remove unneeded loading for async
  18 + if mode == :async
  19 + if relation.respond_to?(:primary_key)
  20 + relation = relation.select(relation.primary_key).except(:includes, :preload)
  21 + elsif relation.respond_to?(:only)
  22 + relation = relation.only(:_id)
  23 + end
  24 + end
  25 +
  26 + if mode == :async && full
  27 + return full_reindex_async(relation)
  28 + end
  29 +
  30 + relation = resume_relation(relation) if resume
  31 +
  32 + reindex_options = {
  33 + mode: mode,
  34 + method_name: method_name,
  35 + full: full
  36 + }
  37 + record_indexer = RecordIndexer.new(index)
  38 +
  39 + in_batches(relation) do |items|
  40 + record_indexer.reindex(items, **reindex_options)
  41 + end
  42 + end
  43 +
  44 + def batches_left
  45 + Searchkick.with_redis { |r| r.scard(batches_key) }
  46 + end
  47 +
  48 + def batch_completed(batch_id)
  49 + Searchkick.with_redis { |r| r.srem(batches_key, batch_id) }
  50 + end
  51 +
  52 + private
  53 +
  54 + def resume_relation(relation)
  55 + if relation.respond_to?(:primary_key)
  56 + # use total docs instead of max id since there's not a great way
  57 + # to get the max _id without scripting since it's a string
  58 + where = relation.arel_table[relation.primary_key].gt(index.total_docs)
  59 + relation = relation.where(where)
  60 + else
  61 + raise Error, "Resume not supported for Mongoid"
  62 + end
  63 + end
  64 +
  65 + def in_batches(relation)
  66 + if relation.respond_to?(:find_in_batches)
  67 + klass = relation.klass
  68 + # remove order to prevent possible warnings
  69 + relation.except(:order).find_in_batches(batch_size: batch_size) do |batch|
  70 + # prevent scope from affecting search_data as well as inline jobs
  71 + # Active Record runs relation calls in scoping block
  72 + # https://github.com/rails/rails/blob/main/activerecord/lib/active_record/relation/delegation.rb
  73 + # note: we could probably just call klass.current_scope = nil
  74 + # anywhere in reindex method (after initial all call),
  75 + # but this is more cautious
  76 + previous_scope = klass.current_scope(true)
  77 + if previous_scope
  78 + begin
  79 + klass.current_scope = nil
  80 + yield batch
  81 + ensure
  82 + klass.current_scope = previous_scope
  83 + end
  84 + else
  85 + yield batch
  86 + end
  87 + end
  88 + else
  89 + klass = relation.klass
  90 + each_batch(relation, batch_size: batch_size) do |batch|
  91 + # prevent scope from affecting search_data as well as inline jobs
  92 + # note: Model.with_scope doesn't always restore scope, so use custom logic
  93 + previous_scope = Mongoid::Threaded.current_scope(klass)
  94 + if previous_scope
  95 + begin
  96 + Mongoid::Threaded.set_current_scope(nil, klass)
  97 + yield batch
  98 + ensure
  99 + Mongoid::Threaded.set_current_scope(previous_scope, klass)
  100 + end
  101 + else
  102 + yield batch
  103 + end
  104 + end
  105 + end
  106 + end
  107 +
  108 + def each_batch(relation, batch_size:)
  109 + # https://github.com/karmi/tire/blob/master/lib/tire/model/import.rb
  110 + # use cursor for Mongoid
  111 + items = []
  112 + relation.all.each do |item|
  113 + items << item
  114 + if items.length == batch_size
  115 + yield items
  116 + items = []
  117 + end
  118 + end
  119 + yield items if items.any?
  120 + end
  121 +
  122 + def batch_size
  123 + @batch_size ||= index.options[:batch_size] || 1000
  124 + end
  125 +
  126 + def full_reindex_async(relation)
  127 + batch_id = 1
  128 + class_name = relation.searchkick_options[:class_name]
  129 +
  130 + in_batches(relation) do |items|
  131 + batch_job(class_name, batch_id, items.map(&:id))
  132 + batch_id += 1
  133 + end
  134 + end
  135 +
  136 + def batch_job(class_name, batch_id, record_ids)
  137 + Searchkick.with_redis { |r| r.sadd(batches_key, batch_id) }
  138 + Searchkick::BulkReindexJob.perform_later(
  139 + class_name: class_name,
  140 + index_name: index.name,
  141 + batch_id: batch_id,
  142 + record_ids: record_ids.map { |v| v.instance_of?(Integer) ? v : v.to_s }
  143 + )
  144 + end
  145 +
  146 + def batches_key
  147 + "searchkick:reindex:#{index.name}:batches"
  148 + end
  149 + end
  150 +end
... ...
lib/searchkick/results.rb
  Diff comments   View file @684eecf
1   -require "forwardable"
2   -
3 1 module Searchkick
4 2 class Results
5 3 include Enumerable
... ... @@ -19,13 +17,11 @@ module Searchkick
19 17 @results ||= with_hit.map(&:first)
20 18 end
21 19  
22   - # TODO return enumerator like with_score
23 20 def with_hit
24   - @with_hit ||= begin
25   - if missing_records.any?
26   - Searchkick.warn("Records in search index do not exist in database: #{missing_records.map { |v| v[:id] }.join(", ")}")
27   - end
28   - with_hit_and_missing_records[0]
  21 + return enum_for(:with_hit) unless block_given?
  22 +
  23 + build_hits.each do |result|
  24 + yield result
29 25 end
30 26 end
31 27  
... ... @@ -157,10 +153,11 @@ module Searchkick
157 153 end
158 154 end
159 155  
160   - # TODO return enumerator like with_score
161 156 def with_highlights(multiple: false)
162   - with_hit.map do |result, hit|
163   - [result, hit_highlights(hit, multiple: multiple)]
  157 + return enum_for(:with_highlights, multiple: multiple) unless block_given?
  158 +
  159 + with_hit.each do |result, hit|
  160 + yield result, hit_highlights(hit, multiple: multiple)
164 161 end
165 162 end
166 163  
... ... @@ -287,7 +284,7 @@ module Searchkick
287 284 end
288 285  
289 286 if hit["highlight"] || options[:highlight]
290   - highlight = Hash[hit["highlight"].to_a.map { |k, v| [base_field(k), v.first] }]
  287 + highlight = hit["highlight"].to_a.to_h { |k, v| [base_field(k), v.first] }
291 288 options[:highlighted_fields].map { |k| base_field(k) }.each do |k|
292 289 result["highlighted_#{k}"] ||= (highlight[k] || result[k])
293 290 end
... ... @@ -302,23 +299,25 @@ module Searchkick
302 299 end
303 300 end
304 301  
  302 + def build_hits
  303 + @build_hits ||= begin
  304 + if missing_records.any?
  305 + Searchkick.warn("Records in search index do not exist in database: #{missing_records.map { |v| v[:id] }.join(", ")}")
  306 + end
  307 + with_hit_and_missing_records[0]
  308 + end
  309 + end
  310 +
305 311 def results_query(records, hits)
  312 + records = Searchkick.scope(records)
  313 +
306 314 ids = hits.map { |hit| hit["_id"] }
307 315 if options[:includes] || options[:model_includes]
308 316 included_relations = []
309 317 combine_includes(included_relations, options[:includes])
310 318 combine_includes(included_relations, options[:model_includes][records]) if options[:model_includes]
311 319  
312   - records =
313   - if defined?(NoBrainer::Document) && records < NoBrainer::Document
314   - if Gem.loaded_specs["nobrainer"].version >= Gem::Version.new("0.21")
315   - records.eager_load(included_relations)
316   - else
317   - records.preload(included_relations)
318   - end
319   - else
320   - records.includes(included_relations)
321   - end
  320 + records = records.includes(included_relations)
322 321 end
323 322  
324 323 if options[:scope_results]
... ... @@ -344,7 +343,7 @@ module Searchkick
344 343  
345 344 def hit_highlights(hit, multiple: false)
346 345 if hit["highlight"]
347   - Hash[hit["highlight"].map { |k, v| [(options[:json] ? k : k.sub(/\.#{@options[:match_suffix]}\z/, "")).to_sym, multiple ? v : v.first] }]
  346 + hit["highlight"].to_h { |k, v| [(options[:json] ? k : k.sub(/\.#{@options[:match_suffix]}\z/, "")).to_sym, multiple ? v : v.first] }
348 347 else
349 348 {}
350 349 end
... ...
lib/tasks/searchkick.rake
  Diff comments   View file @684eecf
... ... @@ -4,9 +4,12 @@ namespace :searchkick do
4 4 class_name = ENV["CLASS"]
5 5 abort "USAGE: rake searchkick:reindex CLASS=Product" unless class_name
6 6  
7   - model = class_name.safe_constantize
8   - abort "Could not find class: #{class_name}" unless model
9   - abort "#{class_name} is not a searchkick model" unless Searchkick.models.include?(model)
  7 + model =
  8 + begin
  9 + Searchkick.load_model(class_name)
  10 + rescue Searchkick::Error => e
  11 + abort e.message
  12 + end
10 13  
11 14 puts "Reindexing #{model.name}..."
12 15 model.reindex
... ...
searchkick.gemspec
  Diff comments   View file @684eecf
... ... @@ -13,9 +13,8 @@ Gem::Specification.new do |spec|
13 13 spec.files = Dir["*.{md,txt}", "{lib}/**/*"]
14 14 spec.require_path = "lib"
15 15  
16   - spec.required_ruby_version = ">= 2.4"
  16 + spec.required_ruby_version = ">= 2.6"
17 17  
18   - spec.add_dependency "activemodel", ">= 5"
19   - spec.add_dependency "elasticsearch", ">= 6", "< 7.14"
  18 + spec.add_dependency "activemodel", ">= 5.2"
20 19 spec.add_dependency "hashie"
21 20 end
... ...
test/aggs_test.rb
  Diff comments   View file @684eecf
... ... @@ -250,7 +250,7 @@ class AggsTest &lt; Minitest::Test
250 250 end
251 251  
252 252 def buckets_as_hash(agg)
253   - Hash[agg["buckets"].map { |v| [v["key"], v["doc_count"]] }]
  253 + agg["buckets"].to_h { |v| [v["key"], v["doc_count"]] }
254 254 end
255 255  
256 256 def store_agg(options, agg_key = "store_id")
... ... @@ -259,9 +259,9 @@ class AggsTest &lt; Minitest::Test
259 259 end
260 260  
261 261 def store_multiple_aggs(options)
262   - Hash[Product.search("Product", **options).aggs.map do |field, filtered_agg|
  262 + Product.search("Product", **options).aggs.to_h do |field, filtered_agg|
263 263 [field, buckets_as_hash(filtered_agg)]
264   - end]
  264 + end
265 265 end
266 266  
267 267 def interval_key
... ...
test/boost_test.rb
  Diff comments   View file @684eecf
... ... @@ -171,8 +171,6 @@ class BoostTest &lt; Minitest::Test
171 171 end
172 172  
173 173 def test_boost_by_indices
174   - skip if cequel?
175   -
176 174 store_names ["Rex"], Animal
177 175 store_names ["Rexx"], Product
178 176  
... ...
test/default_scope_test.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,31 @@
  1 +require_relative "test_helper"
  2 +
  3 +class DefaultScopeTest < Minitest::Test
  4 + def setup
  5 + Band.destroy_all
  6 + end
  7 +
  8 + def test_reindex
  9 + store [
  10 + {name: "Test", active: true},
  11 + {name: "Test 2", active: false}
  12 + ], reindex: false
  13 +
  14 + Band.reindex
  15 + assert_search "*", ["Test"], {load: false}
  16 + end
  17 +
  18 + def test_search
  19 + Band.reindex
  20 + Band.search("*") # test works
  21 +
  22 + error = assert_raises(Searchkick::Error) do
  23 + Band.all.search("*")
  24 + end
  25 + assert_equal "search must be called on model, not relation", error.message
  26 + end
  27 +
  28 + def default_model
  29 + Band
  30 + end
  31 +end
... ...
test/geo_shape_test.rb
  Diff comments   View file @684eecf
... ... @@ -31,23 +31,6 @@ class GeoShapeTest &lt; Minitest::Test
31 31 ]
32 32 end
33 33  
34   - def test_circle
35   - # https://github.com/elastic/elasticsearch/issues/39237
36   - skip unless Searchkick.server_below?("6.6.0")
37   -
38   - assert_search "*", ["Region A"], {
39   - where: {
40   - territory: {
41   - geo_shape: {
42   - type: "circle",
43   - coordinates: {lat: 28.0, lon: 38.0},
44   - radius: "444000m"
45   - }
46   - }
47   - }
48   - }
49   - end
50   -
51 34 def test_envelope
52 35 assert_search "*", ["Region A"], {
53 36 where: {
... ... @@ -144,23 +127,6 @@ class GeoShapeTest &lt; Minitest::Test
144 127 }
145 128 end
146 129  
147   - def test_contains
148   - # CONTAINS query relation not supported
149   - skip unless Searchkick.server_below?("6.6.0")
150   -
151   - assert_search "*", ["Region C"], {
152   - where: {
153   - territory: {
154   - geo_shape: {
155   - type: "envelope",
156   - relation: "contains",
157   - coordinates: [[12, 13], [13, 12]]
158   - }
159   - }
160   - }
161   - }
162   - end
163   -
164 130 def test_latlon
165 131 assert_search "*", ["Region A"], {
166 132 where: {
... ...
test/index_cache_test.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,49 @@
  1 +require_relative "test_helper"
  2 +
  3 +class IndexCacheTest < Minitest::Test
  4 + def setup
  5 + Product.class_variable_get(:@@searchkick_index_cache).clear
  6 + end
  7 +
  8 + def test_default
  9 + object_id = Product.search_index.object_id
  10 + 3.times do
  11 + assert_equal object_id, Product.search_index.object_id
  12 + end
  13 + end
  14 +
  15 + def test_max_size
  16 + starting_ids = object_ids(20)
  17 + assert_equal starting_ids, object_ids(20)
  18 + Product.search_index(name: "other")
  19 + refute_equal starting_ids, object_ids(20)
  20 + end
  21 +
  22 + def test_thread_safe
  23 + object_ids = with_threads { object_ids(20) }
  24 + assert_equal object_ids[0], object_ids[1]
  25 + assert_equal object_ids[0], object_ids[2]
  26 + end
  27 +
  28 + # object ids can differ since threads progress at different speeds
  29 + # test to make sure doesn't crash
  30 + def test_thread_safe_max_size
  31 + with_threads { object_ids(1000) }
  32 + end
  33 +
  34 + private
  35 +
  36 + def object_ids(count)
  37 + count.times.map { |i| Product.search_index(name: "index#{i}").object_id }
  38 + end
  39 +
  40 + def with_threads
  41 + previous = Thread.report_on_exception
  42 + begin
  43 + Thread.report_on_exception = true
  44 + 3.times.map { Thread.new { yield } }.map(&:join).map(&:value)
  45 + ensure
  46 + Thread.report_on_exception = previous
  47 + end
  48 + end
  49 +end
... ...
test/index_test.rb
  Diff comments   View file @684eecf
... ... @@ -58,9 +58,13 @@ class IndexTest &lt; Minitest::Test
58 58 def test_mappings
59 59 store_names ["Dollar Tree"], Store
60 60 assert_equal ["Dollar Tree"], Store.search(body: {query: {match: {name: "dollar"}}}).map(&:name)
61   - mapping = Store.search_index.mapping.values.first["mappings"]
62   - mapping = mapping["store"] if Searchkick.server_below?("7.0.0")
63   - assert_equal "text", mapping["properties"]["name"]["type"]
  61 + mapping = Store.search_index.mapping
  62 + assert_kind_of Hash, mapping
  63 + assert_equal "text", mapping.values.first["mappings"]["properties"]["name"]["type"]
  64 + end
  65 +
  66 + def test_settings
  67 + assert_kind_of Hash, Store.search_index.settings
64 68 end
65 69  
66 70 def test_remove_blank_id
... ... @@ -71,6 +75,18 @@ class IndexTest &lt; Minitest::Test
71 75 Product.reindex
72 76 end
73 77  
  78 + # keep simple for now, but maybe return client response in future
  79 + def test_store_response
  80 + product = Searchkick.callbacks(false) { Product.create!(name: "Product A") }
  81 + assert_nil Product.search_index.store(product)
  82 + end
  83 +
  84 + # keep simple for now, but maybe return client response in future
  85 + def test_bulk_index_response
  86 + product = Searchkick.callbacks(false) { Product.create!(name: "Product A") }
  87 + assert_nil Product.search_index.bulk_index([product])
  88 + end
  89 +
74 90 # TODO move
75 91  
76 92 def test_filterable
... ... @@ -87,7 +103,6 @@ class IndexTest &lt; Minitest::Test
87 103 end
88 104  
89 105 def test_large_value
90   - skip if nobrainer?
91 106 large_value = 1000.times.map { "hello" }.join(" ")
92 107 store [{name: "Product A", text: large_value}], Region
93 108 assert_search "product", ["Product A"], {}, Region
... ... @@ -97,7 +112,6 @@ class IndexTest &lt; Minitest::Test
97 112 end
98 113  
99 114 def test_very_large_value
100   - skip if nobrainer?
101 115 large_value = 10000.times.map { "hello" }.join(" ")
102 116 store [{name: "Product A", text: large_value}], Region
103 117 assert_search "product", ["Product A"], {}, Region
... ...
test/inheritance_test.rb
  Diff comments   View file @684eecf
1 1 require_relative "test_helper"
2 2  
3 3 class InheritanceTest < Minitest::Test
4   - def setup
5   - skip if cequel?
6   - super
7   - end
8   -
9 4 def test_child_reindex
10 5 store_names ["Max"], Cat
11 6 assert Dog.reindex
... ... @@ -120,4 +115,14 @@ class InheritanceTest &lt; Minitest::Test
120 115 end
121 116 assert_includes error.message, "Unknown model"
122 117 end
  118 +
  119 + def test_similar
  120 + store_names ["Dog", "Other dog"], Dog
  121 + store_names ["Not dog"], Cat
  122 +
  123 + dog = Dog.find_by!(name: "Dog")
  124 + assert_equal ["Other dog"], dog.similar(fields: [:name]).map(&:name)
  125 + assert_equal ["Not dog", "Other dog"], dog.similar(fields: [:name], models: [Animal]).map(&:name).sort
  126 + assert_equal ["Not dog"], dog.similar(fields: [:name], models: [Cat]).map(&:name).sort
  127 + end
123 128 end
... ...
test/language_test.rb
  Diff comments   View file @684eecf
... ... @@ -40,26 +40,8 @@ class LanguageTest &lt; Minitest::Test
40 40 end
41 41  
42 42 def test_japanese_search_synonyms
43   - error = assert_raises(Searchkick::Error) do
44   - with_options({language: "japanese", search_synonyms: [["้ฃฒใ‚€", "ๅ–ฐใ‚‰ใ†"]]}) do
45   - end
46   - end
47   - assert_equal "Search synonyms are not supported yet for language", error.message
48   - end
49   -
50   - def test_japanese2
51 43 # requires https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-kuromoji.html
52   - with_options({language: "japanese2"}) do
53   - store_names ["JRๆ–ฐๅฎฟ้ง…ใฎ่ฟ‘ใใซใƒ“ใƒผใƒซใ‚’้ฃฒใฟใซ่กŒใ“ใ†ใ‹"]
54   - assert_language_search "้ฃฒใ‚€", ["JRๆ–ฐๅฎฟ้ง…ใฎ่ฟ‘ใใซใƒ“ใƒผใƒซใ‚’้ฃฒใฟใซ่กŒใ“ใ†ใ‹"]
55   - assert_language_search "jr", ["JRๆ–ฐๅฎฟ้ง…ใฎ่ฟ‘ใใซใƒ“ใƒผใƒซใ‚’้ฃฒใฟใซ่กŒใ“ใ†ใ‹"]
56   - assert_language_search "ๆ–ฐ", []
57   - end
58   - end
59   -
60   - def test_japanese2_search_synonyms
61   - # requires https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-kuromoji.html
62   - with_options({language: "japanese2", search_synonyms: [["้ฃฒใ‚€", "ๅ–ฐใ‚‰ใ†"]]}) do
  44 + with_options({language: "japanese", search_synonyms: [["้ฃฒใ‚€", "ๅ–ฐใ‚‰ใ†"]]}) do
63 45 store_names ["JRๆ–ฐๅฎฟ้ง…ใฎ่ฟ‘ใใซใƒ“ใƒผใƒซใ‚’้ฃฒใฟใซ่กŒใ“ใ†ใ‹"]
64 46 assert_language_search "ๅ–ฐใ‚‰ใ†", ["JRๆ–ฐๅฎฟ้ง…ใฎ่ฟ‘ใใซใƒ“ใƒผใƒซใ‚’้ฃฒใฟใซ่กŒใ“ใ†ใ‹"]
65 47 assert_language_search "ๆ–ฐ", []
... ... @@ -79,7 +61,7 @@ class LanguageTest &lt; Minitest::Test
79 61 end
80 62  
81 63 def test_korean2
82   - skip if Searchkick.server_below?("6.4.0") || ci?
  64 + skip if ci?
83 65  
84 66 # requires https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-nori.html
85 67 with_options({language: "korean2"}) do
... ...
test/log_subscriber_test.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,97 @@
  1 +require_relative "test_helper"
  2 +
  3 +class LogSubscriberTest < Minitest::Test
  4 + def test_create
  5 + output = capture_logs do
  6 + Product.create!(name: "Product A")
  7 + end
  8 + assert_match "Product Store", output
  9 + end
  10 +
  11 + def test_update
  12 + product = Product.create!(name: "Product A")
  13 + output = capture_logs do
  14 + product.reindex(:search_name)
  15 + end
  16 + assert_match "Product Update", output
  17 + end
  18 +
  19 + def test_destroy
  20 + product = Product.create!(name: "Product A")
  21 + output = capture_logs do
  22 + product.destroy
  23 + end
  24 + assert_match "Product Remove", output
  25 + end
  26 +
  27 + def test_bulk
  28 + output = capture_logs do
  29 + Searchkick.callbacks(:bulk) do
  30 + Product.create!(name: "Product A")
  31 + end
  32 + end
  33 + assert_match "Bulk", output
  34 + refute_match "Product Store", output
  35 + end
  36 +
  37 + def test_reindex
  38 + create_products
  39 + output = capture_logs do
  40 + Product.reindex
  41 + end
  42 + assert_match "Product Import", output
  43 + assert_match '"count":3', output
  44 + end
  45 +
  46 + def test_reindex_relation
  47 + # where.not not supported
  48 + skip if mongoid? && Mongoid::VERSION.to_i < 7
  49 +
  50 + products = create_products
  51 + output = capture_logs do
  52 + Product.where.not(id: products.last.id).reindex
  53 + end
  54 + assert_match "Product Import", output
  55 + assert_match '"count":2', output
  56 + end
  57 +
  58 + def test_search
  59 + output = capture_logs do
  60 + Product.search("product").to_a
  61 + end
  62 + assert_match "Product Search", output
  63 + end
  64 +
  65 + def test_multi_search
  66 + output = capture_logs do
  67 + Searchkick.multi_search([Product.search("product")])
  68 + end
  69 + assert_match "Multi Search", output
  70 + end
  71 +
  72 + private
  73 +
  74 + def create_products
  75 + Searchkick.callbacks(false) do
  76 + 3.times.map do
  77 + Product.create!(name: "Product A")
  78 + end
  79 + end
  80 + end
  81 +
  82 + def capture_logs
  83 + previous_logger = ActiveSupport::LogSubscriber.logger
  84 + io = StringIO.new
  85 + begin
  86 + ActiveSupport::LogSubscriber.logger = ActiveSupport::Logger.new(io)
  87 + yield
  88 + io.rewind
  89 + output = io.read
  90 + previous_logger.debug(output) if previous_logger
  91 + puts output if ENV["LOG_SUBSCRIBER"]
  92 + output
  93 + ensure
  94 + ActiveSupport::LogSubscriber.logger = previous_logger
  95 + end
  96 + end
  97 +end
... ...
test/match_test.rb
  Diff comments   View file @684eecf
... ... @@ -34,8 +34,6 @@ class MatchTest &lt; Minitest::Test
34 34 end
35 35  
36 36 def test_percent
37   - # Note: "2% Milk" doesn't get matched in ES below 5.1.1
38   - # This could be a bug since it has an edit distance of 1
39 37 store_names ["1% Milk", "Whole Milk"]
40 38 assert_search "1%", ["1% Milk"]
41 39 end
... ...
test/models/animal.rb
  Diff comments   View file @684eecf
... ... @@ -3,7 +3,5 @@ class Animal
3 3 inheritance: true,
4 4 text_start: [:name],
5 5 suggest: [:name],
6   - index_name: -> { "#{name.tableize}-#{Date.today.year}#{Searchkick.index_suffix}" },
7   - callbacks: :async,
8   - wordnet: ENV["WORDNET"]
  6 + index_name: -> { "#{name.tableize}-#{Date.today.year}#{Searchkick.index_suffix}" }
9 7 end
... ...
test/models/artist.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,7 @@
  1 +class Artist
  2 + searchkick unscope: true
  3 +
  4 + def should_index?
  5 + should_index
  6 + end
  7 +end
... ...
test/models/product.rb
  Diff comments   View file @684eecf
... ... @@ -24,7 +24,13 @@ class Product
24 24  
25 25 attr_accessor :conversions, :user_ids, :aisle, :details
26 26  
  27 + class << self
  28 + attr_accessor :dynamic_data
  29 + end
  30 +
27 31 def search_data
  32 + return self.class.dynamic_data.call if self.class.dynamic_data
  33 +
28 34 serializable_hash.except("id", "_id").merge(
29 35 conversions: conversions,
30 36 user_ids: user_ids,
... ...
test/multi_search_test.rb
  Diff comments   View file @684eecf
... ... @@ -4,8 +4,8 @@ class MultiSearchTest &lt; Minitest::Test
4 4 def test_basic
5 5 store_names ["Product A"]
6 6 store_names ["Store A"], Store
7   - products = Product.search("*", execute: false)
8   - stores = Store.search("*", execute: false)
  7 + products = Product.search("*")
  8 + stores = Store.search("*")
9 9 Searchkick.multi_search([products, stores])
10 10 assert_equal ["Product A"], products.map(&:name)
11 11 assert_equal ["Store A"], stores.map(&:name)
... ... @@ -13,14 +13,14 @@ class MultiSearchTest &lt; Minitest::Test
13 13  
14 14 def test_methods
15 15 result = Product.search("*")
16   - query = Product.search("*", execute: false)
  16 + query = Product.search("*")
17 17 assert_empty(result.methods - query.methods)
18 18 end
19 19  
20 20 def test_error
21 21 store_names ["Product A"]
22   - products = Product.search("*", execute: false)
23   - stores = Store.search("*", order: [:bad_field], execute: false)
  22 + products = Product.search("*")
  23 + stores = Store.search("*", order: [:bad_field])
24 24 Searchkick.multi_search([products, stores])
25 25 assert !products.error
26 26 assert stores.error
... ... @@ -28,13 +28,13 @@ class MultiSearchTest &lt; Minitest::Test
28 28  
29 29 def test_misspellings_below_unmet
30 30 store_names ["abc", "abd", "aee"]
31   - products = Product.search("abc", misspellings: {below: 5}, execute: false)
  31 + products = Product.search("abc", misspellings: {below: 5})
32 32 Searchkick.multi_search([products])
33 33 assert_equal ["abc", "abd"], products.map(&:name)
34 34 end
35 35  
36 36 def test_query_error
37   - products = Product.search("*", order: {bad_column: :asc}, execute: false)
  37 + products = Product.search("*", order: {bad_column: :asc})
38 38 Searchkick.multi_search([products])
39 39 assert products.error
40 40 error = assert_raises(Searchkick::Error) { products.results }
... ...
test/notifications_test.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,25 @@
  1 +require_relative "test_helper"
  2 +
  3 +class NotificationsTest < Minitest::Test
  4 + def test_search
  5 + notifications = capture_notifications do
  6 + Product.search("product").to_a
  7 + end
  8 +
  9 + assert_equal 1, notifications.size
  10 + assert_equal "search.searchkick", notifications.last[:name]
  11 + end
  12 +
  13 + private
  14 +
  15 + def capture_notifications
  16 + notifications = []
  17 + callback = lambda do |name, started, finished, unique_id, payload|
  18 + notifications << {name: name, payload: payload}
  19 + end
  20 + ActiveSupport::Notifications.subscribed(callback, /searchkick/) do
  21 + yield
  22 + end
  23 + notifications
  24 + end
  25 +end
... ...
test/order_test.rb
  Diff comments   View file @684eecf
... ... @@ -11,22 +11,6 @@ class OrderTest &lt; Minitest::Test
11 11 assert_order "product", ["Product A", "Product B", "Product C", "Product D"], order: "name"
12 12 end
13 13  
14   - # TODO no longer map id to _id in Searchkick 5
15   - # since sorting on _id is deprecated in Elasticsearch
16   - def test_id
17   - skip if cequel? || !Searchkick.server_below?("8.0.0")
18   -
19   - store_names ["Product A", "Product B"]
20   - product_a = Product.where(name: "Product A").first
21   - product_b = Product.where(name: "Product B").first
22   - _, stderr = capture_io do
23   - assert_order "product", [product_a, product_b].sort_by { |r| r.id.to_s }.map(&:name), order: {id: :asc}
24   - end
25   - unless Searchkick.server_below?("7.6.0")
26   - assert_match "Loading the fielddata on the _id field is deprecated", stderr
27   - end
28   - end
29   -
30 14 def test_multiple
31 15 store [
32 16 {name: "Product A", color: "blue", store_id: 1},
... ...
test/parameters_test.rb
  Diff comments   View file @684eecf
... ... @@ -6,25 +6,51 @@ class ParametersTest &lt; Minitest::Test
6 6 super
7 7 end
8 8  
9   - def test_where_unpermitted
10   - # TODO raise error in Searchkick 6
11   - store [{name: "Product A", store_id: 1}, {name: "Product B", store_id: 2}]
  9 + def test_options
12 10 params = ActionController::Parameters.new({store_id: 1})
13   - assert_search "product", ["Product A"], where: params
  11 + assert_raises(ActionController::UnfilteredParameters) do
  12 + Product.search("*", **params)
  13 + end
  14 + end
  15 +
  16 + def test_where
  17 + params = ActionController::Parameters.new({store_id: 1})
  18 + assert_raises(ActionController::UnfilteredParameters) do
  19 + Product.search("*", where: params)
  20 + end
14 21 end
15 22  
16 23 def test_where_permitted
17 24 store [{name: "Product A", store_id: 1}, {name: "Product B", store_id: 2}]
18 25 params = ActionController::Parameters.new({store_id: 1})
19   - assert_search "product", ["Product A"], where: params.permit!
  26 + assert_search "product", ["Product A"], where: params.permit(:store_id)
  27 + end
  28 +
  29 + def test_where_value
  30 + store [{name: "Product A", store_id: 1}, {name: "Product B", store_id: 2}]
  31 + params = ActionController::Parameters.new({store_id: 1})
  32 + assert_search "product", ["Product A"], where: {store_id: params[:store_id]}
20 33 end
21 34  
22 35 def test_where_hash
23 36 params = ActionController::Parameters.new({store_id: {value: 10, boost: 2}})
24   - # TODO make TypeError
25   - error = assert_raises Searchkick::InvalidQueryError do
  37 + error = assert_raises(TypeError) do
26 38 assert_search "product", [], where: {store_id: params[:store_id]}
27 39 end
28 40 assert_equal error.message, "can't cast ActionController::Parameters"
29 41 end
  42 +
  43 + def test_aggs_where
  44 + params = ActionController::Parameters.new({store_id: 1})
  45 + assert_raises(ActionController::UnfilteredParameters) do
  46 + Product.search("*", aggs: {size: {where: params}})
  47 + end
  48 + end
  49 +
  50 + def test_aggs_where_smart_aggs_false
  51 + params = ActionController::Parameters.new({store_id: 1})
  52 + assert_raises(ActionController::UnfilteredParameters) do
  53 + Product.search("*", aggs: {size: {where: params}}, smart_aggs: false)
  54 + end
  55 + end
30 56 end
... ...
test/query_test.rb
  Diff comments   View file @684eecf
... ... @@ -3,10 +3,8 @@ require_relative &quot;test_helper&quot;
3 3 class QueryTest < Minitest::Test
4 4 def test_basic
5 5 store_names ["Milk", "Apple"]
6   - query = Product.search("milk", execute: false)
7   - query.body[:query] = {match_all: {}}
  6 + query = Product.search("milk", body: {query: {match_all: {}}})
8 7 assert_equal ["Apple", "Milk"], query.map(&:name).sort
9   - assert_equal ["Apple", "Milk"], query.execute.map(&:name).sort
10 8 end
11 9  
12 10 def test_with_uneffective_min_score
... ... @@ -15,15 +13,15 @@ class QueryTest &lt; Minitest::Test
15 13 end
16 14  
17 15 def test_default_timeout
18   - assert_equal "6s", Product.search("*", execute: false).body[:timeout]
  16 + assert_equal "6s", Product.search("*").body[:timeout]
19 17 end
20 18  
21 19 def test_timeout_override
22   - assert_equal "1s", Product.search("*", body_options: {timeout: "1s"}, execute: false).body[:timeout]
  20 + assert_equal "1s", Product.search("*", body_options: {timeout: "1s"}).body[:timeout]
23 21 end
24 22  
25 23 def test_request_params
26   - assert_equal "dfs_query_then_fetch", Product.search("*", request_params: {search_type: "dfs_query_then_fetch"}, execute: false).params[:search_type]
  24 + assert_equal "dfs_query_then_fetch", Product.search("*", request_params: {search_type: "dfs_query_then_fetch"}).params[:search_type]
27 25 end
28 26  
29 27 def test_debug
... ... @@ -44,7 +42,7 @@ class QueryTest &lt; Minitest::Test
44 42 # body_options
45 43  
46 44 def test_body_options_should_merge_into_body
47   - query = Product.search("*", body_options: {min_score: 1.0}, execute: false)
  45 + query = Product.search("*", body_options: {min_score: 1.0})
48 46 assert_equal 1.0, query.body[:min_score]
49 47 end
50 48  
... ... @@ -75,8 +73,8 @@ class QueryTest &lt; Minitest::Test
75 73  
76 74 assert_equal 2, result.length
77 75  
78   - result.group_by(&:class).each_pair do |klass, records|
79   - assert records.first.association(associations[klass].first).loaded?
  76 + result.group_by(&:class).each_pair do |model, records|
  77 + assert records.first.association(associations[model].first).loaded?
80 78 end
81 79 end
82 80  
... ...
test/reindex_test.rb
  Diff comments   View file @684eecf
... ... @@ -5,40 +5,57 @@ class ReindexTest &lt; Minitest::Test
5 5 store_names ["Product A", "Product B"], reindex: false
6 6  
7 7 product = Product.find_by!(name: "Product A")
8   - product.reindex(refresh: true)
  8 + assert_equal true, product.reindex(refresh: true)
9 9 assert_search "product", ["Product A"]
10 10 end
11 11  
  12 + def test_record_destroyed
  13 + store_names ["Product A", "Product B"]
  14 +
  15 + product = Product.find_by!(name: "Product A")
  16 + product.destroy
  17 + Product.search_index.refresh
  18 + assert_equal true, product.reindex
  19 + end
  20 +
12 21 def test_record_async
13 22 store_names ["Product A", "Product B"], reindex: false
14 23  
15 24 product = Product.find_by!(name: "Product A")
16   - product.reindex(mode: :async)
  25 + perform_enqueued_jobs do
  26 + assert_equal true, product.reindex(mode: :async)
  27 + end
17 28 Product.search_index.refresh
18 29 assert_search "product", ["Product A"]
19 30 end
20 31  
21 32 def test_record_queue
22   - skip unless defined?(ActiveJob) && defined?(Redis)
23   -
24 33 reindex_queue = Product.searchkick_index.reindex_queue
25 34 reindex_queue.clear
26 35  
27 36 store_names ["Product A", "Product B"], reindex: false
28 37  
29 38 product = Product.find_by!(name: "Product A")
30   - product.reindex(mode: :queue)
  39 + assert_equal true, product.reindex(mode: :queue)
31 40 Product.search_index.refresh
32 41 assert_search "product", []
33 42  
34   - Searchkick::ProcessQueueJob.perform_now(class_name: "Product")
  43 + perform_enqueued_jobs do
  44 + Searchkick::ProcessQueueJob.perform_now(class_name: "Product")
  45 + end
35 46 Product.search_index.refresh
36 47 assert_search "product", ["Product A"]
37 48 end
38 49  
39   - def test_relation_inline
40   - skip if nobrainer? || cequel?
  50 + def test_record_index
  51 + store_names ["Product A", "Product B"], reindex: false
41 52  
  53 + product = Product.find_by!(name: "Product A")
  54 + assert_equal true, Product.search_index.reindex([product], refresh: true)
  55 + assert_search "product", ["Product A"]
  56 + end
  57 +
  58 + def test_relation_inline
42 59 store_names ["Product A"]
43 60 store_names ["Product B", "Product C"], reindex: false
44 61 Product.where(name: "Product B").reindex(refresh: true)
... ... @@ -46,41 +63,107 @@ class ReindexTest &lt; Minitest::Test
46 63 end
47 64  
48 65 def test_relation_associations
49   - skip if nobrainer? || cequel?
50   -
51 66 store_names ["Product A"]
52 67 store = Store.create!(name: "Test")
53 68 Product.create!(name: "Product B", store_id: store.id)
54   - store.products.reindex(refresh: true)
  69 + assert_equal true, store.products.reindex(refresh: true)
55 70 assert_search "product", ["Product A", "Product B"]
56 71 end
57 72  
58   - def test_relation_should_index
59   - skip if nobrainer? || cequel?
  73 + def test_relation_scoping
  74 + store_names ["Product A", "Product B"]
  75 + Product.dynamic_data = lambda do
  76 + {
  77 + name: "Count #{Product.count}"
  78 + }
  79 + end
  80 + Product.where(name: "Product A").reindex(refresh: true)
  81 + assert_search "count", ["Count 2"], load: false
  82 + ensure
  83 + Product.dynamic_data = nil
  84 + end
  85 +
  86 + def test_relation_scoping_restored
  87 + # TODO add test for Mongoid
  88 + skip unless activerecord?
  89 +
  90 + assert_nil Product.current_scope
  91 + Product.where(name: "Product A").scoping do
  92 + scope = Product.current_scope
  93 + refute_nil scope
  94 +
  95 + Product.all.reindex(refresh: true)
60 96  
61   - skip "TODO make pass in Searchkick 5"
  97 + # note: should be reset even if we don't do it
  98 + assert_equal scope, Product.current_scope
  99 + end
  100 + assert_nil Product.current_scope
  101 + end
62 102  
  103 + def test_relation_should_index
63 104 store_names ["Product A", "Product B"]
64 105 Searchkick.callbacks(false) do
65 106 Product.find_by(name: "Product B").update!(name: "DO NOT INDEX")
66 107 end
67   - Product.where(name: "DO NOT INDEX").reindex
  108 + assert_equal true, Product.where(name: "DO NOT INDEX").reindex
68 109 Product.search_index.refresh
69 110 assert_search "product", ["Product A"]
70 111 end
71 112  
72 113 def test_relation_async
73   - skip "Not available yet"
  114 + store_names ["Product A"]
  115 + store_names ["Product B", "Product C"], reindex: false
  116 + perform_enqueued_jobs do
  117 + Product.where(name: "Product B").reindex(mode: :async)
  118 + end
  119 + Product.search_index.refresh
  120 + assert_search "product", ["Product A", "Product B"]
  121 + end
  122 +
  123 + def test_relation_async_should_index
  124 + store_names ["Product A", "Product B"]
  125 + Searchkick.callbacks(false) do
  126 + Product.find_by(name: "Product B").update!(name: "DO NOT INDEX")
  127 + end
  128 + perform_enqueued_jobs do
  129 + assert_equal true, Product.where(name: "DO NOT INDEX").reindex(mode: :async)
  130 + end
  131 + Product.search_index.refresh
  132 + assert_search "product", ["Product A"]
74 133 end
75 134  
76 135 def test_relation_queue
77   - skip "Not available yet"
  136 + reindex_queue = Product.searchkick_index.reindex_queue
  137 + reindex_queue.clear
  138 +
  139 + store_names ["Product A"]
  140 + store_names ["Product B", "Product C"], reindex: false
  141 +
  142 + Product.where(name: "Product B").reindex(mode: :queue)
  143 + Product.search_index.refresh
  144 + assert_search "product", ["Product A"]
  145 +
  146 + perform_enqueued_jobs do
  147 + Searchkick::ProcessQueueJob.perform_now(class_name: "Product")
  148 + end
  149 + Product.search_index.refresh
  150 + assert_search "product", ["Product A", "Product B"]
  151 + end
  152 +
  153 + def test_relation_index
  154 + store_names ["Product A"]
  155 + store_names ["Product B", "Product C"], reindex: false
  156 + Product.search_index.reindex(Product.where(name: "Product B"), refresh: true)
  157 + assert_search "product", ["Product A", "Product B"]
78 158 end
79 159  
80 160 def test_full_async
81 161 store_names ["Product A"], reindex: false
82   - reindex = Product.reindex(async: true)
83   - assert_search "product", [], conversions: false
  162 + reindex = nil
  163 + perform_enqueued_jobs do
  164 + reindex = Product.reindex(async: true)
  165 + assert_search "product", [], conversions: false
  166 + end
84 167  
85 168 index = Searchkick::Index.new(reindex[:index_name])
86 169 index.refresh
... ... @@ -94,7 +177,11 @@ class ReindexTest &lt; Minitest::Test
94 177  
95 178 def test_full_async_should_index
96 179 store_names ["Product A", "Product B", "DO NOT INDEX"], reindex: false
97   - reindex = Product.reindex(async: true)
  180 +
  181 + reindex = nil
  182 + perform_enqueued_jobs do
  183 + reindex = Product.reindex(async: true)
  184 + end
98 185  
99 186 index = Searchkick::Index.new(reindex[:index_name])
100 187 index.refresh
... ... @@ -102,6 +189,8 @@ class ReindexTest &lt; Minitest::Test
102 189 end
103 190  
104 191 def test_full_async_wait
  192 + skip "Need to fix for test adapter"
  193 +
105 194 store_names ["Product A"], reindex: false
106 195  
107 196 capture_io do
... ... @@ -113,8 +202,12 @@ class ReindexTest &lt; Minitest::Test
113 202  
114 203 def test_full_async_non_integer_pk
115 204 Sku.create(id: SecureRandom.hex, name: "Test")
116   - reindex = Sku.reindex(async: true)
117   - assert_search "sku", [], conversions: false
  205 +
  206 + reindex = nil
  207 + perform_enqueued_jobs do
  208 + reindex = Sku.reindex(async: true)
  209 + assert_search "sku", [], conversions: false
  210 + end
118 211  
119 212 index = Searchkick::Index.new(reindex[:index_name])
120 213 index.refresh
... ... @@ -135,7 +228,14 @@ class ReindexTest &lt; Minitest::Test
135 228 end
136 229  
137 230 def test_full_resume
138   - assert Product.reindex(resume: true)
  231 + if mongoid?
  232 + error = assert_raises(Searchkick::Error) do
  233 + Product.reindex(resume: true)
  234 + end
  235 + assert_equal "Resume not supported for Mongoid", error.message
  236 + else
  237 + assert Product.reindex(resume: true)
  238 + end
139 239 end
140 240  
141 241 def test_full_refresh
... ... @@ -144,10 +244,10 @@ class ReindexTest &lt; Minitest::Test
144 244  
145 245 def test_full_partial_async
146 246 store_names ["Product A"]
147   - # warn for now
148   - assert_warns "unsupported keywords: :async" do
  247 + error = assert_raises(ArgumentError) do
149 248 Product.reindex(:search_name, async: true)
150 249 end
  250 + assert_match "unsupported keywords: :async", error.message
151 251 end
152 252  
153 253 def test_callbacks_false
... ... @@ -166,8 +266,6 @@ class ReindexTest &lt; Minitest::Test
166 266 end
167 267  
168 268 def test_callbacks_queue
169   - skip unless defined?(ActiveJob) && defined?(Redis)
170   -
171 269 # TODO figure out which earlier test leaves records in index
172 270 Product.reindex
173 271  
... ... @@ -181,7 +279,9 @@ class ReindexTest &lt; Minitest::Test
181 279 assert_search "product", [], load: false, conversions: false
182 280 assert_equal 2, reindex_queue.length
183 281  
184   - Searchkick::ProcessQueueJob.perform_later(class_name: "Product")
  282 + perform_enqueued_jobs do
  283 + Searchkick::ProcessQueueJob.perform_now(class_name: "Product")
  284 + end
185 285 Product.searchkick_index.refresh
186 286 assert_search "product", ["Product A", "Product B"], load: false
187 287 assert_equal 0, reindex_queue.length
... ... @@ -194,13 +294,22 @@ class ReindexTest &lt; Minitest::Test
194 294 assert_search "product", ["Product A", "Product B"], load: false
195 295 assert_equal 2, reindex_queue.length
196 296  
197   - Searchkick::ProcessQueueJob.perform_later(class_name: "Product")
  297 + perform_enqueued_jobs do
  298 + Searchkick::ProcessQueueJob.perform_now(class_name: "Product")
  299 + end
198 300 Product.searchkick_index.refresh
199 301 assert_search "product", ["Product A", "Product C"], load: false
200 302 assert_equal 0, reindex_queue.length
201 303  
202 304 # ensure no error with empty queue
203   - Searchkick::ProcessQueueJob.perform_later(class_name: "Product")
  305 + Searchkick::ProcessQueueJob.perform_now(class_name: "Product")
  306 + end
  307 +
  308 + def test_object_index
  309 + error = assert_raises(Searchkick::Error) do
  310 + Product.search_index.reindex(Object.new)
  311 + end
  312 + assert_equal "Cannot reindex object", error.message
204 313 end
205 314  
206 315 def test_transaction
... ...
test/reindex_v2_job_test.rb
  Diff comments   View file @684eecf
... ... @@ -5,7 +5,7 @@ class ReindexV2JobTest &lt; Minitest::Test
5 5 product = Searchkick.callbacks(false) { Product.create!(name: "Boom") }
6 6 Product.search_index.refresh
7 7 assert_search "*", []
8   - Searchkick::ReindexV2Job.perform_later("Product", product.id.to_s)
  8 + Searchkick::ReindexV2Job.perform_now("Product", product.id.to_s)
9 9 Product.search_index.refresh
10 10 assert_search "*", ["Boom"]
11 11 end
... ... @@ -15,7 +15,7 @@ class ReindexV2JobTest &lt; Minitest::Test
15 15 Product.reindex
16 16 assert_search "*", ["Boom"]
17 17 Searchkick.callbacks(false) { product.destroy }
18   - Searchkick::ReindexV2Job.perform_later("Product", product.id.to_s)
  18 + Searchkick::ReindexV2Job.perform_now("Product", product.id.to_s)
19 19 Product.search_index.refresh
20 20 assert_search "*", []
21 21 end
... ...
test/results_test.rb
  Diff comments   View file @684eecf
1 1 require_relative "test_helper"
2 2  
3 3 class ResultsTest < Minitest::Test
  4 + def test_array_methods
  5 + store_names ["Product A", "Product B"]
  6 + products = Product.search("product")
  7 + assert_equal 2, products.count
  8 + assert_equal 2, products.size
  9 + assert_equal 2, products.length
  10 + assert products.any?
  11 + refute products.empty?
  12 + refute products.none?
  13 + refute products.one?
  14 + assert products.many?
  15 + assert_kind_of Product, products[0]
  16 + assert_kind_of Array, products.slice(0, 1)
  17 + assert_kind_of Array, products.to_ary
  18 + end
  19 +
  20 + def test_with_hit
  21 + store_names ["Product A", "Product B"]
  22 + results = Product.search("product")
  23 + assert_kind_of Enumerator, results.with_hit
  24 + assert_equal 2, results.with_hit.to_a.size
  25 + count = 0
  26 + results.with_hit do |product, hit|
  27 + assert_kind_of Product, product
  28 + assert_kind_of Hash, hit
  29 + count += 1
  30 + end
  31 + assert_equal 2, count
  32 + end
  33 +
4 34 def test_with_score
5 35 store_names ["Product A", "Product B"]
6 36 results = Product.search("product")
... ... @@ -15,13 +45,13 @@ class ResultsTest &lt; Minitest::Test
15 45 assert_equal 2, count
16 46 end
17 47  
18   - def test_model_name_with_klass
  48 + def test_model_name_with_model
19 49 store_names ["Product A", "Product B"]
20 50 results = Product.search("product")
21 51 assert_equal "Product", results.model_name.human
22 52 end
23 53  
24   - def test_model_name_without_klass
  54 + def test_model_name_without_model
25 55 store_names ["Product A", "Product B"]
26 56 results = Searchkick.search("product")
27 57 assert_equal "Result", results.model_name.human
... ...
test/routing_test.rb
  Diff comments   View file @684eecf
... ... @@ -2,15 +2,12 @@ require_relative &quot;test_helper&quot;
2 2  
3 3 class RoutingTest < Minitest::Test
4 4 def test_query
5   - query = Store.search("Dollar Tree", routing: "Dollar Tree", execute: false)
  5 + query = Store.search("Dollar Tree", routing: "Dollar Tree")
6 6 assert_equal query.params[:routing], "Dollar Tree"
7 7 end
8 8  
9 9 def test_mappings
10 10 mappings = Store.searchkick_index.index_options[:mappings]
11   - if Searchkick.server_below?("7.0.0")
12   - mappings = mappings[:store]
13   - end
14 11 assert_equal mappings[:_routing], required: true
15 12 end
16 13  
... ...
test/search_test.rb
  Diff comments   View file @684eecf
... ... @@ -2,19 +2,10 @@ require_relative &quot;test_helper&quot;
2 2  
3 3 class SearchTest < Minitest::Test
4 4 def test_search_relation
5   - _, stderr = capture_io { Product.search("*") }
6   - assert_equal "", stderr
7   - _, stderr = capture_io { Product.all.search("*") }
8   - assert_match "WARNING", stderr
9   - end
10   -
11   - def test_search_relation_default_scope
12   - Band.reindex
13   -
14   - _, stderr = capture_io { Band.search("*") }
15   - assert_equal "", stderr
16   - _, stderr = capture_io { Band.all.search("*") }
17   - assert_match "WARNING", stderr
  5 + error = assert_raises(Searchkick::Error) do
  6 + Product.all.search("*")
  7 + end
  8 + assert_equal "search must be called on model, not relation", error.message
18 9 end
19 10  
20 11 def test_body
... ... @@ -55,24 +46,32 @@ class SearchTest &lt; Minitest::Test
55 46 def test_bad_mapping
56 47 Product.searchkick_index.delete
57 48 store_names ["Product A"]
58   - error = assert_raises(Searchkick::InvalidQueryError) { Product.search "test" }
  49 + error = assert_raises(Searchkick::InvalidQueryError) { Product.search("test").to_a }
59 50 assert_equal "Bad mapping - run Product.reindex", error.message
60 51 ensure
61 52 Product.reindex
62 53 end
63 54  
64 55 def test_missing_index
65   - assert_raises(Searchkick::MissingIndexError) { Product.search("test", index_name: "not_found") }
  56 + assert_raises(Searchkick::MissingIndexError) { Product.search("test", index_name: "not_found").to_a }
66 57 end
67 58  
68 59 def test_unsupported_version
69   - raises_exception = ->(_) { raise Elasticsearch::Transport::Transport::Error, "[500] No query registered for [multi_match]" }
  60 + skip if Searchkick.opensearch?
  61 +
  62 + raises_exception = lambda do |*|
  63 + if defined?(Elastic::Transport)
  64 + raise Elastic::Transport::Transport::Error, "[500] No query registered for [multi_match]"
  65 + else
  66 + raise Elasticsearch::Transport::Transport::Error, "[500] No query registered for [multi_match]"
  67 + end
  68 + end
70 69 Searchkick.client.stub :search, raises_exception do
71   - assert_raises(Searchkick::UnsupportedVersionError) { Product.search("test") }
  70 + assert_raises(Searchkick::UnsupportedVersionError) { Product.search("test").to_a }
72 71 end
73 72 end
74 73  
75 74 def test_invalid_body
76   - assert_raises(Searchkick::InvalidQueryError) { Product.search(body: {boom: true}) }
  75 + assert_raises(Searchkick::InvalidQueryError) { Product.search(body: {boom: true}).to_a }
77 76 end
78 77 end
... ...
test/should_index_test.rb
  Diff comments   View file @684eecf
... ... @@ -39,7 +39,6 @@ class ShouldIndexTest &lt; Minitest::Test
39 39 end
40 40 Product.where(id: product.id).reindex
41 41 Product.searchkick_index.refresh
42   - # TODO fix in Searchkick 5
43   - # assert_search "index", []
  42 + assert_search "index", []
44 43 end
45 44 end
... ...
test/support/activerecord.rb
  Diff comments   View file @684eecf
... ... @@ -63,6 +63,13 @@ ActiveRecord::Schema.define do
63 63  
64 64 create_table :bands do |t|
65 65 t.string :name
  66 + t.boolean :active
  67 + end
  68 +
  69 + create_table :artists do |t|
  70 + t.string :name
  71 + t.boolean :active
  72 + t.boolean :should_index
66 73 end
67 74 end
68 75  
... ... @@ -96,5 +103,9 @@ class Song &lt; ActiveRecord::Base
96 103 end
97 104  
98 105 class Band < ActiveRecord::Base
99   - default_scope { where(name: "Test") }
  106 + default_scope { where(active: true).order(:name) }
  107 +end
  108 +
  109 +class Artist < ActiveRecord::Base
  110 + default_scope { where(active: true).order(:name) }
100 111 end
... ...
test/support/cequel.rb
View file @d6abba8
... ... @@ -1,95 +0,0 @@
1   -cequel =
2   - Cequel.connect(
3   - host: "127.0.0.1",
4   - port: 9042,
5   - keyspace: "searchkick_test",
6   - default_consistency: :all
7   - )
8   -cequel.logger = $logger
9   -cequel.schema.drop! if cequel.schema.exists?
10   -cequel.schema.create!
11   -Cequel::Record.connection = cequel
12   -
13   -class Product
14   - include Cequel::Record
15   -
16   - key :id, :uuid, auto: true
17   - column :name, :text, index: true
18   - column :store_id, :int
19   - column :in_stock, :boolean
20   - column :backordered, :boolean
21   - column :orders_count, :int
22   - column :found_rate, :decimal
23   - column :price, :int
24   - column :color, :text
25   - column :latitude, :decimal
26   - column :longitude, :decimal
27   - column :description, :text
28   - column :alt_description, :text
29   - column :created_at, :timestamp
30   -end
31   -
32   -class Store
33   - include Cequel::Record
34   -
35   - key :id, :timeuuid, auto: true
36   - column :name, :text
37   -
38   - # has issue with id serialization
39   - def search_data
40   - {
41   - name: name
42   - }
43   - end
44   -end
45   -
46   -class Region
47   - include Cequel::Record
48   -
49   - key :id, :timeuuid, auto: true
50   - column :name, :text
51   - column :text, :text
52   -end
53   -
54   -class Speaker
55   - include Cequel::Record
56   -
57   - key :id, :timeuuid, auto: true
58   - column :name, :text
59   -end
60   -
61   -class Animal
62   - include Cequel::Record
63   -
64   - key :id, :timeuuid, auto: true
65   - column :name, :text
66   -
67   - # has issue with id serialization
68   - def search_data
69   - {
70   - name: name
71   - }
72   - end
73   -end
74   -
75   -class Dog < Animal
76   -end
77   -
78   -class Cat < Animal
79   -end
80   -
81   -class Sku
82   - include Cequel::Record
83   -
84   - key :id, :uuid
85   - column :name, :text
86   -end
87   -
88   -class Song
89   - include Cequel::Record
90   -
91   - key :id, :timeuuid, auto: true
92   - column :name, :text
93   -end
94   -
95   -[Product, Store, Region, Speaker, Animal, Sku, Song].each(&:synchronize_schema)
test/support/helpers.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,95 @@
  1 +class Minitest::Test
  2 + include ActiveJob::TestHelper
  3 +
  4 + def setup
  5 + Product.destroy_all
  6 + Store.destroy_all
  7 + Animal.destroy_all
  8 + Speaker.destroy_all
  9 + end
  10 +
  11 + protected
  12 +
  13 + def store(documents, model = default_model, reindex: true)
  14 + if reindex
  15 + documents.shuffle.each do |document|
  16 + model.create!(document)
  17 + end
  18 + model.searchkick_index.refresh
  19 + else
  20 + Searchkick.callbacks(false) do
  21 + documents.shuffle.each do |document|
  22 + model.create!(document)
  23 + end
  24 + end
  25 + end
  26 + end
  27 +
  28 + def store_names(names, model = default_model, reindex: true)
  29 + store names.map { |name| {name: name} }, model, reindex: reindex
  30 + end
  31 +
  32 + # no order
  33 + def assert_search(term, expected, options = {}, model = default_model)
  34 + assert_equal expected.sort, model.search(term, **options).map(&:name).sort
  35 + end
  36 +
  37 + def assert_order(term, expected, options = {}, model = default_model)
  38 + assert_equal expected, model.search(term, **options).map(&:name)
  39 + end
  40 +
  41 + def assert_equal_scores(term, options = {}, model = default_model)
  42 + assert_equal 1, model.search(term, **options).hits.map { |a| a["_score"] }.uniq.size
  43 + end
  44 +
  45 + def assert_first(term, expected, options = {}, model = default_model)
  46 + assert_equal expected, model.search(term, **options).map(&:name).first
  47 + end
  48 +
  49 + def assert_misspellings(term, expected, misspellings = {}, model = default_model)
  50 + options = {
  51 + fields: [:name, :color],
  52 + misspellings: misspellings
  53 + }
  54 + assert_search(term, expected, options, model)
  55 + end
  56 +
  57 + def assert_warns(message)
  58 + _, stderr = capture_io do
  59 + yield
  60 + end
  61 + assert_match "[searchkick] WARNING: #{message}", stderr
  62 + end
  63 +
  64 + def with_options(options, model = default_model)
  65 + previous_options = model.searchkick_options.dup
  66 + begin
  67 + model.searchkick_options.merge!(options)
  68 + model.reindex
  69 + yield
  70 + ensure
  71 + model.searchkick_options.clear
  72 + model.searchkick_options.merge!(previous_options)
  73 + end
  74 + end
  75 +
  76 + def activerecord?
  77 + defined?(ActiveRecord)
  78 + end
  79 +
  80 + def mongoid?
  81 + defined?(Mongoid)
  82 + end
  83 +
  84 + def default_model
  85 + Product
  86 + end
  87 +
  88 + def ci?
  89 + ENV["CI"]
  90 + end
  91 +
  92 + # for Active Job helpers
  93 + def tagged_logger
  94 + end
  95 +end
... ...
test/support/mongoid.rb
  Diff comments   View file @684eecf
... ... @@ -2,7 +2,7 @@ Mongoid.logger = $logger
2 2 Mongo::Logger.logger = $logger if defined?(Mongo::Logger)
3 3  
4 4 Mongoid.configure do |config|
5   - config.connect_to "searchkick_test"
  5 + config.connect_to "searchkick_test", server_selection_timeout: 1
6 6 end
7 7  
8 8 class Product
... ... @@ -71,6 +71,17 @@ class Band
71 71 include Mongoid::Document
72 72  
73 73 field :name
  74 + field :active, type: Mongoid::Boolean
74 75  
75   - default_scope -> { where(name: "Test") }
  76 + default_scope -> { where(active: true).order(name: 1) }
  77 +end
  78 +
  79 +class Artist
  80 + include Mongoid::Document
  81 +
  82 + field :name
  83 + field :active, type: Mongoid::Boolean
  84 + field :should_index, type: Mongoid::Boolean
  85 +
  86 + default_scope -> { where(active: true).order(name: 1) }
76 87 end
... ...
test/support/nobrainer.rb
View file @d6abba8
... ... @@ -1,73 +0,0 @@
1   -NoBrainer.configure do |config|
2   - config.app_name = :searchkick
3   - config.environment = :test
4   -end
5   -
6   -class Product
7   - include NoBrainer::Document
8   - include NoBrainer::Document::Timestamps
9   -
10   - field :id, type: Object
11   - field :name, type: Text
12   - field :in_stock, type: Boolean
13   - field :backordered, type: Boolean
14   - field :orders_count, type: Integer
15   - field :found_rate
16   - field :price, type: Integer
17   - field :color, type: String
18   - field :latitude
19   - field :longitude
20   - field :description, type: String
21   - field :alt_description, type: String
22   -
23   - belongs_to :store, validates: false
24   -end
25   -
26   -class Store
27   - include NoBrainer::Document
28   -
29   - field :id, type: Object
30   - field :name, type: String
31   -end
32   -
33   -class Region
34   - include NoBrainer::Document
35   -
36   - field :id, type: Object
37   - field :name, type: String
38   - field :text, type: Text
39   -end
40   -
41   -class Speaker
42   - include NoBrainer::Document
43   -
44   - field :id, type: Object
45   - field :name, type: String
46   -end
47   -
48   -class Animal
49   - include NoBrainer::Document
50   -
51   - field :id, type: Object
52   - field :name, type: String
53   -end
54   -
55   -class Dog < Animal
56   -end
57   -
58   -class Cat < Animal
59   -end
60   -
61   -class Sku
62   - include NoBrainer::Document
63   -
64   - field :id, type: String
65   - field :name, type: String
66   -end
67   -
68   -class Song
69   - include NoBrainer::Document
70   -
71   - field :id, type: Object
72   - field :name, type: String
73   -end
test/synonyms_test.rb
  Diff comments   View file @684eecf
... ... @@ -31,14 +31,6 @@ class SynonymsTest &lt; Minitest::Test
31 31 assert_search "clorox", ["Clorox Bleach", "Kroger Bleach"], fields: [{name: :word_start}]
32 32 end
33 33  
34   - def test_wordnet
35   - # requires WordNet
36   - skip unless ENV["WORDNET"]
37   -
38   - store_names ["Creature", "Beast", "Dragon"], Animal
39   - assert_search "animal", ["Creature", "Beast"], {}, Animal
40   - end
41   -
42 34 def test_directional
43 35 store_names ["Lightbulb", "Green Onions", "Led"]
44 36 assert_search "led", ["Lightbulb", "Led"]
... ...
test/test_helper.rb
  Diff comments   View file @684eecf
... ... @@ -2,7 +2,6 @@ require &quot;bundler/setup&quot;
2 2 Bundler.require(:default)
3 3 require "minitest/autorun"
4 4 require "minitest/pride"
5   -require "active_support/core_ext" if defined?(NoBrainer)
6 5 require "active_support/notifications"
7 6  
8 7 ENV["RACK_ENV"] = "test"
... ... @@ -14,21 +13,17 @@ end
14 13  
15 14 $logger = ActiveSupport::Logger.new(ENV["VERBOSE"] ? STDOUT : nil)
16 15  
17   -if defined?(OpenSearch)
18   - Searchkick.client = OpenSearch::Client.new
19   -end
20   -
21   -if Searchkick.client.transport.respond_to?(:transport)
22   - Searchkick.client.transport.transport.logger = $logger
23   -else
24   - Searchkick.client.transport.logger = $logger
  16 +if ENV["LOG_TRANSPORT"]
  17 + transport_logger = ActiveSupport::Logger.new(STDOUT)
  18 + if Searchkick.client.transport.respond_to?(:transport)
  19 + Searchkick.client.transport.transport.logger = transport_logger
  20 + else
  21 + Searchkick.client.transport.logger = transport_logger
  22 + end
25 23 end
26 24 Searchkick.search_timeout = 5
27 25 Searchkick.index_suffix = ENV["TEST_ENV_NUMBER"] # for parallel tests
28 26  
29   -# add to elasticsearch-7.0.0/config/
30   -Searchkick.wordnet_path = "wn_s.pl" if ENV["WORDNET"]
31   -
32 27 puts "Running against #{Searchkick.opensearch? ? "OpenSearch" : "Elasticsearch"} #{Searchkick.server_version}"
33 28  
34 29 Searchkick.redis =
... ... @@ -41,16 +36,12 @@ Searchkick.redis =
41 36 I18n.config.enforce_available_locales = true
42 37  
43 38 ActiveJob::Base.logger = $logger
44   -ActiveJob::Base.queue_adapter = :inline
  39 +ActiveJob::Base.queue_adapter = :test
45 40  
46 41 ActiveSupport::LogSubscriber.logger = ActiveSupport::Logger.new(STDOUT) if ENV["NOTIFICATIONS"]
47 42  
48 43 if defined?(Mongoid)
49 44 require_relative "support/mongoid"
50   -elsif defined?(NoBrainer)
51   - require_relative "support/nobrainer"
52   -elsif defined?(Cequel)
53   - require_relative "support/cequel"
54 45 else
55 46 require_relative "support/activerecord"
56 47 end
... ... @@ -70,96 +61,4 @@ Animal.reindex
70 61 Speaker.reindex
71 62 Region.reindex
72 63  
73   -class Minitest::Test
74   - def setup
75   - Product.destroy_all
76   - Store.destroy_all
77   - Animal.destroy_all
78   - Speaker.destroy_all
79   - end
80   -
81   - protected
82   -
83   - def store(documents, klass = default_model, reindex: true)
84   - if reindex
85   - documents.shuffle.each do |document|
86   - klass.create!(document)
87   - end
88   - klass.searchkick_index.refresh
89   - else
90   - Searchkick.callbacks(false) do
91   - documents.shuffle.each do |document|
92   - klass.create!(document)
93   - end
94   - end
95   - end
96   - end
97   -
98   - def store_names(names, klass = default_model, reindex: true)
99   - store names.map { |name| {name: name} }, klass, reindex: reindex
100   - end
101   -
102   - # no order
103   - def assert_search(term, expected, options = {}, klass = default_model)
104   - assert_equal expected.sort, klass.search(term, **options).map(&:name).sort
105   - end
106   -
107   - def assert_order(term, expected, options = {}, klass = default_model)
108   - assert_equal expected, klass.search(term, **options).map(&:name)
109   - end
110   -
111   - def assert_equal_scores(term, options = {}, klass = default_model)
112   - assert_equal 1, klass.search(term, **options).hits.map { |a| a["_score"] }.uniq.size
113   - end
114   -
115   - def assert_first(term, expected, options = {}, klass = default_model)
116   - assert_equal expected, klass.search(term, **options).map(&:name).first
117   - end
118   -
119   - def assert_misspellings(term, expected, misspellings = {}, klass = default_model)
120   - options = {
121   - fields: [:name, :color],
122   - misspellings: misspellings
123   - }
124   - assert_search(term, expected, options, klass)
125   - end
126   -
127   - def assert_warns(message)
128   - _, stderr = capture_io do
129   - yield
130   - end
131   - assert_match "[searchkick] WARNING: #{message}", stderr
132   - end
133   -
134   - def with_options(options, klass = default_model)
135   - previous_options = klass.searchkick_options.dup
136   - begin
137   - klass.searchkick_options.merge!(options)
138   - klass.reindex
139   - yield
140   - ensure
141   - klass.searchkick_options.clear
142   - klass.searchkick_options.merge!(previous_options)
143   - end
144   - end
145   -
146   - def activerecord?
147   - defined?(ActiveRecord)
148   - end
149   -
150   - def nobrainer?
151   - defined?(NoBrainer)
152   - end
153   -
154   - def cequel?
155   - defined?(Cequel)
156   - end
157   -
158   - def default_model
159   - Product
160   - end
161   -
162   - def ci?
163   - ENV["CI"]
164   - end
165   -end
  64 +require_relative "support/helpers"
... ...
test/unscope_test.rb 0 โ†’ 100644
  Diff comments   View file @684eecf
... ... @@ -0,0 +1,40 @@
  1 +require_relative "test_helper"
  2 +
  3 +class UnscopeTest < ActiveSupport::TestCase
  4 + def setup
  5 + @@once ||= Artist.reindex
  6 +
  7 + Artist.unscoped.destroy_all
  8 + end
  9 +
  10 + def test_reindex
  11 + create_records
  12 +
  13 + Artist.reindex
  14 + assert_search "*", ["Test", "Test 2"]
  15 + assert_search "*", ["Test", "Test 2"], {load: false}
  16 + end
  17 +
  18 + def test_relation_async
  19 + create_records
  20 +
  21 + perform_enqueued_jobs do
  22 + Artist.unscoped.reindex(mode: :async)
  23 + end
  24 +
  25 + Artist.search_index.refresh
  26 + assert_search "*", ["Test", "Test 2"]
  27 + end
  28 +
  29 + def create_records
  30 + store [
  31 + {name: "Test", active: true, should_index: true},
  32 + {name: "Test 2", active: false, should_index: true},
  33 + {name: "Test 3", active: false, should_index: false},
  34 + ], reindex: false
  35 + end
  36 +
  37 + def default_model
  38 + Artist
  39 + end
  40 +end
... ...
test/where_test.rb
  Diff comments   View file @684eecf
... ... @@ -15,14 +15,11 @@ class WhereTest &lt; Minitest::Test
15 15 assert_search "product", ["Product A"], where: {user_ids: 2}
16 16 assert_search "product", ["Product A", "Product C"], where: {user_ids: [2, 3]}
17 17  
18   - # due to precision
19   - unless cequel?
20   - # date
21   - assert_search "product", ["Product A"], where: {created_at: {gt: now - 1}}
22   - assert_search "product", ["Product A", "Product B"], where: {created_at: {gte: now - 1}}
23   - assert_search "product", ["Product D"], where: {created_at: {lt: now - 2}}
24   - assert_search "product", ["Product C", "Product D"], where: {created_at: {lte: now - 2}}
25   - end
  18 + # date
  19 + assert_search "product", ["Product A"], where: {created_at: {gt: now - 1}}
  20 + assert_search "product", ["Product A", "Product B"], where: {created_at: {gte: now - 1}}
  21 + assert_search "product", ["Product D"], where: {created_at: {lt: now - 2}}
  22 + assert_search "product", ["Product C", "Product D"], where: {created_at: {lte: now - 2}}
26 23  
27 24 # integer
28 25 assert_search "product", ["Product A"], where: {store_id: {lt: 2}}
... ... @@ -86,14 +83,14 @@ class WhereTest &lt; Minitest::Test
86 83 end
87 84  
88 85 def test_where_string_operators
89   - error = assert_raises RuntimeError do
  86 + error = assert_raises(ArgumentError) do
90 87 assert_search "product", [], where: {store_id: {"lt" => 2}}
91 88 end
92 89 assert_includes error.message, "Unknown where operator"
93 90 end
94 91  
95 92 def test_unknown_operator
96   - error = assert_raises RuntimeError do
  93 + error = assert_raises(ArgumentError) do
97 94 assert_search "product", [], where: {store_id: {contains: "%2%"}}
98 95 end
99 96 assert_includes error.message, "Unknown where operator"
... ... @@ -116,42 +113,31 @@ class WhereTest &lt; Minitest::Test
116 113  
117 114 def test_regexp_not_anchored
118 115 store_names ["abcde"]
119   - # regular expressions are always anchored right now
120   - # TODO change in future release
121   - assert_warns "Regular expressions are always anchored in Elasticsearch" do
122   - assert_search "*", [], where: {name: /abcd/}
123   - end
124   - assert_warns "Regular expressions are always anchored in Elasticsearch" do
125   - assert_search "*", [], where: {name: /bcde/}
126   - end
127   - assert_warns "Regular expressions are always anchored in Elasticsearch" do
128   - assert_search "*", ["abcde"], where: {name: /abcde/}
129   - end
130   - assert_warns "Regular expressions are always anchored in Elasticsearch" do
131   - assert_search "*", ["abcde"], where: {name: /.*bcd.*/}
132   - end
  116 + assert_search "*", ["abcde"], where: {name: /abcd/}
  117 + assert_search "*", ["abcde"], where: {name: /bcde/}
  118 + assert_search "*", ["abcde"], where: {name: /abcde/}
  119 + assert_search "*", ["abcde"], where: {name: /.*bcd.*/}
133 120 end
134 121  
135 122 def test_regexp_anchored
136 123 store_names ["abcde"]
137 124 assert_search "*", ["abcde"], where: {name: /\Aabcde\z/}
138   - assert_warns "Regular expressions are always anchored in Elasticsearch" do
139   - assert_search "*", [], where: {name: /\Abcd/}
140   - end
141   - assert_warns "Regular expressions are always anchored in Elasticsearch" do
142   - assert_search "*", [], where: {name: /bcd\z/}
143   - end
  125 + assert_search "*", ["abcde"], where: {name: /\Aabc/}
  126 + assert_search "*", ["abcde"], where: {name: /cde\z/}
  127 + assert_search "*", [], where: {name: /\Abcd/}
  128 + assert_search "*", [], where: {name: /bcd\z/}
144 129 end
145 130  
146 131 def test_regexp_case
147 132 store_names ["abcde"]
148 133 assert_search "*", [], where: {name: /\AABCDE\z/}
149   - unless case_insensitive_supported?
150   - assert_warns "Case-insensitive flag does not work with Elasticsearch < 7.10" do
  134 + if case_insensitive_supported?
  135 + assert_search "*", ["abcde"], where: {name: /\AABCDE\z/i}
  136 + else
  137 + error = assert_raises(ArgumentError) do
151 138 assert_search "*", [], where: {name: /\AABCDE\z/i}
152 139 end
153   - else
154   - assert_search "*", ["abcde"], where: {name: /\AABCDE\z/i}
  140 + assert_equal "Case-insensitive flag does not work with Elasticsearch < 7.10", error.message
155 141 end
156 142 end
157 143  
... ... @@ -354,9 +340,10 @@ class WhereTest &lt; Minitest::Test
354 340 _, stderr = capture_io do
355 341 assert_search "san", ["San Francisco", "San Antonio"], where: {location: {geo_polygon: {points: polygon}}}
356 342 end
357   - unless Searchkick.server_below?("7.12.0")
358   - assert_match "Deprecated field [geo_polygon] used", stderr
359   - end
  343 + # only warns for elasticsearch gem < 8
  344 + # unless Searchkick.server_below?("7.12.0")
  345 + # assert_match "Deprecated field [geo_polygon] used", stderr
  346 + # end
360 347  
361 348 # Field [location] is not of type [geo_shape] but of type [geo_point] error for previous versions
362 349 unless Searchkick.server_below?("7.14.0")
... ...