github / searchkick

Download as
Browse Code »

Commit fe639ff0f6b11571b549cd86ba463926bfd9133a

Authored by Andrew Kane
1 parent 18a501ce
Exists in master and in 15 other branches allow_missing, conversions_period, conversions_v2, find_in_batches, lazy_index_name, less_deps, nested, reindex_refactor, relation, relation_search, scope_attributes, search_as_you_type, synonyms_ci, test_language, wildcard

Added deep_paging option

Showing 5 changed files with 26 additions and 9 deletions   Show diff stats
CHANGELOG.md
  Diff comments   View file @fe639ff
  1 +## 4.1.2 (unreleased)
  2 +
  3 +- Added `deep_paging` option
  4 +
1 5 ## 4.1.1 (2019-11-19)
2 6  
3 7 - Added `chinese2` and `korean2` languages
... ...
README.md
  Diff comments   View file @fe639ff
... ... @@ -137,8 +137,6 @@ Limit / offset
137 137 limit: 20, offset: 40
138 138 ```
139 139  
140   -**Note:** By default, Elasticsearch [limits pagination](#deep-pagination) to the first 10,000 results for performance
141   -
142 140 Select
143 141  
144 142 ```ruby
... ... @@ -182,6 +180,8 @@ Get the full response from Elasticsearch
182 180 results.response
183 181 ```
184 182  
  183 +**Note:** By default, Elasticsearch [limits paging](#deep-paging-master) to the first 10,000 results for performance. This applies to the total count as well.
  184 +
185 185 ### Boosting
186 186  
187 187 Boost important fields
... ... @@ -1518,20 +1518,20 @@ end
1518 1518 products.clear_scroll
1519 1519 ```
1520 1520  
1521   -## Deep Pagination
  1521 +## Deep Paging [master]
1522 1522  
1523   -By default, Elasticsearch limits pagination to the first 10,000 results [for performance](https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html). We don’t recommend changing this, but if you need to, you can use:
  1523 +By default, Elasticsearch limits paging to the first 10,000 results. [Here’s why](https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html). We don’t recommend changing this, but if you really need all results, you can use:
1524 1524  
1525 1525 ```ruby
1526 1526 class Product < ApplicationRecord
1527   - searchkick settings: {index: {max_result_window: 1000000000}}
  1527 + searchkick deep_paging: true
1528 1528 end
1529 1529 ```
1530 1530  
1531   -And search with:
  1531 +If you just need an accurate total count, you can instead use:
1532 1532  
1533 1533 ```ruby
1534   -Product.search("pears", limit: 1000000000, body_options: {track_total_hits: true})
  1534 +Product.search("pears", body_options: {track_total_hits: true})
1535 1535 ```
1536 1536  
1537 1537 ## Nested Data
... ...
lib/searchkick/index_options.rb
  Diff comments   View file @fe639ff
... ... @@ -456,6 +456,13 @@ module Searchkick
456 456 mappings = mappings.symbolize_keys.deep_merge(custom_mapping.symbolize_keys)
457 457 end
458 458  
  459 + if options[:deep_paging]
  460 + if !settings.dig(:index, :max_result_window) && !settings[:"index.max_result_window"]
  461 + settings[:index] ||= {}
  462 + settings[:index][:max_result_window] = 1_000_000_000
  463 + end
  464 + end
  465 +
459 466 {
460 467 settings: settings,
461 468 mappings: mappings
... ...
lib/searchkick/model.rb
  Diff comments   View file @fe639ff
... ... @@ -3,7 +3,7 @@ module Searchkick
3 3 def searchkick(**options)
4 4 options = Searchkick.model_options.merge(options)
5 5  
6   - unknown_keywords = options.keys - [:_all, :_type, :batch_size, :callbacks, :case_sensitive, :conversions, :default_fields,
  6 + unknown_keywords = options.keys - [:_all, :_type, :batch_size, :callbacks, :case_sensitive, :conversions, :deep_paging, :default_fields,
7 7 :filterable, :geo_shape, :highlight, :ignore_above, :index_name, :index_prefix, :inheritance, :language,
8 8 :locations, :mappings, :match, :merge_mappings, :routing, :searchable, :settings, :similarity,
9 9 :special_characters, :stem, :stem_conversions, :suggest, :synonyms, :text_end,
... ...
lib/searchkick/query.rb
  Diff comments   View file @fe639ff
... ... @@ -232,7 +232,9 @@ module Searchkick
232 232  
233 233 # pagination
234 234 page = [options[:page].to_i, 1].max
235   - per_page = (options[:limit] || options[:per_page] || 10_000).to_i
  235 + # maybe use index.max_result_window in the future
  236 + default_limit = searchkick_options[:deep_paging] ? 1_000_000_000 : 10_000
  237 + per_page = (options[:limit] || options[:per_page] || default_limit).to_i
236 238 padding = [options[:padding].to_i, 0].max
237 239 offset = options[:offset] || (page - 1) * per_page + padding
238 240 scroll = options[:scroll]
... ... @@ -516,6 +518,10 @@ module Searchkick
516 518 # routing
517 519 @routing = options[:routing] if options[:routing]
518 520  
  521 + if searchkick_options[:deep_paging]
  522 + payload[:track_total_hits] = true
  523 + end
  524 +
519 525 # merge more body options
520 526 payload = payload.deep_merge(options[:body_options]) if options[:body_options]
521 527  
... ...