Changelog History
Page 1
-
v5.0.0 Changes
July 20, 2020Major Features and Breaking Changes
๐ Thinking Sphinx v5.0 has one significant change - explicit callbacks - plus drops support for old versions of Rails/Ruby/Sphinx, and adds a few other smaller improvements.
Explicit Callbacks
Previous versions of Thinking Sphinx automatically added callbacks to all ActiveRecord models, for the purpose of persisting changes back to Sphinx (whether that be inserts, updates, or deletions). And while the actual overhead for non-indexed models wasn't super slow, it's still far from ideal.
So now, you need to add callbacks yourself, to just the models you're indexing.
With SQL-backed models (defined using
:with => :active_record
), you'll very likely want to add one of the two following lines inside your model:class Article \< ApplicationRecord# If you're not using delta indices:ThinkingSphinx::Callbacks.append(self, :behaviours =\> [:sql])# If you \*are\* using delta indices:ThinkingSphinx::Callbacks.append(self, :behaviours =\> [:sql, :deltas])end
If you're using real-time indices, you very likely already have callbacks defined in your models, but you can replace them with the new calls:
class Article \< ApplicationRecord# Instead of this...after\_save ThinkingSphinx::RealTime.callback\_for(:article)# use this...ThinkingSphinx::Callbacks.append(self, :behaviours =\> [:real\_time])end
For associated models which still fire real-time callbacks, you can use the
:path
option with the same call:class Comment \< ApplicationRecordbelongs\_to :articleThinkingSphinx::Callbacks.append self,:behaviours =\> [:real\_time],:path=\> [:article]end
And if you're using a custom block with your old real-time callback, you can pass that same block to the new approach as well:
class Article \< ApplicationRecordThinkingSphinx::Callbacks.append(self, :behaviours =\> [:real\_time]) do |instance| # returning an array of instances to index. You could add# custom logic here if you don't want indexing to happen# in some cases.endend
๐ At this point in time, the older callback style for real-time indices will continue to work, but it's still recommended to update your code to the new style instead.
On the off chance you are using SQL-backed indices and you have
attribute_updates
enabled inconfig/thinking_sphinx.yml
, you'll want to specify that in your:behaviours
option:ThinkingSphinx::Callbacks.append(self, :behaviours =\> [:sql, :updates])
Sphinx 2.2.11 or newer is required
๐ Sphinx 2.1 is no longer supported - and ideally, it's best to upgrade any 2.2.x release to 2.2.11.
Sphinx 3.x releases are supported, but there are known issues with indexing SQL-backed indices on a PostgreSQL database (real-time indices are fine though).
As part of this change, Sphinx's docinfo setting is no longer configured, so the
skip_docinfo
setting inconfig/thinking_sphinx.yml
can be removed.When it comes to Manticore as a drop-in replacement for Sphinx, we're testing against the latest 2.x and 3.x releases, which are currently 2.8.2 and 3.4.2 respectively.
๐ Ruby 2.4 or newer is required
๐ Versions of Ruby less than 2.3 are no longer supported, sorry. We're currently testing against 2.4 through to 2.7.
๐ Rails 4.2 or newer is required
๐ It's been a long time coming, but Rails 3.2 (and 4.0 and 4.1) are no longer supported. The current supported versions are 4.2 through to 6.0 (and 6.1 will likely work as well, once it's released).
Other changes to behaviour
- โ Remove internal uses of
send
, replaced withpublic_send
as that's available in all supported Ruby versions. - Custom index_set_class implementations can now expect the
:instances
option to be set alongside:classes
, which is useful in cases to limit the indices returned if you're splitting index data for given classes/models into shards. (Introduced in PR #1171 after discussions with @lunaru in #1166.) - Deletion statements are simplified by avoiding the need to calculate document keys/offsets (@njakobsen via #1134).
- Real-time data is deleted before replacing it, to avoid duplicate data when offsets change (@njakobsen via #1134).
- Use
reference_name
as per customindex_set_class
definitions. Previously, the class method was called onThinkingSphinx::IndexSet
even if a custom subclass was configured. (As per discussions with @kalsan in #1172.) - 0๏ธโฃ Fields and attributes can be overriden - whichever's defined last with a given name is the definition that's used. This is an edge case, but useful if you want to override any of the default fields/indices. (Requested by @kalsan in #1172.)
๐ Bug fixes
None.
- โ Remove internal uses of
-
v4.4.1 Changes
August 23, 2019โฌ๏ธ Upgrading
No breaking or major changes.
๐ Changes to behaviour
- ๐ Automatically remove
app/indices
from Zeitwerk's autoload paths in Rails 6.0 onwards (if using Zeitwerk as the autoloader).
- ๐ Automatically remove
-
v4.4.0 Changes
August 21, 2019โฌ๏ธ Upgrading
No breaking or major changes.
๐ New features
- ๐ Confirmed Rails 6.0 support.
- โ Added ability to have custom real-time index processors (which handles all indices) and populators (which handles a particular index). These are available to get/set via
ThinkingSphinx::RealTime.processor
andThinkingSphinx::RealTime.populator
.
๐จ The processor should accept
call
with two arguments: an array of index objects, and a block to invoke after each index is processed. Here is a simple example for parallel processing of indices:# Add the 'parallel' gem to your Gemfile.ThinkingSphinx::RealTime.processor = Proc.new do |indices, &block| Parallel.map(indices) do |index| puts "Populating index #{index.name}"ThinkingSphinx::RealTime.populator.populate index puts "Populated index #{index.name}" block.call endend
And the populator should respond to
populate
, accepting a single argument which is the index object. Here is a simple example for parallel processing.# Add the 'parallel' gem to your Gemfile.class ParallelPopulatordef self.populate(index) new(index).call enddef initialize(index) @index = index enddef callParallel.each(index.scope.find\_in\_batches) do |instances| transcriber.copy \*instances true # Don't emit any large object because results are accumulatedendActiveRecord::Base.connection.reconnect! endprivateattr\_reader :indexdef [email protected] ||= ThinkingSphinx::RealTime::Transcriber.new index endendThinkingSphinx::RealTime.populator = ParallelPopulator
๐ Instead of building your own procs/classes from scratch, you may instead wish to subclass the default classes to tweak behaviour - or at the very least, both classes are useful as reference points for your own replacements:
These changes were influenced by discussions in #1134 with @njakobsen about parallel processing of real-time indices.
๐ Changes to behaviour
- ๐ Improve failure message when tables don't exist for models associated with Sphinx indices (Kiril Mitov in #1139).
๐ Bug fixes
- 0๏ธโฃ Injected has-many/habtm collection search calls as default extensions to associations in Rails 5+, as it's a more reliable approach in Rails 6.0.0.
-
v4.3.2 Changes
August 03, 2019โฌ๏ธ Upgrading
No breaking or behaviour changes.
๐ Bug fixes
- โช Reverted loading change behaviour from v4.3.1 for Rails v5 (Eduardo J. in #1138).
-
v4.3.1 Changes
August 03, 2019โฌ๏ธ Upgrading
No breaking or behaviour changes.
๐ Bug fixes
- ๐ Fixed loading of index files to work with Rails 6 and Zeitwerk (#1137).
-
v4.3.0 Changes
May 18, 2019โฌ๏ธ Upgrading
No breaking or major changes.
๐ New features
- โ Allow overriding of Sphinx's running state by setting
skip_running_check
to true/false inconfig/thinking_sphinx.yml
for appropriate environments. This is useful when Sphinx commands are interacting with a remote Sphinx daemon. As per discussions in #1131. - Allow skipping of directory creation by setting
skip_directory_creation
to true/false inconfig/thinking_sphinx.yml
for appropriate environments. As per discussions in #1131.
๐ Bug fixes
- ๐ Use ActiveSupport's lock monitor where possible (Rails 5.1.5 onwards) to avoid database deadlocks. Essential investigation by Jonathan del Strother (#1132).
- ๐ Allow facet searching on distributed indices (#1135).
- โ Allow overriding of Sphinx's running state by setting
-
v4.2.0 Changes
March 09, 2019โฌ๏ธ Upgrading
No breaking or major changes.
๐ New features
- Allow changing the default encoding for MySQL database connections from
utf8
to something else via themysql_encoding
setting inconfig/thinking_sphinx.yml
. In the next significant release, the default will change toutf8mb4
(which is supported in MySQL 5.5.3 and newer). - โ Added Rails 6.0 and Manticore 2.8 to the test matrix.
๐ Changes to behaviour
- โ Use Arel's SQL literals for generated order clauses, to avoid warnings from Rails 6.
๐ Bug fixes
- ๐ Fix usage of alternative primary keys in update and deletion callbacks and attribute access.
- Ensure
respond_to?
takes Sphinx scopes into account (Jonathan del Strother in #1124). - โ Add
:excerpts
as a known option for search requests. - ๐ Fix depolymorphed association join construction with Rails 6.0.0.beta2.
- ๐ง Reset ThinkingSphinx::Configuration's cached values when Rails reloads, to avoid holding onto stale references to ActiveRecord models (#1125).
- Don't join against associations in
sql_query
if they're only used by query-sourced properties (Hans de Graaff in #1127).
- Allow changing the default encoding for MySQL database connections from
-
v4.1.0 Changes
December 28, 2018โฌ๏ธ Upgrading
๐ No breaking or major changes, though Ruby 2.2 is now no longer officially supported - but this release will almost certainly still work on it.
๐ New features
- The
:sql
search option can now accept per-model settings with model names as keys. e.g.ThinkingSphinx.search "foo", :sql => {'Article' => {:include => :user}}
(Sergey Malykh in #1120).
๐ Changes to behaviour
- โฌ๏ธ Drop MRI 2.2 from the test matrix, and thus no longer officially supported (though the code will likely continue to work with 2.2 for a while).
- โ Added MRI 2.6, Sphinx 3.1 and Manticore 2.7 to the test matrix.
๐ Bug fixes
- 0๏ธโฃ Real-time indices now work with non-default integer primary keys (alongside UUIDs or other non-integer primary keys).
- The
-
v4.0.0 Changes
April 10, 2018Major Features and Changes
๐ Thinking Sphinx v4.0 has been in development for a little while, and includes some significant changes (as befitting a major release):
๐ Merging Indices
๐ Merging indices is now supported via the new
ts:merge
rake task. This is useful when you're using delta indices as an alternative to runningts:index
regularly to have new/changed records populated into the core indices. Merging should be reliably faster (and it avoids hitting your database to reprocess all the records).๐ Running
ts:index
every now and then to catch any records changed/modified without callbacks is probably wise (perhaps once a day compared to more frequentts:merge
calls).โ Run the daemon on a UNIX socket
If you've got Sphinx and your Rails app all on a single machine, you may want to have the daemon (
searchd
) hosting connections via a UNIX socket instead of a TCP socket. Just set thesocket
value in each appropriate environment withinconfig/thinking_sphinx.yml
(and do not setmysql41
unless you want the daemon to also be available via that TCP port).production: socket: /var/tmp/production.sphinx
๐ This feature is limited to MRI, as JRuby doesn't seem to have a way to connect to UNIX sockets for MySQL-protocol connections.
๐ ActiveRecord 5.2 Support
๐ The new release of ActiveRecord/Rails is happily supported by this release.
๐ Manticore Support
๐ The recent fork of Sphinx known as Manticore is supported, and can be used as a drop-in replacement for Sphinx. In particular, the v2.6.3 release is included in the test matrix.
๐ฅ Breaking Changes
Sphinx 2.1.2 or newer is required
๐ Sphinx 2.0 is no longer supported - make sure you're running at least 2.1.2, but 2.2.11 is recommended if possible.
๐ Ruby 2.2 or newer is required
๐ Versions of Ruby less than 2.2 are no longer supported, sorry.
โ Removed auto-typing of search filter values
If you're filtering via human-entered values (say, via request parameters), then in the past you were allowed to send those string values straight through to Sphinx.
๐ However, Sphinx now supports string filtering, so it's not possible to make assumptions about filter types. Thinking Sphinx v3.4.0 introduced automatic typing of these values, but this was an extra overhead which was far from ideal and was always flagged as temporary.
So, please cast your filter values where appropriate:
Model.search :with =\> {:foo\_id =\> params[:foo\_id]}# should become:Model.search :with =\> {:foo\_id =\> params[:foo\_id].to\_i}
Minor features
- If you want to remove the
docinfo
setting from the generated Sphinx configuration (to avoid warnings in Sphinx 2.2+), addskip_docinfo: true
to each appropriate environment inconfig/thinking_sphinx.yml
. - ๐ Sphinx 3.0 is now supported.
- You can now use relative paths in
config/thinking_sphinx.yml
, but you must also addabsolute_paths: true
to each environment for them to be converted to absolute paths for the generated configuration.
๐ Changes to behaviour
- The INDEX_FILTER environment variable is applied when running ts:index on SQL-backed indices.
- ๐ Useful error messages are now displayed if processing real-time indices is attempted when the daemon isn't running.
- ๐จ Rake task code has been refactored into separate command classes under the hood (which allows for
flying-sphinx
to override when appropriate). - Added
frozen_string_literal: true
pragma comments for safe frozen string literals. - Exceptions are logged when processing real-time indices without halting the processing.
- โก๏ธ Update polymorphic properties to support Rails 5.2.
- ๐ Allow configuration of the index guard approach (e.g.
ThinkingSphinx::Configuration.instance.guarding_strategy = ThinkingSphinx::Guard::None
). - โ Output a warning if guard files exist when calling ts:index.
- โ Delete index guard files as part of ts:rebuild and ts:clear.
๐ Bug fixes
- Don't attempt to interpret indices for models that don't have a database table.
- ๐ Handle situations where no exit code is provided for Sphinx binary calls.
- If you want to remove the
-
v3.4.2 Changes
September 29, 2017โฌ๏ธ Upgrading
๐ No breaking or major changes, just three small fixes and a couple of minor changes.
๐ Changes to behaviour
- ๐ Allow use of deletion callbacks for rollback events.
- โ Remove extra deletion code in the Populator - it's also being done by the real-time rake interface.
๐ Bug fixes
- Real-time callback syntax for namespaced models accepts a string (as was already documented).
- ๐ Fix up logged warnings (and avoiding overwriting the existing warn method).
- โ Add missing search options to known values to avoid incorrect warnings.