Kiba v3.0.0 Release Notes

Release Date: 2020-02-10 // 8 months ago
  • ๐Ÿ’ฅ BREAKING - the kiba CLI is deprecated

    ๐Ÿ—„ The kiba CLI is deprecated in favor of the more modern Kiba.parse programmatic API [#74, #81].

    ๐Ÿ‘ The programmatic API allows everything the "command" mode supported, plus much more, and actually encourage better coding practices. For instance:

    • ๐Ÿ”ง API mode allows to pass live variables (rather than just ENV configuration from command line or JSON configs from files)
    • ๐Ÿ‘ท Doing so permits to wrap resources open/close around running a job
    • โœ… API mode makes it easier to run testing on an ETL process (via minitest/rspec) directly in-process (which allows stubbing/webmock etc), rather than via a command call
    • API mode enforces use of clean modules with explicit loading, rather than polluting the top-level namespace with global methods
    • ๐Ÿ‘ท API mode allows to run jobs from Sidekiq or background job systems, from an HTTP call (if the job is fast), without necessarily waiting for a command line binary to run - this supports more dynamic interactions (e.g. a job is created in reaction to an external event received via HTTP or a websocket)

    ๐Ÿ“š A temporary kiba-legacy-cli gem is available (https://github.com/thbar/kiba-legacy-cli) to ease migration, but the recommendation is really to migrate over and use Kiba.parse directly, as described in the current documentation.

    0๏ธโƒฃ Kiba now defaults to StreamingRunner

    ๐Ÿš€ Introduced in v2.0.0 [#44] to ensure a transform could yield N rows for 1 input row, and improved in v2.5.0 [#57] to help implement "buffering transforms", the StreamingRunner is now made the default to process the jobs [#83].

    This change is expected to be backward compatible and will help with reusability & features of ETL components.

    ๐Ÿ’Ž Ruby compatibility notice

    • ๐Ÿ’Ž Kiba now officially supports MRI Ruby 2.4+ (although 2.3 will still work for now), JRuby 9.2+ or TruffleRuby.
    • ๐Ÿ‘€ You may get warnings with Ruby 2.7 and errors with Ruby 2.8+. See [#85] for status on Ruby 3 keyword arguments support.

Previous changes from v2.5.0

  • Aggregating / buffering transforms

    ๐Ÿš€ A Transform's close can now yield rows (this requires the new StreamingRunner, see v2.0.0 release notes).

    ๐Ÿ‘ This will let component implementers support new types of scenarios:

    • Batch transforms (such as the upcoming Kiba Pro ParallelTransform, or batch SQL lookups)
    • Grouping of rows (including in-memory or db-backed sort, normalisation operations, map operations)

    ๐Ÿ‘€ See #57 for more background & explanations.

    ๐Ÿ’Ž Ruby compatibility notice

    ๐Ÿ’Ž Kiba now requires MRI Ruby 2.3+, JRuby 9.1+ or TruffleRuby.

    โœ… This is done to reduce the testing burden, to encourage users to avoid EOL'ed rubies, and to let me use more recent Ruby features when relevant.

    Other tweaks

    • ๐Ÿ›  Fix incorrect error message when calling transform nil (#73 - thanks @envygeeks for the report).
    • ๐Ÿ›  Fix code & documentation links on Rubygems (#71 - thanks @janko).