EctoCursorBasedStream

Cursor-based streaming of Ecto records, that does not require database transaction.

Gives you a cursor_based_stream/2 function that mimics Ecto.Repo.stream/2 interface.

Advantages in comparison to the standard Ecto.Repo.stream/2:

• streaming can be stopped and continued at any point (by passing option after_cursor: ...),
• works with tables that have milions of records.

Only limitation is that you have to supply a cursor column or columns (by passing option cursor_field: ..., defaults to :id). Such a column(s):

• must have unique values,
• should have a database index. (So that sorting by it, and returning a number of rows larger than x is a performant operation.)

Usage

1. Add ecto_cursor_based_stream to your list of dependencies in mix.exs:

def deps do
  [
    {:ecto_cursor_based_stream, "~> 1.2.0"}
  ]
end

2. Add use EctoCursorBasedStream to the module that uses Ecto.Repo:

defmodule MyRepo do
  use Ecto.Repo
  use EctoCursorBasedStream
end

3. Stream the rows using cursor_based_stream/2:

Post
|> MyRepo.cursor_based_stream()
|> Stream.each(...)
|> Stream.run()

Useful links

• Documentation
• Examples

Contributing

Running tests

Run the following after cloning the repo:

mix deps.get
docker-compose up -d
mix test