A clone of Twitter's Snowflake, built on riak_core
Erlang
102
7 commits
updated May 13, 2015
* riak_id
** Overview
=riak_id= is a very simple application that serves both to
duplicate functionality from Twitter's [[https://github.com/twitter/snowflake]["Snowflake"]] and to teach,
via a very simple example, how to build a [[https://github.com/basho/riak_core][riak_core]] application.
*** Design
The core functionality of =riak_id= is to provide k-sorted unique
identifiers. The structure of the identifiers is very similar to
Snowflake. They are 64-bit integers with the following layout
(from highest to lowest):
| Bits | Description |
|------+-----------------------------------------------|
| 1 | Signedness flag, always 0 |
| 41 | Unix timestamp, down to the millisecond |
| 10 | Top 10 bits of partition number (see Caveats) |
| 12 | Per-partition strictly increasing counter |
|------+-----------------------------------------------|
*** Caveats
- Decent NTP synchronization between machines is needed for
=riak_id= to work as expected.
- Request failures (e.g. from partition, timeout) should simply be
retried. You will eventually get an ID.
- Because only 10 bits are reserved for the partition number,
there is a practical ring-size limit of 1024. This should not
be a problem because the work is mostly CPU-bound.
- So that we can give reasonable guarantees of uniqueness, some
requests may have a minimum latency of 1ms (to allow the
timestamp to increment).
- It will run out of ID space within the century. Snowflake solves
this using the "Twepoch". I didn't bother with that, but it
would be a simple feature to add.
- I wrote =riak_id= to learn, I hope it will be useful to others.
** Quick Start
=riak_id= requires Erlang/OTP, [[https://github.com/basho/rebar][rebar]] and a sane GNU build system.
1) Clone the repository, copy the =rebar= script into the repository.
2) Run =make rel= from the command line.
3) =rel/riak_id/bin/riak_id console=
4) =riak_id:next_id().= from the console to generate IDs.
** Acknowledgments
Thanks to Jesse Newland, Andy Gross, Ryan Zezeski for their work on
the rebar template for riak_core apps, without which I would be
lost. Special thanks to Andy for pointing out some egregious errors
in an early version of the vnode code.
** License
Copyright 2011, Sean Cribbs. Licensed under Apache License 2.0.
Not written in Markdown, so it's shown here as plain text — view it formatted on GitHub.
4 commits
3 commits
Erlang
43.1%
Shell
38.3%
Perl
18.6%
A clone of Twitter's Snowflake, built on riak_core
Erlang
102
7 commits
updated May 13, 2015
* riak_id
** Overview
=riak_id= is a very simple application that serves both to
duplicate functionality from Twitter's [[https://github.com/twitter/snowflake]["Snowflake"]] and to teach,
via a very simple example, how to build a [[https://github.com/basho/riak_core][riak_core]] application.
*** Design
The core functionality of =riak_id= is to provide k-sorted unique
identifiers. The structure of the identifiers is very similar to
Snowflake. They are 64-bit integers with the following layout
(from highest to lowest):
| Bits | Description |
|------+-----------------------------------------------|
| 1 | Signedness flag, always 0 |
| 41 | Unix timestamp, down to the millisecond |
| 10 | Top 10 bits of partition number (see Caveats) |
| 12 | Per-partition strictly increasing counter |
|------+-----------------------------------------------|
*** Caveats
- Decent NTP synchronization between machines is needed for
=riak_id= to work as expected.
- Request failures (e.g. from partition, timeout) should simply be
retried. You will eventually get an ID.
- Because only 10 bits are reserved for the partition number,
there is a practical ring-size limit of 1024. This should not
be a problem because the work is mostly CPU-bound.
- So that we can give reasonable guarantees of uniqueness, some
requests may have a minimum latency of 1ms (to allow the
timestamp to increment).
- It will run out of ID space within the century. Snowflake solves
this using the "Twepoch". I didn't bother with that, but it
would be a simple feature to add.
- I wrote =riak_id= to learn, I hope it will be useful to others.
** Quick Start
=riak_id= requires Erlang/OTP, [[https://github.com/basho/rebar][rebar]] and a sane GNU build system.
1) Clone the repository, copy the =rebar= script into the repository.
2) Run =make rel= from the command line.
3) =rel/riak_id/bin/riak_id console=
4) =riak_id:next_id().= from the console to generate IDs.
** Acknowledgments
Thanks to Jesse Newland, Andy Gross, Ryan Zezeski for their work on
the rebar template for riak_core apps, without which I would be
lost. Special thanks to Andy for pointing out some egregious errors
in an early version of the vnode code.
** License
Copyright 2011, Sean Cribbs. Licensed under Apache License 2.0.
Not written in Markdown, so it's shown here as plain text — view it formatted on GitHub.
4 commits
3 commits
Erlang
43.1%
Shell
38.3%
Perl
18.6%