20

Did the Monero development find LMDB architect and developer Howard Chu because of its interest in LMDB? Or was LMDB introduced to the Monero development team because of Howard Chu's interest in Monero?

Why did Monero choose LMDB over alternatives such as LevelDB RocksDB, HyperLevelDB, Berkeley DB and InfluxDB?

What are the pros and cons of the various database types Monero considered using after the decision was made to stop storing the blockchain in RAM?

25

I wasn't around when the decision to use LMDB was made, but I can present some obvious reasons why such a decision makes sense:

1) at the time they made their decision, there were only 2 embedded key value stores out there with ACID transaction support: BerkeleyDB and LMDB. (RocksDB has recently added it. I haven't tested it to see if it's actually usable. Given that LevelDB has been plagued with bugs even in their snapshot isolation, I wouldn't expect much from RocksDB.)

2) for something with real monetary value, you really don't want a DB that can irrecoverably lose your data. This is something all the LevelDB-using cryptos still don't seem to understand. If it doesn't do real ACID then it's a non-starter.

3) between the only viable choices of BerkeleyDB and LMDB, LMDB is obviously smaller/faster/more reliable. Faced with a lack of 32bit LMDB support, the Monero team adopted both.

4) As key/value stores go, and ignoring ACID for the moment, LMDB offers a number of features that other k/v stores lack, which boosts its efficiency in a cryptocurrency use case. E.g. support for native integer keys, keys with multiple values, fast bulk loading/importing, etc.

Here are some IRC transcripts from the time the choice of DB was being made. It gives you insight into the range of solutions and what factors were considered:

[2014-06-02T16:40:55+0200] <@fluffyp> tacotime: have you ever played with
  hamsterdb?
[2014-06-02T16:41:19+0200] <@tacotime> nope
[2014-06-02T16:41:46+0200] <@tacotime> but i don't wanna mess with any java
  db systems
[2014-06-02T16:42:02+0200] <@fluffyp> tacotime: not java
[2014-06-02T16:42:06+0200] <@fluffyp> c/c++ embedded db
[2014-06-02T16:42:09+0200] <@tacotime> ah okay
[2014-06-02T16:42:13+0200] <@fluffyp> looks like it outperforms leveldb
[2014-06-02T16:42:19+0200] <@fluffyp> has an open source community edition
[2014-06-02T16:42:22+0200] <@tacotime> i wouldn't be surprised
[2014-06-02T16:42:28+0200] <@tacotime> how mature is the software though?
[2014-06-02T16:42:49+0200] <@tacotime> something slow and bugfree is
  better than something fast and buggy when it comes to financial software
[2014-06-02T16:43:09+0200] <@fluffyp> this has been around for years
[2014-06-02T16:43:12+0200] <@fluffyp> since 2010 or so
[2014-06-02T16:44:45+0200] <@tacotime> yeah
[2014-06-02T16:44:54+0200] <@tacotime> a lot of people are using mongodb
  now i feel like
[2014-06-02T16:45:03+0200] <@tacotime> i'm kinda curious about tokumx too
[2014-06-02T16:45:14+0200] <@fluffyp> there's no embeddable version of mongo
  is there?
[2014-06-02T16:45:25+0200] <@tacotime> i don't think so :/
[2014-06-02T16:45:32+0200] <@tacotime> i'm going through the btcdb stuff right
  now and seeing how a db is implemented too
[2014-06-02T16:45:51+0200] <@tacotime> https://github.com/conformal/btcdb
[2014-06-02T16:46:01+0200] <@tacotime> that's a good example of how to do it
  the quick and easy way in go
[2014-06-02T16:47:12+0200] <@tacotime> if you want a fast db you still need to
  keep a lot of the most recent blocks in memory, though
[2014-06-02T16:47:14+0200] <@fluffyp> tacotime: I'm talking about the embedded
  blockchain keyvalue store - one of the devs I was talking to suggested
  hamsterdb over leveldb because it's faster whilst remaining significantly
  robust
[2014-06-02T16:47:26+0200] <@tacotime> and have to be careful with atomicity
[2014-06-02T16:47:28+0200] <@tacotime> ah, ok
[2014-06-02T16:47:37+0200] <@fluffyp> sorry should've clarified that :-P
[2014-06-02T16:47:49+0200] <@tacotime> I think we're talking about the
  same thing
[2014-06-02T16:47:56+0200] <@tacotime> https://github.com/conformal/btcdb/blob/
  master/ldb/leveldb.go
[2014-06-02T16:48:19+0200] <@tacotime> writing an easy keystore like that
  isn't too cluttered and timeconsuming
[2014-06-02T16:48:52+0200] <@tacotime> but regardless of whether you use
  hamsterdb or leveldb, it'll get slow fast if you aren't on an ssd for monero
[2014-06-02T16:48:58+0200] <@fluffyp> yep

And following up a week later:

[2014-06-09T15:55:19+0200] <pen> What is the selected embedded database for
  Monero? I pointed crypto_zoidberg to Sophia. Just wondered about Monero's
  dev choice
[2014-06-09T15:55:44+0200] <@fluffypony> pen: we're still deciding
[2014-06-09T15:57:09+0200] <@fluffypony> if we abstract it properly then embedded
  database choice becomes an academic exercise (i.e.. which one is best for our
  workloads)
[2014-06-09T15:58:25+0200] <pen> leveldb may be a safe choice. However, leveldb
  is not the fast one out there. Let see if BBR dev choice is. I expect that it
  would be LMDB or Sophia
[2014-06-09T15:58:25+0200] <@fluffypony> so many options between kyotocabinet /
  sparkey / tokutek / hamsterdb / hyperdex / symas lmdb / sophia / and tons of
  leveldb forks
[2014-06-09T15:59:40+0200] <pen> hyperdex is not an embeded one. Actually
  hyperdex devs improve leveldb in term of write scalability in a fork
[2014-06-09T15:59:57+0200] <pen> sophia and lmdb are not leveldb forks
[2014-06-09T16:00:17+0200] <pen> sophia is the db that powers Tarantool
[2014-06-09T16:00:25+0200] <othe> mongodb has the best name, i vote for that
[2014-06-09T16:00:44+0200] <@fluffypony> pen: I wasn't implying those are all
  leveldb forks :)
[2014-06-09T16:01:44+0200] <@fluffypony> hyperdex has an embedded db -
  HyperLevelDB (unsurprisingly a leveldb fork)
[2014-06-09T16:02:26+0200] <pen> @othe mongodb is so hyped. And it is not
  designed for an embedded one
[2014-06-09T16:02:34+0200] <@fluffypony> pen: he was just trolling you

My first contact with the Monero project came two months later:

[2014-08-07T11:34:25] fluffypony: heya
[2014-08-07T11:34:32] fluffypony: plz ping me when you're around
[2014-08-07T11:34:48] hyc: what's up?
[2014-08-07T11:34:54] fluffypony: lol that was fast
[2014-08-07T11:35:00] fluffypony: hokay so
[2014-08-07T11:35:04] fluffypony: LMDB
[2014-08-07T11:35:28] fluffypony: how often do you push from the main openldap
  branch to the lmdb gitorious branch?
[2014-08-07T11:35:47] fluffypony: ie. should I be pulling it from the openldb mdb
  branch instead of there?
[2014-08-07T11:36:31] hyc: we normally push simultaneously
[2014-08-07T11:36:55] fluffypony: ok great
[2014-08-07T11:37:06] fluffypony: I'm one of the core team members on Monero, a
  cryptocurrency (not based on Bitcoin)
[2014-08-07T11:37:16] hyc: you can pull from whichever you like. some folks
  complained that they had to pull the entire openldap git repo just to get LMDB,
  so we set up gitorious
[2014-08-07T11:37:19] fluffypony: and we're in the process of moving from a flat
  file in RAM for the blockchain to an embedded DB
[2014-08-07T11:37:27] fluffypony: and I'm DEAD keen on lmdb
[2014-08-07T11:37:35] fluffypony: as it appears it will be most performant for
  our workload
[2014-08-07T11:37:38] hyc: sounds good
[2014-08-07T11:37:48] fluffypony: and it will provide scalability for larger
  instances
[2014-08-07T11:39:06] hyc: yep
[2014-08-07T11:39:29] hyc: and unlike other embedded DBs, it will never corrupt
  or lose your data...
[2014-08-07T11:39:38] fluffypony: yeah and that's the appeal
[2014-08-07T11:39:49] fluffypony: in a distributed consensus system you really
  want to have solid data on-disk
[2014-08-07T11:39:55] fluffypony: because peers are reliant on that
[2014-08-07T11:40:00] hyc: absolutely
  • 2
    Howard, a small clarification: Tarantool doesn't use Sophia any more, we forked it half a year ago and rewrote almost entirely. – Kostja Sep 27 '16 at 21:00
15

LMDB is a fast database that is a lot less brittle than the typical-within-cryptocurrencies leveldb. The only drawback which called for a Berkeley DB fallback was its patchy 32 bit support, but that has now been fixed. Moreover, hyc (LMDB author) is working with the Monero team to ensure LMDB is used the best way. I believe it was Monero that found hyc first, but he'll correct me if I'm wrong :)

This post from hyc is interesting as an overview of LMDB compared with othe databases: https://news.ycombinator.com/item?id=8734356

  • 4
    Heres another good explanation by the LMDB author comparing LMDB to RocksDB (LevelDB): reddit.com/r/Monero/comments/4rdnrg/lmdb_vs_rocksdb/d51egcs – user20 Aug 2 '16 at 13:59
  • 4
    The Monero team found LMDB long before I got involved. They asked me a few usage questions early on. It was only after they had a working backend, and just wanted some help with the 32bit port, that I got directly involved. – hyc Aug 5 '16 at 0:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.