Monero Stack Exchange is a question and answer site for developers and users of the secure, private and untraceable cryptocurrency Monero. It only takes a minute to sign up.
Sign up to join this community
Did the Monero development find LMDB architect and developer Howard Chu because of its interest in LMDB? Or was LMDB introduced to the Monero development team because of Howard Chu's interest in Monero?
Why did Monero choose LMDB over alternatives such as LevelDB RocksDB, HyperLevelDB, Berkeley DB and InfluxDB?
What are the pros and cons of the various database types Monero considered using after the decision was made to stop storing the blockchain in RAM?
I wasn't around when the decision to use LMDB was made, but I can present some obvious reasons why such a decision makes sense:
1) at the time they made their decision, there were only 2 embedded key value stores out there with ACID transaction support: BerkeleyDB and LMDB. (RocksDB has recently added it. I haven't tested it to see if it's actually usable. Given that LevelDB has been plagued with bugs even in their snapshot isolation, I wouldn't expect much from RocksDB.)
2) for something with real monetary value, you really don't want a DB that can irrecoverably lose your data. This is something all the LevelDB-using cryptos still don't seem to understand. If it doesn't do real ACID then it's a non-starter.
3) between the only viable choices of BerkeleyDB and LMDB, LMDB is obviously smaller/faster/more reliable. Faced with a lack of 32bit LMDB support, the Monero team adopted both.
4) As key/value stores go, and ignoring ACID for the moment, LMDB offers a number of features that other k/v stores lack, which boosts its efficiency in a cryptocurrency use case. E.g. support for native integer keys, keys with multiple values, fast bulk loading/importing, etc.
Here are some IRC transcripts from the time the choice of DB was being made. It gives you insight into the range of solutions and what factors were considered:
[2014-06-02T16:40:55+0200] <@fluffyp> tacotime: have you ever played with
hamsterdb?
[2014-06-02T16:41:19+0200] <@tacotime> nope
[2014-06-02T16:41:46+0200] <@tacotime> but i don't wanna mess with any java
db systems
[2014-06-02T16:42:02+0200] <@fluffyp> tacotime: not java
[2014-06-02T16:42:06+0200] <@fluffyp> c/c++ embedded db
[2014-06-02T16:42:09+0200] <@tacotime> ah okay
[2014-06-02T16:42:13+0200] <@fluffyp> looks like it outperforms leveldb
[2014-06-02T16:42:19+0200] <@fluffyp> has an open source community edition
[2014-06-02T16:42:22+0200] <@tacotime> i wouldn't be surprised
[2014-06-02T16:42:28+0200] <@tacotime> how mature is the software though?
[2014-06-02T16:42:49+0200] <@tacotime> something slow and bugfree is
better than something fast and buggy when it comes to financial software
[2014-06-02T16:43:09+0200] <@fluffyp> this has been around for years
[2014-06-02T16:43:12+0200] <@fluffyp> since 2010 or so
[2014-06-02T16:44:45+0200] <@tacotime> yeah
[2014-06-02T16:44:54+0200] <@tacotime> a lot of people are using mongodb
now i feel like
[2014-06-02T16:45:03+0200] <@tacotime> i'm kinda curious about tokumx too
[2014-06-02T16:45:14+0200] <@fluffyp> there's no embeddable version of mongo
is there?
[2014-06-02T16:45:25+0200] <@tacotime> i don't think so :/
[2014-06-02T16:45:32+0200] <@tacotime> i'm going through the btcdb stuff right
now and seeing how a db is implemented too
[2014-06-02T16:45:51+0200] <@tacotime> https://github.com/conformal/btcdb
[2014-06-02T16:46:01+0200] <@tacotime> that's a good example of how to do it
the quick and easy way in go
[2014-06-02T16:47:12+0200] <@tacotime> if you want a fast db you still need to
keep a lot of the most recent blocks in memory, though
[2014-06-02T16:47:14+0200] <@fluffyp> tacotime: I'm talking about the embedded
blockchain keyvalue store - one of the devs I was talking to suggested
hamsterdb over leveldb because it's faster whilst remaining significantly
robust
[2014-06-02T16:47:26+0200] <@tacotime> and have to be careful with atomicity
[2014-06-02T16:47:28+0200] <@tacotime> ah, ok
[2014-06-02T16:47:37+0200] <@fluffyp> sorry should've clarified that :-P
[2014-06-02T16:47:49+0200] <@tacotime> I think we're talking about the
same thing
[2014-06-02T16:47:56+0200] <@tacotime> https://github.com/conformal/btcdb/blob/
master/ldb/leveldb.go
[2014-06-02T16:48:19+0200] <@tacotime> writing an easy keystore like that
isn't too cluttered and timeconsuming
[2014-06-02T16:48:52+0200] <@tacotime> but regardless of whether you use
hamsterdb or leveldb, it'll get slow fast if you aren't on an ssd for monero
[2014-06-02T16:48:58+0200] <@fluffyp> yep
And following up a week later:
[2014-06-09T15:55:19+0200] <pen> What is the selected embedded database for
Monero? I pointed crypto_zoidberg to Sophia. Just wondered about Monero's
dev choice
[2014-06-09T15:55:44+0200] <@fluffypony> pen: we're still deciding
[2014-06-09T15:57:09+0200] <@fluffypony> if we abstract it properly then embedded
database choice becomes an academic exercise (i.e.. which one is best for our
workloads)
[2014-06-09T15:58:25+0200] <pen> leveldb may be a safe choice. However, leveldb
is not the fast one out there. Let see if BBR dev choice is. I expect that it
would be LMDB or Sophia
[2014-06-09T15:58:25+0200] <@fluffypony> so many options between kyotocabinet /
sparkey / tokutek / hamsterdb / hyperdex / symas lmdb / sophia / and tons of
leveldb forks
[2014-06-09T15:59:40+0200] <pen> hyperdex is not an embeded one. Actually
hyperdex devs improve leveldb in term of write scalability in a fork
[2014-06-09T15:59:57+0200] <pen> sophia and lmdb are not leveldb forks
[2014-06-09T16:00:17+0200] <pen> sophia is the db that powers Tarantool
[2014-06-09T16:00:25+0200] <othe> mongodb has the best name, i vote for that
[2014-06-09T16:00:44+0200] <@fluffypony> pen: I wasn't implying those are all
leveldb forks :)
[2014-06-09T16:01:44+0200] <@fluffypony> hyperdex has an embedded db -
HyperLevelDB (unsurprisingly a leveldb fork)
[2014-06-09T16:02:26+0200] <pen> @othe mongodb is so hyped. And it is not
designed for an embedded one
[2014-06-09T16:02:34+0200] <@fluffypony> pen: he was just trolling you
My first contact with the Monero project came two months later:
[2014-08-07T11:34:25] fluffypony: heya
[2014-08-07T11:34:32] fluffypony: plz ping me when you're around
[2014-08-07T11:34:48] hyc: what's up?
[2014-08-07T11:34:54] fluffypony: lol that was fast
[2014-08-07T11:35:00] fluffypony: hokay so
[2014-08-07T11:35:04] fluffypony: LMDB
[2014-08-07T11:35:28] fluffypony: how often do you push from the main openldap
branch to the lmdb gitorious branch?
[2014-08-07T11:35:47] fluffypony: ie. should I be pulling it from the openldb mdb
branch instead of there?
[2014-08-07T11:36:31] hyc: we normally push simultaneously
[2014-08-07T11:36:55] fluffypony: ok great
[2014-08-07T11:37:06] fluffypony: I'm one of the core team members on Monero, a
cryptocurrency (not based on Bitcoin)
[2014-08-07T11:37:16] hyc: you can pull from whichever you like. some folks
complained that they had to pull the entire openldap git repo just to get LMDB,
so we set up gitorious
[2014-08-07T11:37:19] fluffypony: and we're in the process of moving from a flat
file in RAM for the blockchain to an embedded DB
[2014-08-07T11:37:27] fluffypony: and I'm DEAD keen on lmdb
[2014-08-07T11:37:35] fluffypony: as it appears it will be most performant for
our workload
[2014-08-07T11:37:38] hyc: sounds good
[2014-08-07T11:37:48] fluffypony: and it will provide scalability for larger
instances
[2014-08-07T11:39:06] hyc: yep
[2014-08-07T11:39:29] hyc: and unlike other embedded DBs, it will never corrupt
or lose your data...
[2014-08-07T11:39:38] fluffypony: yeah and that's the appeal
[2014-08-07T11:39:49] fluffypony: in a distributed consensus system you really
want to have solid data on-disk
[2014-08-07T11:39:55] fluffypony: because peers are reliant on that
[2014-08-07T11:40:00] hyc: absolutely
LMDB is a fast database that is a lot less brittle than the typical-within-cryptocurrencies leveldb. The only drawback which called for a Berkeley DB fallback was its patchy 32 bit support, but that has now been fixed. Moreover, hyc (LMDB author) is working with the Monero team to ensure LMDB is used the best way. I believe it was Monero that found hyc first, but he'll correct me if I'm wrong :)
This post from hyc is interesting as an overview of LMDB compared with othe databases: https://news.ycombinator.com/item?id=8734356
