Me again. Usual doing/writing balance issues.
The biggest minor release of them all
There were some posts by @blocktrades already, but it's not enough to emphasize how much work was performed to deliver you the new MINOR version.
Versioning
Under the legacy chain, we used versions 0.x.y up to 0.22.y, where x indicated a major release requiring a HardFork, and y referred to a minor release, usually involving bug fixes, improvements, or anything else that did not require consensus changes.
With version 0.23.0, we forked from the legacy chain and began our new journey, during which we achieved a technological breakthrough with the release of version 1.24.0, codenamed Eclipse.
Currently, we are in the HardFork 27 era, and have been using version 1.27.4 for a year while working on a "few details" to improve our API's technological stack.
Let me share some statistics (source: GitHub):
| Metric | Legacy Chain | Hive |
|---|---|---|
| Versions | 0.10.0 - 0.19.0 | 1.27.4 - 1.27.5 |
| Releases | 9 major | 1 minor |
| Commits | 1412 | 1153 |
| Changed files | 312 | 1543 |
| Additions | 32217 | 153683 |
| Deletions | 12192 | 50646 |
| Contributors | 15 | 18 |
And that's just the core Hive, although most of the improvements were made to supporting software.
api.openhive.network upcoming upgrade
I’ve promised to be the last to upgrade, ensuring my node serves as a reliable fallback in case any unexpected issues arise from the recent updates.
So far, so good; I don’t know of any concerning issues that could/should stop me from upgrading.
If you do, speak now, as I’m about to shut down the old stack within a week from now.
If you're unsure whether your application is ready for the new stack, test it using another API endpoint, such as api.hive.blog, which is already on the new stack.
The old way
Based on a hived with all API plugins, including the most resource-hungry: account history,
and hivemind as a python application that syncs via hived’s API and uses a PostgreSQL backend for database storage. Some API operators had separate broadcaster nodes (lightweight consensus node just for broadcast and/or get_block requests), haproxy to handle the load among different instances, jussi for routing requests, redis for caching, nginx for SSL termination and rate limiting, etc.
So it’s very easy and fast to set up if you are me and have all that in your muscle memory after deploying those environments hundreds of times, otherwise it could be a pain in the back(end).
Just for the record, the current old-style hived account history node requires 1.6 TB of storage, of which the block_log occupies 464 GB. The snapshot takes up 1.1 TB, which compresses down to 721 GB. The Hivemind database storage occupies 700 GB, and the shared_memory.bin file needs roughly 22 GB.
All combined takes roughly 2.3TB
The new way
The new way of running API nodes is just letting our hived do what it does best - handle blockchain stuff, and let all the other work which is storing, organizing, and presenting data be handled by software that’s made for it and has been doing it great for decades - PostgreSQL.
We are no longer using the RocksDB account history plugin (well, it’s still very good if you just need to track a few accounts, like exchanges do), instead we use an HAF-based account history API server. Hivemind is also HAF-based so it no longer bothers hived for data. This architecture allows for more fancy specialized APIs, such as a balance tracker and an HAF-based block explorer.
Unfortunately, all that makes it more complex and it could be a challenge to set it up, and to solve that problem, @blocktrades came up with the haf_api_node one-stop repo, which lets you just get what you need and set up with minimal effort and user input.
It still requires time and resources, but it is server resources not human resources.
Out of the box, assuming typical setup (with ZFS compression), datadir takes roughly 2.6TB
Useful stuff https://gtg.openhive.network/
Hivemind dump
After discontinuing v1.27.4, I will no longer update the hivemind dump for non-HAF setups.
The latest dump is hivemind-20240416-v1.26.3-3ba5511b.dump.tar.bz2
Despite the name that contains v1.26.3, it’s compatible with the current hived version. Hivemind 3ba5511b was the latest version to support non-HAF sync.
Because with the new architecture, hivemind is no longer a separate entity, it is not possible to provide just a hivemind snapshot.
It’s the biggest disadvantage of using a new stack, but the other features outweigh this inconvenience.
Account History API Node snapshot
The latest snapshot for the Account History Node is ah-20240417-v1.27.4.tar.bz2.
Obviously, you can still run hived with the RocksDB account history plugin with 1.27.5.
Exchanges (a.k.a. General-purpose) node
The account history plugin is still recommended for exchanges due to its minimal overhead and no requirement for HAF (K.I.S.S.(tm)).
I will continue to serve general-purpose snapshots that include account history for a set of commonly used exchange accounts (and my own, because it’s cool).
TODO
More posts about what’s going on, mirrornet updates, fast node deploy cheat sheet.
Footnotes
Don’t forget to like and subscribe upvote and follow. Or what’s more important - vote for your favorite witnesses (up to 30).