Between 5:00 pm PST on Wednesday, March 8, and 1:24 pm PST on Friday, March 10, approximately 9,400 contributors experienced a delay in receiving their HONEY tokens for the reward period of February 27 through March 5. The initial delay was caused by a failed audit of the HONEY calculations and was prolonged by degraded performance of Hivemapper’s usual Remote Procedure Call (RPC) provider.
Hivemapper operates HONEY rewards on a weekly cycle and usually begins issuing rewards shortly after 5 pm PST on Wednesdays. This week, the timeline was as follows:
Wednesday 3/8, 5:00pm PST: Rewards Pool calculations began
Wednesday 3/8, 10:12pm PST: Community is notified about delays in reward payments
Thursday 3/9, 10:22 am PST: Rewards Pool calculations completed
Thursday 3/9, 4:18 pm PST: Payment completed for Foundation Rewards
Thursday, 3/9, 9:14 pm PST: Payment completed for Map Coverage Rewards
Friday, 3/10, 1:24 pm PST: Payment completed for QA Rewards
Failed Audit of HONEY Calculations
Each week extensive calculations are run and audited to ensure the proper minting of HONEY. The audit of this week’s calculations surfaced a bug in the calculation of Map Consumption Rewards
Although they do not make up a large percentage of rewards today, Map Consumption Rewards are a critical component of the long-term rewards structure for the Hivemapper Network. As customers burn HONEY tokens to consume map imagery, these tokens are redistributed to the contributors who helped map areas where data was consumed.
Instead of skipping over this reward factor and delaying only Map Consumption Rewards, the decision was made to debug the calculations with the expectation it would be an easy fix. In retrospect, this was a mistake that caused all rewards, not just some rewards, to be delayed.
The bug was more complicated than originally expected, especially as a result of the increased scale of HONEY transactions in recent weeks. This is a classic example of “sunk cost fallacy” where the team took too long to make the decision to abandon the Map Consumption Rewards after putting in the time to debug them.
To avoid this mistake going forward, the team is implementing three changes to the rewards calculation and auditing process:
When any reward factors are introduced or modified, Hivemapper will soft launch changes and decouple them from the execution of the current rewards calculations in production.
Other rewards will not be delayed to the extent an isolated element of the calculations fails an audit.
Infrastructure improvements will be made to increase speed and reliability as HONEY transactions scale.
Degraded Performance of Solana Transactions
After completing calculations and final audits on Thursday morning, we noticed transactions were processing much slower than usual due to heavy Solana traffic and degraded performance of our usual Remote Procedure Call (RPC) service.
RPCs allow communications with servers remotely and provide the ability to execute programs in a separate location. In blockchain terms, an RPC allows access to a server node on the specified network and allows you to communicate and interact with that blockchain.
In prior weeks, we’ve been able to process 500 to 700 transactions per second. This throughput has historically been sufficient for the regular weekly rewards cycle.
However, at the lowest point this week, we were processing just 200 transactions per hour. This translates to approximately 11,000 times slower than usual and, at this pace, would have required thousands of hours to process all transactions. To mitigate this significant slowdown, we moved some transactions to a different RPC service this week.
To de-risk this problem going forward, the team is implementing two changes:
Switch RPC providers to Helius
, who is developing a custom solution to support the Hivemapper Network’s large transaction volume securely and reliably.
Have backup RPC options available in the event of unusually slow processing.
Other Enhancements to the HONEY Rewards Process
We hear the community’s calls for increased visibility and timeliness of HONEY rewards each week. As such, the team is expediting an initiative to show “Estimated HONEY” in order to ease anxiety of not knowing where an individual’s rewards stand at a given point in time.
The following is our plan for the Estimated HONEY initiative:
Continuously run calculations to estimate the amount of HONEY an individual’s contributions will earn for a given reward period.
Run simulations to understand the level of confidence we can assign to these estimates prior to making them available to contributors.
Surface “Estimated HONEY” rewards in the section currently labeled “Pending” of the HONEY tab in the Hivemapper mobile application. These estimates could potentially be displayed elsewhere in the future, such as the Network Explorer.
As we build confidence in the estimates, we will research the possibility of prepaying a portion of Estimated HONEY rewards throughout the week instead of rewarding a lump sum after the reward period closes.
Furthermore, the team will improve the mobile app’s notification system to keep contributors informed about HONEY rewards throughout each week.
One more thing…
Did you know we’re hiring
? Put your career on the map by joining the Hivemapper team and solve interesting problems like building a robust HONEY rewards system. We’re breaking ground in the crypto and global mapping industries. Come join us and help make high-quality map data more accessible so that we can collectively understand the world better.