Mining Guide
Want to start mining on Arweave? You've come to the right place! Get set up with this quick and easy guide, and join our awesome network of ever-growing miners.
For any questions and support queries regarding mining on Arweave, we strongly recommend that you join our Discord server as this is the hub of our mining and developer communities. Here you will find plenty of community members and Arweave team members available to help you out 🤖
Miners are responsible for their own compliance with data protection laws (such as GDPR) and other applicable laws in their jurisdiction. Data storage laws vary country to country. Failure to adhere to these laws may entail substantial legal risks for the miner. Please only participate in mining Arweave data if you have understood the legal implications of doing so and consider seeking legal advice.
Install the Miner
Download the .tar.gz archive for your OS from the releases page. Extract the contents of the archive - tar -xzf [release_file].
If your OS/platform architecture is not in the list, check the source code repository README for how to build the miner from source.
Preparation
Preparation: File Descriptors Limit
The number of available file descriptors affects the rate at which your node can process data. Most operating systems default to assigning a low limit for user processes, we recommend increasing it.
These File Descriptors Limit instructions apply to Linux. When running a VDF Server on MacOS, please refer to the VDF guide.
You can check the current limit by executing ulimit -n.
On Linux, to set a bigger global limit, open /etc/sysctl.conf and add the following line:
Execute sysctl -p to make the changes take effect.
You may also need to set a proper limit for the particular user. To set a user-level limit, open /etc/security/limits.conf and add the following line:
Open a new terminal session. To make sure the changes took effect, and the limit was increased, type ulimit -n. You can also change the limit for the current session via ulimit -n 10000000
If the above does not work, set
in both /etc/systemd/user.confand /etc/systemd/system.conf
Preparation: Configuring Large Memory Pages
Mining involves computing 1 RandomX hash and several SHA2 hashes every second for every 3.6 TB mining partition. It is not a lot, but your CPU may nevertheless become a bottleneck when you configure a lot of mining partitions. To maximize your hashing performance, consider configuring huge memory pages in your OS.
On Ubuntu, to see the current values, execute:cat /proc/meminfo | grep HugePages. To set a value, run sudo sysctl -w vm.nr_hugepages=3500. To make the configuration survive reboots, create /etc/sysctl.d/local.conf and put vm.nr_hugepages=3500 there.
The output of cat /proc/meminfo | grep HugePages should then look like this:
AnonHugePages: 0 kB
ShmemHugePages: 0 kB HugePages_Total: 3500 HugePages_Free: 3500 HugePages_Rsvd: 0 HugePages_Surp: 0
If it does not or if there is a "erl_drv_rwlock_destroy" error on startup, reboot the machine.
Finally, tell the miner it can use large pages by specifying enable randomx_large_pageson startup (you can find a complete startup example further in the guide).
Preparation: Data Directory
Create a directory somewhere in your system. We will refer to it as [data_dir] throughout this guide. We recommend having at least 200 GB available on the corresponding disk, although it is possible to configure the node for less space. For mining, you obviously need a lot more space, but the mining data should be stored on separate drives mounted in or symlinked to the folders inside [data_dir].More about it later in this guide.
Preparation: Mining Key
In order to produce and sign a block your mining key needs to be present on your machine. If you want to create a new wallet, run ./bin/create-wallet. The file is then created in [data_dir]/wallets/. Make sure you never share it with anyone! If you want to use an existing wallet, place it under the aforementioned path. Note: when using coordinated mining, the wallet only needs to be present on the exit node.
Preparation: Packing Format
Before you can configure your storage you'll have to decide on a packing format. The legacy packing format, spora_2_6, is still supported, but for new packs we recommend using the new composite format. The composite format includes a packing difficulty option that allows miners to spend more time while packing data in returns for lower disk read rates while mining. When you see composite.1 that's short-hand for "composite packing format with a packing difficulty of 1". This table summarizes the differences:
| Packing Format | Time to pack (benchmarked to spora_2_6) | Disk read rate per partition when mining against a full replica |
|---|---|---|
| 1x | 200 MiB/s |
| 1x | 50 MiB/s |
| 2x | 25 MiB/s |
| 3x | 16.6667 MiB/s |
| 4x | 12.5 MiB/s |
... | ... | ... |
| 32x | 1.5625 MiB/s |
If we assume that a good quality enterprised hard disk drive can sustain 200 MiB/s read rate, then with spora_2_6 you could only store a single 4TB partition per 4TB HDD. However with composite.1 you could conceivably store and mine 4x 4TB partitions on a single 16TB HDD. Or, if you find that your configuration can't sustain a 200 MiB/s read rate for each drive, you could pack to a higher difficulty level to lower the required read rate.
Note: The effective hashrate for a full replica packed to any of the supported packing formats is the same. A miner who has packed a full replica to spora_2_6 or composite.1 or composite.32 can expect to find the same number of blocks on average, but with the higher difficulty miner reading fewer chunks from their storage per second. This allows the miner to use larger hard drives in their setup, without increasing the necessary bandwidth between disk and CPU.
Also note: When mining, all storage modules within the same replica must be packed to the same packing format and difficulty level. For example, a single miner will not be able to build a solution involving chunks from storage_module_1_addr.1 and storage_module_2_addr.2 even if the packing address is the same.
The packing format you select will influence what hardware configuration provides the best return. For more information on mining hardware see the Mining Hardware Guide.
Preparation: Storage Setup
The Arweave dataset is logically partitioned into collections of 3.6 TB "mining partitions". You will store some or all of those mining partitions on your miner in "storage modules". Storage modules can be any size (smaller or larger than the default 3.6TB mining partition size), but many miners opt to align their storage modules with the Arweave partitions.
To setup your storage modules, the first step is to create a folder inside [data_dir]/storage_modules/ for each of the storage modules that you intend to mine with. Storage module folder names should use the following pattern: storage_module[_storage_module_size]_[storage_module_index]_[packing] where packing is either {mining address}.{packing difficulty} (where packing difficulty is an integer), or unpacked. The default storage module size is 3.6TB - in that case specifying storage_module_size is optional.
For any given storage_module_size, you should allow for an additional 10% metadata overhead (such as merkle proofs). This is why, for the default 3.6 TB storage_module_size, we recommend reserving 4 TB of space. The storage modules are indexed sequentially starting from 0, with 0 being the very first 3.6 TB (or storage_module_size) worth of data stored on Arweave, and ranging up to or beyond the current Arweave dataset size (weave_size). You can choose which mining partitions you store by indicating their storage_module_index in the folder name.
For example, to set up a storage module with the very first mining partition in the weave with the default 3.6 TB storage_module_size (packed with your mining address), create the folder such as [data_dir]/storage_modules/storage_module_0_[your_mining_address].1. If you were to store 2 TB storage modules, create the folder such as: [data_dir]/storage_modules/storage_module_2000000000000_0_[your_mining_address].1.
After creating the relevant folders for your chosen partitions, mount your drives onto them. E.g.,
Make sure you replace /dev/sda with the name of your drive (lsblk), [data_dir] - with the absolute path to your data folder, and [your_mining_address] - with your mining address.
If you have a drive already mounted elsewhere, you may create a symbolic link instead:
If you have a RAID setup with a lot of space, you can create a symlink link from the [data_dir]/storage_modules folder.
A few important notes about the storage modules:
Having two or more storage modules that store the same mining partition (say, the partition at index 0 more than once) with the same mining address does not increase your mining performance. Also, it is more profitable mine a complete replica (all mining partitions) of the weave packed with a single address than mine off an equal amount of data packed with different mining addresses. Currently, we only support one mining address per node.
If you want to copy the contents of a storage module elsewhere, restart the node without the corresponding
storage_modulecommand line parameter, copy the data, and restart the node with thestorage_moduleparameter again. You can attach the copied data as a storage module to another node. Just make sure to not copy while the node is interacting with this storage module. Do NOT mine on several nodes with the same mining address simultaneously (see the warning below.)Make sure the disks with the storage modules have sufficient available space for both the data iteself and metadata (10% of the size of the data). Note that
disk_spacecommand line parameter does NOT apply to the storage modules.If you created storage modules with custom
storage_module_sizeas mentioned above, make sure to specify thestorage_module_sizein your command line invocation as follows:storage_module [storage_module_index],[storage_module_size],[your_mining_address].1The module will sync data with the weave offsets betweenstorage_module_index * storage_module_size(in bytes) and(storage_module_index + 1) * storage_module_sizeat folder[data_dir]/storage_modules/storage_module[_storage_module_size]_[storage_module_index]_[your_mining_address].1.The specified mining partition index does not have to be under the current weave size. This makes it possible to configure storage modules in advance. Once the weave data grows sufficiently large to start filling the mining partition at the specified index, the node will begin placing the new data in the already configured storage module.
It is very dangerous to have two or more nodes mine independently using the same mining address. If they find and publish blocks simultaneously, the network will slash your rewards and revoke the mining permission of the mining address! In order to have multipler nodes use the same mining address they must be configured to use coordinated mining. See the Coordinated Mining Guide for more information.
Copying Data Across Storage Modules
When a node starts, it copies (and packs, if required) the data from one storage module to another, in the case when there are two or more intersecting storage modules. For example, if you specify storage_module 11,unpacked storage_module 11,[mining_address].1 and there is some data in the "unpacked" module that is absent from the "mining address" module, the data will be packed with this mining address and stored in 11,[mining_address].1.
If you want to repack a storage module, do not rename the existing one - renaming will not cause repacking, create a new storage module instead.
Unpacked Storage Modules
If you want to sync many replicas of the weave, it makes sense to first create an "unpacked" replica. Then, packing for each mining address will be two times faster compared to repacking a replica packed with another mining address. To configure a storage module for storing unpacked data, specify "unpacked" instead of the mining address.
For example, to sync an unpacked partition 12, specify storage_module 12,unpacked on startup. As with the other storage modules, make sure the [data_dir]/storage_modules/storage_module_12_unpacked folder resides on the desired disk (if you do not create the directory in advance, the node will create it for you so the data will end up on the disk [data_dir]/storage_modulesis mounted to.) After the replica is synced, you can copy it to the other machines where its contents would be copied and packed for the storage modules you configure there.
Preparation: RAM
Minimum: 8 GB + 1 GB per mining partition
Recommended: 8 GB + 2 GB per mining partition
The node determines the amount of chunks to read in memory while mining automatically. If your node runs out of memory anyway, try setting the mining_server_chunk_cache_size_limit option in the command line (specify the number of 256 KiB to cache).
Preparation: CPU
We can broadly outline three tasks computing units solve in Arweave:
Packing and unpacking data
Executing the VDF
Calculating storage proofs
For more information on what hardware to use for your miner, please see the Mining Hardware Guide.
1. Packing
Packing mostly consists of executing RandomX instructions so the faster your CPU computes RandomX hashes, the faster you can pack. Note that packing a single 256 KiB chunk using the spora_2_6 or composite.1 format takes about 30 times longer than computing one RandomX hash. Once you have packed a dataset, you do not necessarily have to keep the powerful process around. You can control the maximum allowed packing rate with the packing_rate start command parameter.
2. VDF
The VDF controls the speed of mining with new mining "seeds" available at 1 second intervals. To keep up with the network your CPU must be able to maintain this 1 second cadence while calculating the VDF. For that the CPU needs to support hardware SHA2 acceleration. Additional cores will not improve VDF performance as VDF hash calculations are sequential and therefore limited to a single thread on a single core.
For more information on VDF, including connecting to a VDF server or running your own VDF server, see Mining VDF.
Running the Miner
Now you’re ready to start the mining process by running the following command from the Arweave directory. An example with one storage module (covering partition 0):
Replace YOUR-MINING-ADDRESS with the address of the wallet you would like to credit when you find a block!
Tip: Avoid killing the arweave process if at all possible. I.e. don't do kill -9 arweave or kill -9 beam or kill -9 erl. To stop the arweave process, use ./bin/stop and then wait for as long as you can for the node to shutdown gracefully. Sometimes if can take a while for the node to shutdown, which we realize is frustrating, but if you kill the node abruptly it can cause rocksdb corruption that can be difficult to recover from. In the worst case you may need to resync and repack a partition.
An example with several storage modules (covering partitions 21, 22, 23):
For more examples see: Mining Examples
In order to protect your machine from material that may be illegal in your country, you should use a content policy when mining Arweave. Content policies can be generated using the Shepherd tool. Shepherd allows you to create your own content policies for the content that you would like to store on your Arweave node, abiding by your moral and legal requirements.
In order to help you get started quickly, @ArweaveTeam provides an NSFW content filter which you can load by adding the following to your Arweave start command:
transaction_blacklist_url http://shepherd-v.com/nsfw.txt
If you would like to see a log of your miner’s activity, you can run ./bin/logs -f in the Arweave directory in a different terminal. Sometimes it is helpful to look at the debug logs which are written if the node is started with the debug flag in the command line - ./bin/debug-logs -f
The mining console should eventually look like this:
Receiving Mining Rewards
When you mine a block, the console shows:
Approximately 20 minutes later, you should see
Note that occasionally your block won't be confirmed (the chain chooses a different fork).
You do not immediately receive the block reward after mining a block. There is a delay in the release of block rewards for miners by approximately thirty days or 30 * 24 * 30 blocks. Your node does not need to stay online in order to receive your reserved mining rewards. This mechanism is designed to discourage signing the same block several times and several competitive forks in general - the network detects these cases and may slash the reserved rewards and revoke the mining permission from the corresponding mining address. Also, the mechanism incentivizes miners to be aligned with the network for at least the medium-term.
To see the total number of Winston (divide by 1000_000_000_000 to get the AR value) reseved for you address, browse to https://arweave.net/wallet/[your-mining-address]/reserved_rewards_total.
Staying in Sync
Watch for the following warnings in your mining console:
If you see them shortly after joining the network, see if they disappear in a few minutes - everything might be fine then. Otherwise, it is likely your processor cannot keep up with VDF computation or there are network connection issues. While VDF execution is done by a single core/thread, the validation of the VDF checkpoints in a block header can be done in parallel (with multiple threads). To speed up VDF validation, try restarting the node with a higher value for max_vdf_validation_thread_count (e.g., the number of CPU threads - 1).
Stopping the Miner
To stop the node, run ./bin/stop or kill the OS process (kill -sigterm <pid> or pkill <name>). Sending a SIGKILL (kill -9) is not recommended.
Defragmenting Storage
Due to Arweave node specifics (storing data in the sparse files), the read throughput during mining after the initial sync might be suboptimal on some disks. In the performance reports printed in the console you can see the estimated optimal performance in MiB/s, per configured storage module. The first number estimates the optimum on a small dataset, the second - on the dataset close in size to the weave size. If the actual performance of a storage module is noticeably lower, consider running a defragmentation procedure to improve your mining performance on this module. (Re)start the miner with the following parameters (in this example, the storage module storing the partition 8 will be defragmented):
The defragmentation is performed before startup. Only chunk files larger than defragmentation_trigger_threshold bytes and those which have grown by more than 10% since the last defragmenation of this module will be updated. Note the defragmentation may take a lot of time.
Troubleshooting
Make sure your node is accessible on the Internet
An important part of the mining process is discovering blocks mined by other miners. Your node needs to be accessible from anywhere on the Internet so that your peers can connect with you and share their blocks.
To check if your node is publicly accessible, browse to http://[Your Internet IP]:1984. You can obtain your public IP here, or by running curl ifconfig.me/ip. If you specified a different port when starting the miner, replace "1984" anywhere in these instructions with your port. If you can not access the node, you need to set up TCP port forwarding for incoming HTTP requests to your Internet IP address on port 1984 to the selected port on your mining machine. For more details on how to set up port forwarding, consult your ISP or cloud provider.
If the node is not accessible on the Internet, the miner functions but is significantly less efficient.
Staying up to Date
Join our Discord server
Once you are successfully mining on the Arweave, you will need to stay up to date with new releases. Check the #announcements channel on the Arweave Miners discord server to learn about new releases. We will announce any steps you need to take to stay up to speed - particularly updates that require you to perform an action within a certain time period in order to stay in sync with the network.
Last updated