Learning-Bitcoin-from-the-Command-Line/03_4_Understanding_the_Descriptor_Wallet.md

3.4: Understanding the Descriptor Wallet

You've got your wallet set up, but before we go further we're going to take a moment to really understand it.

Understand HD Wallets

Private keys are what make the Bitcoin world go round. They're used to generate public keys, which are the foundation of addresses, and they're also used to control those addresses. One private key creates one public key which creates one address. Once upon a time, the Bitcoin Core wallet managed this by holding on to a "bag of keys". A new, unrelated private key would be created every time a new address was desired. But bags of keys are big, inefficient, and prone to loss. That's where the HD wallet came in.

The HD wallet, which is short for the Hierarchical Deterministic Wallet, was defined in BIP-32. It's a hierarchical design where a single "seed" generates an extended private key, which includes not just the private key but also a "chain code" that can be used to create descendents of that key. The key and chain code can be used to deterministically generate chains of keys (and therefore addresses) for a variety of purposes. A Bitcoin HD wallet will typically have individual chains of keys (and addresses) for a variety of different address types. But, they can all be restored from that seed (or from that master extended private key) because of their determinism: the addresses are always created in the same way provided that you have the same starting points (which include your master extended private key and a specific index [0,1,...,n] for a specific type of address).

📖 What is a BIP? A BIP is a Bitcoin Improvement Proposal. It's an in-depth suggestion for a change to the Bitcoin Core code. Often, when a BIP has been sufficiently discussed and updated, it will become an actual part of the Bitcoin Core code. BIP-32 is one of many examples.

Understand Descriptor Wallets

Most of this course presumes that you're working entirely from a single node where you manage your own wallet, sending and receiving payments with the addresses created by that wallet. However, that's not necessarily how the larger Bitcoin ecosystem works. There, you're more likely to be moving addresses between wallets (often on different devices) and even setting up wallets to watch over funds controlled by different wallets.

HD wallets offered a big step forward for managing this sort of interoperability, because they allowed you to load a single seed (or master extended private key) into a new wallet rather than having to move over a whole bag of keys. A few formats were quickly introduced to make it easy to move HD wallets, starting with seed phrases and the xpub and xprv formats. But they quickly proved inadequate. Seed phrases only defined the seed, not what it was used for. The xprv (and xpub) improved on that by defining the root of an HD tree, but they were very specific to a certain type of address. When a new address type was created, a new format was needed, resulting in the yprv and ypub formats. Then a third address type resulted in the creation of zprv and zpub. The system was quickly growing unmanageable.

A new format was needed that described not just the extended keys, but also which address types they would be used to create. Because if you didn't have that, a new wallet would have to test a master extended public key against every possible type of address and that was going to be very inefficient and time-consuming and still introduced the likelihood of losing funds.

📖 What is a seed phase? A collection of mnemonic words can define a seed. That seed is then used to generate an extended private key. Seed phrases are not currently used by Bitcoin Core, but they are in wide use in the larger Bitcoin ecosystem. Future chapters will talk about how to use third-party tools to transform seed phrases into keys that can be imported into bitcoin-cli.

📖 What is xprv? Xprv stands for extended private key. This is the combination of a private key and a chain code. It's a private key that a whole sequence of children private keys can be derived from.

📖 What is xpub? Xpub stands for extended public key. This is the combination of a public key and a chain code. It's a public key that a whole sequence of children public keys can be derived from.

Enter, at last, the descriptor wallet. A descriptor wallet collects together "output descriptors" (sometimes called "wallet descriptors"), which each either define one address or, with a special "ranged descriptor", a whole array of addresses, each at a separate index. They do so through the specification of a format that includes: one or more nested functions (which define how to unlock the Bitcoin at the address), a derivation path (which defines the purpose of an address, which mostly links it to a specific standard), either the master extended public key or the master extended private key, and a checksum to make sure that nothing has been corrupted.

📖 What is a Derivation Path? When you have hierarchical keys, you need to be able to define individual keys as descendents of the master key. For example [0] is the 0th key of the master key, [0/1] is the first son of the 0th key, [0/1/1] is the first son of the first son of the 0th key. Some keys also contain a ' or h after the number, to show they're hardened, which protects them from a specific attack that could otherwise be used to derive a private key from a public key. You don't need to worry about the specifics, other than the fact that a derivation path like [0/1/1/0/0] describes a path down through a hierarchical tree and that descriptor wallets run specific calculations to deterministically determine the right address for a specific position in a tree. A derivation path defines a key, which means that a key represents a derivation path. They're equivalent.

The derivation path allows you to calculate the right key from the master extended key, but it's the introduction of functions into descriptors that makes them particularly powerful, because they allows descriptors to serve a number of different types of past, present, and future addresses (which we'll meet in §4.1).

⚠️ VERSION WARNING: Modern Bitcoin wallets use descriptor wallets stored in SQLite. Older, "classic" wallets were instead bags of keys, stored in BDB (Berkeley Database) format. The classic files can currently still be opened by bitcoin-cli but you wouldn't want to create something new in that format.

Examine Descriptors with listdescriptors

Your Bitcoin Core wallet is a descriptor wallet because it creates its Bitcoin addresses from "ranged descriptors". You can look at them with bitcoin-cli listdescriptors:

$ bitcoin-cli listdescriptors
{
  "wallet_name": "",
  "descriptors": [
    {
      "desc": "pkh([e18dae20/44h/1h/0h]tpubDCsocyjrtJLXKJ3atFwKf6FiPheuVNV27B1swsGiNvk4cuPhVTsCDvusSNcH8thnS68FPFotgHqo9FHNGrvhtx6ZqdbWBQTWrCgK9xous17/0/*)#5q6np9at",
      "timestamp": 1770329126,
      "active": true,
      "internal": false,
      "range": [
        0,
        999
      ],
      "next": 0,
      "next_index": 0
    },
    {
      "desc": "pkh([e18dae20/44h/1h/0h]tpubDCsocyjrtJLXKJ3atFwKf6FiPheuVNV27B1swsGiNvk4cuPhVTsCDvusSNcH8thnS68FPFotgHqo9FHNGrvhtx6ZqdbWBQTWrCgK9xous17/1/*)#95ljusdn",
      "timestamp": 1770329126,
      "active": true,
      "internal": true,
      "range": [
        0,
        999
      ],
      "next": 0,
      "next_index": 0
    },
    {
      "desc": "sh(wpkh([e18dae20/49h/1h/0h]tpubDCmuiGVie2yYNXiwi6ymC2qamb7RdEXKeo8HZtkLAtaCcLHFGvHBY4W3Fx9rTjxYdxkb1kvkLp86oDvdxmYjvrVXpcVG2XptmYu8rYsAirP/0/*))#jdj8slnk",
      "timestamp": 1770329126,
      "active": true,
      "internal": false,
      "range": [
        0,
        999
      ],
      "next": 0,
      "next_index": 0
    },
    {
      "desc": "sh(wpkh([e18dae20/49h/1h/0h]tpubDCmuiGVie2yYNXiwi6ymC2qamb7RdEXKeo8HZtkLAtaCcLHFGvHBY4W3Fx9rTjxYdxkb1kvkLp86oDvdxmYjvrVXpcVG2XptmYu8rYsAirP/1/*))#8vu3gqxf",
      "timestamp": 1770329127,
      "active": true,
      "internal": true,
      "range": [
        0,
        999
      ],
      "next": 0,
      "next_index": 0
    },
    {
      "desc": "tr([e18dae20/86h/1h/0h]tpubDC5ZrtRv5CURANE3oWS1cK7mE9t93KnZfUApDs3dg8xxDuGMsuSejCLeh4yKeqkvHknRboWEctS2BSGQxXhsCx1dgPCTCwjF4a9mgq999Af/0/*)#h4xmum5n",
      "timestamp": 1770329126,
      "active": true,
      "internal": false,
      "range": [
        0,
        999
      ],
      "next": 0,
      "next_index": 0
    },
    {
      "desc": "tr([e18dae20/86h/1h/0h]tpubDC5ZrtRv5CURANE3oWS1cK7mE9t93KnZfUApDs3dg8xxDuGMsuSejCLeh4yKeqkvHknRboWEctS2BSGQxXhsCx1dgPCTCwjF4a9mgq999Af/1/*)#xpr6pwyt",
      "timestamp": 1770329127,
      "active": true,
      "internal": true,
      "range": [
        0,
        999
      ],
      "next": 0,
      "next_index": 0
    },
    {
      "desc": "wpkh([e18dae20/84h/1h/0h]tpubDC4ujMbsd9REzpGk3gnTjkrfJFw1NnvCpx6QBbLj3CHBzcLmVzssTVP8meRAM1WW4pZnK6SCCPGyzi9eMfzSXoeFMNprqtgxG71VRXTmetu/0/*)#3658f8sn",
      "timestamp": 1770329126,
      "active": true,
      "internal": false,
      "range": [
        0,
        1001
      ],
      "next": 3,
      "next_index": 3
    },
    {
      "desc": "wpkh([e18dae20/84h/1h/0h]tpubDC4ujMbsd9REzpGk3gnTjkrfJFw1NnvCpx6QBbLj3CHBzcLmVzssTVP8meRAM1WW4pZnK6SCCPGyzi9eMfzSXoeFMNprqtgxG71VRXTmetu/1/*)#qw3x5jqt",
      "timestamp": 1770329127,
      "active": true,
      "internal": true,
      "range": [
        0,
        999
      ],
      "next": 0,
      "next_index": 0
    }
  ]
}

Wow, that's a lot! But it's really just a listing of eight descriptors (desc) with a bunch of additional information on each. And those eight descriptors actually represent a large number of addresses that can deterministically be created on the fly.

• There are four types of addresses, which we'll meet in the next chapter (pkh, sh, tr, and wpkh).
• Each type of address supports external addresses (for receiving funds from other wallets) and internal addresses (for sending change back to this wallet), as defined by internal.
• Each type of address has its own key (e.g., tpubDCsocyjrtJLXKJ3atFwKf6FiPheuVNV27B1swsGiNvk4cuPhVTsCDvusSNcH8thnS68FPFotgHqo9FHNGrvhtx6ZqdbWBQTWrCgK9xous17).
• Each type of address supports a range of addresses, initially running 0 to 999.

In other words, even though you have 8 descriptors, that's replacing thousands of addresses. Thatt's a huge boon for backups (when you want to protect your funds) and for moving control of your funds from one wallet-app to another.

With that understood, we can look more closely at one of the descriptors:

{
  "desc": "wpkh([e18dae20/84h/1h/0h]tpubDC4ujMbsd9REzpGk3gnTjkrfJFw1NnvCpx6QBbLj3CHBzcLmVzssTVP8meRAM1WW4pZnK6SCCPGyzi9eMfzSXoeFMNprqtgxG71VRXTmetu/0/*)#3658f8sn",
  "timestamp": 1770329126,
  "active": true,
  "internal": false,
  "range": [
    0,
    1001
  ],
  "next": 3,
  "next_index": 3
},

This contains:

• desc: The descriptor.
• timestamp: When the descriptor was created.
• active: Is the descriptor still in use for creating new addresses? (It could have been superseded by a new master extended key, for example when encrypting the wallet.)
• internal: Is this a descriptor for internal addresses (for change)?
• range: For ranged descriptors, what's the range?
• next, next_index: What is the next address to create for this descriptor? In this example, the next one is 3 because three addresses have been created, as we'll see in the next chapter.

As for the descriptor itself, let's break that down further:

• Function: wpkh. The function that is used to create an address from that key. In this cases it's wpkh. That stands for "Witness Public Key Hash," which is one of the methods used to unlock a Bech32 address, which we'll meet in the next chapter.
• Fingerprint: e18dae20. This is a fingerprint of the master extended public key. It tells you which secret was used to generate this address. The fingerprint is not necessary to generate the keys and addresses for a derivation, it's just helpful if you need to go back and find the secret that generated your extended keys.
• Derivation Path: /84h/1h/0h. This describes what part of an HD wallet is being exported. This is the 0th child key of the 1st child of the 84th child in the HD tree. The various levels in the derivation path have very specific meanings as defined in BIP-44: /purpose/coin_type/account/. The purpose of this derivation path is "84", which means that it follows BIP-84, which describes WPKH derivation. The coin type is "1", which means that it's a testnet or signet coin. (A mainnet coin could would be "0".) The account is "0", as it's the only account in our wallet.
• Key: tpubDC4ujMbsd9REzpGk3gnTjkrfJFw1NnvCpx6QBbLj3CHBzcLmVzssTVP8meRAM1WW4pZnK6SCCPGyzi9eMfzSXoeFMNprqtgxG71VRXTmetu. This is the signet or testnet extended master public key that was used to generate this derived key. (A private key could be here instead. A public key would demonstrate how to watch this series of addresses, while a private key would show to control them.)
• Range: /0/*. These are actually the final two parts of the derivation path, which are defined by BIP-44 as change/address_index. The "0" says it's an external address. (An internal or change address would be "1".) The * says it's a ranged address, which means that it's defining a whole set of WPKH addresses that could be created.
• #3658f8sn". This is a checksum showing the descriptor isn't corrupted.

So that's what everything means in a descriptor.

📖 Private Keys Possible. You could instead run bitcoin-cli listdescriptors true if you want your descriptor list to include the private keys instead of the public keys.

Examine Descriptors with getdescriptorinfo

Descriptors can be also be examined with another command, bitcoin-cli getdescriptorinfo:

$ bitcoin-cli getdescriptorinfo "wpkh([e18dae20/84h/1h/0h]tpubDC4ujMbsd9REzpGk3gnTjkrfJFw1NnvCpx6QBbLj3CHBzcLmVzssTVP8meRAM1WW4pZnK6SCCPGyzi9eMfzSXoeFMNprqtgxG71VRXTmetu/0/*)#3658f8sn"
{
  "descriptor": "wpkh([e18dae20/84h/1h/0h]tpubDC4ujMbsd9REzpGk3gnTjkrfJFw1NnvCpx6QBbLj3CHBzcLmVzssTVP8meRAM1WW4pZnK6SCCPGyzi9eMfzSXoeFMNprqtgxG71VRXTmetu/0/*)#3658f8sn",
  "checksum": "3658f8sn",
  "isrange": true,
  "issolvable": true,
  "hasprivatekeys": false
}

This is a pretty sparse description that includes the descriptor and its checksum and confirms that it's a ranged descriptor (isrange), that our wallet has the private key (issolvable), and that the private key isn't included in the descriptor and that there's instead a public key (hasprivatekeys). Beyond that information, getdescriptorinfo serves two other purposes:

1. If you don't have the checksum (which is required for other descriptor-related bitcoin-cli commands), you can enter the descriptor into getdescriptorinfo without it, and it'll be calculated for you.

$ bitcoin-cli getdescriptorinfo "wpkh([e18dae20/84h/1h/0h]tpubDC4ujMbsd9REzpGk3gnTjkrfJFw1NnvCpx6QBbLj3CHBzcLmVzssTVP8meRAM1WW4pZnK6SCCPGyzi9eMfzSXoeFMNprqtgxG71VRXTmetu/0/*)"
{
  "descriptor": "wpkh([e18dae20/84h/1h/0h]tpubDC4ujMbsd9REzpGk3gnTjkrfJFw1NnvCpx6QBbLj3CHBzcLmVzssTVP8meRAM1WW4pZnK6SCCPGyzi9eMfzSXoeFMNprqtgxG71VRXTmetu/0/*)#3658f8sn",
  "checksum": "3658f8sn",
  "isrange": true,
  "issolvable": true,
  "hasprivatekeys": false
}

2. If your descriptor is invalid, getdescriptorinfo will tell you:

$ bitcoin-cli getdescriptorinfo "cow([e18dae20/84h/1h/0h]tpubDC4ujMbsd9REzpGk3gnTjkrfJFw1NnvCpx6QBbLj3CHBzcLmVzssTVP8meRAM1WW4pZnK6SCCPGyzi9eMfzSXoeFMNprqtgxG71VRXTmetu/0/*)"
error code: -5
error message:
'cow([e18dae20/84h/1h/0h]tpubDC4ujMbsd9REzpGk3gnTjkrfJFw1NnvCpx6QBbLj3CHBzcLmVzssTVP8meRAM1WW4pZnK6SCCPGyzi9eMfzSXoeFMNprqtgxG71VRXTmetu/0/*)' is not a valid descriptor function

Import a Descriptor

The really important feature of descriptors is that you can take them to another (remote) machine and import them. This is done with the importdescriptors command. The following example shows the import of the private-key version of our BIP-84 ranged descriptor into another wallet:

$ bitcoin-cli importdescriptors '[{ "desc": "wpkh(tprv8ZgxMBicQKsPd1dP4NpsFDpsLUCnZ7oyn4UEbYLw7if1EDVCxMgfSzAwP3aCr1YeRvX9GtGvHsCLdrM7zaDyh33jEj7joQoEeNEyJaSYm5p/84h/1h/0h/0/*)#grdqnase", "timestamp":1770329126, "active": true, "range": [0,10] }]'
[
  {
    "success": true
  }
]

You'll note that this is a much more complex bitcoin-cli command than anything we've used before. It requires the input of a JSON array with a variety of different variables. (Which is a pain.) The desc is that private-key descriptor, the timestamp says how much of the blockchain to rescan, the range says how much of the range to import, and the active says that this descriptor can be used to create new addresses. After importing it, this descriptor becomes the main (active) one for your wallet, which will be used to derive new addresses.

Summary: Understanding the Descriptor

Descriptor wallets define large numbers of keys with a single simple description for each type of address in your wallet. Currently, there are eight ranged descriptors in each Bitcoin Core Wallet, defining internal and external addresses for the four in-use types of Bitcoin addresses.

🔥 What is the power of descriptors? Descriptors allow you to import and export sets of keys and addresses. That's great if you want to move between different wallets. As a developer, they also allow you to build up the precise sort of addresses that you're interested in creating.

What's Next?

MOove on to "addresses" with Chapter Three: Preparing Your Bitcoin Addresses.