flywithu/Learning-Bitcoin-from-the-Command-Line
1
1
Fork 0
mirror of https://github.com/ChristopherA/Learning-Bitcoin-from-the-Command-Line.git synced 2025-06-08 16:36:32 +00:00
Code Issues Releases Wiki Activity

Update 4_2__Interlude_Using_JQ.md

Browse Source
This commit is contained in:
Shannon Appelcline 2017-04-03 15:48:21 -07:00 committed by GitHub
parent 5cf798dc96
commit 053c1da879
1 changed files with 42 additions and 31 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

73
4_2__Interlude_Using_JQ.md
View File

@ -8,7 +8,7 @@ Creating a raw transaction revealed how more complex bitcoin-cli results can't e
JQ is available from a [Github repository](https://stedolan.github.io/jq/). Just download for Linux, OS X, or Windows, as appropriate.
Once you've downloaded the binary, you can install it on your system:
Once you've downloaded the binary, you can install it on your system. If you're working on a Debian VPS as we suggest, your installation will look like this:
```
$ mv jq-linux64 jq
$ sudo /usr/bin/install -m 0755 -o root -g root -t /usr/local/bin jq
@ -19,7 +19,7 @@ _What is JQ?_ The repository explains it best, saying "jq is like sed for JSON d
**Usage Example:** _Capture the hex from a signed raw transaction._
In the previous section, the use of `signrawtransaction` offered the first example of not being able to easily capture data into variables due to the use of JSON:
In the previous section, the use of `signrawtransaction` offered an example of not being able to easily capture data into variables due to the use of JSON output:
```
$ bitcoin-cli signrawtransaction $rawtxhex
{
@ -29,28 +29,31 @@ $ bitcoin-cli signrawtransaction $rawtxhex
```
Fortunately, JQ can easily capture data of that sort!
To use JQ, run `jq` at the backend of a pipe, and always use the standard invocation of `jq -r '.'`. The `-r` tells JQ to produce raw output, which will work for command-line variables, while the `.` tells jq to ouput. We protect those argument in `' '` because we'll need that protection later as our `jq` invocations get more complex.
To use JQ, run `jq` at the backend of a pipe, and always use the standard invocation of `jq -r '.'`. The `-r` tells JQ to produce raw output, which will work for command-line variables, while the `.` tells jq to output. We protect that argument in `' '` because we'll need that protection later as our `jq` invocations get more complex.
To capture a specific value from a JSON object, you just list the key after the `.`:
```
$ bitcoin-cli signrawtransaction $rawtxhex | jq -r '.hex'
0200000001735dfa1584b930a78ad2c1d6db72dd2a80ae5e5d552ad97e19f1d50d41fdd6d8000000006a47304402202210ce4b2a037da02622c380278cd79fec4e0e016e66f3eb894a2dcbb9ee998f02202cac167e6abdbbf08af139fb7c6b86e9d2e58e5516cd566ae2d54953ead9923b012102111bb978a3c93a00038ae344a1a017d7fee8a9be9d0558b5793ce6f440704a96ffffffff01b0e78604000000001976a914e7c1345fc8f87c68170b3aa798a956c2fe6a9eff88ac00000000
```
With that tool in hand, you can capture information from JSON objects to command-line variables, as usual:
With that tool in hand, you can capture information from JSON objects to command-line variables:
```
$ signedtx=$(bitcoin-cli signrawtransaction $rawtxhex | jq -r '.hex')
$ echo $signedtx
0200000001735dfa1584b930a78ad2c1d6db72dd2a80ae5e5d552ad97e19f1d50d41fdd6d8000000006a47304402202210ce4b2a037da02622c380278cd79fec4e0e016e66f3eb894a2dcbb9ee998f02202cac167e6abdbbf08af139fb7c6b86e9d2e58e5516cd566ae2d54953ead9923b012102111bb978a3c93a00038ae344a1a017d7fee8a9be9d0558b5793ce6f440704a96ffffffff01b0e78604000000001976a914e7c1345fc8f87c68170b3aa798a956c2fe6a9eff88ac00000000
```
You can then use those variables easily and without error:
```
$ bitcoin-cli sendrawtransaction $signedtx
3f9ccb6e16663e66dc119de1866610cc4f7a83079bfec2abf0598ed3adf10a78
```
## Use JQ to Access a JSON Object Value in an Array by Key
## Use JQ to Access Single JSON Object Values in an Array by Key
**Usage Example:** _Capture the txid and vout for a selected UTXO._
Grabbing data out of a JSON object is easy, but if that JSON object is in a JSON array? The `listunspent` command offers a great example, because it'll usually contain a number of different transactions.
Grabbing data out of a JSON object is easy, but what if that JSON object is in a JSON array? The `listunspent` command offers a great example, because it'll usually contain a number of different transactions. What if you want to capture specific information from _one_ of them?
When working with an array, the first thing you need to do is tell JQ which index to access. For example, you might have looked through your transactions in `listunspent` and decided that you wanted to work with the first of them. You would then use `'.[0]'` to access that first element.
When working with a JSON array, the first thing you need to do is tell JQ which index to access. For example, you might have looked through your transactions in `listunspent` and decided that you wanted to work with the first of them. You use `'.[0]'` to access that first element. The `[]` says that we're referencing a JSON array and the `0` says we want the 0th index.
```
$ bitcoin-cli listunspent | jq -r '.[0]'
{
@ -65,11 +68,13 @@ $ bitcoin-cli listunspent | jq -r '.[0]'
"solvable": true
}
```
You can then capture an individual value from that selected array by (1) using a pipe _within_ the JQ arguments; and then (2) requesting the specific value afterward, just like in the previous example. The following would capture the `txid` from the 0th JSON object in the JSON array produced by `listunspent`:
You can then capture an individual value from that selected array by (1) using a pipe _within_ the JQ arguments; and then (2) requesting the specific value afterward, as in the previous example. The following would capture the `txid` from the 0th JSON object in the JSON array produced by `listunspent`:
```
~$ bitcoin-cli listunspent | jq -r '.[0] | .txid'
2b5f5798359e0e23e02764588166f222d4ce056419dec83c743b72aad171d708
```
Carefully note how the `' 's` go around the whole JQ expression _including_ the pipe.
This method can be used to fill in variables for a UTXO that you want to use:
```
$ newtxid=$(bitcoin-cli listunspent | jq -r '.[0] | .txid')
@ -81,11 +86,11 @@ $ echo $newvout
```
Voila! We could now create a new raw transaction using our 0th UTXO as an input, without having to type in any of the UTXO info by hand!
## Use JQ to Access Multiple JSON Object Values in an Array by Key
## Use JQ to Access Matching JSON Object Values in an Array by Key
**Usage Example:** _List the value of all unspent UTXOs._
Instead of accessing a specific value in a specific JSON object, you can instead access all of a specific value across all the JSON objects. This is done with `.[]`, where no index is specified. For example, this would list all unspent funds:
Instead of accessing a single, specific value in a specific JSON object, you could instead access all of a specific value across all the JSON objects. This is done with `.[]`, where no index is specified. For example, this would list all unspent funds:
```
$ bitcoin-cli listunspent | jq -r '.[] | .amount'
0.76
@ -109,9 +114,9 @@ $ bitcoin-cli getbalance
**Usage Example:** _List usage information for all UTXOs._
JQ is great for capturing individual elements from JSON objects and arrays and placing those elements into variables. That will be its prime use in future sections. However, it can also be used to cut down huge amounts of information output by `bitcoin-cli` into reasonable amounts of information.
JQ can easily capture individual elements from JSON objects and arrays and place those elements into variables. That will be its prime use in future sections. However, it can also be used to cut down huge amounts of information output by `bitcoin-cli` into reasonable amounts of information.
The following shows you the most important information on your UTXOs:
For example, you might want to see a listing of all your UTXOs (`.[]`) and get a listing of all of their most important information (`.txid, .vout, .amount`):
```
$ bitcoin-cli listunspent | jq -r '.[] | .txid, .vout, .amount'
2b5f5798359e0e23e02764588166f222d4ce056419dec83c743b72aad171d708
@ -124,11 +129,11 @@ ec0598918f6f5476cb90365651e8a2724ef26f949290bbf196f41ed96092a52f
0
1.95
```
This can make it easy to decide what UTXOs to spend in a raw transaction, but it's not very pretty.
This makes it easy to decide which UTXOs to spend in a raw transaction, but it's not very pretty.
However, JQ lets you be fancy. You can use `{}`s to create new JSON objects (either for additional parsing or for pretty output). You also get to define a new key for each of your values. The resulting output should be much more intuitive and less prone to error (though obviously, less useful for dumping info straight into variables).
Fortunately, JQ also lets you be fancy. You can use `{}`s to create new JSON objects (either for additional parsing or for pretty output). You also get to define the name of the new key for each of your values. The resulting output should be much more intuitive and less prone to error (though obviously, less useful for dumping info straight into variables).
The following example shows the exact same parsing of `listunspent`, but with the abridgement of each old JSON object rebuilt as a new JSON object, with all of the new values named with their old keys:
The following example shows the exact same parsing of `listunspent`, but with the each old JSON object rebuilt as a new, abridged JSON object, with all of the new values named with their old keys:
```
$ bitcoin-cli listunspent | jq -r '.[] | { txid: .txid, vout: .vout, amount: .amount }'
{
@ -168,9 +173,9 @@ You could of course rename your new keys as you see fit. There's nothing magic i
```
## Use JQ to Access JSON Objects by Looked-Up Value
**Usage Example:** _Automatically look up a UTXO being used in a transaction._
**Usage Example:** _Automatically look up UTXOs being used in a transaction._
The JQ lookups so far have been fairly simple: you use a key to lookup one or more values in a JSON object or array. But what if you instead want to lookup a value in a JSON object ... by another value? This sort of indirect lookup has real applicability when you're working with transactions built on existing UTXOs. For example, it can allow you to calculate the sum value of the UTXOs being used in a transaction, something that is vitally important.
The JQ lookups so far have been fairly simple: you use a key to look up one or more values in a JSON object or array. But what if you instead want to look up a value in a JSON object ... by another value? This sort of indirect lookup has real applicability when you're working with transactions built on existing UTXOs. For example, it can allow you to calculate the sum value of the UTXOs being used in a transaction, something that is vitally important.
This example uses the following raw transaction. Note that this is a more complex raw transaction with two inputs and two outputs. We'll learn about making those in a few sections; for now, it's necessary to be able to offer robust examples. Note that unlike our previous examples, this one has two objects in its `vin` array and two in its `vout` array.
```
@ -236,7 +241,7 @@ $ bitcoin-cli decoderawtransaction $rawtxhex
### Retrieve the Value(s)
Assume that we know exactly how this transaction is constructed: we know that it uses two UTXOs as input. To retrieve the txid for the two UTXOs, we could `jq` to the .vin key-value, then to its 0th array, then to that array's .txid key-value. Then we could do the same with the 1st array. Easy:
Assume that we know exactly how this transaction is constructed: we know that it uses two UTXOs as input. To retrieve the txid for the two UTXOs, we could use `jq` to look up the transaction's .vin value, then reference the .vin's 0th array, then that array's .txid value. Afterward, we could do the same with the 1st array. Easy:
```
$ usedtxid1=$(bitcoin-cli decoderawtransaction $rawtxhex | jq -r '.vin | .[0] | .txid')
$ echo $usedtxid1
@ -247,7 +252,7 @@ ec0598918f6f5476cb90365651e8a2724ef26f949290bbf196f41ed96092a52f
```
However, it would be better to have a general case that _automatically_ saved all the txids of our UTXOs.
We already know that we can access all of the `.txid`s by using an `.[]` array value:
We already know that we can access all of the `.txid`s by using an `.[]` array value. We can use that to build a general .txid lookup:
```
usedtxid=($(bitcoin-cli decoderawtransaction $rawtxhex | jq -r '.vin | .[] | .txid'))
$ echo ${usedtxid[0]}
@ -255,13 +260,13 @@ $ echo ${usedtxid[0]}
$ echo ${usedtxid[1]}
ec0598918f6f5476cb90365651e8a2724ef26f949290bbf196f41ed96092a52f
```
The only real trick here is how we saved the information to the bash shell. Rather than saving to a variable with `$(commands)`, we instead saved to an array with `($(commands))`. We were then able to access the individual bash array elements with a `${variable[n]}` construction. We can instead access the whole array with `${variable[@]}`, which we'll use in the next section. (Yeah, no one ever said bash was pretty.)
The only real trick here is how we saved the information to the bash shell. Rather than saving to a variable with `$(command)`, we instead saved to an array with `($(command))`. We were then able to access the individual bash array elements with a `${variable[n]}` construction. We could instead access the whole array with `${variable[@]}`, which we'll use in the next section. (Yeah, no one ever said bash was pretty.)
### Retrieve the Related Object(s)
We can now move back to `listunspent`. To find our transactions, we need to look through the entire JSON array (`[]`). We can then choose (`select`) individual JSON objects that include (`contains`) our txids. (The `select` and `contains` arguments show off some of the complexity of JSON, but for now just know that this particular invocation will work.)
We can now use this information to reference UTXOs in `listunspent`. To find the information on the UTXOs being used by the raw transaction, we need to look through the entire JSON array (`[]`). We can then choose (`select`) individual JSON objects that include (`contains`) our txids. (The `select` and `contains` arguments show off some of the complexity of JSON, but for now just know that this particular invocation will work.)
Here we are picking out our two UTXOs, one at a time:
To start simply, this picks out the two UTXOs one at a time:
```
$ bitcoin-cli listunspent | jq -r '.[] | select (.txid | contains("'${usedtxid[0]}'"))'
{
@ -288,7 +293,7 @@ $ bitcoin-cli listunspent | jq -r '.[] | select (.txid | contains("'${usedtxid[1
"solvable": true
}
```
A simple Bash for-loop can instead give you _all_ of your UTXOs:
A simple bash for-loop can instead give you _all_ of your UTXOs:
```
for txid in ${usedtxid[@]}; do bitcoin-cli listunspent | jq -r '.[] | select (.txid | contains("'$txid'"))'; done
{
@ -319,30 +324,36 @@ for txid in ${usedtxid[@]}; do bitcoin-cli listunspent | jq -r '.[] | select (.t
**Usage Example:** _Automatically calculate the value of the UTXOs used in a transaction._
You can now go one step further, and request the .amount (or any other JSON key-value) from the objects you're retrieving. For this example, we're still using that `$usedtxid` variable we set:
You can now go one step further, and request the .amount (or any other JSON key-value) from the UTXOs you're retrieving.
This example uses the `$usedtxid` array that was set as follows:
```
$ usedtxid=($(bitcoin-cli decoderawtransaction $rawtxhex | jq -r '.vin | .[] | .txid'))
```
You can look at them individual with a `for` script:
You can use it to look at .amounts individually with a bash `for` script:
```
$ for txid in ${usedtxid[@]}; do bitcoin-cli listunspent | jq -r '.[] | select (.txid | contains("'$txid'")) | .amount'; done
0.76
3.9
```
You can sum them up with an `awk` script, to really see how much money is in the UTXOs your transaction is spending:
Alternatively, you can sum up the .amounts with an `awk` script, to really see how much money is in the UTXOs that the transaction is spending:
```
$ usedtxid=($(bitcoin-cli decoderawtransaction $rawtxhex | jq -r '.vin | .[] | .txid'))
$ for txid in ${usedtxid[@]}; do bitcoin-cli listunspent | jq -r '.[] | select (.txid | contains("'$txid'")) | .amount'; done | awk '{s+=$1} END {print s}'
4.66
```
## Use JQ for Complex Calculations
**Usage Example:** _Calculate the fee for a real transaction._
**Usage Example:** _Calculate the fee for a transaction._
A few more lines of code that calculate the value of the vout, then subtract the vout from the vin can be used to calculate the transaction fee.
To figure out the complete transaction fee at this point just requires determining out how much money is going through the .vout:
```
$ bitcoin-cli decoderawtransaction $rawtxhex | jq -r '.vout [] | .value' | awk '{s+=$1} END {print s}'
4.6505
```
Then, you subtract the .vout .amount (4.6505) from the .vin .amount (4.66) and you have a fee calculator!
Putting it all together gives us:
Putting it all together gives us a complete calculator in just four lines of script:
```
$ usedtxid=($(bitcoin-cli decoderawtransaction $rawtxhex | jq -r '.vin | .[] | .txid'))
$ btcin=$(for txid in ${usedtxid[@]}; do bitcoin-cli listunspent | jq -r '.[] | select (.txid | contains("'$txid'")) | .amount'; done | awk '{s+=$1} END {print s}')
@ -352,7 +363,7 @@ $ echo "$btcin-$btcout"| /usr/bin/bc
```
And that's also a good example of why you double-check your fees: we'd intended to send a transaction fee of 5,000 satoshis, but sent 95,000 satoshis instead. Whoops!
> **WARNING:** This was not an intentional lesson. We really messed up our calculation of the transaction fee when we wrote our raw transaction. It's *that* easy, then your money is gone.
> **WARNING:** This was not an intentional lesson. We genuinelly messed up our calculation of the transaction fee when we wrote our raw transaction. It's *that* easy, then your money is gone.
For more JSON magic (and if any of this isn't clear), please read the [JSON Manual](https://stedolan.github.io/jq/manual/) and the [JSON Cookbook](https://github.com/stedolan/jq/wiki/Cookbook). We'll be regularly using JQ in future examples.