What is Tracing? 🤔
Tracing typically involves capturing and recording information about the system to debug or perform a real-time analysis of the system. It helps identify bugs in the system.
Do you know when we add print messages to the code to debug it? We are tracing the program with our own “tracing system”. The debug messages added to the code are static tracepoints, and our “trace system” might send the debug messages to the program output log file (debug.log in Bitcoin).
Well, using print statements is a popular way of doing that, but it does not scale as every time we need to trace some part of your code, we need to add some debug statements and build it again.
We don’t have to add debug statements every time we need to trace some part of the code because we have built-in tools and frameworks that can do for us, interesting isn’t it? 😃
For example, we can collect the data for tracing, which can be done through kprobes, tracepoints, uprobes, dtrace probes/usdt and many more.
You can learn more about it here.
Let's see how it helps in Bitcoin.
Terminology
Tracing: It records every trace events
Trace events: A captured piece of information
Tracing Framework: Tools that help in Tracing the system, extracting the events captured and returning them to the Userspace
Probe: An instrumentation point in software or hardware that generates events that can execute aggregation/summaries/statistic programs
StaticTracing: Hardcoded instrumentation points in code that are fixed, they may be provided as part of a stable API and documented
Tracing in Bitcoin
To connect to the Bitcoin network, you’ll need a Bitcoin client which here is Bitcoin Core, there are others but we’ll focus here only on Bitcoin Core. However, Bitcoin Core already has a log file to track or query the information. However, producing real-time insights about the stuff happening in the network requires parsing frequently which can be resource-consuming which can be done easily with Tracepoints. Tracepoints in Bitcoin Core will benefit everyone ranging from developers to company managers to monitor their nodes, debug and also benchmark all the changes that have been implemented.
The tracing in Bitcoin Core is done with USDT (User-Space Statically Defined Tracing), a kind of interface provided in Bitcoin Core for tracing.
The USDT has four main parts:
There are some tracing scripts (written in Python) that compile the eBPF program and load the eBPF program into the kernel VM
The eBPF program hooks into the Tracepoint(s) which are there in Bitcoin-core
When the Tracepoint is called it is passed to the eBPF program
The eBPF program processes the arguments and later returns the data to the tracing scripts, the tracing scripts display the information to the user
The above-mentioned was some fundamental information about how tracing works in Bitcoin Core. We'll see each point in detail and later get a hands-on demo with tracing.
You might wonder why we don't use some external tools such as Wireshark, Dynatrace, etc. This has been greatly answered here.
Ad-hoc logging could be implemented for Tracing purposes, why use USDT?
We can make use of ad-hoc logging everywhere for debugging but this is an advanced concept that might be difficult for beginners to set up and resource consumption would come into play since we need to store the data that the logger has captured in some folder(ipaddr_rcv.dat, messages_sent.dat, etc), this would also impact the overall performance of the node. So with this, we can take advantage of USDT which is already present in Linux Kernel.
Learn more about it here.
Userspace static tracepoints give lots of flexibility without invasive logging code. It's more flexible than ad-hoc logging code, allowing you to interact with many different aspects of the system without having to enable per-subsystem logging says William Casarine.
Why eBPF? 🤔
One might wonder why eBPF as a tracing framework is used when there are a lot of other options such as ftrace, perfevents, etc which is because eBPF provides programmable capabilities also known as eBPF Programs (often written in C, we will likely use tool that generates that program for you). Then the eBPF program can be attached to Kernel for listening to the events.
Prerequisites
Bitcoin-Core (see the previous article for that)
Linux System (Kernel version above 4.x for eBPF, check using
uname -r.)
Why not use Logging (log file -> debug.log)
Bitcoin Core already has a file for storing all the logs i.e. debug.log then why do we need tracing?
Who do you think is the consumer of the log file?
The humans right? But the tracing can be useful for the system as well, we can parse the log file and use it by the system but it's not ideal since log messages can change and it also affects the resource consumption as we will have to parse again and again for real-time analysis!!
Use-Cases for Tracing
The first tracepoints that have been merged were #22006, you can see the developers testing out the tracepoints and giving some feedback. PR #20981 shows the discussion, thoughts and ideas for additional tracepoints that can be implemented in Bitcoin.
Developers and researchers have been using the tracepoints to monitor their nodes and detect any anomalies in the network. In the same way, I am working on a peer-observer(developed by 0xB10C) project during my Summer Of Bitcoin period which identifies the behaviour of peers that have been connected to me that use the tracepoints available in Bitcoin Core. However, this project also uses some additional tracepoints that are currently being reviewed by the developers of Bitcoin Core i.e. #25830. We'll see how we can set up peer-observer in my next article.
On 28th May 2024, one of the nodes that was monitored by 0xB10C was under attack. We noticed that node X had 100% CPU usage in theb-msghandthread. X has peerblockfilters and peerbloomfilters enabled. Turns out that after announcing its new IP with support for NODE_BLOOM, a bunch of "Schildbach's Bitcoin Wallets" (/bitcoinj:0.16.2/Bitcoin Wallet:10.14/) decided to connect and request merkleblocks from X peer. Bloom filters(BIP37) is known to be expensive and a DoS vector, which was picked up by Monitoring and Alerting. This caused the inbound connection to be dropped by a significant amount and was alerted by the alerting system.
All this could be possible due to tracepoints available in Bitcoin Core.
Other use cases include benchmarking of the performance of martinus PR #22702 by 0xB10C and jamesob, they have used the tracepoints available in Bitcoin core.
In summary, the tracepoints included in Bitcoin Core were very helpful for the developers and researchers for debugging, monitoring and benchmarking the code changes in the system, these are based on the Linux kernel and do not require any intrusive changes and they have very few implications on performance if not used.
Let's Trace!! ✊
Before starting with tracing Bitcoin, make sure you have your Bitcoin Core running.
Check with this command inside your Bitcoin Core repo:
./src/bitcoin-cli -getinfo
It should show like this:
Note: I have usedSignethere, you can useMainnettoo!
If you reached here, you're ready to dive in for tracing. Install this dependency for USDT,
sudo apt install systemtap-sdt-dev
Now, we need to configure the make file so that it enables the USDT,
./configure --enable-usdt
Now build and compile the bitcoins,
make install
make
Take your coffee and wind it down till it gets compiled.
Re-run your bitcoind after the compilation is completed,
./src/bitcoind -daemon
Your Bitcoin-core client is ready for tracing 🎉🎉
Check if the Tracepoints are Configured
gdb ./src/bitcoind
You should see some tracepoints such as utxos, feefilter.
Note: You'll need gdb for this
Run Tracing Scripts
Some tracing scripts are already present and written in Python, which you can see here.
sudo ./contrib/tracing/p2p_monitor.py ./src/bitcoind
Note:
make sure the***./src/bitcoind***is the location for the binary file that was just compiled after the configurational(usdt) changes
run the above-mentioned command with sudo as we need root privileges
We should see something like this,
This shows the real-time connected peers to our node, ADDR is the IP Address, TYPE shows the connection type(inbound or outbound), and INBOUND and OUTBOUND show the inbound and outbound message bandwidth exchanged between the nodes.
We can also expand each peer to see the messages that have been exchanged using the Enter or Space button on your keyboard,
You can also try out other scripts present in the Bitcoin Core repo, such as the log_raw_p2p_msgs.py, which displays all the raw p2p messages exchanged between your peers. Try running it and see the output.
Credits
I would like to thank 0xB10C for reviewing this article and providing valuable suggestions. Additionally, I extend my appreciation to Sudeshna for offering helpful feedback.