Interesting article about diving into debugging of TCP stack by Chris Stevens, Michał Switakowski, Ivan Sadikov, Winglung Ngai and Evan Ye. It is about their experience with analysis that determined the TCP stack was the source of the degradation observed on their platform. Their platform was experiencing network performance regressions due to Linux patches for the TCP SACKs vulnerabilities.
The article then goes in details how their went about:
- A failing benchmark – became 6x slower
- Debugging TCP connections swith netstat and ss
- SACKs in the TCP trace (tcpdump)
- Dissecting the TCP stack
- Lessons learned
Benchmarks are just numbers; logging is vital. The authors were also impressed with Linux ss command. The ss command is a tool used to dump socket statistics and displays information in similar fashion (although simpler and faster) to netstat.
It provides far more detail about a socket’s state and made the retransmission failures obvious. It also helps to make better sense of the TCP trace as it too was missing the expected retransmissions. Sweet!
[Read More]