May 13, 2013 - by Mr. Max Bruning
Robert Mustacchi of Joyent suggested I write about a tool that he wrote that allows one to simulate network problems. The tool is
ipdadm(1M), the internet packet disturber. The tool allows you to specify as a percentage: packet drops, packet corruption, and delays. This can be very useful when you want to test the effect of these on the behavior of a networked application.
The tool works on all network interfaces of a given zone. Be careful that you do not use this in a place where you can not undo your actions. For instance, on my first time trying this, I specified 20% of packets are to be corrupted in a zone that I was ssh-ed into. My
ssh session dropped immediately. Fortunately, I had access to the global zone on the machine and was able to change the corruption rate back to 0%. (Note also that with access to the global zone, you can use
zlogin to login to the zone where you are using
ipdadm). Currently there is no way to specify a specific interface, IP address, or port number. The tool works on all interfaces being used within a zone.
Also note that the tool is full duplex. So if you request a 20 microsecond delay on packets, the delay is both for incoming and outgoing packets.
Here are some examples, run from the global zone.
# ipdadm -z 73e886b7-77a5-4081-aaf6-83382e3bdced info zone 73e886b7-77a5-4081-aaf6-83382e3bdced does not exist or has no ipd actions enabled #
Currently, no ipd actions are set for the zone. From a different machine:
$ scp words firstname.lastname@example.org:/var/tmp/ words 100% 202KB 201.8KB/s 00:01 $ scp words email@example.com:/var/tmp/ words 100% 202KB 201.8KB/s 00:01
Let's set the drop rate to 10% and try again.
# ipdadm -z 73e886b7-77a5-4081-aaf6-83382e3bdced drop 10 # ipdadm -z 73e886b7-77a5-4081-aaf6-83382e3bdced info ipd information for zone 73e886b7-77a5-4081-aaf6-83382e3bdced: corrupt: 0% chance of packet corruption drop: 10% chance of packet drop delay: 0 microsecond delay per packet #
And from the other machine:
$ scp words firstname.lastname@example.org:/var/tmp/ words 100% 202KB 22.4KB/s 00:09 $ scp words email@example.com:/var/tmp/ words 100% 202KB 40.4KB/s 00:05 $
A 10% drop rate resulted in 5 to 10 times longer. At a 30% drop rate,
scp stalled after
copying 71% of the file, and did not finish. Running
netstat(1M) shows retransmissions (to be expected with packets being dropped).
# netstat -s 5 | grep tcpRetrans tcpRetransSegs = 76 tcpRetransBytes = 52932 tcpRetransSegs = 0 tcpRetransBytes = 0 tcpRetransSegs = 2 tcpRetransBytes = 752 tcpRetransSegs = 0 tcpRetransBytes = 0 ...
The following removes the drops and sets a delay of 100 milliseconds.
# ipdadm -z 73e886b7-77a5-4081-aaf6-83382e3bdced remove drop # ipdadm -z 73e886b7-77a5-4081-aaf6-83382e3bdced delay 100000 #
scp to the zone with the delay set.
$ scp words firstname.lastname@example.org:/var/tmp words 100% 202KB 28.8KB/s 00:07
Setting corruption does not work well with
Here is output from
scp with corrupt set to 20%.
$ scp words email@example.com:/var/tmp/ hash mismatch key_verify failed for server_host_key lost connection $
Setting drop to 100% mimics the effect of a network partition, (i.e., the network looks to be down. A useful experiment to determine how well your application behaves in a network outage is to set the drop rate to 100% in the system running your SAMP or nodestack server software.
There also exists a node module and library, both written by Keith Wesolowski at https://github.com/wesolows/node-ipd and https://github.com/joyent/illumos-joyent/tree/master/usr/src/lib/libipd respectively. These allow one to use
ipdadm from within a Node.JS script or from within another program.
The implementation of
ipdadm is in two parts: a kernel module (
ipd), that gets installed as a packet filter hook (note that this refers to a paper entitled "Packet Filter Hooks Design, 3 Sep 2006", but that paper seems to have been lost to the internet) Also see Packet sniffing with packet filtering hooks; and, a user level command (
ipdadm), that controls the kernel module.
ipdadm is not currently in illumos, i.e., only SmartOS, and is restricted to exclusive IP stack zones (i.e., not for shared IP stacks).
Robert says it will be pushed upstream to illumos, but he's been busy. Source for the
ipd kernel module is available at usr/src/uts/common/inet/ipd/. The
ipd.c file in this directory has a long comment at the beginning explaining how
ipd works. Source for the
ipdadm user level command is at usr/src/cmd/ipdadm/.
Note that you can snoop and see dropped packets actually arriving, but outbound dropped packets are not seen. This is because
ipd intercepts packets at the IP layer, not below. And, as mentioned earlier, packet drops, corruption, and delays are done full-duplex (i.e., both directions).
For information about other such enhancements that Joyent has made to SmartOS (some of which have been pushed up to illumos), see SmartOS Additions/Modifications to illumos.