Hervey Allen Carlos Vicente Network Startup Resource Center

Hervey Allen Carlos Vicente Network Startup Resource Center Registry Operations Curriculum Network Performance Definitions and Analysis

Network Performance Metrics Planning performance management Metrics Network Systems Services Definitions

Planning What's the intention? Baselining, Defend Troubleshooting, Planning growth yourself from accusations -”it's the network!” Who is the information for? Administration, How NOC, customers to structure and present the information Reach: Can I measure everything? Impact on devices (measurements and measuring) Balance between amount of information and time to get it

Metrics Network performance metrics Channel capacity, nominal & effective Channel utilization Delay and jitter Packet loss and errors

Metrics System performance metrics Availability Memory, CPU Utilization, load, I/O wait, etc. Service performance metrics Wait time / Delay Availability How can I justify maintaining the service? Who is using it? How often? Economic value? Other value?

Common Network Performance Measurements Relative to traffic: Bits per second Packets per second Unicast vs. non-unicast packets Errors Dropped Flows per second Round Jitter packets trip time (RTT) (variation between packet RTT)

Nominal Channel Capacity The maximum number of bits that can be transmitted for a unit of time (eg: bits per second) Depends on: Bandwidth of the physical medium Cable Electromagnetic waves Processing capacity for each transmission element Efficiency Channel of algorithms in use to access medium encoding and compression

Effective Channel Capacity Always a fraction of the nominal channel capacity Dependent on: Additional Device overhead of protocols in each layer limitations on both ends Flow control algorithm efficiency, etc. For example: TCP

Channel Utilization What fraction of the nominal channel capacity is actually in use Important! Future planning What utilization growth rate am I seeing? For when should I plan on buying additional capacity? Where should I invest for my updates? Problem resolution Where are my bottlenecks, etc.

95th Percentile th 95 Percentile The smallest value that is larger than 95% of the values in a given sample This means that 95% of the time the channel utilization is equal to or less than this value Or rather, the peaks are discarded from consideration Why is this important in networks? Gives you an idea of the standard, sustained channel utilization. ISPs use this measure to bill customers with “larger” connections.

95th Percentile

Bits per second vs Packets p.s.

End-to-end Delay The time required to transmit a packet along its entire path Created by an application, handed over to the OS, passed to a network card (NIC), encoded, transmitted over a physical medium (copper, fibre, air), received by an intermediate device (switch, router), analyzed, retransmitted over another medium, etc. The most common measurement uses ping for total roundtrip-time (RTT).

Historical Measurement of Delay

Types of Delay Causes of end-to-end delay: Processor delays Buffer delays Transmission delays Propagation delays

Processing Delay Required time to analyze a packet header and decide where to send the packet (eg. a routing decision) - Inside a router this depends on the number of entries in the routing table, the implementation of data structures, hardware in use, etc. This can include error verification / checksumming (i.e. IPv4, IPv6 header checksum)

Queuing Delay Queuing Delay The time a packet is enqueued until it is transmitted The number of packets waiting in the queue will depend on traffic intensity and of the type of traffic Router queue algorithms try to adapt delays to specific preferences, or impose equal delay on all traffic.

Transmission Delay Transmission Delay The time required to push all the bits in a packet on the transmission medium in use For N Number of bits, S Size of packet, d delay d S/N For example, to transmit 1024 bits using Fast Ethernet (100Mbps): d 1024/1x10e8 10.24 micro seconds

Propagation Delay Once a bit is 'pushed' on to the transmission medium, the time required for the bit to propagate to the end of its physical trajectory The velocity of propagation of the circuit depends mainly on the actual distance of the physical circuit In the majority of cases this is close to the speed of light. For d distance, s propagation velocity PD d/s

Transmission vs. Propagation Can be confusing at first Consider this example: Two 100 Mbps circuits - 1 km of optic fiber - Via satellite with a distance of 30 km between the base and the satellite For two packets of the same size which will have the larger transmission delay? Propagation delay?

Packet Loss Occurs due to the fact that buffers are not infinite in size - When a packet arrives to a buffer that is full the packet is discarded. - Packet loss, if it must be corrected, is resolved at higher levels in the network stack (transport or application layers) - Loss correction using retransmission of packets can cause yet more congestion if some type of (flow) control is not used (to inform the source that it's pointless to keep sending more packets at the present time)

Jitter

Flow Control and Congestion Flow Control and Congestion Limits the transmission amount (rate) because the receiver cannot process packets at the same rate that packets are arriving. Limit the amount sent (transmission rate) because of loss or delays in the circuit.

Controls in TCP IP (Internet Protocol) implements service that not connection oriented. - There is no mechanism in IP to deal with packet loss. TCP (Transmission Control Protocol) implements flow and congestion control. - Only on the ends as the intermediate nodes at the network level do not talk TCP

Congestion vs. Flow in TCP Congestion vs. Flow in TCP Flow: controlled by window size (RcvWindow), which is sent by the receiving end. Congestion: controlled by the value of the congestion window (Congwin) Maintained independently by the sender This varies based on the detection of packets lost - Timeout or receiving three ACKs repeated Behaviors: - Additive Increments / Multiplicative Decrements (AIMD) - Slow Start - React to timeout events

Different TCP Congestion Control Algorithms

Questions? ?

Local Analysis Questions? As we know. Before we blame the network, let's verify whether the problem is ours. What can go wrong locally? Hardware problems Excessive load (CPU, memory, I/O) What's considered 'normal'? Use analysis tools frequently Become familiar with the normal state and values for your machine. It is essential to maintain history SNMP agents and databases

Local Analysis Linux Performance Analysis Three main categories: Processes Memory Processes that are executing (running) Processes that are waiting (sleeping) waiting their turn blocked Real Virtual I/O (Input/Output) Storage Network

Key Indicators Insufficent CPU Number of processes waiting to execute is always high High CPU utilization (load avg.) Insufficient memory Very little free memory Lots of swap activity (swap in, swap out) Slow I/O Lots of blocked processes High number of block transfers

Local Analysis Luckily, in Unix there are dozens of useful tools that give us lots of useful information about our machine Some of the more well-known include: vmstat top lsof netstat - tcpdump - wireshark (ethereal) - iptraf - iperf

vmstat Show periodic summary information about processes, memory, pagin, I/O, CPU state, etc vmstat -options delay count # vmstat 2 # vmstat 2 procs -----------memory------------swap-- -----io---- --system-- ----cpu---- ----cpu---rprocs b -----------memory---------swpd free buff cache ---swap-si so -----io---bi bo --system-in cs us sy id wa r b swpd free buff cache si so bi bo in cs us sy 73 id wa 2 0 209648 25552 571332 2804876 0 0 3 4 3 3 15 11 0 209648 24680 25552571332 5713322804900 2804876 2 2 0 0209648 00 00 0 3 444 4 273 379356 3161516116873 0 0 209648 25216 24680571336 5713322804904 2804900 444 439 27346735 79356161610167468 0 0 1 2 0 0209648 00 00 6 0 1234 209648 25212 25216571336 5713362804904 2804904 439100282 46735 17 16 21 10 62 74 00 1 1 0 0209648 00 00 0 6 1234 22 159 1 0 209648 25212 571336 2804904 0 0 0 22 159 100282 17 21 2 0 209648 25196 571348 2804912 0 0 0 500 270 82455 14 18 68 620 0 2 0 209648 25196 571348 2804912 0 0 0 500 243 27077480 82455161415186968 0 0 1 0 209648 25192 571348 2804912 0 0 0 272 209648 25880 25192571360 5713482804916 2804912 272 255 24383619 77480161614156969 0 0 2 1 0 0209648 00 00 0 0 444 2 0 209648 25880 571360 2804916 0 0 0 444 255 83619161618146669 0 0 2 0 209648 25872 571360 2804920 0 0 0 178 220 90521 2 0 209648 25872 571360 2804920 0 0 0 178 220 90521 16 18 66 0

top Basic performance tool for Unix/Linux environments Periodically show a list of system performance statistics: CPU use RAM and SWAP memory usage Load average (cpu utilization) Information by process

top cont. Information by process (most relevant columns shown): PID: Process ID USER: user running (owner) of the process %CPU: Percentage of CPU utilization by the process since the last sample %MEM: Percentage of physical memory (RAM) used by the process TIME: Total CPU time used by the process since it was started

Load Average Average number of active processes in the last 1, 5 and 15 minutes A simple yet useful measurement Depending on the machine the acceptable range considered to be normal can vary: Multi-processor machines can handle more active processes per unit of time (than single processor machines)

top Some useful interactive keyboard commands for top f F , u k d,s : Add or remove columns : Specify which column to order by : Move the column on which we order : Specify a specific user : Specify a process to kill (stop) : Change the display update interval

netstat Show us information about: Network connections Routing tables Interface (NIC) statistics Multicast group members

netstat Some useful options -n: Show addresses, ports and userids in numeric form -r: Routing table -s: Statistics by protocol -i: Status of interfaces -l: Listening sockets --tcp, --udp: Specify the protocol -A: Address family [inet inet6 unix etc.] -p: Show the name of each process for each port -c: Show output/results continuously

netstat Examples: # netstat -n --tcp -c # netstat -n --tcp -cservers) Active Internet connections (w/o Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Proto Recv-Q Send-Q Local Address tcp 0 272 ::ffff:192.188.51.40:22 tcp 272 ::ffff:192.188.51.40:22 tcp 00 0 ::ffff:192.188.51.40:22 tcp 0 0 ::ffff:192.188.51.40:22 Foreign Address State Foreign Address State ::ffff:128.223.60.27:60968 ESTABLISHED ::ffff:128.223.60.27:60968 ESTABLISHED ESTABLISHED ::ffff:128.223.60.27:53219 ::ffff:128.223.60.27:53219 ESTABLISHED # netstat -lnp --tcp # netstat -lnp --tcp Active Internet connections (only servers) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Proto Recv-Q Send-Q Local Address tcp 0 0 0.0.0.0:199 tcp 0.0.0.0:199 tcp 00 0 00.0.0.0:3306 tcp 0 0 0.0.0.0:3306 Foreign Address Foreign Address 0.0.0.0:* 0.0.0.0:* 0.0.0.0:* 0.0.0.0:* State State LISTEN LISTEN LISTEN LISTEN PID/Program name PID/Program name 11645/snmpd 11645/snmpd 1997/mysqld 1997/mysqld # netstat -ic # netstat -ic table Kernel Interface Kernel Interface Iface MTU MettableRX-OK RX-ERR RX-DRP RX-OVR Iface MTU Met RX-OK RX-ERR eth0 1500 0 2155901 0 RX-DRP 0 RX-OVR 0 1500 0 0 2155901 loeth0 16436 18200 00 00 00 lo 16436 0 18200 0 0 eth0 1500 0 2155905 0 0 00 1500 0 0 2155905 loeth0 16436 18200 00 00 00 lo 16436 0 18200 0 0 eth0 1500 0 2155907 0 0 00 eth0 1500 0 2155907 0 0 lo 16436 0 18200 0 0 00 lo 16436 0 0 2155910 18200 eth0 1500 00 00 00 eth0 1500 0 2155910 0 0 lo 16436 0 18200 0 0 00 lo 16436 0 0 2155913 18200 eth0 1500 00 00 00 eth0 1500 0 2155913 0 0 0 TX-OK TX-ERR TX-DRP TX-OVR Flg TX-OK TX-ERR Flg 339116 0 TX-DRP 0 TX-OVR 0 BMRU 339116 BMRU 18200 00 00 0 0LRU 18200 0 0 0 LRU 339117 0 0 0 BMRU 339117 BMRU 18200 00 00 0 0LRU 18200 0 0 0 LRU 339120 0 0 0 BMRU 339120 0 0 0 BMRU 18200 0 0 0 LRU 18200 LRU 339122 00 00 0 0BMRU 339122 0 0 0 BMRU 18200 0 0 0 LRU 18200 LRU 339124 00 00 0 0BMRU 339124 0 0 0 BMRU

netstat cont. Examples: # netstat –tcp –listening --program # netstat –tcp –listening --program Active Internet connections (only servers) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Proto Recv-Q Send-Q Local Address tcp 0 0 *:5001 tcp *:5001 tcp 00 0 0localhost:mysql tcp 0 0 localhost:mysql tcp 0 0 *:www tcp 0 0 *:www tcp 0 0 t60-2.local:domain tcp t60-2.local:domain tcp 00 0 0t60-2.local:domain tcp 0 0 t60-2.local:domain tcp 0 0 t60-2.local:domain tcp 0 0 t60-2.local:domain tcp 0 0 localhost:domain tcp localhost:domain tcp 00 0 0localhost:ipp tcp 0 0 localhost:ipp tcp 0 0 localhost:smtp tcp 0 0 localhost:smtp tcp 0 0 localhost:953 tcp 0 0 localhost:953 tcp 0 0 *:https tcp *:https tcp6 00 0 0[::]:ftp tcp6 0 0 [::]:ftp tcp6 0 0 [::]:domain tcp6 0 0 [::]:domain tcp6 0 0 [::]:ssh tcp6 [::]:ssh tcp6 00 0 0[::]:3000 tcp6 0 0 [::]:3000 tcp6 0 0 ip6-localhost:953 tcp6 0 0 ip6-localhost:953 tcp6 0 0 [::]:3005 tcp6 0 0 [::]:3005 Foreign Address Foreign Address *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* *:* [::]:* [::]:* [::]:* [::]:* [::]:* [::]:* [::]:* [::]:* [::]:* [::]:* [::]:* [::]:* State State LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN PID/Program name PID/Program name 13598/iperf 13598/iperf 5586/mysqld 5586/mysqld 7246/apache2 7246/apache2 5378/named 5378/named 5378/named 5378/named 5378/named 5378/named 5378/named 5378/named 5522/cupsd 5522/cupsd 6772/exim4 6772/exim4 5378/named 5378/named 7246/apache2 7246/apache2 7185/proftpd 7185/proftpd 5378/named 5378/named 5427/sshd 5427/sshd 17644/ntop 17644/ntop 5378/named 5378/named 17644/ntop 17644/ntop

netstat cont. sudo netstat -atup sudo netstat -atup Active Internet connections (servers and established) (if run as root PID/Program name is included) Active Internet connections (servers and established) (if run as root PID/Program name is included) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 *:35586 *:* LISTEN 2540/ekpd tcp 0 0 *:35586 *:* LISTEN 2540/ekpd tcp 0 0 localhost:mysql *:* LISTEN 2776/mysqld tcp localhost:mysql *:* LISTEN 2776/mysqld tcp 00 0 0*:www *:* LISTEN 14743/apache2 tcp 0 0 *:www *:* LISTEN 14743/apache2 tcp 0 0 d229-231.uoregon:domain *:* LISTEN 2616/named tcp 0 0 d229-231.uoregon:domain *:* LISTEN 2616/named tcp 0 0 *:ftp *:* LISTEN 3408/vsftpd tcp *:ftp *:* LISTEN 3408/vsftpd tcp 00 0 0localhost:domain *:* LISTEN 2616/named tcp 0 0 localhost:domain *:* LISTEN 2616/named tcp 0 0 *:ssh *:* LISTEN 2675/sshd tcp 0 0 *:ssh *:* LISTEN 2675/sshd tcp 0 0 localhost:ipp *:* LISTEN 3853/cupsd tcp 0 0 localhost:ipp *:* LISTEN 3853/cupsd tcp 0 0 localhost:smtp *:* LISTEN 3225/exim4 tcp localhost:smtp *:* LISTEN 3225/exim4 tcp 00 0 0localhost:953 *:* LISTEN 2616/named tcp 0 0 localhost:953 *:* LISTEN 2616/named tcp 0 0 *:https *:* LISTEN 14743/apache2 tcp 0 0 *:https *:* LISTEN 14743/apache2 tcp6 0 0 [::]:domain [::]:* LISTEN 2616/named tcp6 [::]:domain [::]:* LISTEN 2616/named tcp6 00 0 0[::]:ssh [::]:* LISTEN 2675/sshd tcp6 0 0 [::]:ssh [::]:* LISTEN 2675/sshd tcp6 0 0 ip6-localhost:953 [::]:* LISTEN 2616/named tcp6 0 0 ip6-localhost:953 [::]:* LISTEN 2616/named udp 0 0 *:50842 *:* 3828/avahi-daemon: udp *:50842 *:* 3828/avahi-daemon: udp 00 0 0localhost:snmp *:* 3368/snmpd udp 0 0 localhost:snmp *:* 3368/snmpd udp 0 0 d229-231.uoregon:domain *:* 2616/named udp 0 0 d229-231.uoregon:domain *:* 2616/named udp 0 0 localhost:domain *:* 2616/named udp 0 0 localhost:domain *:* 2616/named udp 0 0 *:bootpc *:* 13237/dhclient udp *:bootpc *:* 13237/dhclient udp 00 0 0*:mdns *:* 3828/avahi-daemon: udp 0 0 *:mdns *:* 3828/avahi-daemon: udp 0 0 d229-231.uoregon.ed:ntp *:* 3555/ntpd udp 0 0 d229-231.uoregon.ed:ntp *:* 3555/ntpd udp 0 0 localhost:ntp *:* 3555/ntpd udp localhost:ntp *:* 3555/ntpd udp 00 0 0*:ntp *:* 3555/ntpd udp 0 0 *:ntp *:* 3555/ntpd udp6 0 0 [::]:domain [::]:* 2616/named udp6 0 0 [::]:domain [::]:* 2616/named udp6 0 0 fe80::213:2ff:fe1f::ntp [::]:* 3555/ntpd udp6 fe80::213:2ff:fe1f::ntp[::]:* [::]:* 3555/ntpd udp6 00 0 0ip6-localhost:ntp 3555/ntpd udp6 0 0 ip6-localhost:ntp [::]:* 3555/ntpd udp6 0 0 [::]:ntp [::]:* 3555/ntpd udp6 0 0 [::]:ntp [::]:* 3555/ntpd

lsof (LiSt of Open Files) lsof is particularly useful because in Unix everything is a file: unix sockets, ip sockets, directories, etc. Allows you to associate open files by: -p: PID (Process ID) -i : A network address (protocol:port) -u: A user

lsof Example: First, using netstat -ln –tcp determine that port 6010 is open and waiting for a connection (LISTEN) ##netstat -ln --tcp netstat -ln --tcp Active Internet connections (only servers) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Proto Recv-Q Send-Q Local Address tcp tcp 0 tcp tcp 0 Foreign Address Foreign Address State State 0 0 127.0.0.1:6010 0 127.0.0.1:6010 0.0.0.0:* 0.0.0.0:* LISTEN LISTEN 0 0 127.0.0.1:6011 0 127.0.0.1:6011 0.0.0.0:* 0.0.0.0:* LISTEN LISTEN

lsof Determine what process has the port (6010) open and what other resources are being used: # lsof -i tcp:6010 # lsof PID -i tcp:6010 COMMAND USER FD COMMAND 10301 PID root USER sshd sshd sshd sshd 10301root root 10301 10301 root # lsof -p 10301 # lsof -p COMMAND PID10301 USER COMMAND10301 PIDroot USER sshd sshd 10301 root sshd 10301 root sshd 10301root root sshd 10301 sshd 10301 root sshd 10301 root sshd 10301 root sshd 10301 root sshd 10301root root sshd 10301 sshd 10301 root sshd 10301 root sshd 10301root root sshd 10301 sshd 10301 root sshd 10301 root sshd 10301 root sshd 10301 root sshd 10301root root sshd 10301 sshd 10301 root sshd 10301 root sshd 10301root root sshd 10301 sshd 10301 root . . TYPE DEVICE SIZE NODE NAME FD TYPE53603 DEVICE SIZE NAME 6u IPv4 TCPNODE localhost.localdomain:x11-ssh-offset (LISTEN) IPv4 53604 53603 7u6u IPv6 7u IPv6 53604 FD cwdFD cwd rtd rtd txt txt mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem TYPE TYPE DIR DIR DIR DIR REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG TCP[::1]:x11-ssh-offset localhost.localdomain:x11-ssh-offset (LISTEN) TCP (LISTEN) TCP [::1]:x11-ssh-offset (LISTEN) DEVICE SIZE NODE NAME DEVICE SIZE NODE 8,2 4096 2 / NAME 8,2 4096 8,2 4096 2 2/ / 8,2 379720 40961422643 2/usr/sbin/sshd / 8,2 8,2 379720 1422643 /usr/sbin/sshd 8,2 32724 1437533 /usr/lib/libwrap.so.0.7.6 8,2 32724 1437533 /usr/lib/libwrap.so.0.7.6 8,2 15088 3080329 /lib/libutil-2.4.so 8,2 75632 150881414093 3080329/usr/lib/libz.so.1.2.3 /lib/libutil-2.4.so 8,2 8,2 75632 1414093 /usr/lib/libz.so.1.2.3 8,2 96040 3080209 /lib/libnsl-2.4.so 8,2 100208 960401414578 3080209/usr/lib/libgssapi krb5.so.2.2 /lib/libnsl-2.4.so 8,2 8,2 100208 1414578 /usr/lib/libgssapi krb5.so.2.2 8,2 11684 1414405 /usr/lib/libkrb5support.so.0.0 8,2 11684 1414405 /usr/lib/libkrb5support.so.0.0 8,2 10368 3080358 /lib/libsetrans.so.0 8,2 103683080231 3080358/lib/libcom err.so.2.1 /lib/libsetrans.so.0 8,2 7972 8,2 7972 3080231 /lib/libcom err.so.2.1 8,2 30140 1420233 /usr/lib/libcrack.so.2.8.0 8,2 11168 301403080399 1420233/lib/security/pam succeed if.so /usr/lib/libcrack.so.2.8.0 8,2 8,2 11168 3080399 /lib/security/pam succeed if.so

lsof cont. What network services am I running? # lsof -i # lsof -iPID COMMAND USER FD COMMAND PID USER 50u FD firefox 4429 hervey firefox 4429 hervey 128.223.60.21:www (ESTABLISHED 50u 128.223.60.21:www (ESTABLISHED named 5378 bind 20u named 5378 bind 21u 20u named 5378 bind named 5378 bind 21u sshd 5427 root 3u sshd 5427 root cupsd 5522 root 3u3u cupsd 5522 root mysqld 5586 mysql 10u3u mysqld 5586 mysql 10u snmpd 6477 snmp 8u snmpd 6477Debian-exim snmp exim4 6772 3u8u exim4 6772 Debian-exim ntpd 6859 ntp 16u3u ntpd 6859 ntp 16u ntpd 6859 ntp 17u ntpd 6859 ntp 17u ntpd 6859 ntp 18u ntpd 6859 ntp 19u 18u ntpd 6859 ntp ntpd 6859 ntp 19u proftpd 7185 proftpd 1u proftpd 7185 proftpd apache2 7246 www-data 3u1u apache2 7246 www-data apache2 7246 www-data 4u3u apache2 7246 www-data 4u . . iperf 13598 root 3u iperf 13598 root apache2 27088 www-data 3u3u apache2 27088 27088 www-data apache2 www-data 4u3u apache2 27088 www-data 4u TYPE DEVICE SIZE NODE NAME TYPE1875852 DEVICE SIZE TCP NODE192.168.179.139:56890NAME IPv4 IPv4 1875852 TCP 192.168.179.139:56890IPv6 13264 IPv6 13267 13264 IPv4 IPv4 13267 IPv6 13302 IPv61983466 13302 IPv4 IPv4 1983466 IPv4 13548 IPv4 14633 13548 IPv4 IPv4 14675 14633 IPv4 IPv4 14675 IPv4 14743 IPv4 14743 IPv6 14744 IPv6 14744 IPv6 14746 IPv6 14747 14746 IPv6 IPv6 14747 IPv6 15718 IPv6 15718 IPv4 15915 IPv4 15917 15915 IPv4 IPv4 15917 TCP *:domain (LISTEN) TCPlocalhost:domain *:domain (LISTEN) TCP (LISTEN) TCP localhost:domain (LISTEN) TCP *:ssh (LISTEN) TCPlocalhost:ipp *:ssh (LISTEN) TCP (LISTEN) TCP localhost:ipp (LISTEN) TCP localhost:mysql (LISTEN) TCPlocalhost:snmp localhost:mysql (LISTEN) UDP UDPlocalhost:smtp localhost:snmp(LISTEN) TCP TCP localhost:smtp (LISTEN) UDP *:ntp UDP *:ntp UDP *:ntp UDP *:ntp UDP [fe80::250:56ff:fec0:8]:ntp UDPip6-localhost:ntp [fe80::250:56ff:fec0:8]:ntp UDP UDP ip6-localhost:ntp TCP *:ftp (LISTEN) TCP *:ftp(LISTEN) (LISTEN) TCP *:www TCP*:https *:www (LISTEN) TCP (LISTEN) TCP *:https (LISTEN) IPv4 1996053 IPv4 1996053 IPv4 15915 IPv4 15917 15915 IPv4 IPv4 15917 TCP *:5001 (LISTEN) TCP*:www *:5001 (LISTEN) TCP (LISTEN) TCP*:https *:www (LISTEN) TCP (LISTEN) TCP *:https (LISTEN)

tcpdump Show received packet headers by a given interface. Optionally filter using boolean expressions. Allows you to write information to a file for later analysis. Requires administrator (root) privileges to use since you must configure network interfaces (NICs) to be in “promiscuous” mode.

tcpdump Some useful options: -i : -l : Specify the interface (ex: -i eth0) Make stdout line buffered (view as you capture) -v, -vv, -vvv: Display more information -n : Don't convert addresses to names (avoid DNS) -nn : Don't translate port numbers -w : Write raw packets to a file -r : Read packets from a file created by '-w'

tcpdump Boolean expressions: Using the 'AND', 'OR', 'NOT' operators Expressions consist of one, or more, primtives, which consist of a qualifier and an ID (name or number): Expression :: [NOT] primitive [ AND OR NOT primitive .] primitive :: qualifier name number qualifier :: type address protocol type :: host net port port range address :: src dst protocol :: ether fddi tr wlan ip ip6 arp rarp decnet tcp udp

tcpdump Examples: Show all HTTP traffic that originates from 192.168.1.1 # tcpdump -lnXvvv port 80 and src host 192.168.1.1 # tcpdump -lnXvvv port 80 and src host 192.168.1.1 - Show all traffic originating from 192.168.1.1 except SSH # tcpdump -lnXvvv src host 192.168.1.1 and not port 22 # tcpdump -lnXvvv src host 192.168.1.1 and not port 22

Wireshark Wireshark is a graphical packet analyser based on libpcap, the same library that tcpdump utilizes for capturing and storing packets The graphical interface has some advantages, including: Hierarchical visualization by protocol (drill-down) Follow a TCP “conversation” (Follow TCP Stream) Colors to distinguish traffic types Lots of statistics, graphs, etc.

Wireshark Wireshark is what came after Ethereal. The combination of tcpdump and wireshark can be quite powerful. For example: # tcpdump -i eth1 -A -s1500 -2 dump.log port 21 sudo wireshark -r dump.log

Wireshark

iptraf Many measurable statistics and functions By protocol/port By packet size Generates logs Utilizes DNS to translate addresses Advantages Simplicity Menu-based (uses “curses”) Flexible configuration

iptraf You can run it periodically in the background (-B) It allows you, for example, to run as a cron job to periodically analyze logs. Generate alarms Save in a data base Has a great name. “Interactive Colorful IP LAN Monitor” etc. Example: iptraf -i eth1

iptraf –i eth0 Sample iptraf output from the above command:

iperf To measure network throughput between two points iperf has two modes, server and client Easy to use Great to help determine optimal TCP parameters TCP window size (socket buffer) MTU maximum segment size See man iperf for more

iperf Using UDP you can generate packet loss and jitter reports You can run multiple parallel sessions using threads Supports IPv6

iperf parameters Usage: iperf [-s -c host] [options] Usage: iperf [-s -c[-v --version] host] [options] iperf [-h --help] iperf [-h --help] [-v --version] Client/Server: Client/Server: -f, --format [kmKM] format to report: Kbits, Mbits, KBytes, MBytes --format# [kmKM] format to report: Mbits,reports KBytes, MBytes -i, -f, --interval seconds between periodicKbits, bandwidth --interval # seconds between periodic bandwidth -l, -i, --len #[KM] length of buffer to read or write (defaultreports 8 KB) -l,--print mss --len #[KM] length of buffer to readsegment or write (default 8 KB) -m, print TCP maximum size (MTU - TCP/IP header) --print mss TCP maximum segment size (MTU - TCP/IP header) -p,-m, --port # serverprint port to listen on/connect to --port # use server to listen to -u,-p, --udp UDPport rather thanon/connect TCP --udp UDPwindow rather than -w,-u, --window #[KM]useTCP size TCP (socket buffer size) --window #[KM] TCP window size (socket buffer size) -B,-w, --bind host bind to host , an interface or multicast address --bind host forbind to host , an interface multicast address -C,-B, --compatibility use with older versions doesornot sent extra msgs --compatibility usemaximum with older segment versions does not sent msgs -M,-C, --mss # setfor TCP size (MTU - 40extra bytes) --mss # setset TCP segment sizeAlgorithm (MTU - 40 bytes) -N,-M, --nodelay TCP nomaximum delay, disabling Nagle's --nodelay set no delay,todisabling Nagle's Algorithm -V,-N, --IPv6Version SetTCP the domain IPv6 -V, --IPv6Version Set the domain to IPv6 Server specific: Server specific: run in server mode -s, --server --server run in insingle serverthreaded mode UDP mode -U,-s, --single udp run --single udp runrun singleas threaded UDP mode -D,-U, --daemon theinserver a daemon -D, --daemon run the server as a daemon Client specific: Client specific: #[KM] for UDP, bandwidth to send at in bits/sec -b, --bandwidth -b, --bandwidth #[KM]1 Mbit/sec, for UDP,implies bandwidth (default -u) to send at in bits/sec (default implies -u) -c, --client host run 1 inMbit/sec, client mode, connecting to host --client host in client mode, connecting to host -d,-c, --dualtest Do arun bidirectional test simultaneously --dualtest Do a bidirectional test simultaneously -n,-d, --num #[KM] number of bytes to transmit (instead of -t) --num #[KM] of bytes transmit (instead of -t) -r,-n, --tradeoff Do a number bidirectional test to individually --tradeoff Do in a bidirectional individually -t,-r, --time # time seconds to test transmit for (default 10 secs) --time name # time in seconds transmit for (default 10 secs) -F,-t,--fileinput input the data totobe transmitted from a file --fileinput name data to be transmitted from a file -I, -F, --stdin input theinput datathe to be transmitted from stdin --stdin input the data tobidirectional be transmitted stdin -L,-I,--listenport # port to recieve testsfrom back on --listenport port to of recieve bidirectional tests back on -P,-L, --parallel # # number parallel client threads to run #time-to-live, number parallel (default client threads to run -T,-P, --ttl--parallel # forofmulticast 1) -T, --ttl # time-to-live, for multicast (default 1)

iperf - TCP iperf -s iperf -s ----------------------------------------------------------------------------------------------------------------------Server listening on TCP port 5001 Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) TCP window size: 85.3 KByte (default) ----------------------------------------------------------------------------------------------------------------------[ 4] local 128.223.157.19 port 5001 connected with 201.249.107.39 port 39601 [ 4] local 128.223.157.19 port 5001 connected with 201.249.107.39 port 39601 [ 4] 0.0-11.9 sec 608 KBytes 419 Kbits/sec [ 4] 0.0-11.9 sec 608 KBytes 419 Kbits/sec ----------------------------------------------------------------------------------------------------------------------# iperf -c nsrc.org # iperf -c nsrc.org ----------------------------------------------------------------------------------------------------------------------Client connecting to nsrc.org, TCP port 5001 Client connecting to nsrc.org, TCP port 5001 TCP window size: 16.0 KByte (default) TCP window size: 16.0 KByte (default) ----------------------------------------------------------------------------------------------------------------------[ 3] local 192.168.1.170 port 39601 connected with 128.223.157.19 port 5001 [ 3] local 192.168.1.170 port 39601 connected with 128.223.157.19 port 5001 [ 3] 0.0-10.3 sec 608 KBytes 485 Kbits/sec [ 3] 0.0-10.3 sec 608 KBytes 485 Kbits/sec

iperf - UDP # iperf -c host1 -u -b100M # iperf -c host1 -u -b100M ------------------------------------------------------------ -----------------------------------------------------------Client connecting to nsdb, UDP port 5001 Client connecting to nsdb, UDP port 5001 Sending 1470 byte datagrams Sending 1470 byte 106 datagrams UDP buffer size: KByte (default) UDP buffer size: 106 KByte (default) -----------------------------------------------------------[ -----------------------------------------------------------3] local 128.223.60.27 port 39606 connected with 128.223.250.135 port 5001 [ 3] 0.0-10.0 local 128.223.60.27 port 39606 connected with 128.223.250.135 port 5001 [ 3] sec 114 MBytes 95.7 Mbits/sec 3]Sent 0.0-10.0 sec 114 MBytes 95.7 Mbits/sec [ [ 3] 81377 datagrams 3]Server Sent 81377 datagrams [ [ 3] Report: [ 3] Server Report: [ 3] 0.0-10.0 sec 114 MBytes 95.7 Mbits/sec 0.184 ms 1/81378 (0.0012%) [ 3] 0.0-10.0 sec 114 MBytes 95.7 Mbits/sec 0.184 ms 1/81378 (0.0012%) iperf -s -u -i 1 iperf -s -u -i 1 ------------------------------------------------------------ -----------------------------------------------------------Server listening on UDP port 5001 Server listening ondatagrams UDP port 5001 Receiving 1470 byte Receiving 1470 byte datagrams UDP buffer size: 108 KByte (default) UDP buffer size: 108 KByte (default) ----------------------------------------------------------------------------------------------------------------------[ 3] local 128.223.250.135 port 5001 connected with 128.223.60.27 port 39606 3] 0.0local1.0 128.223.250.135 port 95.4 5001 Mbits/sec connected with 128.223.60.27 port 39606 [ [ 3] sec 11.4 MBytes 0.184 ms 0/ 8112 (0%) 3] 1.00.0-2.0 1.0sec sec 11.4 11.4MBytes MBytes 95.7 95.4Mbits/sec Mbits/sec 0.177 0.184ms ms 0/8141 8112(0%) (0%) [ [ 3] 0/ [ 3] 1.02.0 sec 11.4 MBytes 95.7 Mbits/sec 0.177 ms 0/ 8141 (0%) [ 3] 2.0- 3.0 sec 11.4 MBytes 95.6 Mbits/sec 0.182 ms 0/ 8133 (0%) [ 3] 2.0- 3.0 sec 11.4 MBytes 95.6 Mbits/sec 0.182 ms 0/ 8133 (0%) . [ . 3] 8.0- 9.0 sec 11.4 MBytes 95.7 Mbits/sec 0.177 ms 0/ 8139 (0%) [ 3] 8.09.0 sec 11.4 MBytes 95.7 Mbits/sec 0.177 ms 0/8137 8139(0%) (0%) [ 3] 9.0-10.0 sec 11.4 MBytes 95.7 Mbits/sec 0.180 ms 0/ 3] 0.0-10.0 9.0-10.0sec sec 11.4 MBytes 95.7 95.7 Mbits/sec Mbits/sec 0.184 0.180 ms ms 0/ 8137 (0.0012%) (0%) [ [ 3] 114 MBytes 1/81378 [ 3] 0.0-10.0 sec 114 MBytes 95.7 Mbits/sec 0.184 ms 1/81378 (0.0012%)

Bibliography Monitoring Virtual Memory with vmstat http://www.linuxjournal.com/article/8178 How to use TCPDump http://www.erg.abdn.ac.uk/users/alastair/tcpdump.html linux command tcpdump example http://smartproteam.com/linux-tutorials/linux-command-tcpdump/ simple usage of tcpdump http://linux.byexamples.com/archives/283/simple-usage-of-tcpdump/ TCPDUMP Command man page with examples http://www.cyberciti.biz/howto/question/man/tcpdump-man-page-with-examples.php TCPDump Tutorial http://inst.eecs.berkeley.edu/ ee122/fa06/projects/tcpdump-6up.pdf