Troubleshooting Tool: Using the FortiOS built-in packet sniffer

Article

Introduction Sniffer Basics Basic sniffing command Example 1 : Simple trace Example 2 : Simple trace Filter Functionality Example 3 : Trace with filters Introduction All FortiGate units have a powerful packet sniffer on board. If you know tcpdump you should feel comfortable using the FortiGate Sniffer. See the related article "Packet capture (sniffer) tips" for additional sniffer tips. Scope : All FortiOS Note :   Other Fortinet appliances also providing a CLI sniffer : FortiAnalyzer - FortiMail - FortiManager                       DMZ                        |                        |                   +-----------+   ----internal----| FortiGate |---external-----                   +-----------+ Sniffer Basics The packet sniffer "sits" in the FortiGate and can sniff traffic on a specific Interface or on all Interfaces. There are 3 different Level of Information, also known as Verbose Levels 1 to 3, where verbose 1 shows less information and verbose 3 shows the most information. Verbose 4, 5 and 6 would additionally provide the interface details Verbose levels in detail: 1: print header of packets 2: print header and data from IP of packets 3: print header and data from Ethernet of packets 4: print header of packets with interface name 5: print header and data from IP of packets with interface name 6: print header and data from Ethernet of packets with interface nameThis article walks through some examples and different levels of verbosity to show the different possibilities for debugging. Basic sniffing command All Packet sniffing commands start like: # diag sniffer packet <interface> <'filter'> <verbose> <count> a Where... <interface> can be an Interface name or "any" for all Interfaces. <'filter'> is a very powerful filter functionality which will be described in more detail.<verbose> means the level of verbosity as described already. <count> the number of packets the sniffer reads before stopping. a introduced in release 3.0 MR6, this setting allows display of absolute time stamp Example 1: Simple Trace Sniff 3 packets of all traffic with verbose Level 4 on internal Interface # diag sniffer packet internal none 4 3 internal in 192.168.0.1.22 -> 192.168.0.30.1144: psh 2859918764 ack 1949135261internal in 192.168.0.1.22 -> 192.168.0.30.1144: psh 2859918816 ack 1949135261internal out 192.168.0.30.1144 -> 192.168.0.1.22: ack 2859918884 As you can see we caught some Packets in the middle of a communication. Because the 192.168.0.1 IP Address uses Port 22 (192.168.0.1.22) we can assume that we've caught some Packets from a running SSH Session. The "none" variable means 'no filter applies', "4" means 'verbose 4' and "3" means 'catch 3 packets and stop'. Example 2: Simple Trace Sniff 3 packets of all traffic with verbose Level 4 on Internal interface # diag sniffer packet internal none 4 3 internal out 192.168.0.30.1156 -> 192.168.0.1.80: syn 2164883624internal in 192.168.0.1.80 -> 192.168.0.30.1156: syn 3792179542 ack 2164883625internal out 192.168.0.30.1156 -> 192.168.0.1.80: ack 3792179543 Apparently we caught some more interesting information, just when a TCP session was being set up. 192.168.0.30 tries to connect to 192.168.0.1 on Port 80 with a syn and gets a syn ack back. Finally the session is acknowledged and established after the 3-way TCP handshake. With information level set to Verbose 4, we see a summary of Source and Destination IP Address, as well as Source and Destination Port. We can also see the corresponding TCP Sequence numbers. If you don't enter a <count> value, the Sniffer runs forever until you stop it with <CTRL C> Hint: For further investigation it's always a good idea to log to a file. If you're using Putty (a free SSH client for Windows) you can easily log all Output to a file which you can search/sort/process. Verbose 5 and Verbose 6 levels: Verbose 5 contains much more information 1. The IP Header as we've already seen in Verbose 4 2. The Payload of the IP packet itself An Output of Verbose 5 looks like this: # diag sniffer packet internal none 5 1 internal in 192.168.0.1.22 -> 192.168.0.30.1144: psh 2867817048 ack 1951061933 0x0000 4510 005c 8eb1 4000 4006 2a6b c0a8 0001 E..\..@.@.*k.... 0x0010 c0a8 001e 0016 0478 aaef 6a58 744a d7ad .......x..jXtJ.. 0x0020 5018 0b5c 8ab9 0000 9819 880b f465 62a8 P..\.........eb. 0x0030 3eaf 3804 3fee 2555 8deb 24da dd0d c684 >.8.?.%U..$..... 0x0040 08a9 7907 202d 5898 a85c facb 8c0a f9e5 ..y..-X..\...... 0x0050 bd9c b649 5318 7fc5 c415 5a59 ...IS.....ZY Notice the in/ out parameter after internal interface that will confirm the direction of the packet entering or leaving the interface. Verbose 6, finally, even includes Ethernet (Ether Frame) Information. A script is available (fgt2eth.pl), which will convert a captured verbose 6 output, into a file that can be read and decoded by Ethereal/Wireshark. See the end of this article for details. Use of absolute time stamp in sniffer trace will report the absolute system time (no time zone) in packet summary: # diag sniffer packet internal none 4 2 a 2010-06-02 10:23:17.170751 port1 out arp who-has 192.168.1.110 tell 192.168.1.103 2010-06-02 10:23:19.077409 port1 in arp who-has 192.168.1.120 tell 192.168.1.2 Hint: Below is the format that Technical Support will usually request when attempting to analyze a problem as it includes full packet content, as well as absolute time stamp, in order to correlate packets with other system events. # diag sniffer packet any <'filter'> 6 0 a Filter Functionality As already mentioned: diag sniffer includes a powerful filter functionality that will be described here. FortiOS tells us: <filter> filter for sniffer Syntax: '[[src|dst] host<IP1>] [[src|dst] host<IP2>] [[arp|ip|gre|esp|udp|tcp] [port_no]] [[arp|ip|gre|esp|udp|tcp] [port_no]]' If a second host is specified, only the traffic between the 2 hosts will be displayed. <filter> flexible logical filters for sniffer (or "none"). For example: To print udp 1812 traffic between forti1 and either forti2 or forti3 'udp and port 1812 and host forti1 and (forti2 or forti3)' Imagine you only want to sniff the traffic from one PC to another PC. Without Filter the sniffer will display all packets which is far too much and painful to debug. Example 3: Trace with Filters To see what's going on between two PCs (or a PC and a FortiGate),(Don't forget to put your filter expressions in single quotes ' ' ): # diag sniffer packet internal 'src host 192.168.0.130 and dst host 192.168.0.1' 1 192.168.0.130.3426 -> 192.168.0.1.80: syn 1325244087 192.168.0.130.3426 -> 192.168.0.1.80: ack 3483111190 192.168.0.130.3426 -> 192.168.0.1.80: psh 1325244088 ack 3483111190 192.168.0.130.1035 -> 192.168.0.1.53: udp 26 192.168.0.130.1035 -> 192.168.0.1.53: udp 42 192.168.0.130.1035 -> 192.168.0.1.53: udp 42 192.168.0.130 -> 192.168.0.1: icmp: echo request 192.168.0.130.3426 -> 192.168.0.1.80: psh 1325244686 ack 3483111190 192.168.0.130 -> 192.168.0.1: icmp: echo request Assuming there is a lot of traffic on the wire, this filter command will only display traffic (but all traffic) from Source 192.168.0.130 to Destination 192.168.0.1. It will NOT show traffic to 192.168.0.130 (for example the ICMP reply) because we said ' src host 192.168.0.130 and dst host 192.168.0.1' As you can see we also captured some other things like ICMP or DNS queries from a PC. If we're just interested in a specific type of traffic (let's say TCP Traffic only) we need to change our filter command slightly like this: # diag sniffer packet internal 'src host 192.168.0.130 and dst host 192.168.0.1 and tcp' 1 192.168.0.130.3569 -> 192.168.0.1.23: syn 1802541497 192.168.0.1.23 -> 192.168.0.130.3569: syn 4238146022 ack 1802541498 192.168.0.130.3569 -> 192.168.0.1.23: ack 4238146023 Though ICMP (ping) was also running, the trace only shows the TCP part. As we can see the Destination is: 192.168.0.1.23 which is IP 192.168.0.1 on Port 23. Apparently we found a Telnet Session to 192.168.0.1 right during initial setup. The same the other way around: # diag sniffer packet internal 'host 192.168.0.130 and icmp' 1 192.168.0.130 -> 192.168.0.1: icmp: echo request 192.168.0.1 -> 192.168.0.130: icmp: echo reply In this example we're sniffing for ICMP only, to and from 192.168.0.130 Another useful feature is logical combination. Let us assume you want to sniff for ICMP and TCP only (but not for UDP, ARP, etc). You can combine protocols in the following manner: # diag sniffer packet internal 'host 192.168.0.130 and (icmp or tcp)' 1 This sniff will display all tcp or icmp traffic to and from host 192.168.0.30, in verbose 1 level. Now we are going to limit the sniffer even more: We want to sniff traffic between 2 hosts, but only TCP and only port 80. # diag sniffer packet internal 'host 192.168.0.130 and 192.168.0.1 and tcp port 80' 1 192.168.0.130.3625 -> 192.168.0.1.80: syn 2057246590 192.168.0.1.80 -> 192.168.0.130.3625: syn 3291168205 ack 2057246591 192.168.0.130.3625 -> 192.168.0.1.80: ack 3291168206 192.168.0.130.3625 -> 192.168.0.1.80: psh 2057246591 ack 3291168206 192.168.0.1.80 -> 192.168.0.130.3625: ack 2057247265 A logical "and" is used in this command between 192.168.0.130 and 192.168.0.1 so that only packets containing both these host addresses will be seen. Even if telnet and ssh is running between the two hosts, we only see port 80 TCP traffic. Filtered can be used to display packets based on their content, using hexadecimal byte position. Match TTL = 1 # diagnose sniffer packet port2 "ip[8:1] = 0x01" Match Source IP address = 192.168.1.2: # diagnose sniffer packet internal "(ether[26:4]=0xc0a80102)" Match Source MAC = 00:09:0f:89:10:ea # diagnose sniffer packet internal "(ether[6:4]=0x00090f89) and (ether[10:2]=0x10ea)" Match Destination MAC = 00:09:0f:89:10:ea # diagnose sniffer packet internal "(ether[0:4]=0x00090f89) and (ether[4:2]=0x10ea)" Match ARP packets only # diagnose sniffer packet internal "ether proto 0x0806" TCP or UDP flags can be addressed using the following: Match packets with RST flag set: # diagnose sniffer packet internal "tcp[13] & 4 != 0" Match packets with SYN flag set: # diagnose sniffer packet internal "tcp[13] & 2 != 0" Match packets with SYN-ACK flag set: # diagnose sniffer packet internal "tcp[13] = 18" Also attached is the fgt2eth.pl script that will convert a verbose level 3 or 6 sniffer output, into a file readable and decodable by Ethereal/Wireshark. The fgt2eth.exe file is also attached to this article, this file is outdated and is not supported but may provide some guidance. Note: The attached script is provided "as is", it is not supported by Technical Support. $ ./fgt2eth.pl  Version : Dec 19 2014 Usage : fgt2eth.pl -in <input_file_name> Mandatory argument are :     -in  <input_file>     Specify the file to convert (FGT verbose 3 text file) Optional arguments are :     -help                 Display help only     -version              Display script version and date     -out <output_file>    Specify the output file (Ethereal readable)                 By default <input_file>.pcap is used                 - will start wireshark for realtime follow-up     -lines <lines>        Only convert the first <lines> lines     -demux            Create one pcap file per interface (verbose 6 only)     -debug                Turns on debug mode