Archive

Posts Tagged ‘network’

Auto-Discovery with NMAP

June 26th, 2017 Comments off

Discover

I created this script in order to use it as a feed for enterprise management tools such as Nagios.   It’s a bit difficult to get a handle on auto-discovery tools within an enterprise management tool when it discovers what can be an overwhelming number of hosts.  NMAP is smart enough to translate the MAC address into a vendor if it can.

This script called discover uses the NMAP tool on Linux (tested on Centos).


#! /bin/bash
#
# Name: discover
#
# Garland Joseph, garland.joseph@gmail.com
# Date: June 2017
#
# Auto-discover on subnet using nmap, can be fed into something like nagios
# as a seed file after proper formatting.
#
# ----

if [[ -z ${1} ]]
then
cat <<EOD
$0 <subnet>
where
subnet, by example, is something like 192.168.1.0/24
EOD
exit
fi

nmap -sn ${1} | awk '
BEGIN{ printf("%-16s| %-18s| %-35s| %-30s\n","IP","MAC","NAME","VENDOR") }
/^Nmap scan report/ {
NAME=$5
x=$NF
gsub("[()]","",x)
IP=x
}

/^MAC Address/ {
MAC=$3
split($0,a,"(")
split(a[2],b,")")
VENDOR=b[1]
printf("%-16s| %-18s| %-35s| %-30s\n",IP,MAC,NAME,VENDOR)
}'

Here is an example of the output

[root@localhost ~]# ./discover 192.168.1.0/24
IP            | MAC               | NAME                               | VENDOR
192.168.1.1   | C8:D7:19:DE:54:2E | NyaRaePrimary                      | Cisco Consumer Products
192.168.1.100 | B8:27:EB:72:2A:4A | kodi1.grandenetworks.net           | Raspberry Pi Foundation
192.168.1.101 | 00:1F:3B:75:7F:EB | 192.168.1.101                      | Intel Corporate
192.168.1.102 | 6C:3B:E5:76:96:A5 | HP-Printer.grandenetworks.net      | Hewlett Packard
192.168.1.105 | A8:47:4A:AC:8F:89 | 192.168.1.105                      | Unknown
192.168.1.106 | F0:7D:68:0A:7C:8A | OllieMaeJoseph.grandenetworks.net  | D-Link
192.168.1.107 | 58:82:A8:81:C3:A6 | XboxOne                            | Unknown
192.168.1.109 | 28:56:5A:39:ED:FD | BRW28565A39EDFD.grandenetworks.net | Unknown
192.168.1.110 | 64:20:0C:90:24:D9 | Garlands-iPad.grandenetworks.net   | Apple
192.168.1.112 | 7C:D1:C3:17:0C:58 | Apple-TV.grandenetworks.net        | Apple
192.168.1.115 | B8:27:EB:12:DA:AC | kodi2.grandenetworks.net           | Raspberry Pi Foundation
192.168.1.118 | A4:77:33:8E:CE:C2 | Chromecast.grandenetworks.net      | Google
192.168.1.119 | 6C:AD:F8:5D:3A:D6 | 192.168.1.119                      | Azurewave Technologies
192.168.1.131 | F0:7D:68:0A:7A:D5 | EmmaEdwards.grandenetworks.net     | D-Link
192.168.1.135 | 28:10:7B:0C:3A:71 | EarlEdwards.grandenetworks.net     | D-Link International
192.168.1.136 | 00:09:B0:D6:A8:2A | 192.168.1.136                      | Onkyo
192.168.1.138 | 28:10:7B:0C:3A:74 | LeeJoseph.grandenetworks.net       | D-Link International
192.168.1.145 | D4:3D:7E:EF:93:99 | obama                              | Micro-Star Int'l Co
192.168.1.147 | E4:3E:D7:44:21:8F | LGwebOSTV.grandenetworks.net       | Unknown

Here is a Diagram of my home LAN as revealed by the Network Status Map in Nagios

Categories: Networking Tags: , , , , , ,

u2ucap: circular trace, stop when event detected

May 1st, 2017 Comments off

This script is part of a series of scripts that perform packet capture between two endpoints.  In this case the endpoints are two UNIX/LINUX machines.  The script was tested with the “source endpoint” as a Redhat machine and a “target endpoint” as an Ubuntu machine.  The circular traces are started on each machine and stopped whenever an event is detected.  In this case the event is to monitor a file for a particular string.

Requirements: tcpdump

# Author: Garland R. Joseph, garland.joseph@gmail.com
#   Date: May 2017
#
# This script is offered as is.  It is designed to
# run a circular trace using tcpdump on UNIX systems.
#
# You will either have to manually enter the password
# for the root account on the remote system or setup
# ssh keys from prompt-less access.
#
# Default file size is 2 x 1 GB per endpoint.
#
# The traces will stop once a key string SEARCH_STRING is
# found in LOG_FILE.
#
# Note:   Some UNIX systems line LINUX Fedora will
#         result in permsission denied when using
#         tcpdump  -W and -C options and writing to / or /root.
#
#         Tested on Source endpoint Redhat and Target endpoint Ubuntu.
# -----

#
# Defaults
#

USAGE="u2ucap [-v]  [ -c capture_file ] [ -w secs ] -h remote_host -l log_file -s search_string"
DEBUG=false
CAPTURE_FILE="/tmp/capture"
SLEEP_TIME="5"   #seconds
SEARCH_STRING=""
LOG_FILE=""
REMOTE_HOST=""
TCPDUMPCMD="tcpdump -C 1 -W 2 -w ${CAPTURE_FILE}"

#
# Process command line arguments
#

while getopts ":vc:w:l:s:h:" opts
do
 case ${opts} in
   v) DEBUG=true ;;
   c) CAPTURE_FILE=${OPTARG} ;;
   w) SLEEP_TIME=${OPTARG} ;;
   s) SEARCH_STRING=${OPTARG}  ;;
   l) LOG_FILE=${OPTARG} ;;
   h) REMOTE_HOST=${OPTARG} ;;
   ":") echo "Please specify a value for ${OPTARG}" ; exit ;;
  \?) echo "${OPTARG} is not a valid switch" ; echo "${USAGE}" ; exit;;
 esac
done

#
# Insure required values have been specified, check for existence of
# log file, getops should handle case of no values for -l and -s.
# A sanity check in the event getopts varies per unix
#

if [[ -z ${SEARCH_STRING} || -z ${LOG_FILE} || -z ${REMOTE_HOST} ]]
then
 echo ${USAGE}
 exit
fi

if ! [[ -f ${LOG_FILE} ]]
then
 echo "File ${LOG_FILE} does not exist"
 exit
fi

#
# Start trace on remote host
#

ssh root@${REMOTE_HOST} ${TCPDUMPCMD} 2>/dev/null 1>/dev/null &
REMOTE_PID=`ssh root@${REMOTE_HOST} ps -aef | egrep tcpdump | egrep -v grep | awk '{print $2}'`
${DEBUG} && echo "${0}-I-REMOTE_PID, remote pid is ${REMOTE_PID}."
#
# Start trace on this host
#
#${TCPDUMPCMD} 2>/dev/null 1>/dev/null &
exec ${TCPDUMPCMD} 2>/dev/null 1>/dev/null &
LOCAL_PID=$!
${DEBUG} && echo "${0}-I-LOCAL_PID, local pid is ${LOCAL_PID}."

#
# Monitor log file
#

old_count=`grep -c ${SEARCH_STRING} ${LOG_FILE}`
(( new_count=old_count ))
(( i = 0 ))
while (( old_count == new_count ))
do
   (( i++ ))
   ${DEBUG} && echo "${0}-F-SLEEP, sleeping ${SLEEP_TIME}, iteration ${i}."
   sleep ${SLEEP_TIME}
   new_count=`grep -c ${SEARCH_STRING} ${LOG_FILE}`
done

#
#  At this point, search string has been found, stop traces
#

###${DEBUG} && set -x
kill ${LOCAL_PID}
ssh root@${REMOTE_HOST} kill ${REMOTE_PID}
####${DEBUG} && set +x

#
# Verbose only reminders
#

####${DEBUG} && echo "Consult files ${CAPTURE_FILE} on local and remote host."

exit

Anatomy of a Home Network

June 20th, 2016 Comments off

I wrote this document for my father in 2001 in order to explain some of the more important elements in his home network. This also helps him whenever he calls into the Time Warner support line. Technical support is problematic due to the number of groups responsible for support and the division of these groups involved in managing the collection of network resources (DNS servers, gateways, etc).

The problem is exacerbated when these support groups don’t have clear communication paths allowing for systematic problem determination and resolution (How many times have you called into support and have to enter your customer information each time you are transferred to a different group and repeat the problem?).

Understanding your network and the resources you use, can help you direct support groups and bypass the overhead that comes with talking to different people. In addition you may be able to provide yourself a backup plan for a particular resource. For example, if the DNS servers go down and you know the IP address of a particular FTP file site you use, you can enter the IP address and still perform the file transfer.

Basic Physical/Logical Relationship

The left side shows a pseudo-logical breakdown and the right side shows my father’s workstation setup.

Physical/Logical Setup

 

Gateways and Network

The following diagrams illustrate the relationship between the physical devices (Linksys switch and Toshiba DSL Cable modem)  in relationship to the networks.  The critical points in the following diagrams are the network characteristics of eah network and in paticular, the various gateways and IP addresses that control transmission between networks.  The diagrams are trivialized.

                            

Break-down of the Time-Warner Network and Network Resources

The Time-Warner network consists of a number of subnets.  The table that follows reveal critical network resources (i.e. DHCP server) within the Time-Warner network.

Network Resources

Network Resources Summarized:
Network ResourceIP AddressMAC AddressFunctionNetwork
Primary DNS Server24.93.35.32unknownDomain Name Services; Resolves hostnames to IPs and vice-versa.Time Warner
Secondary DNS Server24.93.35.33unknownDomain Name Services; Resolves hostnames to IPs and vice-versa.Time Warner
Gateway to Time Warner's Closest Subnet66.68.112.100-30-7B-FF-48-54GatewayTime Warner
Toshiba DSL Cable ModemN/A--pass-thru00-04-75-3F-CF-04Bridge from ISDN to EthernetTime Warner
Mail Server (mail.rr.com)24.30.218.18unknownMail smtp and pop servicesTime Warner
Linksys WAN Port66.68.116.96*00-04-5A-DB-74-68Interface to Time Warner's closest subnetTime Warner
Linksys LAN Port 192.168.1.100-04-5A-DB-74-67GatewayjosephNET

Useful Network Commands

The following table shows useful network commands in understanding the various networks.

CommandCommand OptionsResults/Impact
arp -a-a = display arp tableShows the MAC addresses w/ corresponding IP for network entities (on same subnet); helps to to resolve duplicate IP addresses
ping Where is something like 24.93.35.33.Tells you if you have connectivity to a host. If the host is on the same subnet, an the MAC address of the host is cached in the arp cache.
netstatwith no optionsDisplays all active connections, ip addresses, and port information
netstat -a-a = display all connections and listening ports, hostnames are includedList of ports and connections. Use this to see if a particular client connection is actually made; for example, if you see port 80, this means a web server connection; port 443 would be an SSL connection...port corresponds to applications.
netstat -an-n = display all connections and listening ports, ipaddress are includedList of ports and connections. Use this to see if a particular client connection is actually made; for example, if you see port 80, this means a web server connection; port 443 would be an SSL connection...port corresponds to applications. -n means DNS is not used to look up IP addresses, so this command is faster than just -an
netstat -e-e = display Ethernet statisticsUse this for performance issues
netstat -es-e = Ethernet stats
-s = display by protocol
Use this for performance issues.
netstat -r-r = routing tabledisplay routing table
route printequivalent to netstat -rdisplay routing table
ipconfigN/Adisplay adapter IP configuration

Example: Using ping -r to Understand the Network

Here is an example of using ping -r to decipher the network.

ping -r example

 

Categories: Networking Tags: , ,

Network Socket Table

June 2nd, 2016 No comments

 

Preface

The purpose of this document is to give a list of commands that can be executed from the command line during networking or system problem determination. These commands focus on parsing the network socket table (namely: netstat -a). The socket table is arguably a common denominator for any system (regardless of operating system) that participates in the TCP/IP protocol.

Both UNIX/LINUX and Windows (Powershell) commands are given. It is important to understand the context of the command before entering it; more so than constructing the command line syntax. Once you understand the context, you can simply cut and paste the command.

The ability to quickly parse the socket table allows the administrator to be responsive during times of crisis.

What follows is a scenario that shows:

· the value of the quick-use of the socket table

· insight into the left-side and the right-side of the table

· a summary of the context, and finally

· a list of commands in both UNIX/LINUX and Powershell.

Click Here for and Index of UNIX and Powershell Commands


Scenario

It was my first day on the job. I just moved from Austin, Texas to Greensboro, North Carolina to work for a large financial institution. I was fresh from a stable IBM support environment. I was excited to tackle real world issues again.

The environment was a distributed computing environment featuring different operating systems: OS400, AIX, Linux and Windows. A few dedicated AIX systems were also used as part of the network infrastructure… different from the traditional roles for AIX systems: like DB2, Oracle, WebSphere, etc. This was before F5/BigIP devices…. dedicated to load balancing, Data Power devices that bridge multi-level protocols at wire speed, or Cisco’s defining support of vlans in VMware environments. In other words, using AIX or LINUX operating systems to perform functions that have now migrated to “network devices”.

There were 2-twin AIX systems that were used as proxy servers (IBM-B50, 2U-rack mounted units). The network engineer said they were load-balanced.

So…I logged into both machines to for look and after a few minutes I said “well, I don’t think they are load balanced”. The network engineer asked, “what do you mean?”. I said, “well, I only see 3 connections on this partner, and about 1000 connections on the other partner.” These load-balanced proxy servers had been running for about two years.

I simply logged on to both systems and did

netstat -an | egrep <PROXY-SERVER-PORT> | wc -l

Same company, about a week later…this time there was a distributed application hanging on the AIX side. I traced the problem down to a connection to an OS400 machine (via the socket table). I asked the Application team to check the application processes on the OS400 system and they told me that the OS400 machine was not relevant. It wasn’t until I was in a room with the application folks explaining the socket table that they finally looked at the OS400 and found the issue. I pointed to the IP address of the OS400 appearing in the socket table.

So, I love the socket table. It’s one of the first areas I consider whenever I walk into a new environment. It’s a system administrator’s quick insight into applications in a distributed environment. Any host system that participates in the TCP/IP protocol is going to have this table.


The Left-side versus the Right-Side

I think of the socket table as having a left side and a right side. I frequently “slice” (using the Unix cut or awk command) the table into two and then parse the output. The left side will show the local IP addresses of the system (as well as the loop back). The right side shows the IP addresses of machines connecting to this machine.

The following diagram shows a search for ESTABLISHED connections…

image

The following shows “slicing” the table to evaluate remote connection information.

image

In this case our local host has 10 connections to port 5201 on host 192.168.1.101.   I ran the iperf tool in server mode on host 192.168.1.101 and started the client on the local host requesting 10 connections (the iperf tool is used to test network bandwidth).   We also have one connection to port 22 on 192.168.1.101.  Port 22 is the ssh server. This means I have ssh’d into machine 192.168.1.101 from the local host.

 

Context

I always parse the socket table to get a sense of what a server is doing anytime I am called into any system issue, especially when I am looking at a server for the first time. If it’s an issue with an application that works over the network, I always track down the port associated with the application. Here is a partial list of things I look for when parsing the socket table and the context:

Item Context
Ports in LISTEN state This gives an idea of how many server types of applications on the host.
Ports not in LISTEN state This gives an idea of current connection from remote hosts.
Ports in some sort of CLOSE state for a long time. Repetitive use of this command and counting them will show if some connections are hanging in close state.
Ports in SYN state for a long time Repetitive use of this command and counting them will show if some connections are hanging in SYN state.
Ports in WAIT states for a long time Repetitive use of this command and counting them will show if some connections are hanging in WAIT state.
Number of Ports accessing an application Server Port Repetitive use of this command and filtering about a specific port will show the application server load network-wise.
Number of unique hosts connecting to this server Cutting the table in half (vertically, column-wise) and looking at the right side of the socket table reveals this information.  It helps to pipe this output to sort and a count tool.
A list of connections that don’t have 0 in send and receive queues (UNIX only). Repetitive use of this command and observing if the send/receive queues are non-zero can indicate an application is not processing.

 

The Commands


1. What tcp sockets are open?

UNIX

netstat -an | egrep -I “^tcp.*LISTEN”

1a

netstat -an | egrep -I “^tcp.*LISTEN” | awk ‘{split($4,a,”:”);print a[2]}’

1b


Windows (Powershell)

netstat -an | select-string “LISTEN”

image

netstat -an | select-string “LISTEN” | %{$i=%{$_.Line.Split(“:”)};$j=$i[1] -replace ‘\s+’,’ ‘;$k=$j.Split(” “);$k[0]} | where {$_ -ne “”}

image


2. What TCP ports are in an ESTABLISHED state

UNIX

netstat -an | egrep ESTABLISH

2a

netstat -an | egrep ESTABLISH | awk ‘{split($4,a,”:”);print a[2]}’

2b

Windows (Powershell)

netstat -an | select-string “ESTABLISHED” | %{$i=%{$_.Line.Split(“:”)};$j=$i[1] -replace ‘\s+’,’ ‘;$k=$j.Split(” “);$k[0]} | where {$_ -ne “”}

image


3. What TCP ports are in some sort of Wait State?

UNIX

nestat –an | egrep WAIT

3a

 

Windows (Powershell)

netstat -an | select-string “WAIT”

image


4. How many connections are in some sort of WAIT state?

UNIX

netstat -an | egrep WAIT | wc -l

4a

Windows (Powershell)

netstat -an | select-string “WAIT” | where {$_ -ne “”} | Measure-Object –Line

image


5. What are the IP address and ports of the remote machines that are in some sort of WAIT state to this machine?

UNIX

nestat -an | egrep WAIT | awk ‘{print $4}’

5a

Windows (Powershell)

netstat -an | select-string “WAIT” | %{$_ -replace ‘\s+’,’ ‘ } | %{$i=$_.Split(” “);$i[3]}

image


6. What are the IP address (no ports) of the remote machines that are in some sort of WAIT state to this machine?

UNIX

netstat -an | egrep WAIT | awk ‘{print($4,a,”:”);print a[1]}’

6a

Windows (Powershell)

netstat -an | select-string “WAIT” | %{$_ -replace ‘\s+’,’ ‘ } | %{$i=$_.Split(” “);$i[3]} | %{$i=$_.Split(“:”);$i[0]}

image


7. What are the unique IP address of remote connections in some sort of WAIT state?

UNIX

netstat -an | egrep  WAIT | awk ‘{split($4,a,”:”);print a[1]}’ | sort -u

7a

Windows (Powershell)

netstat -an | select-string “WAIT” | %{$_ -replace ‘\s+’,’ ‘ } | %{$i=$_.Split(” “);$i[3]} | %{$i=$_.Split(“:”);$i[0]} | Sort-Object | Get-Unique

image