Archive for June, 2016

Anatomy of a Home Network

June 20th, 2016 Comments off

I wrote this document for my father in 2001 in order to explain some of the more important elements in his home network. This also helps him whenever he calls into the Time Warner support line. Technical support is problematic due to the number of groups responsible for support and the division of these groups involved in managing the collection of network resources (DNS servers, gateways, etc).

The problem is exacerbated when these support groups don’t have clear communication paths allowing for systematic problem determination and resolution (How many times have you called into support and have to enter your customer information each time you are transferred to a different group and repeat the problem?).

Understanding your network and the resources you use, can help you direct support groups and bypass the overhead that comes with talking to different people. In addition you may be able to provide yourself a backup plan for a particular resource. For example, if the DNS servers go down and you know the IP address of a particular FTP file site you use, you can enter the IP address and still perform the file transfer.

Basic Physical/Logical Relationship

The left side shows a pseudo-logical breakdown and the right side shows my father’s workstation setup.

Physical/Logical Setup


Gateways and Network

The following diagrams illustrate the relationship between the physical devices (Linksys switch and Toshiba DSL Cable modem)  in relationship to the networks.  The critical points in the following diagrams are the network characteristics of eah network and in paticular, the various gateways and IP addresses that control transmission between networks.  The diagrams are trivialized.


Break-down of the Time-Warner Network and Network Resources

The Time-Warner network consists of a number of subnets.  The table that follows reveal critical network resources (i.e. DHCP server) within the Time-Warner network.

Network Resources

Network Resources Summarized:
Network ResourceIP AddressMAC AddressFunctionNetwork
Primary DNS Server24.93.35.32unknownDomain Name Services; Resolves hostnames to IPs and vice-versa.Time Warner
Secondary DNS Server24.93.35.33unknownDomain Name Services; Resolves hostnames to IPs and vice-versa.Time Warner
Gateway to Time Warner's Closest Subnet66.68.112.100-30-7B-FF-48-54GatewayTime Warner
Toshiba DSL Cable ModemN/A--pass-thru00-04-75-3F-CF-04Bridge from ISDN to EthernetTime Warner
Mail Server ( smtp and pop servicesTime Warner
Linksys WAN Port66.68.116.96*00-04-5A-DB-74-68Interface to Time Warner's closest subnetTime Warner
Linksys LAN Port

Useful Network Commands

The following table shows useful network commands in understanding the various networks.

CommandCommand OptionsResults/Impact
arp -a-a = display arp tableShows the MAC addresses w/ corresponding IP for network entities (on same subnet); helps to to resolve duplicate IP addresses
ping Where is something like you if you have connectivity to a host. If the host is on the same subnet, an the MAC address of the host is cached in the arp cache.
netstatwith no optionsDisplays all active connections, ip addresses, and port information
netstat -a-a = display all connections and listening ports, hostnames are includedList of ports and connections. Use this to see if a particular client connection is actually made; for example, if you see port 80, this means a web server connection; port 443 would be an SSL connection...port corresponds to applications.
netstat -an-n = display all connections and listening ports, ipaddress are includedList of ports and connections. Use this to see if a particular client connection is actually made; for example, if you see port 80, this means a web server connection; port 443 would be an SSL connection...port corresponds to applications. -n means DNS is not used to look up IP addresses, so this command is faster than just -an
netstat -e-e = display Ethernet statisticsUse this for performance issues
netstat -es-e = Ethernet stats
-s = display by protocol
Use this for performance issues.
netstat -r-r = routing tabledisplay routing table
route printequivalent to netstat -rdisplay routing table
ipconfigN/Adisplay adapter IP configuration

Example: Using ping -r to Understand the Network

Here is an example of using ping -r to decipher the network.

ping -r example


Categories: Networking Tags: , ,

Network Socket Table

June 2nd, 2016 No comments



The purpose of this document is to give a list of commands that can be executed from the command line during networking or system problem determination. These commands focus on parsing the network socket table (namely: netstat -a). The socket table is arguably a common denominator for any system (regardless of operating system) that participates in the TCP/IP protocol.

Both UNIX/LINUX and Windows (Powershell) commands are given. It is important to understand the context of the command before entering it; more so than constructing the command line syntax. Once you understand the context, you can simply cut and paste the command.

The ability to quickly parse the socket table allows the administrator to be responsive during times of crisis.

What follows is a scenario that shows:

· the value of the quick-use of the socket table

· insight into the left-side and the right-side of the table

· a summary of the context, and finally

· a list of commands in both UNIX/LINUX and Powershell.

Click Here for and Index of UNIX and Powershell Commands


It was my first day on the job. I just moved from Austin, Texas to Greensboro, North Carolina to work for a large financial institution. I was fresh from a stable IBM support environment. I was excited to tackle real world issues again.

The environment was a distributed computing environment featuring different operating systems: OS400, AIX, Linux and Windows. A few dedicated AIX systems were also used as part of the network infrastructure… different from the traditional roles for AIX systems: like DB2, Oracle, WebSphere, etc. This was before F5/BigIP devices…. dedicated to load balancing, Data Power devices that bridge multi-level protocols at wire speed, or Cisco’s defining support of vlans in VMware environments. In other words, using AIX or LINUX operating systems to perform functions that have now migrated to “network devices”.

There were 2-twin AIX systems that were used as proxy servers (IBM-B50, 2U-rack mounted units). The network engineer said they were load-balanced.

So…I logged into both machines to for look and after a few minutes I said “well, I don’t think they are load balanced”. The network engineer asked, “what do you mean?”. I said, “well, I only see 3 connections on this partner, and about 1000 connections on the other partner.” These load-balanced proxy servers had been running for about two years.

I simply logged on to both systems and did

netstat -an | egrep <PROXY-SERVER-PORT> | wc -l

Same company, about a week later…this time there was a distributed application hanging on the AIX side. I traced the problem down to a connection to an OS400 machine (via the socket table). I asked the Application team to check the application processes on the OS400 system and they told me that the OS400 machine was not relevant. It wasn’t until I was in a room with the application folks explaining the socket table that they finally looked at the OS400 and found the issue. I pointed to the IP address of the OS400 appearing in the socket table.

So, I love the socket table. It’s one of the first areas I consider whenever I walk into a new environment. It’s a system administrator’s quick insight into applications in a distributed environment. Any host system that participates in the TCP/IP protocol is going to have this table.

The Left-side versus the Right-Side

I think of the socket table as having a left side and a right side. I frequently “slice” (using the Unix cut or awk command) the table into two and then parse the output. The left side will show the local IP addresses of the system (as well as the loop back). The right side shows the IP addresses of machines connecting to this machine.

The following diagram shows a search for ESTABLISHED connections…


The following shows “slicing” the table to evaluate remote connection information.


In this case our local host has 10 connections to port 5201 on host   I ran the iperf tool in server mode on host and started the client on the local host requesting 10 connections (the iperf tool is used to test network bandwidth).   We also have one connection to port 22 on  Port 22 is the ssh server. This means I have ssh’d into machine from the local host.



I always parse the socket table to get a sense of what a server is doing anytime I am called into any system issue, especially when I am looking at a server for the first time. If it’s an issue with an application that works over the network, I always track down the port associated with the application. Here is a partial list of things I look for when parsing the socket table and the context:

Item Context
Ports in LISTEN state This gives an idea of how many server types of applications on the host.
Ports not in LISTEN state This gives an idea of current connection from remote hosts.
Ports in some sort of CLOSE state for a long time. Repetitive use of this command and counting them will show if some connections are hanging in close state.
Ports in SYN state for a long time Repetitive use of this command and counting them will show if some connections are hanging in SYN state.
Ports in WAIT states for a long time Repetitive use of this command and counting them will show if some connections are hanging in WAIT state.
Number of Ports accessing an application Server Port Repetitive use of this command and filtering about a specific port will show the application server load network-wise.
Number of unique hosts connecting to this server Cutting the table in half (vertically, column-wise) and looking at the right side of the socket table reveals this information.  It helps to pipe this output to sort and a count tool.
A list of connections that don’t have 0 in send and receive queues (UNIX only). Repetitive use of this command and observing if the send/receive queues are non-zero can indicate an application is not processing.


The Commands

1. What tcp sockets are open?


netstat -an | egrep -I “^tcp.*LISTEN”


netstat -an | egrep -I “^tcp.*LISTEN” | awk ‘{split($4,a,”:”);print a[2]}’


Windows (Powershell)

netstat -an | select-string “LISTEN”


netstat -an | select-string “LISTEN” | %{$i=%{$_.Line.Split(“:”)};$j=$i[1] -replace ‘\s+’,’ ‘;$k=$j.Split(” “);$k[0]} | where {$_ -ne “”}


2. What TCP ports are in an ESTABLISHED state


netstat -an | egrep ESTABLISH


netstat -an | egrep ESTABLISH | awk ‘{split($4,a,”:”);print a[2]}’


Windows (Powershell)

netstat -an | select-string “ESTABLISHED” | %{$i=%{$_.Line.Split(“:”)};$j=$i[1] -replace ‘\s+’,’ ‘;$k=$j.Split(” “);$k[0]} | where {$_ -ne “”}


3. What TCP ports are in some sort of Wait State?


nestat –an | egrep WAIT



Windows (Powershell)

netstat -an | select-string “WAIT”


4. How many connections are in some sort of WAIT state?


netstat -an | egrep WAIT | wc -l


Windows (Powershell)

netstat -an | select-string “WAIT” | where {$_ -ne “”} | Measure-Object –Line


5. What are the IP address and ports of the remote machines that are in some sort of WAIT state to this machine?


nestat -an | egrep WAIT | awk ‘{print $4}’


Windows (Powershell)

netstat -an | select-string “WAIT” | %{$_ -replace ‘\s+’,’ ‘ } | %{$i=$_.Split(” “);$i[3]}


6. What are the IP address (no ports) of the remote machines that are in some sort of WAIT state to this machine?


netstat -an | egrep WAIT | awk ‘{print($4,a,”:”);print a[1]}’


Windows (Powershell)

netstat -an | select-string “WAIT” | %{$_ -replace ‘\s+’,’ ‘ } | %{$i=$_.Split(” “);$i[3]} | %{$i=$_.Split(“:”);$i[0]}


7. What are the unique IP address of remote connections in some sort of WAIT state?


netstat -an | egrep  WAIT | awk ‘{split($4,a,”:”);print a[1]}’ | sort -u


Windows (Powershell)

netstat -an | select-string “WAIT” | %{$_ -replace ‘\s+’,’ ‘ } | %{$i=$_.Split(” “);$i[3]} | %{$i=$_.Split(“:”);$i[0]} | Sort-Object | Get-Unique