Table of Contents

Troubleshooting Internetwork Performance

Sporadic Service Availability and Poor AppleTalk Internetwork Performance
Poor Bridging Performance over Serial Lines or LANs
Poor DECnet Performance over Serial Lines or LANs
Slow Performance and Intermittent Loss of Connections over RSRB
Slow Performance over ISO CLNS
Poor Novell Server Performance over Router in an IPX LAN Internetwork
Poor Novell Server Performance over Router in a WAN
Slow Performance in TCP/IP Internetworks
Slow TCP/IP Performance Despite Multiple Paths
Load Balancing Problem Example
Slow Host or Network Response over a WAN or Serial Link
Dropped Connections over a WAN or Serial Link
Poor XNS Server Performance over Router in a LAN Internetwork
Poor XNS Server Performance over Router in a WAN
Switching-Support Matrices

Troubleshooting Internetwork Performance

This chapter focuses on common symptoms associated with poor performance in internetworks, possible causes of those symptoms, and general suggestions for identifying, isolating, and resolving problem causes. The symptom modules in this chapter pertain to performance-related problems for the various protocols and technologies addressed in this publication.

Performance symptoms discussed in this chapter include the following:

• Sporadic Service Availability and Poor AppleTalk Internetwork Performance
  

• Poor Bridging Performance over Serial Lines or LANs
  

• Poor DECnet Performance over Serial Lines or LANs
  

• Slow Performance and Intermittent Loss of Connections over RSRB
  

• Slow Performance over ISO CLNS
  

• Poor Novell Server Performance over Router in an IPX LAN Internetwork
  

• Poor Novell Server Performance over Router in a WAN
  

• Slow Performance in TCP/IP Internetworks
  

• Slow TCP/IP Performance Despite Multiple Paths
  

• Slow Host or Network Response over a WAN or Serial Link
  

• Dropped Connections over a WAN or Serial Link
  

• Poor XNS Server Performance over Router in a LAN Internetwork
  

• Poor XNS Server Performance over Router in a WAN
  

• Switching-Support Matrices
  

Sporadic Service Availability and Poor AppleTalk Internetwork Performance

Symptom: Connectivity to AppleTalk services over an internetwork is unpredictable and generally slow. Table 15-1 outlines possible causes and suggested actions when service availability is unpredictable and performance is slow on an AppleTalk internetwork.

Table 15-1 : AppleTalk: Sporadic Service Availability and Poor Performance

Possible Causes	Suggested Actions
ZIP storm	Step 1 Examine the output of the show appletalk traffic EXEC command to determine the number of ZIP requests being sent; repeat after 30 seconds. Step 2 If the number of ZIP requests is greater than 10 and increasing, a ZIP storm is probably occurring. Step 3 Use the show appletalk route EXEC command to see whether there is a network that is described as having "no zone set." If you find such a network, a node in that network is probably not responding to ZIP requests, which results in a ZIP storm. Step 4 Determine why the node is not responding to ZIP requests.
Duplicate network numbers	Step 1 The network that exhibits this symptom is likely to contain duplicate network numbers equidistant from the point at which problems are observed. Either change the network number of the afflicted network or remove AppleTalk from the associated interface. In either case, the original network number associated with the interface should disappear from the internetwork within a few minutes. If it persists, you probably have found the duplicate network. Step 2 If you changed the network number on the interface, no further action is required. If not, change it now (making sure it is unique). Remember to reenter the zone name and any other interface configurations for AppleTalk on that interface.
Unexpected back door	Step 1 Inspect the internetwork for any bridges that may link networks that are routing AppleTalk. Step 2 If any bridges (including routers configured for bridging) are found, set all bridges to forward nonrouted protocols and filter routed protocols. Step 3 Use the show appletalk route and show interfaces EXEC commands to monitor routes and neighbors. Step 4 If networks continue to be associated with the wrong interfaces, consult your router technical support representative for more assistance.

Poor Bridging Performance over Serial Lines or LANs

Symptom: Users complain of slow host response and dropped connections. Bridging traffic itself might not be heavy, but links also might be routing other protocols. Measurable symptoms include input and output drops, increasing 5-minute input/output rates, and unusually high increases in collision counts. Table 15-2 outlines possible causes and suggested actions when bridging performance is poor over WAN links.

Table 15-2 : Bridging: Poor Performance over Serial or WAN Links

Possible Causes	Suggested Actions
Overutilized serial line bandwidth	Step 1 Use the show interfaces EXEC command to determine whether output drops are occurring; check the output for high values in the 5-minute input and output packet rate fields. If any output drops are found, bandwidth is probably insufficient. If the input and output rates indicate values close to the media maximum throughput for the line, the line is saturated. Step 2 Use bridging filters to reduce traffic. Step 3 If local-area transport (LAT) is being bridged and the problem persists, try LAT compression. Step 4 Increase bandwidth or add parallel lines (combined with the bridge-group group circuit number interface configuration command) to increase available bandwidth.
Excessive traffic on LAN	Step 1 Use the show interfaces EXEC command to determine whether output drops are occurring; check the 5-minute input and output packet rate for unusually high values; see if the collision counter is incrementing at a high rate. If any output drops are found, bandwidth is probably insufficient. Collisions and unusually high 5-minute input and output rates also indicate that the media is saturated. Step 2 Segment the network using internetworking devices (either routers or bridges, depending on your network and the situation). Step 3 Implement bridging filters to prevent unnecessary traffic from flooding local segments.
Unstable media, or network device has hardware problem	Step 1 Use the show interfaces EXEC command to determine whether interface resets are occurring or whether transition counters are incrementing. Step 2 Check the physical connections of all suspect devices; check modems for proper attachment and functionality; check for noisy lines; check appliques and other router or bridge hardware. Refer to the media and hardware troubleshooting discussions in the "Troubleshooting Router Startup Problems" and the "Troubleshooting Serial Line Problems" chapters. Step 3 Remove and replace defective network interface cards or other devices.

Poor DECnet Performance over Serial Lines or LANs

Symptom: Users complain of slow host response and dropped connections. Table 15-3 outlines possible causes and suggested actions when DECnet performance is poor over serial lines or LAN.

Table 15-3 : DECnet: Poor Performance over Serial Lines or LANs

Possible Causes	Suggested Actions
Overutilized bandwidth	Step 1 Use the show interfaces EXEC command to determine whether input and output queues are full. Step 2 Use the show decnet traffic EXEC command to determine whether packets are being received and forwarded. If the number of received packets is greater than the number of forwarded packets and if the queues are full, congestion is probably the problem. Step 3 To reduce traffic overhead, increase hello timer or update timers to the same value on all routers in the network. Step 4 Implement the priority queuing and buffer changes described in the section "Slow Host or Network Response over a WAN or Serial Link" later in this chapter.
Network device has hardware problem and is sending out large amounts of incorrect data over LAN	Step 1 Use a LAN traffic monitor to collect information about the amount and type of data transferred from and received by each node in the network for a period of time. Step 2 Compare data from all systems. Step 3 Use a network analyzer (or perform a binary search) to isolate the problem nodes that are generating excessive data. Step 4 Remove and replace defective network interface cards or devices.

Slow Performance and Intermittent Loss of Connections over RSRB

Symptom: Users complain about connection loss at peak traffic periods when trying to connect to resources on the other side of a router configured for remote source-route bridging (RSRB). Table 15-4 outlines a possible cause and suggested actions when performance is slow and connectivity is intermittent over RSRB connections.

Table 15-4 : Bridging: Slow Performance and Intermittent Loss of Connections over RSRB

Possible Cause

Suggested Actions

Busy router; high CPU utilization when using TCP encapsulation

Step 1 Use the show processes EXEC command to determine CPU utilization. Look for CPU utilization higher than 50 percent. High CPU utilization can cause RSRB sessions to time out when TCP encapsulation is used. Step 2 Check the configuration for the local-ack keyword in the source-bridge remote-peer global configuration command. Step 3 If it is missing, add the local-ack keyword. Step 4 Consider using the source-bridge fst-peername global configuration command to implement fast sequenced transport (FST) on the link.

Slow Performance over ISO CLNS

Symptom: Users complain about poor performance and slow response in a large ISO Connectionless Network Service (CLNS) network. Table 15-5 lists possible causes and suggested actions when performance is slow over an ISO CLNS network.

Table 15-5 : ISO CLNS: Slow Performance over ISO CLNS Network

Possible Causes	Suggested Actions
Busy router; duplicate routing updates; congested network	Step 1 Use the show processes EXEC command to see CPU utilization. Look for CPU utilization higher than 50 percent. Step 2 Check the configuration for multiple ISO-Interior Gateway Routing Protocol (IGRP) processes configured on a single interface. Multiple ISO-IGRP processes on a single interface cause different Level 2 routing updates to be sent out on the interface. In a large network, the additional bandwidth of these unnecessary updates can degrade performance. Step 3 In the router configuration, remove all but one IGRP routing process per interface.
Multihomed area is too large; busy router	Step 1 Use the write terminal privileged EXEC command to look for the assignment of multiple area addresses. Step 2 Use the show clns routes and show clns neighbors EXEC commands (for ISO-IGRP) or show isis database EXEC command (for Intermediate System-to-Intermediate System [IS-IS]) to display areas and routes. Step 3 Verify that your network topology does not extend multihomed areas farther than necessary. When an area extends farther than necessary, Level 1 traffic increases. The additional packet processing can degrade performance. Step 4 Reconfigure the areas as required.

Poor Novell Server Performance over Router in an IPX LAN Internetwork

Symptom: Users complain that sessions drop at peak traffic periods when they are trying to connect to resources on the other side of a router configured to route Novell Internetwork Packet Exchange (IPX). Table 15-6 outlines possible causes and suggested actions when a Novell server performs poorly over a router in a Novell IPX LAN internetwork.

Table 15-6 : IPX: Poor Novell Server Performance over Router in an IPX LAN Internetwork

Possible Causes	Suggested Actions
Excessive traffic; collisions causing session drops (Ethernet problem only)	Step 1 Use a protocol analyzer to look for collisions in excess of normal acceptable conditions (varies for specific site). As an alternative, use the show interfaces EXEC command to get a rough estimate of the collision count. Step 2 Examine the protocol analyzer output to determine bandwidth utilization. The protocol analyzer output will provide the most accurate reading of dynamic traffic information. As an alternative, you can run the Novell load monitor command at a server console to get an approximate idea of bandwidth utilization. If bandwidth utilization detected by the analyzer averages 15 to 20 percent or higher, you are likely to have a load-related performance problem. Step 3 If you see that collisions are increasing steadily with a higher-than-expected bandwidth utilization, consider segmenting the network with additional bridges or routers.
Insufficient bandwidth on Token Ring to handle traffic	Step 1 Upgrade from 4- to 16-Mbps Token Ring throughout the network. Step 2 If performance is still inadequate, consider segmenting the network with additional bridges or routers.

Poor Novell Server Performance over Router in a WAN

Symptom: Users complain about sessions dropping at peak traffic periods when trying to connect to resources on the other side of a router configured to route Novell IPX over a WAN or serial link. Table 15-7 outlines a possible cause and suggested actions when a Novell server performs poorly over a router in a Novell IPX WAN internetwork.

Table 15-7 : IPX: Poor Novell Server Performance over Router in an IPX WAN Internetwork

Possible Cause

Suggested Actions

Other protocol dominates CPU time

Step 1 Use the show processes EXEC command to look for large numbers appearing in the Runtime (ms) and Invoked fields for certain protocols. A protocol that has a value that is 10 times or greater than the value indicated for Novell traffic is a likely suspect. When this kind of condition exists, Novell traffic is not getting adequate access to the CPU, and performance is affected. Step 2 Use the show interfaces EXEC command to look for a high level of output drops. Step 3 If you see output drops, and another protocol is dominating CPU time (indicated in the show processes Runtime (ms) field), use priority queuing to force the router to give priority to Novell traffic. More information about priority queuing is provided in the "Troubleshooting Serial Line Problems" chapter. Step 4 If priority queuing does not improve performance, add bandwidth by implementing a higher-speed line or by adding additional lines of the same speed. If you add additional lines, use the ipx maximum-paths global configuration command to specify the number of paths.

Slow Performance in TCP/IP Internetworks

Symptom: TCP/IP internetwork performance is slow, with poor host response, spotty connection service, and generally slow file transfers. Packets might be dropped. Table 15-8 outlines possible causes and suggested actions when performance is slow in a TCP/IP internetwork.

Table 15-8 : TCP/IP: Slow Performance in TCP/IP Internetworks

Possible Causes	Suggested Actions
Bad network link, which results in dropped or lost packets	Step 1 Issue the ping command along entire length of the path to determine where packets are being dropped. Step 2 Perform serial debugging or other media debugging. For media and hardware diagnostic information, refer to the "Troubleshooting Router Startup Problems" chapter. For more specific information about serial debugging, refer to the "Troubleshooting Serial Line Problems" chapter. Step 3 Replace hardware or add bandwidth as necessary.
Access list applied to one link, but not another (when there are multiple paths to a destination)	Step 1 Refer to the section "Slow TCP/IP Performance Despite Multiple Paths" later in this chapter.
Congested link	Step 1 Determine whether the link is indeed congested. For media and hardware diagnostic information, refer to the "Troubleshooting Router Startup Problems" chapter. For more specific information about serial debugging, refer to the "Troubleshooting Serial Line Problems" chapter. Step 2 Apply priority queuing if feasible. Step 3 If you cannot implement priority queuing or if priority queuing does not help, add bandwidth or additional routers.

Slow TCP/IP Performance Despite Multiple Paths

Symptom: Despite multiple paths from one network to another and apparently sufficient bandwidth, performance over the links is poor, and traffic does not appear to be getting through some of the links. Although this can be considered a connectivity problem, it manifests itself as a performance issue. Table 15-9 outlines possible causes and suggested actions when TCP/IP performance is slow despite multiple paths.

Table 15-9 : TCP/IP: Slow TCP/IP Performance Despite Multiple Paths

Possible Causes	Suggested Actions
Misconfigured access lists where multiple paths and one or more access lists block access to one or more routes	Step 1 Use the ping and trace EXEC commands to determine where traffic is stopping. This procedure works best when standard access lists are used. Step 2 If ping or trace packets are stopped along the way, check the specific router for access lists. Step 3 If an access list is found, disable the list and monitor traffic through the router, using the ping and trace EXEC commands. Step 4 If ping and trace packets get through after removing the access list, you might need to add explicit permit statements to the access list to allow the blocked traffic type. If extended access lists are specified, ping and trace packets might get through, even though intended traffic is not getting through. Step 5 If ping packets get through, attach a network analyzer along the path where problems occur to see where the dropped packet type was last seen. The next node is the most likely suspect. Step 6 Remove any access list, use the protocol being blocked, and monitor traffic through the router.
Bad interface or media hardware	Step 1 For media and hardware diagnostic information, refer to the "Troubleshooting Router Startup Problems" chapter. For more information about serial debugging, refer to the "Troubleshooting Serial Line Problems" chapter. Step 2 Perform serial debugging or other media debugging. Step 3 Replace hardware or add bandwidth as necessary.
Load balancing problem (see Figure 15-1, following this table)	Step 1 Use the show interfaces and show ip traffic EXEC commands and ping out to the destination to determine where traffic is being dropped. Step 2 At the point of congestion, relieve traffic problems by adding a router in parallel or by increasing the bandwidth of the link. Step 3 If you cannot add a router or bandwidth, try adjusting the hop count on the congested router. This is done using the offset-list router configuration command to add a hop to a route received from a particular router. Step 4 Use the distance router configuration command to set the administrative distance for a particularly slow route.

Load Balancing Problem Example

Figure 15-1 illustrates a situation where two routes might be equivalent in terms of hop count from Host-D to Far-Net, but due to the level of traffic (associated with Router-B and the Data Center), the route through Router-A is administratively preferred. In this case, both routes look equally good to Host-D, so without any configuration modifications, Host-D can use either RouterA or Router-B to communicate with Far-Net. However, as outlined in the load balancing problem discussion in
Table 15-9, several options are available to force traffic from Host-D (intended for FarNet) to go through Router-A.

Figure 15-1 : Load Balancing Problem Map

Slow Host or Network Response over a WAN or Serial Link

Symptom: As with similar loss of connection problems, users complain about very slow host and network responsiveness at peak traffic periods over a WAN or serial link.

Obtain the following information when troubleshooting load-related connection problems:

• Observe the output of the show interfaces serial EXEC command on both ends of the serial line, and evaluate error counters.
  

• If you see input errors, refer to the section "Evaluating Input Errors" in the "Troubleshooting Serial Line Problems" chapter for details about isolating the sources of input errors.
  

Table 15-10 outlines possible causes and suggested actions when host or network response is slow over a WAN or serial link.

Table 15-10 : WAN: Slow Host or Network Response over a WAN or Serial Link

Possible Causes	Suggested Actions
Noisy serial line	Step 1 Use the show interfaces serial EXEC command to determine whether input errors are increasing. Step 2 If input errors appear and are increasing, diagnose the serial line as described in the "Troubleshooting Serial Line Problems" chapter.
Overutilized bandwidth	Step 1 Use the show interfaces serial EXEC command to determine whether input errors are increasing. Step 2 If input errors do not appear, the problem is related to congestion. Step 3 Turn off fast switching on the affected interface. Step 4 Check the applications that are being run, especially for very large file transfers scheduled at particular times of day. Step 5 If large transfers occur, set up priority queuing. (Priority queuing requires that the protocol allow flow control.) Step 6 Rearrange the timing of file transfers so that links are not overused during normal business hours. Step 7 Add bandwidth and consider using dial backup over the new link for applications that are taking excessive bandwidth on existing links. Step 8 Adjust buffer size. For details, see the section "Adjusting Buffers to Ease Overutilized Serial Links," in the "Troubleshooting Serial Line Problems" chapter.
Hardware in the serial link is unreliable	Step 1 Use a serial analyzer to troubleshoot the serial line or perform the tests described in the sections "Using Extended ping Tests to Troubleshoot Serial Lines" and "CSU and DSU Loopback Tests" in the "Troubleshooting Serial Line Problems" chapter. Step 2 Replace hardware as necessary.
Carrier is automatically rerouting T1 trunk lines	Step 1 Contact the long-line carrier service to determine whether rerouting is occurring. Step 2 Ensure that the carrier provides a dedicated circuit if automatic switching is causing performance problems.

Dropped Connections over a WAN or Serial Link

Symptom: Users complain about dropped connections and the inability to make host connections at peak traffic periods. One example of this problem is in an environment that features bridged DEC LAT traffic and multiple routed protocols. Data entry input (or other application requests) from users might be getting buffered at the end of an already long input queue and eventually one end of the connection times out. Table 15-11 outlines possible causes and suggested actions when connections are dropped over a serial or WAN interconnection.

Table 15-11 : WAN: Dropped Connections over a WAN or Serial Link

Possible Cause	Suggested Actions
Noisy serial line	Step 1 Use the show interfaces serial EXEC command to determine whether input errors are increasing. Step 2 If input errors appear and are increasing, diagnose the serial line as described in the "Troubleshooting Serial Line Problems" chapter.
Overutilized bandwidth	Step 1 If input errors do not appear, the problem is related to congestion. Step 2 Turn off fast switching on the affected interface. Step 3 Check the applications being run, especially for very large file transfers scheduled at particular times of day. Step 4 If you find large, scheduled file transfers, set up priority queuing. (Priority queuing requires that the protocol allow flow control.) Step 5 When bridging LAT, consider using the bridgegroup group lat-compression interface configuration command to reduce bandwidth by implementing LAT compression. Step 6 Rearrange the timing of file transfers so that links are not overused during normal business hours. Step 7 Add bandwidth and consider using dial backup over the new link for applications that are taking excessive bandwidth on existing links. Step 8 Adjust buffer size. For details, see "Adjusting Buffers to Ease Overutilized Serial Links," in the "Troubleshooting Serial Line Problems" chapter.
Hardware in the serial link is unreliable	Step 1 Use a serial analyzer to troubleshoot the serial line or perform the tests described in the sections "Using Extended ping Tests to Troubleshoot Serial Lines" and "CSU and DSU Loopback Tests" in the "Troubleshooting Serial Line Problems" chapter. Step 2 Replace hardware as necessary.
Inadequate bandwidth	Step 1 After checking all of the above, examine the output of the show interfaces serial EXEC command. If, after these actions, load is still about 80 percent, the line is inadequate for traffic requirements. Step 2 Add another serial line.

Poor XNS Server Performance over Router in a LAN Internetwork

Symptom: Users complain about sessions dropping at peak traffic periods when trying to connect to resources on the other side of a router configured to route XNS. Table 15-12 outlines possible causes and suggested actions when XNS server performance is poor over a router in a LAN internetwork.

Table 15-12 : XNS: Poor XNS Server Performance over Router in a LAN Internetwork

Possible Causes	Suggested Actions
Excessive traffic; collisions causing session drops (Ethernet only)	Step 1 Use a protocol analyzer to examine traffic to look for collisions in excess of normal acceptable conditions (varies for specific site). As an alternative, use the show interfaces EXEC command to get a rough estimate of the collision count. Step 2 Examine the output of the protocol analyzer to determine bandwidth utilization. Output from a protocol analyzer will provide a more accurate reading of dynamic traffic information. If the protocol analyzer detects bandwidth utilization of 15 to 20 percent (on average) or higher, you are likely to have a load-related performance problem. Step 3 If you see a higher-than-expected bandwidth utilization and if collisions are increasing steadily, consider segmenting the network with additional bridges or routers.
Insufficient bandwidth on Token Ring to handle traffic	Step 1 Upgrade from 4- to 16-Mbps Token Ring throughout network. Step 2 If performance is still inadequate, consider segmenting the network with additional bridges or routers.

Poor XNS Server Performance over Router in a WAN

Symptom: Users complain that sessions drop at peak traffic periods when they are trying to connect to resources on the other side of a router that is configured to route XNS over a WAN or serial link. Table 15-13 outlines a possible cause and suggested actions when an XNS server performs poorly over a router in a WAN internetwork.

Table 15-13 : XNS: Poor XNS Server Performance over Router in a WAN

Possible Cause

Suggested Actions

Another protocol dominates CPU time

Step 1 Use the show processes EXEC command to look for large numbers in the Runtime (ms) and Invoked fields for certain protocols. An example would be a protocol that has a value that is 10 times or greater than the value indicated for XNS traffic. When this kind of condition exists, XNS traffic is not getting adequate access to the CPU, and performance is affected. Step 2 Use the show interfaces EXEC command to look for a high level of output drops. Step 3 If you see output drops and another protocol is dominating CPU time (indicated in the show process Runtime [ms] field), use priority queuing to force the system to handle XNS traffic over other protocols. More information about priority queuing is provided in the "Troubleshooting Serial Line Problems" chapter. Step 4 If priority queuing does not improve performance, add bandwidth by implementing a higher-speed line or by adding additional lines of same speed. If you add additional lines, use the xns maximum-paths global configuration command to specify the number of paths

Switching-Support Matrices

The overall performance of the network can vary according to the switching mechanism used for a given protocol, interface type, software version, or encapsulation method. Furthermore, the type of switching that is available on a particular platform depends on the version of the installed software and on the individual cards installed in the router.

The following matrices compare protocols and network media types, and define the switching methods available for each combination based on the particular Cisco Internetwork Operating System (Cisco IOS) release you are running.