How to find broadcast storm have on the Nortel Switched Firewall Accelerator NSF 6616 or 6614?

Symptom:
VRRP log messages indicating the Accelerators are changing from BACKUP TO MASTER.
       NOTICE: Accelerator 00:60:cf:ae:08:00's vrrp status changed to MASTER
Default gateways are reported as down or not in use
    Default gateway 1 (192.168.0.1) is down
Acceleration is switching from on to off to on again
     CRITICAL: Acceleration Status changed from ACCELERATING to NOT ACCELERATING


Cause:
Broadcast packets are not handled by the SPs.  Instead, broadcast packets get forwarded to the MP for processing. A broadcast storm can bring the MP to 100% utilization.  When the MP is running at 100% dealing with the Broadcast packets, it will not be able to send VRRP updates, default gateway health checks, accelerator health checks, and others and so on.

Problem Resolution

Fix:
The first step is to determine that you are under a Broadcast Storm.
1) Check the port Stats by running /stats/port #/if on the Accelerator.  Here is the output

Interface statistics for port 3:
                    ifHCIn Counters       ifHCOut Counters
Octets:                            0                      0
UcastPkts:                      1019                  0
BroadcastPkts:              12602991             0
MulticastPkts:             78065             0
Discards:                        566                     0
Errors:                             0                0
ifInUnknownProtos:                0

In the above capture, you will see the BroadcastPkts count is much higher than the UcastPkts or the MulticastPkts.  Normal traffic typically has the Ucast as the highest count.

2) You can also check the MP Stats at the time of the attack with the /stats/mp/cpu command.  Here is a sample output showing the MP at 100% utilization.

>> MP-specific Statistics# cpu
------------------------------------------------------------------
CPU utilization:
cpuAUtil1Second:            100%  cpuBUtil1Second:           100%
cpuAUtil4Seconds:           100%  cpuBUtil4Seconds:          100%
cpuAUtil64Seconds:          97%   cpuBUtil64Seconds:         98%

3) Determine the source of the Broadcast Storms.  Set up a Sniffer on the suspected port and look for broadcast packets.  Broadcast packets do not have IP addresses, only source and destination MAC. In a Broadcast packet, the destination MAC is FF:FF:FF:FF:FF:FF

4) When the source is determined, you will need to troubleshoot the offending device to find out why it is sending a Broadcast Storm