Extreme IO Latency Caused by a Poorly Seated Fiber Cable
One of my customers was experiencing some extreme IO latency in their environment: in the hundreds of ms. Obviously, there was some pain associated with this issue, and so they asked for help. This environment had 3 HP c7000 chassis and 2 different SANs; the issue was affecting every host accessing every LUN, so we decided that the issue must be in the fiber channel fabric somewhere. After poking around, we quickly realized that the firmware on the Brocade switches was from January 2013, so was quite old. I pulled up top on each switch and saw no appreciable CPU or Memory usage. Next, I looked at porterrshow to see if there were any problems on the switches; each one had a single port that had a ridiculous number of Enc Out errors. I cleared the error stat counters by using portstatsclear -i 0-33 and then issued another porterrshow , and found that we had roughly 100,000 Enc Out errors on that port each second. Coincidentally, each switch had 1 p...