Make sure you choose the right Twinax cables for your project. Cables ARE important.
Last year, our company had some courses tailored for some of us. These courses were provided by one of the main Cisco Solution Learning Partners in Spain.
To be honest, I was expecting them to be more like I’ve seen on Internetwork Expert ine.com. They were far from that level of expertise and superb organization of thoughts. Let’s say that they were not that bad altogether.
Anyway, the courses were a mixture of stuff from DCUCI, DCUFI and UCS troubleshooting. When it was time to talk about the Twinax cables and how convenient a solution they are, the guy told us a story about a previous experience he had with one of his customers.
Long story short, they had spent about 2 weeks trying to find out why some servers with QLogic CNA cards were not able to communicate with the rest of the network. They found out at the end that they were using the wrong cables… There were no soecific details about the model of the QLogic cards or the cables.
I heard this story twice, so I thought it was a good idea to share it with some of my colleagues.
The new project
No more than 6 months later, I was assigned a project that involved a couple of Nexus 5000, 5548UP to be more precise, and their integration with an existing Ethernet network and Fibre channel fabric.
The customer had also acquired from us several QLogic 10GB CNA cards for IBM system x servers. I actually had no way to know the precise model of CNA card until I opened one of the boxes. The QLogic precise model was QLE8142-CU.
I have to say that finding documents with relevant information about the great amount of QLogic models is not that easy. Here’s a brief description of the QLE8342: “QLE8342-CU; Dual-port 10GbE Ethernet to PCIe Converged Network Adapter with empty SFP+ cages (meant for use with copper cables”.
Funnily enough, I found it in here, which seems to be an Asian retail store website. Other websites or PDF documents that I found, those from QLogic or IBM only spoke about the QLogic 2600 or the 9100 adapter series, but not the more specific model and description. Then again, maybe I was not good enough at searching.
Vmware ESXi hypervisors were to be installed in all of these IBM system x servers and the servers were meant to connect to the LAN and SAN using both of their CNA adapters simultaneously.
This looked good. It was my first FCoE implementation and it involved some of the Data Center technology that I had been paying around in the past months, namely vPC, Fibre Channel in Nexus switches and a bunch of VMware ESXi hosts (they had an old cluster running vSphere 4, but I had 5.1 and 5.5 in my own lab).
We agreed on the physical and logical design, both for the Ethernet network and the SAN and started with the installation.
We were done setting up the 2 Nexus 5K and the vPC configuration and wanted to test with the first couple of QLogic CNA cards installed in one of the IBM system x servers.
To or surprise, we were not able to bring the links up.
We double and triple checked the configs of the Nexus switches.
We were told in those courses that if there was a VSAN mismatch between the N and F ports, the FC link would never come up, but this was FCoE we were dealing with here.
We went nuts trying to update the ESXi drivers and the firmware of the CNA cards, but that didn’t help. We were only getting strange, inconsistent results because after one of these updates, they told us that one of the links had gone up (we weren’t there to see it though).
We asked around but no one seemed to know what was going on.
It wasn’t until one of our finest engineers back then (he’s sadly no longer with us) went through some hard to find literature about QLogic cards that we realised that this QLogic CNA model in particular worked ONLY with twinax cables supplied by QLogic… Ours had been supplied by Cisco.
This still did not provide a precise solution, but now that we knew there was something fishy with the cables, we followed this lead and kept searching.
I finally came accross with this:
A simple CNA link in Wikipedia where it said:
“QLogic offers CNAs via their QLogic 8200 & 8300 series Converged Network Adapters. They offer single and dual port PCI cards with copper or optical fibre interfaces.”
A PDF from IBM that stated:
“The QLogic 10Gb CNA for IBM System x® (PN 42C1800) is a PCI Express 2.0 x8 10Gb Converged Network Adapter with two SFP+ cages. The adapter can support
either SFP+ Multimode Fiber SR optical modules or SFP+ Active Copper cables. Two SFP+ cages for either SFP+ Fiber SR or SFP+ Active Copper”
The keyword here is “Active”. There’s active and passive Twinax cables. It actually does NOT matter what vendor provides them.
Obviously, the customer was not very happy about this. So we had our sales guy fix his mistake by rushing into buying the right cables for these picky QLogic cards.
It was not easy or cheap, but they finally got some Active Twinax cables supplied by IBM. We tested with these and immediately the links went up.
It turned out out that the twinax cables bought from Cisco were passive and these QLogic cards did not like that.
Pay attention to the problems others have had in the past and listen to them!