We were troubleshooting a couple of WiSM2 modules inserted into a couple of Cisco Catalyst 6509-VE in VSS mode with a couple of Cisco 1602 and 2602 Access Points physically connected to a Catalyst 3850.
The goal was to associate both APs to the WLCs (which are in HA-SSO redundancy mode).
An Infoblox grid was configured as the DHCP server for the scope of the Vlan where the APs were connected. We followed these guidelines from Cisco to configure DHCP options 43 and 60 for an Cisco Aironet 2600 AP.
The string we used for the “vendor-cass-identifier” (option 60) was “Cisco AP c2600” and the string for the “vendor-encapsulated-options” (option 43) was “f104ac1049f1“, which is the cancatenated HEX string for decimal 241 Type, 1*4 Length and 172.16.73.241 Value.
We reset the ports of the Cat 3850 where the access points were connected and, to our surprise, the links did not come back up, they showed now as “not connect” in the output of a “show interface status”.
The log started to get flooded with these messages:
*Jul 10 04:45:52.500: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/44: PD removed
*Jul 10 04:45:52.501: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/44: Power Controller reports an unknown error
We googled these error messages and all we found was that when they appeared on different platforms, such as PoE capable Cisco 3570 or 3960 switches, the solution was sometimes either performing a software upgrade or a reboot, but mostly configuring “power inline delay ….” at an interface level. The delay option is unfortunately not available in a Catalyst 3850 running IOS-XE 3.2.2:
Cat3850(config-if)#power inline ?
auto Automatically detect and power inline devices
never Never apply inline power
police Police the power drawn on the port
port Configure Port Power Level
static High priority inline power interface
By issuing repeatedly a “show power inline” one could tell that the state of the power given to the ports was rather strange because the operational status kept flapping between “ok” and “faulty”.
We had another issue while trying to see the spanning tree instance of the access Vlan and STP port state where the ports of these 2 APs had been placed. We ultimately realised that the reason was that we had reached the limit of the supported pvst instances of the Catalyst 3850′s… It turns out that you are only able to configure up to 128 Vlans, meaning that, yes, only 128 pvst instances are allowed.
After restting a couple more times the ports, we decided that a reboot of the 3850 could not hurt and we gave it a shot. When the switch came back up, we noticed that the ports had been granted power and that the ports were up again:
*Jul 11 03:55:30.367: %ILPOWER-7-DETECT: Interface Gi1/0/44: Power Device detected: IEEE PD
*Jul 11 03:55:31.418: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/44, changed state to down
*Jul 11 03:55:31.479: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/44: Power granted
*Jul 11 03:55:35.366: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/44, changed state to up
Cat3850#sh int statu | i 4
Gi1/0/4 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/14 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/24 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/25 notconnect 140 auto auto 10/100/1000BaseTX
Gi1/0/34 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/37 notconnect 140 auto auto 10/100/1000BaseTX
Gi1/0/39 connected 140 a-full a-100 10/100/1000BaseTX
Gi1/0/40 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/41 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/42 notconnect 1 auto auto 10/100/1000BaseTX
Gi1/0/43 AP 1602I connected 80 a-full a-1000 10/100/1000BaseTX
Gi1/0/44 AP 2602E connected 80 a-full a-1000 10/100/1000BaseTX
Gi1/0/45 notconnect 140 auto auto 10/100/1000BaseTX
Gi1/0/46 notconnect 140 auto auto 10/100/1000BaseTX
Gi1/0/47 notconnect 140 auto auto 10/100/1000BaseTX
Gi1/0/48 notconnect 1 auto auto 10/100/1000BaseTX
Cat3850#show power inline | i 4
1 450.0 30.0 420.0
Gi1/0/4 auto off 0.0 n/a n/a 30.0
Gi1/0/14 auto off 0.0 n/a n/a 30.0
Gi1/0/24 auto off 0.0 n/a n/a 30.0
Gi1/0/34 auto off 0.0 n/a n/a 30.0
Gi1/0/38 auto on 4.0 Ieee PD 1 30.0
Gi1/0/40 auto off 0.0 n/a n/a 30.0
Gi1/0/41 auto off 0.0 n/a n/a 30.0
Gi1/0/42 auto off 0.0 n/a n/a 30.0
Gi1/0/43 auto on 13.0 AIR-CAP1602I-E-K9 3 30.0
Gi1/0/44 auto on 13.0 AIR-CAP2602E-E-K9 3 30.0
Gi1/0/45 auto off 0.0 n/a n/a 30.0
Gi1/0/46 auto off 0.0 n/a n/a 30.0
Gi1/0/47 auto off 0.0 n/a n/a 30.0
Gi1/0/48 auto off 0.0 n/a n/a 30.0
The APs did not associate to the WLCs as we had expected, but they at least got their IP via DHCP and we were able to ping them.
The problem now lies within the way the Infoblox inserts DHCP options 43 and 60 into the DHCP offer and how the Cisco APs interpret it. If anyone has a similar setup and has been able to get it up and running, please contribute, suggestions will be greatly appreciated.
Solution to the PoE issue: A simple reboot to correctly allocate power.