Author Topic: Onboard nic not working  (Read 5680 times)

power9mm

  • Full Member
  • ***
  • Posts: 103
  • Karma: +8/-3
    • View Profile
Onboard nic not working
« on: January 29, 2023, 02:35:07 pm »
I can't get any network access with the onboard nic despite the light showing its active. tried this with opensuse and fedora. luckily i had a pcie nic on hand.

meklort

  • Newbie
  • *
  • Posts: 19
  • Karma: +16/-0
    • View Profile
    • GitHub
Re: Onboard nic not working
« Reply #1 on: January 29, 2023, 03:24:54 pm »
Can you check to the firmware version on the NIC? This can be done using fwupd on your machine (fwupdmgr get-devices).

Older firmware version had some issues with starting up. If it's out of date (latest is 0.6.12), please update with "fwupdmgr update" and check again. Note that you'll need to completely disconnect power from the machine for about 30 seconds after the update before the new firmware really starts. A good order is (1) update, (2) reboot, (3) shutdown, (4) disconnect power for 30 seconds (until all lights on the motherboard are off), and (5) reconnect and boot up again.

- Which network ports have you tried? The port above the usb connector is shared between the BMC and the OS. You might try a different port to see if there's a different behavior.
- Was the BMC able to get an ip address before the OS started? This can be checked in your router / dhcp server if a lease was given. If the BMC worked then this may be a handoff issue to the OS.
- Can you also confirm that you are not trying to use jumbo frames? The firmware has an issue with jumbo frames that I haven't looked at yet, but the symptoms are that the the firmware will reset the connection once every couple of seconds - you'll still get some packets in and out, but only for a few seconds at a time.


power9mm

  • Full Member
  • ***
  • Posts: 103
  • Karma: +8/-3
    • View Profile
Re: Onboard nic not working
« Reply #2 on: January 29, 2023, 03:39:48 pm »
i tried the two dual ethernet ports, not the BMC port. they show as detected upon bootup. i dont know what the firmware is but it should be recent as this was a board from the latest shipment. I haven't attempted any BMC access as i do not have a crossover cable currently.

power9mm

  • Full Member
  • ***
  • Posts: 103
  • Karma: +8/-3
    • View Profile
Re: Onboard nic not working
« Reply #3 on: January 29, 2023, 06:16:21 pm »
fwupdmgr doesnt appear to list the onboard NICs. I see them mentioned during boot however..

the motherboard rev is 1.01


meklort

  • Newbie
  • *
  • Posts: 19
  • Karma: +16/-0
    • View Profile
    • GitHub
Re: Onboard nic not working
« Reply #4 on: January 30, 2023, 12:24:56 am »
You don't see something like the following via fwupdmgr (note that I'm running an unreleased version)?
Code: [Select]
IBM C1P9S01 REV 1.01

├─NetXtreme BCM5719 Gigabit Ethernet PCIe:
│     Device ID:          7de5ffdca08fa52d95fd4bb42aa5d07a4b35d2dd
│     Current version:    0.6.38
│     Vendor:             Broadcom Inc. and subsidiaries (PCI:0x14E4)
│     Release Branch:     oss-firmware
│     GUIDs:              30fe13b6-aa73-5d8c-a19f-c7b600f0117a ← PCI\VEN_14E4&DEV_1657
│                         cd40a56d-7fbf-57ef-9502-50e9f35d9b1b ← PCI\VEN_14E4&DEV_1657&REV_00
│                         5d8b12bf-1973-58fc-8b31-3e50fe31954d ← PCI\VEN_14E4&DEV_1657&SUBSYS_14E41981
│                         df899e77-b408-51b3-acd3-e91343344bfb ← PCI\VEN_14E4&DEV_1657&SUBSYS_14E41981&REV_00
│     Device Flags:       • Updatable
│                         • Supported on remote server
│                         • Needs a reboot after installation
│                         • Cryptographic hash verification is available
│                         • Device will backup firmware before installing
│                         • Unsigned Payload
│   
├─88SE9235 PCIe 2.0 x2 4-port SATA 6 Gb/s Controller:
│     Device ID:          024ec185fcba9289f4336862423686455165d68a

In any case, if that doesn't show anything, can you run "ethtool  -I enP4p1s0f0" (enP4p1s0f0 is correct on Fedora, may be different on your OS):
Code: [Select]
root@bb /h/meklort [1]# ethtool  -i enP4p1s0f0
driver: tg3
version: 6.2.0-0.rc4.31.fc38.ppc64le
firmware-version: stage1-0.6.38 NCSI v0.6.0.38
expansion-rom-version:
bus-info: 0004:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Finally, do note that there's shouldn't be a need for a crossover cable - the port support (and has enabled) Auto-MDI/MDIX, so if you don't want to plug it into your switch, you should be able to plug a computer directly into it and still have it negotiate properly.


power9mm

  • Full Member
  • ***
  • Posts: 103
  • Karma: +8/-3
    • View Profile
Re: Onboard nic not working
« Reply #5 on: January 30, 2023, 09:10:20 am »
ive only seen the broadcom come up during OS install with the listed drives tge3, but it didn't show up at all with fwupmgr. i'll try ethtool

power9mm

  • Full Member
  • ***
  • Posts: 103
  • Karma: +8/-3
    • View Profile
Re: Onboard nic not working
« Reply #6 on: January 30, 2023, 07:11:10 pm »
https://i.imgur.com/mCPsi14.png

this is what i could find with lspci, the listed device name you gave didn't work with ethtool.

meklort

  • Newbie
  • *
  • Posts: 19
  • Karma: +16/-0
    • View Profile
    • GitHub
Re: Onboard nic not working
« Reply #7 on: January 31, 2023, 01:40:09 am »
Are they showing up at all past lspci (good that it sees them)? What does dmesg say? How about ip or ifconfig? Provide as many details as you are willing to.

This should give some details (may need sudo depending on the distribution):
Code: [Select]
dmesg | grep tg3 | sed s/"MAC address.*"/"MAC Address <mac removed>"/g

For reference, this is what I see, showing the three ports / network interfaces and that port 2 is online (the bmc port):
Code: [Select]
meklort@bb ~> dmesg | grep tg3 | sed s/"MAC address.*"/"MAC Address <mac removed>"/g
[    2.271519] tg3 0004:01:00.0: enabling device (0140 -> 0142)
[    2.306172] tg3 0004:01:00.0 eth0: Tigon3 [partno(BCM95719) rev 5719001] (PCI Express) MAC Address <mac removed>
[    2.306206] tg3 0004:01:00.0 eth0: attached PHY is serdes (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[    2.306236] tg3 0004:01:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[    2.306254] tg3 0004:01:00.0 eth0: dma_rwctrl[00000000] dma_mask[64-bit]
[    2.306421] tg3 0004:01:00.1: enabling device (0140 -> 0142)
[    2.336139] tg3 0004:01:00.1 eth1: Tigon3 [partno(BCM95719) rev 5719001] (PCI Express) MAC Address <mac removed>
[    2.336173] tg3 0004:01:00.1 eth1: attached PHY is serdes (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[    2.336203] tg3 0004:01:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[    2.336220] tg3 0004:01:00.1 eth1: dma_rwctrl[00000000] dma_mask[64-bit]
[    2.336402] tg3 0004:01:00.2: enabling device (0140 -> 0142)
[    2.374594] tg3 0004:01:00.2 eth2: Tigon3 [partno(BCM95719) rev 5719001] (PCI Express) MAC Address <mac removed>
[    2.374621] tg3 0004:01:00.2 eth2: attached PHY is serdes (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[    2.374650] tg3 0004:01:00.2 eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[    2.374677] tg3 0004:01:00.2 eth2: dma_rwctrl[00000000] dma_mask[64-bit]
[    2.377649] tg3 0004:01:00.1 enP4p1s0f1: renamed from eth1
[    2.449801] tg3 0004:01:00.0 enP4p1s0f0: renamed from eth0
[    2.529154] tg3 0004:01:00.2 enP4p1s0f2: renamed from eth2
[  125.902784] tg3 0004:01:00.2 enP4p1s0f2: Link is up at 1000 Mbps, full duplex
[  125.902806] tg3 0004:01:00.2 enP4p1s0f2: Flow control is off for TX and off for RX
[  125.902818] tg3 0004:01:00.2 enP4p1s0f2: EEE is enabled

power9mm

  • Full Member
  • ***
  • Posts: 103
  • Karma: +8/-3
    • View Profile
Re: Onboard nic not working
« Reply #8 on: January 31, 2023, 10:16:07 pm »
this is what i get

meklort

  • Newbie
  • *
  • Posts: 19
  • Karma: +16/-0
    • View Profile
    • GitHub
Re: Onboard nic not working
« Reply #9 on: February 01, 2023, 10:29:21 pm »
Ah, that's very interesting. The tg3.c source code has ENODEV being reported in tg3_phy_probe. That routine has to cooperate with the APE firmware for everything to work right, so now I'm wondering what the APE firmware log is showing.

In any case, that's not expected, can you provide more details on your setup? What are you connecting to? What cables are you using? etc.

If you are willing to setup a dev environment, you can get a dump of the ape console and see what the firmware is trying to do. You'd need to compile the ape console utility in the firmware repo here: https://github.com/meklort/bcm5719-fw or use the prebuilt binary included in the release package, if you don't want to setup the environment yourself).

It'd also be worth connecting the BMC port to a switch / router / etc and seeing if you're able to talk to it before the OS starts up. My first guess is that the APE firmware is getting locked up somewhere (possibly also talking to the PHY), and not releasing the PHY resources for Linux.

In any case, the APE console output would be very useful to debug it.

power9mm

  • Full Member
  • ***
  • Posts: 103
  • Karma: +8/-3
    • View Profile
Re: Onboard nic not working
« Reply #10 on: February 02, 2023, 12:54:07 pm »
Yeah.

it's just a blackbird with an 8 core cpu. Vega 64, 2.5 SSD and a pcie nic. Nothing special. its just hard wired into a librecmc router which is just wired to a modem. Cat6 cable i think.
I can try connecting to the BMC with an airgapped laptop I suppose. I don't want to connect that port to any internet connected devices.

meklort

  • Newbie
  • *
  • Posts: 19
  • Karma: +16/-0
    • View Profile
    • GitHub
Re: Onboard nic not working
« Reply #11 on: February 04, 2023, 10:51:22 am »
So, this is what the state as I see it:

- Linux appeases to be unable to acquire a lock in the hardware. (see https://github.com/torvalds/linux/blob/master/drivers/net/ethernet/broadcom/tg3.c#L15439)
- The APE or RX firmware in the part has priority over Linux
- The APE or RX firmware likely has acquired the lock, and failed to release it due to an infinite loop / PHY hardware not responding how the firmware expected it to.
(Note: If you have an older version of the firmware, the RX firmware acquired the lock for the endpoint it was running on, however this failed to work in some cases, and is disabled in newer firmware. The APE firmware still grabs a lock as well, however this should be for one port only, not all of them, and so I don't expect this to be where the issue is)

All that said, when the lock failed to be acquired, Linux stopped initializing the device, and so you don't get the eth port showing up.
This is also why fwupd and ethtool failed to work - they depend on the tg3 driver being loaded.

To fix this, you'll need to do one of a two things:
- Try connecting to a different external device like a switch before going to the router and see if you get a different result. This is probably the simplest option if it works (unlikely due to all ports failing to initialize)
- If you're unable to get Linux to ever see the device, you'll need to install the development tools for the OSS firmware and at that point there are a couple of options.
(1) You can try loading the proprietary firmware image and see if that works. if it does *not* work, you'll need to RMA with Raptor as it's a hardware issue. If, on the other had, the proprietary firmware works, you can leave it as-is, or, we can flash the latest oss fw release and continue to (2)
(2) You/We can try debugging the issue. You'll need to provide dumps of the registers using the bcmregtool utility. That'll let me know if the firmware is spinning / locked up, the firmware version, etc and if there's an easy fix.

vikings.thum

  • Newbie
  • *
  • Posts: 43
  • Karma: +17/-0
    • View Profile
    • Vikings
Re: Onboard nic not working
« Reply #12 on: February 22, 2023, 04:04:59 am »
Is your BMC not working as well?
I've had the same issue with one of my Blackbirds and contacted RaptorCS;
Quote
I know there was a manufacturing process change to fix data corruption seen on the first handful of boards, it's very possible this one shipped out prior to the change.
They fixed it by providing an update bundle which I then flashed. When you contact them, provide your ports MAC addresses so can embed that in the image.
They provided the following commands to flash the new image:
Code: [Select]
set +e
echo 0004:01:00.1 > /sys/bus/pci/drivers/tg3/unbind
set -e
bcmflash -t raw -i 1 -r bcm5719.rom[
After powering down and removing the system from your wall socket, boot it up again and it should work as expected.
https://shop.vikings.net
XMPP: thum@jabber.vikings.net
Libera.Chat IRC: #vikings (handle: 'thum')

power9mm

  • Full Member
  • ***
  • Posts: 103
  • Karma: +8/-3
    • View Profile
Re: Onboard nic not working
« Reply #13 on: February 22, 2023, 01:16:14 pm »
I haven't tried the BMC yet. I suppose I will now since you brought that up.