network card to reduce attack surface?

[n2vi]

I have updated the BMC firmware in the past [see 2021-03-21 post], but as time passes I'm uncomfortable with the attack surface exposed by the BMC listening on the motherboard network ports. My current solution is to unplug from those and instead add a network card on the PCI bus. I still have serial connections for BMC and POWER for doing system administration.

Is there a simpler way to achieve this? Perhaps a BMC configuration trick that disables NC-SI?

[Borley]

IIRC the BMC is only remote accessible through the third RJ45 port (adjoined to rear panel USB on Blackbird).

The C1P9S01 BMC is attached to network port 3 via NCSI, and is configured to request an IP address via DHCP

It should be safe to just avoid using that port.

[n2vi]

Thanks for the quick response!

I should have clarified in my question that I'm running TalosII and definitely can ssh to the BMC through the main ports. I don't have a Blackbird, so can't say what is possible there but if I'm looking at the correct schematic off Raptor's site it would seem to have the same issue.

It is the bmcweb process I see running on the BMC that particularly prompted me to be nervous, but I haven't investigated that in detail. A potential pre-auth ssh vulnerability is enough to make me want the air-gap anyway.

[n2vi]

It might be enough (from the BMC serial port) to say "ifconfig eth0 down", which I've done.
I'm not enough of an expert in the firmware's network stack to feel confident that's all I need.

[power9mm]

so to clarify, the BMC on the blackbird is isolated and not accessible if one has network access on the other two ethernet ports. correct?

[Hasturtium]

so to clarify, the BMC on the blackbird is isolated and not accessible if one has network access on the other two ethernet ports. correct?

By all accounts that’s correct- the official documentation says that much, and it’s repeated on several pages on the official wiki.

[AdamJoseph]

so to clarify, the BMC on the blackbird is isolated and not accessible if one has network access on the other two ethernet ports. correct?

I'm not sure about the Blackbird, but please note that this is most definitely not true for Talos2.

The Talos2 BMC is connected directly to the management interface of the two-port ethernet chip, and there is nothing you can do to prevent an attacker with control of the BMC from having total control over both network adapters.

All of my Talos2 machines use separate PCIe cards for networking as a result of this unfortunate situation.  Hopefully Arctic Tern will eventually allow me to re-pinstrap the BMC and hold its reset pin in the asserted state so I can go back to using the on-motherboard ethernet ports.

[meklort]

The original post is a bit old, but since there now activity, I figured I should add a few clarifications.

so to clarify, the BMC on the blackbird is isolated and not accessible if one has network access on the other two ethernet ports. correct?

By all accounts that’s correct- the official documentation says that much, and it’s repeated on several pages on the official wiki.

To clarify this a bit more, as it's partially correct:
- The proprietary firmware, if you are still on it, technically allows all ports to be used for network traffic on the BMC. The latest BMC firmware is configured to select only the correct port for the Blackbird of Talos II, however in some cases, this could malfunction. It is also relatively easy to reconfigure this one connected to the BMC.
- If you are using the open source firmware, this is configured to only connect to the specified port at build time, as such, the BMC cannot communicate on a separate port mistakenly. There are of course ways for the BMC to turn on the host, and then instruct the POWER9 CPU to flash the NIC firmware, but that's not something that the BMC can do as easily as the option with the proprietary code.

I'm not sure about the Blackbird, but please note that this is most definitely not true for Talos2.

The Talos2 BMC is connected directly to the management interface of the two-port ethernet chip, and there is nothing you can do to prevent an attacker with control of the BMC from having total control over both network adapters.

All of my Talos2 machines use separate PCIe cards for networking as a result of this unfortunate situation.  Hopefully Arctic Tern will eventually allow me to re-pinstrap the BMC and hold its reset pin in the asserted state so I can go back to using the on-motherboard Ethernet ports.

Technically speaking, the only way that the BMC can take control of the network controller is by loading a custom host OS image, that then talks to the device. The BMC does not have a way to re-flash the network card firmware directly, nor does it have a way to load new firmware on the device directly. This can only be done from the host, which the BMC does have full control of. Your model of adding a second Ethernet card does make things harder, but the BMC can still take control of this by replacing the host image.
The general threat model for the Talos II and blackbird is that the BMC is in control of the host, and not the other way around, ado so this is how things are designed. The BMC can always compromise the host.

Is there a simpler way to achieve this? Perhaps a BMC configuration trick that disables NC-SI?

You can disable network access to the BMC a couple of ways:
- Remove the firmware on the NIC, specifically the APE. This will disable the BMC from being able to access the network (without first re flashing the firmware)
- Build a custom version of the firmware that disables NC-SI. At this point, there's not much benefit of running any firmware, but it's still an option.
- Use the open source firmware and *don't* use port 0 (Talos II), or port 2 (blackbird), as those are the ports configured for NC-SI access to the network.

[AdamJoseph]

The general threat model for the Talos II and blackbird is that the BMC is in control of the host, and not the other way around, ado so this is how things are designed. The BMC can always compromise the host.

This is a weak threat model that some people choose to adopt for convenience's sake.  If this were the most general threat model, the SEEPROM and many other features would be pointless.

the BMC can still take control of this by replacing the host image.

In a properly secured system this is at worst a denial-of-service attack.  See https://wiki.raptorcs.com/wiki/Secure_Boot_with_your_own_keys

Technically speaking, the only way that the BMC can take control of the network controller is by loading a custom host OS image,

Take a look at the Talos2 schematics: T2P9D01, v3.8, page 96.  The NCSI management interface for the ethernet chip goes to the BMC.  This is essentially the management interface for the tiny switch inside the BCM5719.  It can select the filtering criteria (vlan, mac address, etc) for which packets get passed to the BMC.  Or it can set no filtering at all in order to let the BMC sniff all traffic.

Think about it: both "tcpdump" and "ip link set dev address $MAC_ADDRESS" work on the BMC.  You can sniff traffic and inject packets on either ethernet port from the BMC, without "loading a custom host OS image".

[meklort]

Take a look at the Talos2 schematics: T2P9D01, v3.8, page 96.  The NCSI management interface for the ethernet chip goes to the BMC.  This is essentially the management interface for the tiny switch inside the BCM5719.  It can select the filtering criteria (vlan, mac address, etc) for which packets get passed to the BMC.  Or it can set no filtering at all in order to let the BMC sniff all traffic.

Think about it: both "tcpdump" and "ip link set dev address $MAC_ADDRESS" work on the BMC.  You can sniff traffic and inject packets on either ethernet port from the BMC, without "loading a custom host OS image".

Yes, I understand how NC-SI works and how it is used to connect the BMC and the NIC.

That said, there is *no* switch in the NIC, and it doesn't operate the way you discribed. The NC-SI traffic is handheld by firmware and not by hardware.

You are correct that NC-SI allows setting the VLAN, MAC, etc, however this is done by sending a command to the APE on the BCM5719, which then sets the appropriate registers to enable it.
Please se here for how packets from the BMC are handled: https://github.com/meklort/bcm5719-fw/blob/main/ape/main.c#L193
and here for how the NC-SI packets are handled, such as the set MAC command: https://github.com/meklort/bcm5719-fw/blob/main/libs/NCSI/ncsi.c#L530
You may also note that the open source firmware does not support setting a different VLAN as well, butin any case, as the firmware is open source, you can always explicitly lock it down to a specific MAC and VLAN.

In this case, there is no checking that the requested MAC is different from the host MAC, and so I will add that in a future release, as it's a very good point.

[AdamJoseph]

In this case, there is no checking that the requested MAC is different from the host MAC, and so I will add that in a future release, as it's a very good point.

Stuff like this is my big-picture concern.

The ASPEED BMC has its fingers in a lot of pies.  Removing it from the trusted computing base is much less error prone than trying to plug all the different holes in it.  We're not even sure we know about all the holes.

It's great that we now have control firwmware-level control of the BCM5719!  Back when I bought my Talos2s that was not the case, and it annoyed me, a lot.  This is an awesome development, but it doesn't solve all the problems that come from assuming the ASPEED BMC is trustworthy.

[ClassicHasClass]

The BMC question is what Kestrel should ultimately solve.