From 5b7c4cabbb65f5c469464da6c5f614cbd7f730f2 Mon Sep 17 00:00:00 2001 From: Linus Torvalds Date: Tue, 21 Feb 2023 18:24:12 -0800 Subject: Merge tag 'net-next-6.3' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next Pull networking updates from Jakub Kicinski: "Core: - Add dedicated kmem_cache for typical/small skb->head, avoid having to access struct page at kfree time, and improve memory use. - Introduce sysctl to set default RPS configuration for new netdevs. - Define Netlink protocol specification format which can be used to describe messages used by each family and auto-generate parsers. Add tools for generating kernel data structures and uAPI headers. - Expose all net/core sysctls inside netns. - Remove 4s sleep in netpoll if carrier is instantly detected on boot. - Add configurable limit of MDB entries per port, and port-vlan. - Continue populating drop reasons throughout the stack. - Retire a handful of legacy Qdiscs and classifiers. Protocols: - Support IPv4 big TCP (TSO frames larger than 64kB). - Add IP_LOCAL_PORT_RANGE socket option, to control local port range on socket by socket basis. - Track and report in procfs number of MPTCP sockets used. - Support mixing IPv4 and IPv6 flows in the in-kernel MPTCP path manager. - IPv6: don't check net.ipv6.route.max_size and rely on garbage collection to free memory (similarly to IPv4). - Support Penultimate Segment Pop (PSP) flavor in SRv6 (RFC8986). - ICMP: add per-rate limit counters. - Add support for user scanning requests in ieee802154. - Remove static WEP support. - Support minimal Wi-Fi 7 Extremely High Throughput (EHT) rate reporting. - WiFi 7 EHT channel puncturing support (client & AP). BPF: - Add a rbtree data structure following the "next-gen data structure" precedent set by recently added linked list, that is, by using kfunc + kptr instead of adding a new BPF map type. - Expose XDP hints via kfuncs with initial support for RX hash and timestamp metadata. - Add BPF_F_NO_TUNNEL_KEY extension to bpf_skb_set_tunnel_key to better support decap on GRE tunnel devices not operating in collect metadata. - Improve x86 JIT's codegen for PROBE_MEM runtime error checks. - Remove the need for trace_printk_lock for bpf_trace_printk and bpf_trace_vprintk helpers. - Extend libbpf's bpf_tracing.h support for tracing arguments of kprobes/uprobes and syscall as a special case. - Significantly reduce the search time for module symbols by livepatch and BPF. - Enable cpumasks to be used as kptrs, which is useful for tracing programs tracking which tasks end up running on which CPUs in different time intervals. - Add support for BPF trampoline on s390x and riscv64. - Add capability to export the XDP features supported by the NIC. - Add __bpf_kfunc tag for marking kernel functions as kfuncs. - Add cgroup.memory=nobpf kernel parameter option to disable BPF memory accounting for container environments. Netfilter: - Remove the CLUSTERIP target. It has been marked as obsolete for years, and we still have WARN splats wrt races of the out-of-band /proc interface installed by this target. - Add 'destroy' commands to nf_tables. They are identical to the existing 'delete' commands, but do not return an error if the referenced object (set, chain, rule...) did not exist. Driver API: - Improve cpumask_local_spread() locality to help NICs set the right IRQ affinity on AMD platforms. - Separate C22 and C45 MDIO bus transactions more clearly. - Introduce new DCB table to control DSCP rewrite on egress. - Support configuration of Physical Layer Collision Avoidance (PLCA) Reconciliation Sublayer (RS) (802.3cg-2019). Modern version of shared medium Ethernet. - Support for MAC Merge layer (IEEE 802.3-2018 clause 99). Allowing preemption of low priority frames by high priority frames. - Add support for controlling MACSec offload using netlink SET. - Rework devlink instance refcounts to allow registration and de-registration under the instance lock. Split the code into multiple files, drop some of the unnecessarily granular locks and factor out common parts of netlink operation handling. - Add TX frame aggregation parameters (for USB drivers). - Add a new attr TCA_EXT_WARN_MSG to report TC (offload) warning messages with notifications for debug. - Allow offloading of UDP NEW connections via act_ct. - Add support for per action HW stats in TC. - Support hardware miss to TC action (continue processing in SW from a specific point in the action chain). - Warn if old Wireless Extension user space interface is used with modern cfg80211/mac80211 drivers. Do not support Wireless Extensions for Wi-Fi 7 devices at all. Everyone should switch to using nl80211 interface instead. - Improve the CAN bit timing configuration. Use extack to return error messages directly to user space, update the SJW handling, including the definition of a new default value that will benefit CAN-FD controllers, by increasing their oscillator tolerance. New hardware / drivers: - Ethernet: - nVidia BlueField-3 support (control traffic driver) - Ethernet support for imx93 SoCs - Motorcomm yt8531 gigabit Ethernet PHY - onsemi NCN26000 10BASE-T1S PHY (with support for PLCA) - Microchip LAN8841 PHY (incl. cable diagnostics and PTP) - Amlogic gxl MDIO mux - WiFi: - RealTek RTL8188EU (rtl8xxxu) - Qualcomm Wi-Fi 7 devices (ath12k) - CAN: - Renesas R-Car V4H Drivers: - Bluetooth: - Set Per Platform Antenna Gain (PPAG) for Intel controllers. - Ethernet NICs: - Intel (1G, igc): - support TSN / Qbv / packet scheduling features of i226 model - Intel (100G, ice): - use GNSS subsystem instead of TTY - multi-buffer XDP support - extend support for GPIO pins to E823 devices - nVidia/Mellanox: - update the shared buffer configuration on PFC commands - implement PTP adjphase function for HW offset control - TC support for Geneve and GRE with VF tunnel offload - more efficient crypto key management method - multi-port eswitch support - Netronome/Corigine: - add DCB IEEE support - support IPsec offloading for NFP3800 - Freescale/NXP (enetc): - support XDP_REDIRECT for XDP non-linear buffers - improve reconfig, avoid link flap and waiting for idle - support MAC Merge layer - Other NICs: - sfc/ef100: add basic devlink support for ef100 - ionic: rx_push mode operation (writing descriptors via MMIO) - bnxt: use the auxiliary bus abstraction for RDMA - r8169: disable ASPM and reset bus in case of tx timeout - cpsw: support QSGMII mode for J721e CPSW9G - cpts: support pulse-per-second output - ngbe: add an mdio bus driver - usbnet: optimize usbnet_bh() by avoiding unnecessary queuing - r8152: handle devices with FW with NCM support - amd-xgbe: support 10Mbps, 2.5GbE speeds and rx-adaptation - virtio-net: support multi buffer XDP - virtio/vsock: replace virtio_vsock_pkt with sk_buff - tsnep: XDP support - Ethernet high-speed switches: - nVidia/Mellanox (mlxsw): - add support for latency TLV (in FW control messages) - Microchip (sparx5): - separate explicit and implicit traffic forwarding rules, make the implicit rules always active - add support for egress DSCP rewrite - IS0 VCAP support (Ingress Classification) - IS2 VCAP filters (protos, L3 addrs, L4 ports, flags, ToS etc.) - ES2 VCAP support (Egress Access Control) - support for Per-Stream Filtering and Policing (802.1Q, 8.6.5.1) - Ethernet embedded switches: - Marvell (mv88e6xxx): - add MAB (port auth) offload support - enable PTP receive for mv88e6390 - NXP (ocelot): - support MAC Merge layer - support for the the vsc7512 internal copper phys - Microchip: - lan9303: convert to PHYLINK - lan966x: support TC flower filter statistics - lan937x: PTP support for KSZ9563/KSZ8563 and LAN937x - lan937x: support Credit Based Shaper configuration - ksz9477: support Energy Efficient Ethernet - other: - qca8k: convert to regmap read/write API, use bulk operations - rswitch: Improve TX timestamp accuracy - Intel WiFi (iwlwifi): - EHT (Wi-Fi 7) rate reporting - STEP equalizer support: transfer some STEP (connection to radio on platforms with integrated wifi) related parameters from the BIOS to the firmware. - Qualcomm 802.11ax WiFi (ath11k): - IPQ5018 support - Fine Timing Measurement (FTM) responder role support - channel 177 support - MediaTek WiFi (mt76): - per-PHY LED support - mt7996: EHT (Wi-Fi 7) support - Wireless Ethernet Dispatch (WED) reset support - switch to using page pool allocator - RealTek WiFi (rtw89): - support new version of Bluetooth co-existance - Mobile: - rmnet: support TX aggregation" * tag 'net-next-6.3' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next: (1872 commits) page_pool: add a comment explaining the fragment counter usage net: ethtool: fix __ethtool_dev_mm_supported() implementation ethtool: pse-pd: Fix double word in comments xsk: add linux/vmalloc.h to xsk.c sefltests: netdevsim: wait for devlink instance after netns removal selftest: fib_tests: Always cleanup before exit net/mlx5e: Align IPsec ASO result memory to be as required by hardware net/mlx5e: TC, Set CT miss to the specific ct action instance net/mlx5e: Rename CHAIN_TO_REG to MAPPED_OBJ_TO_REG net/mlx5: Refactor tc miss handling to a single function net/mlx5: Kconfig: Make tc offload depend on tc skb extension net/sched: flower: Support hardware miss to tc action net/sched: flower: Move filter handle initialization earlier net/sched: cls_api: Support hardware miss to tc action net/sched: Rename user cookie and act cookie sfc: fix builds without CONFIG_RTC_LIB sfc: clean up some inconsistent indentings net/mlx4_en: Introduce flexible array to silence overflow warning net: lan966x: Fix possible deadlock inside PTP net/ulp: Remove redundant ->clone() test in inet_clone_ulp(). ... --- Documentation/admin-guide/thunderbolt.rst | 336 ++++++++++++++++++++++++++++++ 1 file changed, 336 insertions(+) create mode 100644 Documentation/admin-guide/thunderbolt.rst (limited to 'Documentation/admin-guide/thunderbolt.rst') diff --git a/Documentation/admin-guide/thunderbolt.rst b/Documentation/admin-guide/thunderbolt.rst new file mode 100644 index 000000000..2ed79f41a --- /dev/null +++ b/Documentation/admin-guide/thunderbolt.rst @@ -0,0 +1,336 @@ +.. SPDX-License-Identifier: GPL-2.0 + +====================== + USB4 and Thunderbolt +====================== +USB4 is the public specification based on Thunderbolt 3 protocol with +some differences at the register level among other things. Connection +manager is an entity running on the host router (host controller) +responsible for enumerating routers and establishing tunnels. A +connection manager can be implemented either in firmware or software. +Typically PCs come with a firmware connection manager for Thunderbolt 3 +and early USB4 capable systems. Apple systems on the other hand use +software connection manager and the later USB4 compliant devices follow +the suit. + +The Linux Thunderbolt driver supports both and can detect at runtime which +connection manager implementation is to be used. To be on the safe side the +software connection manager in Linux also advertises security level +``user`` which means PCIe tunneling is disabled by default. The +documentation below applies to both implementations with the exception that +the software connection manager only supports ``user`` security level and +is expected to be accompanied with an IOMMU based DMA protection. + +Security levels and how to use them +----------------------------------- +The interface presented here is not meant for end users. Instead there +should be a userspace tool that handles all the low-level details, keeps +a database of the authorized devices and prompts users for new connections. + +More details about the sysfs interface for Thunderbolt devices can be +found in ``Documentation/ABI/testing/sysfs-bus-thunderbolt``. + +Those users who just want to connect any device without any sort of +manual work can add following line to +``/etc/udev/rules.d/99-local.rules``:: + + ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1" + +This will authorize all devices automatically when they appear. However, +keep in mind that this bypasses the security levels and makes the system +vulnerable to DMA attacks. + +Starting with Intel Falcon Ridge Thunderbolt controller there are 4 +security levels available. Intel Titan Ridge added one more security level +(usbonly). The reason for these is the fact that the connected devices can +be DMA masters and thus read contents of the host memory without CPU and OS +knowing about it. There are ways to prevent this by setting up an IOMMU but +it is not always available for various reasons. + +Some USB4 systems have a BIOS setting to disable PCIe tunneling. This is +treated as another security level (nopcie). + +The security levels are as follows: + + none + All devices are automatically connected by the firmware. No user + approval is needed. In BIOS settings this is typically called + *Legacy mode*. + + user + User is asked whether the device is allowed to be connected. + Based on the device identification information available through + ``/sys/bus/thunderbolt/devices``, the user then can make the decision. + In BIOS settings this is typically called *Unique ID*. + + secure + User is asked whether the device is allowed to be connected. In + addition to UUID the device (if it supports secure connect) is sent + a challenge that should match the expected one based on a random key + written to the ``key`` sysfs attribute. In BIOS settings this is + typically called *One time saved key*. + + dponly + The firmware automatically creates tunnels for Display Port and + USB. No PCIe tunneling is done. In BIOS settings this is + typically called *Display Port Only*. + + usbonly + The firmware automatically creates tunnels for the USB controller and + Display Port in a dock. All PCIe links downstream of the dock are + removed. + + nopcie + PCIe tunneling is disabled/forbidden from the BIOS. Available in some + USB4 systems. + +The current security level can be read from +``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is +the Thunderbolt domain the host controller manages. There is typically +one domain per Thunderbolt host controller. + +If the security level reads as ``user`` or ``secure`` the connected +device must be authorized by the user before PCIe tunnels are created +(e.g the PCIe device appears). + +Each Thunderbolt device plugged in will appear in sysfs under +``/sys/bus/thunderbolt/devices``. The device directory carries +information that can be used to identify the particular device, +including its name and UUID. + +Authorizing devices when security level is ``user`` or ``secure`` +----------------------------------------------------------------- +When a device is plugged in it will appear in sysfs as follows:: + + /sys/bus/thunderbolt/devices/0-1/authorized - 0 + /sys/bus/thunderbolt/devices/0-1/device - 0x8004 + /sys/bus/thunderbolt/devices/0-1/device_name - Thunderbolt to FireWire Adapter + /sys/bus/thunderbolt/devices/0-1/vendor - 0x1 + /sys/bus/thunderbolt/devices/0-1/vendor_name - Apple, Inc. + /sys/bus/thunderbolt/devices/0-1/unique_id - e0376f00-0300-0100-ffff-ffffffffffff + +The ``authorized`` attribute reads 0 which means no PCIe tunnels are +created yet. The user can authorize the device by simply entering:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized + +This will create the PCIe tunnels and the device is now connected. + +If the device supports secure connect, and the domain security level is +set to ``secure``, it has an additional attribute ``key`` which can hold +a random 32-byte value used for authorization and challenging the device in +future connects:: + + /sys/bus/thunderbolt/devices/0-3/authorized - 0 + /sys/bus/thunderbolt/devices/0-3/device - 0x305 + /sys/bus/thunderbolt/devices/0-3/device_name - AKiTiO Thunder3 PCIe Box + /sys/bus/thunderbolt/devices/0-3/key - + /sys/bus/thunderbolt/devices/0-3/vendor - 0x41 + /sys/bus/thunderbolt/devices/0-3/vendor_name - inXtron + /sys/bus/thunderbolt/devices/0-3/unique_id - dc010000-0000-8508-a22d-32ca6421cb16 + +Notice the key is empty by default. + +If the user does not want to use secure connect they can just ``echo 1`` +to the ``authorized`` attribute and the PCIe tunnels will be created in +the same way as in the ``user`` security level. + +If the user wants to use secure connect, the first time the device is +plugged a key needs to be created and sent to the device:: + + # key=$(openssl rand -hex 32) + # echo $key > /sys/bus/thunderbolt/devices/0-3/key + # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized + +Now the device is connected (PCIe tunnels are created) and in addition +the key is stored on the device NVM. + +Next time the device is plugged in the user can verify (challenge) the +device using the same key:: + + # echo $key > /sys/bus/thunderbolt/devices/0-3/key + # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized + +If the challenge the device returns back matches the one we expect based +on the key, the device is connected and the PCIe tunnels are created. +However, if the challenge fails no tunnels are created and error is +returned to the user. + +If the user still wants to connect the device they can either approve +the device without a key or write a new key and write 1 to the +``authorized`` file to get the new key stored on the device NVM. + +De-authorizing devices +---------------------- +It is possible to de-authorize devices by writing ``0`` to their +``authorized`` attribute. This requires support from the connection +manager implementation and can be checked by reading domain +``deauthorization`` attribute. If it reads ``1`` then the feature is +supported. + +When a device is de-authorized the PCIe tunnel from the parent device +PCIe downstream (or root) port to the device PCIe upstream port is torn +down. This is essentially the same thing as PCIe hot-remove and the PCIe +toplogy in question will not be accessible anymore until the device is +authorized again. If there is storage such as NVMe or similar involved, +there is a risk for data loss if the filesystem on that storage is not +properly shut down. You have been warned! + +DMA protection utilizing IOMMU +------------------------------ +Recent systems from 2018 and forward with Thunderbolt ports may natively +support IOMMU. This means that Thunderbolt security is handled by an IOMMU +so connected devices cannot access memory regions outside of what is +allocated for them by drivers. When Linux is running on such system it +automatically enables IOMMU if not enabled by the user already. These +systems can be identified by reading ``1`` from +``/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection`` attribute. + +The driver does not do anything special in this case but because DMA +protection is handled by the IOMMU, security levels (if set) are +redundant. For this reason some systems ship with security level set to +``none``. Other systems have security level set to ``user`` in order to +support downgrade to older OS, so users who want to automatically +authorize devices when IOMMU DMA protection is enabled can use the +following ``udev`` rule:: + + ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1" + +Upgrading NVM on Thunderbolt device, host or retimer +---------------------------------------------------- +Since most of the functionality is handled in firmware running on a +host controller or a device, it is important that the firmware can be +upgraded to the latest where possible bugs in it have been fixed. +Typically OEMs provide this firmware from their support site. + +There is also a central site which has links where to download firmware +for some machines: + + `Thunderbolt Updates `_ + +Before you upgrade firmware on a device, host or retimer, please make +sure it is a suitable upgrade. Failing to do that may render the device +in a state where it cannot be used properly anymore without special +tools! + +Host NVM upgrade on Apple Macs is not supported. + +Once the NVM image has been downloaded, you need to plug in a +Thunderbolt device so that the host controller appears. It does not +matter which device is connected (unless you are upgrading NVM on a +device - then you need to connect that particular device). + +Note an OEM-specific method to power the controller up ("force power") may +be available for your system in which case there is no need to plug in a +Thunderbolt device. + +After that we can write the firmware to the non-active parts of the NVM +of the host or device. As an example here is how Intel NUC6i7KYK (Skull +Canyon) Thunderbolt controller NVM is upgraded:: + + # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem + +Once the operation completes we can trigger NVM authentication and +upgrade process as follows:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate + +If no errors are returned, the host controller shortly disappears. Once +it comes back the driver notices it and initiates a full power cycle. +After a while the host controller appears again and this time it should +be fully functional. + +We can verify that the new NVM firmware is active by running the following +commands:: + + # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate + 0x0 + # cat /sys/bus/thunderbolt/devices/0-0/nvm_version + 18.0 + +If ``nvm_authenticate`` contains anything other than 0x0 it is the error +code from the last authentication cycle, which means the authentication +of the NVM image failed. + +Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN`` +depend on the order they are registered in the NVMem subsystem. N in +the name is the identifier added by the NVMem subsystem. + +Upgrading on-board retimer NVM when there is no cable connected +--------------------------------------------------------------- +If the platform supports, it may be possible to upgrade the retimer NVM +firmware even when there is nothing connected to the USB4 +ports. When this is the case the ``usb4_portX`` devices have two special +attributes: ``offline`` and ``rescan``. The way to upgrade the firmware +is to first put the USB4 port into offline mode:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline + +This step makes sure the port does not respond to any hotplug events, +and also ensures the retimers are powered on. The next step is to scan +for the retimers:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan + +This enumerates and adds the on-board retimers. Now retimer NVM can be +upgraded in the same way than with cable connected (see previous +section). However, the retimer is not disconnected as we are offline +mode) so after writing ``1`` to ``nvm_authenticate`` one should wait for +5 or more seconds before running rescan again:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan + +This point if everything went fine, the port can be put back to +functional state again:: + + # echo 0 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline + +Upgrading NVM when host controller is in safe mode +-------------------------------------------------- +If the existing NVM is not properly authenticated (or is missing) the +host controller goes into safe mode which means that the only available +functionality is flashing a new NVM image. When in this mode, reading +``nvm_version`` fails with ``ENODATA`` and the device identification +information is missing. + +To recover from this mode, one needs to flash a valid NVM image to the +host controller in the same way it is done in the previous chapter. + +Networking over Thunderbolt cable +--------------------------------- +Thunderbolt technology allows software communication between two hosts +connected by a Thunderbolt cable. + +It is possible to tunnel any kind of traffic over a Thunderbolt link but +currently we only support Apple ThunderboltIP protocol. + +If the other host is running Windows or macOS, the only thing you need to +do is to connect a Thunderbolt cable between the two hosts; the +``thunderbolt-net`` driver is loaded automatically. If the other host is +also Linux you should load ``thunderbolt-net`` manually on one host (it +does not matter which one):: + + # modprobe thunderbolt-net + +This triggers module load on the other host automatically. If the driver +is built-in to the kernel image, there is no need to do anything. + +The driver will create one virtual ethernet interface per Thunderbolt +port which are named like ``thunderbolt0`` and so on. From this point +you can either use standard userspace tools like ``ifconfig`` to +configure the interface or let your GUI handle it automatically. + +Forcing power +------------- +Many OEMs include a method that can be used to force the power of a +Thunderbolt controller to an "On" state even if nothing is connected. +If supported by your machine this will be exposed by the WMI bus with +a sysfs attribute called "force_power". + +For example the intel-wmi-thunderbolt driver exposes this attribute in: + /sys/bus/wmi/devices/86CCFD48-205E-4A77-9C48-2021CBEDE341/force_power + + To force the power to on, write 1 to this attribute file. + To disable force power, write 0 to this attribute file. + +Note: it's currently not possible to query the force power state of a platform. -- cgit v1.2.3