From 5b7c4cabbb65f5c469464da6c5f614cbd7f730f2 Mon Sep 17 00:00:00 2001 From: Linus Torvalds Date: Tue, 21 Feb 2023 18:24:12 -0800 Subject: Merge tag 'net-next-6.3' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next Pull networking updates from Jakub Kicinski: "Core: - Add dedicated kmem_cache for typical/small skb->head, avoid having to access struct page at kfree time, and improve memory use. - Introduce sysctl to set default RPS configuration for new netdevs. - Define Netlink protocol specification format which can be used to describe messages used by each family and auto-generate parsers. Add tools for generating kernel data structures and uAPI headers. - Expose all net/core sysctls inside netns. - Remove 4s sleep in netpoll if carrier is instantly detected on boot. - Add configurable limit of MDB entries per port, and port-vlan. - Continue populating drop reasons throughout the stack. - Retire a handful of legacy Qdiscs and classifiers. Protocols: - Support IPv4 big TCP (TSO frames larger than 64kB). - Add IP_LOCAL_PORT_RANGE socket option, to control local port range on socket by socket basis. - Track and report in procfs number of MPTCP sockets used. - Support mixing IPv4 and IPv6 flows in the in-kernel MPTCP path manager. - IPv6: don't check net.ipv6.route.max_size and rely on garbage collection to free memory (similarly to IPv4). - Support Penultimate Segment Pop (PSP) flavor in SRv6 (RFC8986). - ICMP: add per-rate limit counters. - Add support for user scanning requests in ieee802154. - Remove static WEP support. - Support minimal Wi-Fi 7 Extremely High Throughput (EHT) rate reporting. - WiFi 7 EHT channel puncturing support (client & AP). BPF: - Add a rbtree data structure following the "next-gen data structure" precedent set by recently added linked list, that is, by using kfunc + kptr instead of adding a new BPF map type. - Expose XDP hints via kfuncs with initial support for RX hash and timestamp metadata. - Add BPF_F_NO_TUNNEL_KEY extension to bpf_skb_set_tunnel_key to better support decap on GRE tunnel devices not operating in collect metadata. - Improve x86 JIT's codegen for PROBE_MEM runtime error checks. - Remove the need for trace_printk_lock for bpf_trace_printk and bpf_trace_vprintk helpers. - Extend libbpf's bpf_tracing.h support for tracing arguments of kprobes/uprobes and syscall as a special case. - Significantly reduce the search time for module symbols by livepatch and BPF. - Enable cpumasks to be used as kptrs, which is useful for tracing programs tracking which tasks end up running on which CPUs in different time intervals. - Add support for BPF trampoline on s390x and riscv64. - Add capability to export the XDP features supported by the NIC. - Add __bpf_kfunc tag for marking kernel functions as kfuncs. - Add cgroup.memory=nobpf kernel parameter option to disable BPF memory accounting for container environments. Netfilter: - Remove the CLUSTERIP target. It has been marked as obsolete for years, and we still have WARN splats wrt races of the out-of-band /proc interface installed by this target. - Add 'destroy' commands to nf_tables. They are identical to the existing 'delete' commands, but do not return an error if the referenced object (set, chain, rule...) did not exist. Driver API: - Improve cpumask_local_spread() locality to help NICs set the right IRQ affinity on AMD platforms. - Separate C22 and C45 MDIO bus transactions more clearly. - Introduce new DCB table to control DSCP rewrite on egress. - Support configuration of Physical Layer Collision Avoidance (PLCA) Reconciliation Sublayer (RS) (802.3cg-2019). Modern version of shared medium Ethernet. - Support for MAC Merge layer (IEEE 802.3-2018 clause 99). Allowing preemption of low priority frames by high priority frames. - Add support for controlling MACSec offload using netlink SET. - Rework devlink instance refcounts to allow registration and de-registration under the instance lock. Split the code into multiple files, drop some of the unnecessarily granular locks and factor out common parts of netlink operation handling. - Add TX frame aggregation parameters (for USB drivers). - Add a new attr TCA_EXT_WARN_MSG to report TC (offload) warning messages with notifications for debug. - Allow offloading of UDP NEW connections via act_ct. - Add support for per action HW stats in TC. - Support hardware miss to TC action (continue processing in SW from a specific point in the action chain). - Warn if old Wireless Extension user space interface is used with modern cfg80211/mac80211 drivers. Do not support Wireless Extensions for Wi-Fi 7 devices at all. Everyone should switch to using nl80211 interface instead. - Improve the CAN bit timing configuration. Use extack to return error messages directly to user space, update the SJW handling, including the definition of a new default value that will benefit CAN-FD controllers, by increasing their oscillator tolerance. New hardware / drivers: - Ethernet: - nVidia BlueField-3 support (control traffic driver) - Ethernet support for imx93 SoCs - Motorcomm yt8531 gigabit Ethernet PHY - onsemi NCN26000 10BASE-T1S PHY (with support for PLCA) - Microchip LAN8841 PHY (incl. cable diagnostics and PTP) - Amlogic gxl MDIO mux - WiFi: - RealTek RTL8188EU (rtl8xxxu) - Qualcomm Wi-Fi 7 devices (ath12k) - CAN: - Renesas R-Car V4H Drivers: - Bluetooth: - Set Per Platform Antenna Gain (PPAG) for Intel controllers. - Ethernet NICs: - Intel (1G, igc): - support TSN / Qbv / packet scheduling features of i226 model - Intel (100G, ice): - use GNSS subsystem instead of TTY - multi-buffer XDP support - extend support for GPIO pins to E823 devices - nVidia/Mellanox: - update the shared buffer configuration on PFC commands - implement PTP adjphase function for HW offset control - TC support for Geneve and GRE with VF tunnel offload - more efficient crypto key management method - multi-port eswitch support - Netronome/Corigine: - add DCB IEEE support - support IPsec offloading for NFP3800 - Freescale/NXP (enetc): - support XDP_REDIRECT for XDP non-linear buffers - improve reconfig, avoid link flap and waiting for idle - support MAC Merge layer - Other NICs: - sfc/ef100: add basic devlink support for ef100 - ionic: rx_push mode operation (writing descriptors via MMIO) - bnxt: use the auxiliary bus abstraction for RDMA - r8169: disable ASPM and reset bus in case of tx timeout - cpsw: support QSGMII mode for J721e CPSW9G - cpts: support pulse-per-second output - ngbe: add an mdio bus driver - usbnet: optimize usbnet_bh() by avoiding unnecessary queuing - r8152: handle devices with FW with NCM support - amd-xgbe: support 10Mbps, 2.5GbE speeds and rx-adaptation - virtio-net: support multi buffer XDP - virtio/vsock: replace virtio_vsock_pkt with sk_buff - tsnep: XDP support - Ethernet high-speed switches: - nVidia/Mellanox (mlxsw): - add support for latency TLV (in FW control messages) - Microchip (sparx5): - separate explicit and implicit traffic forwarding rules, make the implicit rules always active - add support for egress DSCP rewrite - IS0 VCAP support (Ingress Classification) - IS2 VCAP filters (protos, L3 addrs, L4 ports, flags, ToS etc.) - ES2 VCAP support (Egress Access Control) - support for Per-Stream Filtering and Policing (802.1Q, 8.6.5.1) - Ethernet embedded switches: - Marvell (mv88e6xxx): - add MAB (port auth) offload support - enable PTP receive for mv88e6390 - NXP (ocelot): - support MAC Merge layer - support for the the vsc7512 internal copper phys - Microchip: - lan9303: convert to PHYLINK - lan966x: support TC flower filter statistics - lan937x: PTP support for KSZ9563/KSZ8563 and LAN937x - lan937x: support Credit Based Shaper configuration - ksz9477: support Energy Efficient Ethernet - other: - qca8k: convert to regmap read/write API, use bulk operations - rswitch: Improve TX timestamp accuracy - Intel WiFi (iwlwifi): - EHT (Wi-Fi 7) rate reporting - STEP equalizer support: transfer some STEP (connection to radio on platforms with integrated wifi) related parameters from the BIOS to the firmware. - Qualcomm 802.11ax WiFi (ath11k): - IPQ5018 support - Fine Timing Measurement (FTM) responder role support - channel 177 support - MediaTek WiFi (mt76): - per-PHY LED support - mt7996: EHT (Wi-Fi 7) support - Wireless Ethernet Dispatch (WED) reset support - switch to using page pool allocator - RealTek WiFi (rtw89): - support new version of Bluetooth co-existance - Mobile: - rmnet: support TX aggregation" * tag 'net-next-6.3' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next: (1872 commits) page_pool: add a comment explaining the fragment counter usage net: ethtool: fix __ethtool_dev_mm_supported() implementation ethtool: pse-pd: Fix double word in comments xsk: add linux/vmalloc.h to xsk.c sefltests: netdevsim: wait for devlink instance after netns removal selftest: fib_tests: Always cleanup before exit net/mlx5e: Align IPsec ASO result memory to be as required by hardware net/mlx5e: TC, Set CT miss to the specific ct action instance net/mlx5e: Rename CHAIN_TO_REG to MAPPED_OBJ_TO_REG net/mlx5: Refactor tc miss handling to a single function net/mlx5: Kconfig: Make tc offload depend on tc skb extension net/sched: flower: Support hardware miss to tc action net/sched: flower: Move filter handle initialization earlier net/sched: cls_api: Support hardware miss to tc action net/sched: Rename user cookie and act cookie sfc: fix builds without CONFIG_RTC_LIB sfc: clean up some inconsistent indentings net/mlx4_en: Introduce flexible array to silence overflow warning net: lan966x: Fix possible deadlock inside PTP net/ulp: Remove redundant ->clone() test in inet_clone_ulp(). ... --- Documentation/networking/representors.rst | 259 ++++++++++++++++++++++++++++++ 1 file changed, 259 insertions(+) create mode 100644 Documentation/networking/representors.rst (limited to 'Documentation/networking/representors.rst') diff --git a/Documentation/networking/representors.rst b/Documentation/networking/representors.rst new file mode 100644 index 000000000..ee1f5cd54 --- /dev/null +++ b/Documentation/networking/representors.rst @@ -0,0 +1,259 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================= +Network Function Representors +============================= + +This document describes the semantics and usage of representor netdevices, as +used to control internal switching on SmartNICs. For the closely-related port +representors on physical (multi-port) switches, see +:ref:`Documentation/networking/switchdev.rst `. + +Motivation +---------- + +Since the mid-2010s, network cards have started offering more complex +virtualisation capabilities than the legacy SR-IOV approach (with its simple +MAC/VLAN-based switching model) can support. This led to a desire to offload +software-defined networks (such as OpenVSwitch) to these NICs to specify the +network connectivity of each function. The resulting designs are variously +called SmartNICs or DPUs. + +Network function representors bring the standard Linux networking stack to +virtual switches and IOV devices. Just as each physical port of a Linux- +controlled switch has a separate netdev, so does each virtual port of a virtual +switch. +When the system boots, and before any offload is configured, all packets from +the virtual functions appear in the networking stack of the PF via the +representors. The PF can thus always communicate freely with the virtual +functions. +The PF can configure standard Linux forwarding between representors, the uplink +or any other netdev (routing, bridging, TC classifiers). + +Thus, a representor is both a control plane object (representing the function in +administrative commands) and a data plane object (one end of a virtual pipe). +As a virtual link endpoint, the representor can be configured like any other +netdevice; in some cases (e.g. link state) the representee will follow the +representor's configuration, while in others there are separate APIs to +configure the representee. + +Definitions +----------- + +This document uses the term "switchdev function" to refer to the PCIe function +which has administrative control over the virtual switch on the device. +Typically, this will be a PF, but conceivably a NIC could be configured to grant +these administrative privileges instead to a VF or SF (subfunction). +Depending on NIC design, a multi-port NIC might have a single switchdev function +for the whole device or might have a separate virtual switch, and hence +switchdev function, for each physical network port. +If the NIC supports nested switching, there might be separate switchdev +functions for each nested switch, in which case each switchdev function should +only create representors for the ports on the (sub-)switch it directly +administers. + +A "representee" is the object that a representor represents. So for example in +the case of a VF representor, the representee is the corresponding VF. + +What does a representor do? +--------------------------- + +A representor has three main roles. + +1. It is used to configure the network connection the representee sees, e.g. + link up/down, MTU, etc. For instance, bringing the representor + administratively UP should cause the representee to see a link up / carrier + on event. +2. It provides the slow path for traffic which does not hit any offloaded + fast-path rules in the virtual switch. Packets transmitted on the + representor netdevice should be delivered to the representee; packets + transmitted by the representee which fail to match any switching rule should + be received on the representor netdevice. (That is, there is a virtual pipe + connecting the representor to the representee, similar in concept to a veth + pair.) + This allows software switch implementations (such as OpenVSwitch or a Linux + bridge) to forward packets between representees and the rest of the network. +3. It acts as a handle by which switching rules (such as TC filters) can refer + to the representee, allowing these rules to be offloaded. + +The combination of 2) and 3) means that the behaviour (apart from performance) +should be the same whether a TC filter is offloaded or not. E.g. a TC rule +on a VF representor applies in software to packets received on that representor +netdevice, while in hardware offload it would apply to packets transmitted by +the representee VF. Conversely, a mirred egress redirect to a VF representor +corresponds in hardware to delivery directly to the representee VF. + +What functions should have a representor? +----------------------------------------- + +Essentially, for each virtual port on the device's internal switch, there +should be a representor. +Some vendors have chosen to omit representors for the uplink and the physical +network port, which can simplify usage (the uplink netdev becomes in effect the +physical port's representor) but does not generalise to devices with multiple +ports or uplinks. + +Thus, the following should all have representors: + + - VFs belonging to the switchdev function. + - Other PFs on the local PCIe controller, and any VFs belonging to them. + - PFs and VFs on external PCIe controllers on the device (e.g. for any embedded + System-on-Chip within the SmartNIC). + - PFs and VFs with other personalities, including network block devices (such + as a vDPA virtio-blk PF backed by remote/distributed storage), if (and only + if) their network access is implemented through a virtual switch port. [#]_ + Note that such functions can require a representor despite the representee + not having a netdev. + - Subfunctions (SFs) belonging to any of the above PFs or VFs, if they have + their own port on the switch (as opposed to using their parent PF's port). + - Any accelerators or plugins on the device whose interface to the network is + through a virtual switch port, even if they do not have a corresponding PCIe + PF or VF. + +This allows the entire switching behaviour of the NIC to be controlled through +representor TC rules. + +It is a common misunderstanding to conflate virtual ports with PCIe virtual +functions or their netdevs. While in simple cases there will be a 1:1 +correspondence between VF netdevices and VF representors, more advanced device +configurations may not follow this. +A PCIe function which does not have network access through the internal switch +(not even indirectly through the hardware implementation of whatever services +the function provides) should *not* have a representor (even if it has a +netdev). +Such a function has no switch virtual port for the representor to configure or +to be the other end of the virtual pipe. +The representor represents the virtual port, not the PCIe function nor the 'end +user' netdevice. + +.. [#] The concept here is that a hardware IP stack in the device performs the + translation between block DMA requests and network packets, so that only + network packets pass through the virtual port onto the switch. The network + access that the IP stack "sees" would then be configurable through tc rules; + e.g. its traffic might all be wrapped in a specific VLAN or VxLAN. However, + any needed configuration of the block device *qua* block device, not being a + networking entity, would not be appropriate for the representor and would + thus use some other channel such as devlink. + Contrast this with the case of a virtio-blk implementation which forwards the + DMA requests unchanged to another PF whose driver then initiates and + terminates IP traffic in software; in that case the DMA traffic would *not* + run over the virtual switch and the virtio-blk PF should thus *not* have a + representor. + +How are representors created? +----------------------------- + +The driver instance attached to the switchdev function should, for each virtual +port on the switch, create a pure-software netdevice which has some form of +in-kernel reference to the switchdev function's own netdevice or driver private +data (``netdev_priv()``). +This may be by enumerating ports at probe time, reacting dynamically to the +creation and destruction of ports at run time, or a combination of the two. + +The operations of the representor netdevice will generally involve acting +through the switchdev function. For example, ``ndo_start_xmit()`` might send +the packet through a hardware TX queue attached to the switchdev function, with +either packet metadata or queue configuration marking it for delivery to the +representee. + +How are representors identified? +-------------------------------- + +The representor netdevice should *not* directly refer to a PCIe device (e.g. +through ``net_dev->dev.parent`` / ``SET_NETDEV_DEV()``), either of the +representee or of the switchdev function. +Instead, it should implement the ``ndo_get_devlink_port()`` netdevice op, which +the kernel uses to provide the ``phys_switch_id`` and ``phys_port_name`` sysfs +nodes. (Some legacy drivers implement ``ndo_get_port_parent_id()`` and +``ndo_get_phys_port_name()`` directly, but this is deprecated.) See +:ref:`Documentation/networking/devlink/devlink-port.rst ` for the +details of this API. + +It is expected that userland will use this information (e.g. through udev rules) +to construct an appropriately informative name or alias for the netdevice. For +instance if the switchdev function is ``eth4`` then a representor with a +``phys_port_name`` of ``p0pf1vf2`` might be renamed ``eth4pf1vf2rep``. + +There are as yet no established conventions for naming representors which do not +correspond to PCIe functions (e.g. accelerators and plugins). + +How do representors interact with TC rules? +------------------------------------------- + +Any TC rule on a representor applies (in software TC) to packets received by +that representor netdevice. Thus, if the delivery part of the rule corresponds +to another port on the virtual switch, the driver may choose to offload it to +hardware, applying it to packets transmitted by the representee. + +Similarly, since a TC mirred egress action targeting the representor would (in +software) send the packet through the representor (and thus indirectly deliver +it to the representee), hardware offload should interpret this as delivery to +the representee. + +As a simple example, if ``PORT_DEV`` is the physical port representor and +``REP_DEV`` is a VF representor, the following rules:: + + tc filter add dev $REP_DEV parent ffff: protocol ipv4 flower \ + action mirred egress redirect dev $PORT_DEV + tc filter add dev $PORT_DEV parent ffff: protocol ipv4 flower skip_sw \ + action mirred egress mirror dev $REP_DEV + +would mean that all IPv4 packets from the VF are sent out the physical port, and +all IPv4 packets received on the physical port are delivered to the VF in +addition to ``PORT_DEV``. (Note that without ``skip_sw`` on the second rule, +the VF would get two copies, as the packet reception on ``PORT_DEV`` would +trigger the TC rule again and mirror the packet to ``REP_DEV``.) + +On devices without separate port and uplink representors, ``PORT_DEV`` would +instead be the switchdev function's own uplink netdevice. + +Of course the rules can (if supported by the NIC) include packet-modifying +actions (e.g. VLAN push/pop), which should be performed by the virtual switch. + +Tunnel encapsulation and decapsulation are rather more complicated, as they +involve a third netdevice (a tunnel netdev operating in metadata mode, such as +a VxLAN device created with ``ip link add vxlan0 type vxlan external``) and +require an IP address to be bound to the underlay device (e.g. switchdev +function uplink netdev or port representor). TC rules such as:: + + tc filter add dev $REP_DEV parent ffff: flower \ + action tunnel_key set id $VNI src_ip $LOCAL_IP dst_ip $REMOTE_IP \ + dst_port 4789 \ + action mirred egress redirect dev vxlan0 + tc filter add dev vxlan0 parent ffff: flower enc_src_ip $REMOTE_IP \ + enc_dst_ip $LOCAL_IP enc_key_id $VNI enc_dst_port 4789 \ + action tunnel_key unset action mirred egress redirect dev $REP_DEV + +where ``LOCAL_IP`` is an IP address bound to ``PORT_DEV``, and ``REMOTE_IP`` is +another IP address on the same subnet, mean that packets sent by the VF should +be VxLAN encapsulated and sent out the physical port (the driver has to deduce +this by a route lookup of ``LOCAL_IP`` leading to ``PORT_DEV``, and also +perform an ARP/neighbour table lookup to find the MAC addresses to use in the +outer Ethernet frame), while UDP packets received on the physical port with UDP +port 4789 should be parsed as VxLAN and, if their VSID matches ``$VNI``, +decapsulated and forwarded to the VF. + +If this all seems complicated, just remember the 'golden rule' of TC offload: +the hardware should ensure the same final results as if the packets were +processed through the slow path, traversed software TC (except ignoring any +``skip_hw`` rules and applying any ``skip_sw`` rules) and were transmitted or +received through the representor netdevices. + +Configuring the representee's MAC +--------------------------------- + +The representee's link state is controlled through the representor. Setting the +representor administratively UP or DOWN should cause carrier ON or OFF at the +representee. + +Setting an MTU on the representor should cause that same MTU to be reported to +the representee. +(On hardware that allows configuring separate and distinct MTU and MRU values, +the representor MTU should correspond to the representee's MRU and vice-versa.) + +Currently there is no way to use the representor to set the station permanent +MAC address of the representee; other methods available to do this include: + + - legacy SR-IOV (``ip link set DEVICE vf NUM mac LLADDR``) + - devlink port function (see **devlink-port(8)** and + :ref:`Documentation/networking/devlink/devlink-port.rst `) -- cgit v1.2.3