From 5b7c4cabbb65f5c469464da6c5f614cbd7f730f2 Mon Sep 17 00:00:00 2001 From: Linus Torvalds Date: Tue, 21 Feb 2023 18:24:12 -0800 Subject: Merge tag 'net-next-6.3' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next Pull networking updates from Jakub Kicinski: "Core: - Add dedicated kmem_cache for typical/small skb->head, avoid having to access struct page at kfree time, and improve memory use. - Introduce sysctl to set default RPS configuration for new netdevs. - Define Netlink protocol specification format which can be used to describe messages used by each family and auto-generate parsers. Add tools for generating kernel data structures and uAPI headers. - Expose all net/core sysctls inside netns. - Remove 4s sleep in netpoll if carrier is instantly detected on boot. - Add configurable limit of MDB entries per port, and port-vlan. - Continue populating drop reasons throughout the stack. - Retire a handful of legacy Qdiscs and classifiers. Protocols: - Support IPv4 big TCP (TSO frames larger than 64kB). - Add IP_LOCAL_PORT_RANGE socket option, to control local port range on socket by socket basis. - Track and report in procfs number of MPTCP sockets used. - Support mixing IPv4 and IPv6 flows in the in-kernel MPTCP path manager. - IPv6: don't check net.ipv6.route.max_size and rely on garbage collection to free memory (similarly to IPv4). - Support Penultimate Segment Pop (PSP) flavor in SRv6 (RFC8986). - ICMP: add per-rate limit counters. - Add support for user scanning requests in ieee802154. - Remove static WEP support. - Support minimal Wi-Fi 7 Extremely High Throughput (EHT) rate reporting. - WiFi 7 EHT channel puncturing support (client & AP). BPF: - Add a rbtree data structure following the "next-gen data structure" precedent set by recently added linked list, that is, by using kfunc + kptr instead of adding a new BPF map type. - Expose XDP hints via kfuncs with initial support for RX hash and timestamp metadata. - Add BPF_F_NO_TUNNEL_KEY extension to bpf_skb_set_tunnel_key to better support decap on GRE tunnel devices not operating in collect metadata. - Improve x86 JIT's codegen for PROBE_MEM runtime error checks. - Remove the need for trace_printk_lock for bpf_trace_printk and bpf_trace_vprintk helpers. - Extend libbpf's bpf_tracing.h support for tracing arguments of kprobes/uprobes and syscall as a special case. - Significantly reduce the search time for module symbols by livepatch and BPF. - Enable cpumasks to be used as kptrs, which is useful for tracing programs tracking which tasks end up running on which CPUs in different time intervals. - Add support for BPF trampoline on s390x and riscv64. - Add capability to export the XDP features supported by the NIC. - Add __bpf_kfunc tag for marking kernel functions as kfuncs. - Add cgroup.memory=nobpf kernel parameter option to disable BPF memory accounting for container environments. Netfilter: - Remove the CLUSTERIP target. It has been marked as obsolete for years, and we still have WARN splats wrt races of the out-of-band /proc interface installed by this target. - Add 'destroy' commands to nf_tables. They are identical to the existing 'delete' commands, but do not return an error if the referenced object (set, chain, rule...) did not exist. Driver API: - Improve cpumask_local_spread() locality to help NICs set the right IRQ affinity on AMD platforms. - Separate C22 and C45 MDIO bus transactions more clearly. - Introduce new DCB table to control DSCP rewrite on egress. - Support configuration of Physical Layer Collision Avoidance (PLCA) Reconciliation Sublayer (RS) (802.3cg-2019). Modern version of shared medium Ethernet. - Support for MAC Merge layer (IEEE 802.3-2018 clause 99). Allowing preemption of low priority frames by high priority frames. - Add support for controlling MACSec offload using netlink SET. - Rework devlink instance refcounts to allow registration and de-registration under the instance lock. Split the code into multiple files, drop some of the unnecessarily granular locks and factor out common parts of netlink operation handling. - Add TX frame aggregation parameters (for USB drivers). - Add a new attr TCA_EXT_WARN_MSG to report TC (offload) warning messages with notifications for debug. - Allow offloading of UDP NEW connections via act_ct. - Add support for per action HW stats in TC. - Support hardware miss to TC action (continue processing in SW from a specific point in the action chain). - Warn if old Wireless Extension user space interface is used with modern cfg80211/mac80211 drivers. Do not support Wireless Extensions for Wi-Fi 7 devices at all. Everyone should switch to using nl80211 interface instead. - Improve the CAN bit timing configuration. Use extack to return error messages directly to user space, update the SJW handling, including the definition of a new default value that will benefit CAN-FD controllers, by increasing their oscillator tolerance. New hardware / drivers: - Ethernet: - nVidia BlueField-3 support (control traffic driver) - Ethernet support for imx93 SoCs - Motorcomm yt8531 gigabit Ethernet PHY - onsemi NCN26000 10BASE-T1S PHY (with support for PLCA) - Microchip LAN8841 PHY (incl. cable diagnostics and PTP) - Amlogic gxl MDIO mux - WiFi: - RealTek RTL8188EU (rtl8xxxu) - Qualcomm Wi-Fi 7 devices (ath12k) - CAN: - Renesas R-Car V4H Drivers: - Bluetooth: - Set Per Platform Antenna Gain (PPAG) for Intel controllers. - Ethernet NICs: - Intel (1G, igc): - support TSN / Qbv / packet scheduling features of i226 model - Intel (100G, ice): - use GNSS subsystem instead of TTY - multi-buffer XDP support - extend support for GPIO pins to E823 devices - nVidia/Mellanox: - update the shared buffer configuration on PFC commands - implement PTP adjphase function for HW offset control - TC support for Geneve and GRE with VF tunnel offload - more efficient crypto key management method - multi-port eswitch support - Netronome/Corigine: - add DCB IEEE support - support IPsec offloading for NFP3800 - Freescale/NXP (enetc): - support XDP_REDIRECT for XDP non-linear buffers - improve reconfig, avoid link flap and waiting for idle - support MAC Merge layer - Other NICs: - sfc/ef100: add basic devlink support for ef100 - ionic: rx_push mode operation (writing descriptors via MMIO) - bnxt: use the auxiliary bus abstraction for RDMA - r8169: disable ASPM and reset bus in case of tx timeout - cpsw: support QSGMII mode for J721e CPSW9G - cpts: support pulse-per-second output - ngbe: add an mdio bus driver - usbnet: optimize usbnet_bh() by avoiding unnecessary queuing - r8152: handle devices with FW with NCM support - amd-xgbe: support 10Mbps, 2.5GbE speeds and rx-adaptation - virtio-net: support multi buffer XDP - virtio/vsock: replace virtio_vsock_pkt with sk_buff - tsnep: XDP support - Ethernet high-speed switches: - nVidia/Mellanox (mlxsw): - add support for latency TLV (in FW control messages) - Microchip (sparx5): - separate explicit and implicit traffic forwarding rules, make the implicit rules always active - add support for egress DSCP rewrite - IS0 VCAP support (Ingress Classification) - IS2 VCAP filters (protos, L3 addrs, L4 ports, flags, ToS etc.) - ES2 VCAP support (Egress Access Control) - support for Per-Stream Filtering and Policing (802.1Q, 8.6.5.1) - Ethernet embedded switches: - Marvell (mv88e6xxx): - add MAB (port auth) offload support - enable PTP receive for mv88e6390 - NXP (ocelot): - support MAC Merge layer - support for the the vsc7512 internal copper phys - Microchip: - lan9303: convert to PHYLINK - lan966x: support TC flower filter statistics - lan937x: PTP support for KSZ9563/KSZ8563 and LAN937x - lan937x: support Credit Based Shaper configuration - ksz9477: support Energy Efficient Ethernet - other: - qca8k: convert to regmap read/write API, use bulk operations - rswitch: Improve TX timestamp accuracy - Intel WiFi (iwlwifi): - EHT (Wi-Fi 7) rate reporting - STEP equalizer support: transfer some STEP (connection to radio on platforms with integrated wifi) related parameters from the BIOS to the firmware. - Qualcomm 802.11ax WiFi (ath11k): - IPQ5018 support - Fine Timing Measurement (FTM) responder role support - channel 177 support - MediaTek WiFi (mt76): - per-PHY LED support - mt7996: EHT (Wi-Fi 7) support - Wireless Ethernet Dispatch (WED) reset support - switch to using page pool allocator - RealTek WiFi (rtw89): - support new version of Bluetooth co-existance - Mobile: - rmnet: support TX aggregation" * tag 'net-next-6.3' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next: (1872 commits) page_pool: add a comment explaining the fragment counter usage net: ethtool: fix __ethtool_dev_mm_supported() implementation ethtool: pse-pd: Fix double word in comments xsk: add linux/vmalloc.h to xsk.c sefltests: netdevsim: wait for devlink instance after netns removal selftest: fib_tests: Always cleanup before exit net/mlx5e: Align IPsec ASO result memory to be as required by hardware net/mlx5e: TC, Set CT miss to the specific ct action instance net/mlx5e: Rename CHAIN_TO_REG to MAPPED_OBJ_TO_REG net/mlx5: Refactor tc miss handling to a single function net/mlx5: Kconfig: Make tc offload depend on tc skb extension net/sched: flower: Support hardware miss to tc action net/sched: flower: Move filter handle initialization earlier net/sched: cls_api: Support hardware miss to tc action net/sched: Rename user cookie and act cookie sfc: fix builds without CONFIG_RTC_LIB sfc: clean up some inconsistent indentings net/mlx4_en: Introduce flexible array to silence overflow warning net: lan966x: Fix possible deadlock inside PTP net/ulp: Remove redundant ->clone() test in inet_clone_ulp(). ... --- drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.c | 514 +++++++++++++++++++++++++++ 1 file changed, 514 insertions(+) create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.c (limited to 'drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.c') diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.c new file mode 100644 index 000000000..62079f0e3 --- /dev/null +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.c @@ -0,0 +1,514 @@ +/* + * Copyright 2022 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ +#include +#include + +#include "amdgpu_ring_mux.h" +#include "amdgpu_ring.h" +#include "amdgpu.h" + +#define AMDGPU_MUX_RESUBMIT_JIFFIES_TIMEOUT (HZ / 2) +#define AMDGPU_MAX_LAST_UNSIGNALED_THRESHOLD_US 10000 + +static const struct ring_info { + unsigned int hw_pio; + const char *ring_name; +} sw_ring_info[] = { + { AMDGPU_RING_PRIO_DEFAULT, "gfx_low"}, + { AMDGPU_RING_PRIO_2, "gfx_high"}, +}; + +static struct kmem_cache *amdgpu_mux_chunk_slab; + +static inline struct amdgpu_mux_entry *amdgpu_ring_mux_sw_entry(struct amdgpu_ring_mux *mux, + struct amdgpu_ring *ring) +{ + return ring->entry_index < mux->ring_entry_size ? + &mux->ring_entry[ring->entry_index] : NULL; +} + +/* copy packages on sw ring range[begin, end) */ +static void amdgpu_ring_mux_copy_pkt_from_sw_ring(struct amdgpu_ring_mux *mux, + struct amdgpu_ring *ring, + u64 s_start, u64 s_end) +{ + u64 start, end; + struct amdgpu_ring *real_ring = mux->real_ring; + + start = s_start & ring->buf_mask; + end = s_end & ring->buf_mask; + + if (start == end) { + DRM_ERROR("no more data copied from sw ring\n"); + return; + } + if (start > end) { + amdgpu_ring_alloc(real_ring, (ring->ring_size >> 2) + end - start); + amdgpu_ring_write_multiple(real_ring, (void *)&ring->ring[start], + (ring->ring_size >> 2) - start); + amdgpu_ring_write_multiple(real_ring, (void *)&ring->ring[0], end); + } else { + amdgpu_ring_alloc(real_ring, end - start); + amdgpu_ring_write_multiple(real_ring, (void *)&ring->ring[start], end - start); + } +} + +static void amdgpu_mux_resubmit_chunks(struct amdgpu_ring_mux *mux) +{ + struct amdgpu_mux_entry *e = NULL; + struct amdgpu_mux_chunk *chunk; + uint32_t seq, last_seq; + int i; + + /*find low priority entries:*/ + if (!mux->s_resubmit) + return; + + for (i = 0; i < mux->num_ring_entries; i++) { + if (mux->ring_entry[i].ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT) { + e = &mux->ring_entry[i]; + break; + } + } + + if (!e) { + DRM_ERROR("%s no low priority ring found\n", __func__); + return; + } + + last_seq = atomic_read(&e->ring->fence_drv.last_seq); + seq = mux->seqno_to_resubmit; + if (last_seq < seq) { + /*resubmit all the fences between (last_seq, seq]*/ + list_for_each_entry(chunk, &e->list, entry) { + if (chunk->sync_seq > last_seq && chunk->sync_seq <= seq) { + amdgpu_fence_update_start_timestamp(e->ring, + chunk->sync_seq, + ktime_get()); + amdgpu_ring_mux_copy_pkt_from_sw_ring(mux, e->ring, + chunk->start, + chunk->end); + mux->wptr_resubmit = chunk->end; + amdgpu_ring_commit(mux->real_ring); + } + } + } + + del_timer(&mux->resubmit_timer); + mux->s_resubmit = false; +} + +static void amdgpu_ring_mux_schedule_resubmit(struct amdgpu_ring_mux *mux) +{ + mod_timer(&mux->resubmit_timer, jiffies + AMDGPU_MUX_RESUBMIT_JIFFIES_TIMEOUT); +} + +static void amdgpu_mux_resubmit_fallback(struct timer_list *t) +{ + struct amdgpu_ring_mux *mux = from_timer(mux, t, resubmit_timer); + + if (!spin_trylock(&mux->lock)) { + amdgpu_ring_mux_schedule_resubmit(mux); + DRM_ERROR("reschedule resubmit\n"); + return; + } + amdgpu_mux_resubmit_chunks(mux); + spin_unlock(&mux->lock); +} + +int amdgpu_ring_mux_init(struct amdgpu_ring_mux *mux, struct amdgpu_ring *ring, + unsigned int entry_size) +{ + mux->real_ring = ring; + mux->num_ring_entries = 0; + + mux->ring_entry = kcalloc(entry_size, sizeof(struct amdgpu_mux_entry), GFP_KERNEL); + if (!mux->ring_entry) + return -ENOMEM; + + mux->ring_entry_size = entry_size; + mux->s_resubmit = false; + + amdgpu_mux_chunk_slab = kmem_cache_create("amdgpu_mux_chunk", + sizeof(struct amdgpu_mux_chunk), 0, + SLAB_HWCACHE_ALIGN, NULL); + if (!amdgpu_mux_chunk_slab) { + DRM_ERROR("create amdgpu_mux_chunk cache failed\n"); + return -ENOMEM; + } + + spin_lock_init(&mux->lock); + timer_setup(&mux->resubmit_timer, amdgpu_mux_resubmit_fallback, 0); + + return 0; +} + +void amdgpu_ring_mux_fini(struct amdgpu_ring_mux *mux) +{ + struct amdgpu_mux_entry *e; + struct amdgpu_mux_chunk *chunk, *chunk2; + int i; + + for (i = 0; i < mux->num_ring_entries; i++) { + e = &mux->ring_entry[i]; + list_for_each_entry_safe(chunk, chunk2, &e->list, entry) { + list_del(&chunk->entry); + kmem_cache_free(amdgpu_mux_chunk_slab, chunk); + } + } + kmem_cache_destroy(amdgpu_mux_chunk_slab); + kfree(mux->ring_entry); + mux->ring_entry = NULL; + mux->num_ring_entries = 0; + mux->ring_entry_size = 0; +} + +int amdgpu_ring_mux_add_sw_ring(struct amdgpu_ring_mux *mux, struct amdgpu_ring *ring) +{ + struct amdgpu_mux_entry *e; + + if (mux->num_ring_entries >= mux->ring_entry_size) { + DRM_ERROR("add sw ring exceeding max entry size\n"); + return -ENOENT; + } + + e = &mux->ring_entry[mux->num_ring_entries]; + ring->entry_index = mux->num_ring_entries; + e->ring = ring; + + INIT_LIST_HEAD(&e->list); + mux->num_ring_entries += 1; + return 0; +} + +void amdgpu_ring_mux_set_wptr(struct amdgpu_ring_mux *mux, struct amdgpu_ring *ring, u64 wptr) +{ + struct amdgpu_mux_entry *e; + + spin_lock(&mux->lock); + + if (ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT) + amdgpu_mux_resubmit_chunks(mux); + + e = amdgpu_ring_mux_sw_entry(mux, ring); + if (!e) { + DRM_ERROR("cannot find entry for sw ring\n"); + spin_unlock(&mux->lock); + return; + } + + /* We could skip this set wptr as preemption in process. */ + if (ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT && mux->pending_trailing_fence_signaled) { + spin_unlock(&mux->lock); + return; + } + + e->sw_cptr = e->sw_wptr; + /* Update cptr if the package already copied in resubmit functions */ + if (ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT && e->sw_cptr < mux->wptr_resubmit) + e->sw_cptr = mux->wptr_resubmit; + e->sw_wptr = wptr; + e->start_ptr_in_hw_ring = mux->real_ring->wptr; + + /* Skip copying for the packages already resubmitted.*/ + if (ring->hw_prio > AMDGPU_RING_PRIO_DEFAULT || mux->wptr_resubmit < wptr) { + amdgpu_ring_mux_copy_pkt_from_sw_ring(mux, ring, e->sw_cptr, wptr); + e->end_ptr_in_hw_ring = mux->real_ring->wptr; + amdgpu_ring_commit(mux->real_ring); + } else { + e->end_ptr_in_hw_ring = mux->real_ring->wptr; + } + spin_unlock(&mux->lock); +} + +u64 amdgpu_ring_mux_get_wptr(struct amdgpu_ring_mux *mux, struct amdgpu_ring *ring) +{ + struct amdgpu_mux_entry *e; + + e = amdgpu_ring_mux_sw_entry(mux, ring); + if (!e) { + DRM_ERROR("cannot find entry for sw ring\n"); + return 0; + } + + return e->sw_wptr; +} + +/** + * amdgpu_ring_mux_get_rptr - get the readptr of the software ring + * @mux: the multiplexer the software rings attach to + * @ring: the software ring of which we calculate the readptr + * + * The return value of the readptr is not precise while the other rings could + * write data onto the real ring buffer.After overwriting on the real ring, we + * can not decide if our packages have been excuted or not read yet. However, + * this function is only called by the tools such as umr to collect the latest + * packages for the hang analysis. We assume the hang happens near our latest + * submit. Thus we could use the following logic to give the clue: + * If the readptr is between start and end, then we return the copy pointer + * plus the distance from start to readptr. If the readptr is before start, we + * return the copy pointer. Lastly, if the readptr is past end, we return the + * write pointer. + */ +u64 amdgpu_ring_mux_get_rptr(struct amdgpu_ring_mux *mux, struct amdgpu_ring *ring) +{ + struct amdgpu_mux_entry *e; + u64 readp, offset, start, end; + + e = amdgpu_ring_mux_sw_entry(mux, ring); + if (!e) { + DRM_ERROR("no sw entry found!\n"); + return 0; + } + + readp = amdgpu_ring_get_rptr(mux->real_ring); + + start = e->start_ptr_in_hw_ring & mux->real_ring->buf_mask; + end = e->end_ptr_in_hw_ring & mux->real_ring->buf_mask; + if (start > end) { + if (readp <= end) + readp += mux->real_ring->ring_size >> 2; + end += mux->real_ring->ring_size >> 2; + } + + if (start <= readp && readp <= end) { + offset = readp - start; + e->sw_rptr = (e->sw_cptr + offset) & ring->buf_mask; + } else if (readp < start) { + e->sw_rptr = e->sw_cptr; + } else { + /* end < readptr */ + e->sw_rptr = e->sw_wptr; + } + + return e->sw_rptr; +} + +u64 amdgpu_sw_ring_get_rptr_gfx(struct amdgpu_ring *ring) +{ + struct amdgpu_device *adev = ring->adev; + struct amdgpu_ring_mux *mux = &adev->gfx.muxer; + + WARN_ON(!ring->is_sw_ring); + return amdgpu_ring_mux_get_rptr(mux, ring); +} + +u64 amdgpu_sw_ring_get_wptr_gfx(struct amdgpu_ring *ring) +{ + struct amdgpu_device *adev = ring->adev; + struct amdgpu_ring_mux *mux = &adev->gfx.muxer; + + WARN_ON(!ring->is_sw_ring); + return amdgpu_ring_mux_get_wptr(mux, ring); +} + +void amdgpu_sw_ring_set_wptr_gfx(struct amdgpu_ring *ring) +{ + struct amdgpu_device *adev = ring->adev; + struct amdgpu_ring_mux *mux = &adev->gfx.muxer; + + WARN_ON(!ring->is_sw_ring); + amdgpu_ring_mux_set_wptr(mux, ring, ring->wptr); +} + +/* Override insert_nop to prevent emitting nops to the software rings */ +void amdgpu_sw_ring_insert_nop(struct amdgpu_ring *ring, uint32_t count) +{ + WARN_ON(!ring->is_sw_ring); +} + +const char *amdgpu_sw_ring_name(int idx) +{ + return idx < ARRAY_SIZE(sw_ring_info) ? + sw_ring_info[idx].ring_name : NULL; +} + +unsigned int amdgpu_sw_ring_priority(int idx) +{ + return idx < ARRAY_SIZE(sw_ring_info) ? + sw_ring_info[idx].hw_pio : AMDGPU_RING_PRIO_DEFAULT; +} + +/*Scan on low prio rings to have unsignaled fence and high ring has no fence.*/ +static int amdgpu_mcbp_scan(struct amdgpu_ring_mux *mux) +{ + struct amdgpu_ring *ring; + int i, need_preempt; + + need_preempt = 0; + for (i = 0; i < mux->num_ring_entries; i++) { + ring = mux->ring_entry[i].ring; + if (ring->hw_prio > AMDGPU_RING_PRIO_DEFAULT && + amdgpu_fence_count_emitted(ring) > 0) + return 0; + if (ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT && + amdgpu_fence_last_unsignaled_time_us(ring) > + AMDGPU_MAX_LAST_UNSIGNALED_THRESHOLD_US) + need_preempt = 1; + } + return need_preempt && !mux->s_resubmit; +} + +/* Trigger Mid-Command Buffer Preemption (MCBP) and find if we need to resubmit. */ +static int amdgpu_mcbp_trigger_preempt(struct amdgpu_ring_mux *mux) +{ + int r; + + spin_lock(&mux->lock); + mux->pending_trailing_fence_signaled = true; + r = amdgpu_ring_preempt_ib(mux->real_ring); + spin_unlock(&mux->lock); + return r; +} + +void amdgpu_sw_ring_ib_begin(struct amdgpu_ring *ring) +{ + struct amdgpu_device *adev = ring->adev; + struct amdgpu_ring_mux *mux = &adev->gfx.muxer; + + WARN_ON(!ring->is_sw_ring); + if (ring->hw_prio > AMDGPU_RING_PRIO_DEFAULT) { + if (amdgpu_mcbp_scan(mux) > 0) + amdgpu_mcbp_trigger_preempt(mux); + return; + } + + amdgpu_ring_mux_start_ib(mux, ring); +} + +void amdgpu_sw_ring_ib_end(struct amdgpu_ring *ring) +{ + struct amdgpu_device *adev = ring->adev; + struct amdgpu_ring_mux *mux = &adev->gfx.muxer; + + WARN_ON(!ring->is_sw_ring); + if (ring->hw_prio > AMDGPU_RING_PRIO_DEFAULT) + return; + amdgpu_ring_mux_end_ib(mux, ring); +} + +void amdgpu_ring_mux_start_ib(struct amdgpu_ring_mux *mux, struct amdgpu_ring *ring) +{ + struct amdgpu_mux_entry *e; + struct amdgpu_mux_chunk *chunk; + + spin_lock(&mux->lock); + amdgpu_mux_resubmit_chunks(mux); + spin_unlock(&mux->lock); + + e = amdgpu_ring_mux_sw_entry(mux, ring); + if (!e) { + DRM_ERROR("cannot find entry!\n"); + return; + } + + chunk = kmem_cache_alloc(amdgpu_mux_chunk_slab, GFP_KERNEL); + if (!chunk) { + DRM_ERROR("alloc amdgpu_mux_chunk_slab failed\n"); + return; + } + + chunk->start = ring->wptr; + list_add_tail(&chunk->entry, &e->list); +} + +static void scan_and_remove_signaled_chunk(struct amdgpu_ring_mux *mux, struct amdgpu_ring *ring) +{ + uint32_t last_seq = 0; + struct amdgpu_mux_entry *e; + struct amdgpu_mux_chunk *chunk, *tmp; + + e = amdgpu_ring_mux_sw_entry(mux, ring); + if (!e) { + DRM_ERROR("cannot find entry!\n"); + return; + } + + last_seq = atomic_read(&ring->fence_drv.last_seq); + + list_for_each_entry_safe(chunk, tmp, &e->list, entry) { + if (chunk->sync_seq <= last_seq) { + list_del(&chunk->entry); + kmem_cache_free(amdgpu_mux_chunk_slab, chunk); + } + } +} + +void amdgpu_ring_mux_end_ib(struct amdgpu_ring_mux *mux, struct amdgpu_ring *ring) +{ + struct amdgpu_mux_entry *e; + struct amdgpu_mux_chunk *chunk; + + e = amdgpu_ring_mux_sw_entry(mux, ring); + if (!e) { + DRM_ERROR("cannot find entry!\n"); + return; + } + + chunk = list_last_entry(&e->list, struct amdgpu_mux_chunk, entry); + if (!chunk) { + DRM_ERROR("cannot find chunk!\n"); + return; + } + + chunk->end = ring->wptr; + chunk->sync_seq = READ_ONCE(ring->fence_drv.sync_seq); + + scan_and_remove_signaled_chunk(mux, ring); +} + +bool amdgpu_mcbp_handle_trailing_fence_irq(struct amdgpu_ring_mux *mux) +{ + struct amdgpu_mux_entry *e; + struct amdgpu_ring *ring = NULL; + int i; + + if (!mux->pending_trailing_fence_signaled) + return false; + + if (mux->real_ring->trail_seq != le32_to_cpu(*mux->real_ring->trail_fence_cpu_addr)) + return false; + + for (i = 0; i < mux->num_ring_entries; i++) { + e = &mux->ring_entry[i]; + if (e->ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT) { + ring = e->ring; + break; + } + } + + if (!ring) { + DRM_ERROR("cannot find low priority ring\n"); + return false; + } + + amdgpu_fence_process(ring); + if (amdgpu_fence_count_emitted(ring) > 0) { + mux->s_resubmit = true; + mux->seqno_to_resubmit = ring->fence_drv.sync_seq; + amdgpu_ring_mux_schedule_resubmit(mux); + } + + mux->pending_trailing_fence_signaled = false; + return true; +} -- cgit v1.2.3