kernel_samsung_a53x/net/ipv4
Gabriel Krisman Bertazi 6b1a4d01d6 udp: Avoid call to compute_score on multiple sites
[ Upstream commit 50aee97d15113b95a68848db1f0cb2a6c09f753a ]

We've observed a 7-12% performance regression in iperf3 UDP ipv4 and
ipv6 tests with multiple sockets on Zen3 cpus, which we traced back to
commit f0ea27e7bfe1 ("udp: re-score reuseport groups when connected
sockets are present").  The failing tests were those that would spawn
UDP sockets per-cpu on systems that have a high number of cpus.

Unsurprisingly, it is not caused by the extra re-scoring of the reused
socket, but due to the compiler no longer inlining compute_score, once
it has the extra call site in udp4_lib_lookup2.  This is augmented by
the "Safe RET" mitigation for SRSO, needed in our Zen3 cpus.

We could just explicitly inline it, but compute_score() is quite a large
function, around 300b.  Inlining in two sites would almost double
udp4_lib_lookup2, which is a silly thing to do just to workaround a
mitigation.  Instead, this patch shuffles the code a bit to avoid the
multiple calls to compute_score.  Since it is a static function used in
one spot, the compiler can safely fold it in, as it did before, without
increasing the text size.

With this patch applied I ran my original iperf3 testcases.  The failing
cases all looked like this (ipv4):
	iperf3 -c 127.0.0.1 --udp -4 -f K -b $R -l 8920 -t 30 -i 5 -P 64 -O 2

where $R is either 1G/10G/0 (max, unlimited).  I ran 3 times each.
baseline is v6.9-rc3. harmean == harmonic mean; CV == coefficient of
variation.

ipv4:
                 1G                10G                  MAX
	    HARMEAN  (CV)      HARMEAN  (CV)    HARMEAN     (CV)
baseline 1743852.66(0.0208) 1725933.02(0.0167) 1705203.78(0.0386)
patched  1968727.61(0.0035) 1962283.22(0.0195) 1923853.50(0.0256)

ipv6:
                 1G                10G                  MAX
	    HARMEAN  (CV)      HARMEAN  (CV)    HARMEAN     (CV)
baseline 1729020.03(0.0028) 1691704.49(0.0243) 1692251.34(0.0083)
patched  1900422.19(0.0067) 1900968.01(0.0067) 1568532.72(0.1519)

This restores the performance we had before the change above with this
benchmark.  We obviously don't expect any real impact when mitigations
are disabled, but just to be sure it also doesn't regresses:

mitigations=off ipv4:
                 1G                10G                  MAX
	    HARMEAN  (CV)      HARMEAN  (CV)    HARMEAN     (CV)
baseline 3230279.97(0.0066) 3229320.91(0.0060) 2605693.19(0.0697)
patched  3242802.36(0.0073) 3239310.71(0.0035) 2502427.19(0.0882)

Cc: Lorenz Bauer <lmb@isovalent.com>
Fixes: f0ea27e7bfe1 ("udp: re-score reuseport groups when connected sockets are present")
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Reviewed-by: Kuniyuki Iwashima <kuniyu@amazon.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-11-19 12:26:55 +01:00
..
bpfilter Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
netfilter netfilter: complete validation of user input 2024-11-19 11:32:20 +01:00
af_inet.c inet: read sk->sk_family once in inet_recv_error() 2024-11-18 12:13:25 +01:00
ah4.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
arp.c arp: Prevent overflow in arp_req_get(). 2024-11-18 22:25:42 +01:00
bpf_tcp_ca.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
cipso_ipv4.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
datagram.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
devinet.c ipv4: properly combine dev_base_seq and ipv4.dev_addr_genid 2024-11-18 22:25:41 +01:00
esp4.c net: ipv4: fix return value check in esp_remove_trailer 2024-11-08 11:26:10 +01:00
esp4_offload.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
fib_frontend.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
fib_lookup.h Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
fib_notifier.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
fib_rules.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
fib_semantics.c ipv4/fib: send notify when delete source address routes 2024-11-08 11:26:17 +01:00
fib_trie.c ipv4/fib: send notify when delete source address routes 2024-11-08 11:26:17 +01:00
fou.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
gre_demux.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
gre_offload.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
icmp.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
igmp.c bpf: net: Change do_ip_getsockopt() to take the sockptr_t argument 2024-11-19 08:44:49 +01:00
inet_connection_sock.c tcp: properly terminate timers for kernel sockets 2024-11-19 09:22:44 +01:00
inet_diag.c inet_diag: annotate data-races around inet_diag_table[] 2024-11-19 08:44:38 +01:00
inet_fragment.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
inet_hashtables.c net: remove duplicate reuseport_lookup functions 2024-11-19 12:26:55 +01:00
inet_timewait_sock.c tcp: Fix NEW_SYN_RECV handling in inet_twsk_purge() 2024-11-19 11:32:40 +01:00
inetpeer.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
ip_forward.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
ip_fragment.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
ip_gre.c erspan: make sure erspan_base_hdr is present in skb->head 2024-11-19 09:22:47 +01:00
ip_input.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
ip_options.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
ip_output.c net: ipv4: fix a memleak in ip_setup_cork 2024-11-18 12:13:22 +01:00
ip_sockglue.c bpf: net: Change do_ip_getsockopt() to take the sockptr_t argument 2024-11-19 08:44:49 +01:00
ip_tunnel.c Revert "net: ip_tunnel: make sure to pull inner header in ip_tunnel_rcv()" 2024-11-19 09:03:26 +01:00
ip_tunnel_core.c tunnels: fix out of bounds access when building IPv6 PMTU error 2024-11-18 12:13:24 +01:00
ip_vti.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
ipcomp.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
ipconfig.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
ipip.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
ipmr.c ipmr: fix incorrect parameter validation in the ip_mroute_getsockopt() function 2024-11-19 08:44:50 +01:00
ipmr_base.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
Kconfig Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
Makefile Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
metrics.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
netfilter.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
netlink.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
nexthop.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
ping.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
proc.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
protocol.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
raw.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
raw_diag.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
route.c ipv4: check for NULL idev in ip_route_use_hint() 2024-11-19 11:32:37 +01:00
syncookies.c tcp: fix cookie_init_timestamp() overflows 2024-11-18 11:42:50 +01:00
sysctl_net_ipv4.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp.c tcp: defer shutdown(SEND_SHUTDOWN) for TCP_SYN_RECV sockets 2024-11-19 11:32:45 +01:00
tcp_bbr.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_bic.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_bpf.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_cdg.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_cong.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_cubic.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_dctcp.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_dctcp.h Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_diag.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_fastopen.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_highspeed.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_htcp.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_hybla.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_illinois.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_input.c tcp: defer shutdown(SEND_SHUTDOWN) for TCP_SYN_RECV sockets 2024-11-19 11:32:45 +01:00
tcp_ipv4.c tcp: Use refcount_inc_not_zero() in tcp_twsk_unique(). 2024-11-19 11:32:45 +01:00
tcp_lp.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_metrics.c tcp_metrics: do not create an entry from tcp_init_metrics() 2024-11-18 11:42:49 +01:00
tcp_minisocks.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_nv.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_offload.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_output.c tcp: defer shutdown(SEND_SHUTDOWN) for TCP_SYN_RECV sockets 2024-11-19 11:32:45 +01:00
tcp_rate.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_recovery.c tcp: fix excessive TLP and RACK timeouts from HZ rounding 2024-11-08 11:26:10 +01:00
tcp_scalable.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_timer.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_ulp.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_vegas.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_vegas.h Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_veno.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_westwood.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tcp_yeah.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
tunnel4.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
udp.c udp: Avoid call to compute_score on multiple sites 2024-11-19 12:26:55 +01:00
udp_bpf.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
udp_diag.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
udp_impl.h Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
udp_offload.c net: gro: add flush check in udp_gro_receive_segment 2024-11-19 11:32:43 +01:00
udp_tunnel_core.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
udp_tunnel_nic.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
udp_tunnel_stub.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
udplite.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
xfrm4_input.c xfrm: Preserve vlan tags for transport mode software GRO 2024-11-19 11:32:45 +01:00
xfrm4_output.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
xfrm4_policy.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
xfrm4_protocol.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
xfrm4_state.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00
xfrm4_tunnel.c Import A536BXXU9EXDC 2024-06-15 16:02:09 -03:00