Add hw_stats field to several notifier structures to communicate to the
drivers that HW statistics should be configured for nexthops within a given
group.
Signed-off-by: Ido Schimmel <idosch@nvidia.com>
Reviewed-by: David Ahern <dsahern@kernel.org>
Signed-off-by: Petr Machata <petrm@nvidia.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Add netlink support for reading NH group stats.
This data is only for statistics of the traffic in the SW datapath. HW
nexthop group statistics will be added in the following patches.
Emission of the stats is keyed to a new op_stats flag to avoid cluttering
the netlink message with stats if the user doesn't need them:
NHA_OP_FLAG_DUMP_STATS.
Co-developed-by: Petr Machata <petrm@nvidia.com>
Signed-off-by: Petr Machata <petrm@nvidia.com>
Signed-off-by: Ido Schimmel <idosch@nvidia.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Add nexthop group entry stats to count the number of packets forwarded
via each nexthop in the group. The stats will be exposed to user space
for better data path observability in the next patch.
The per-CPU stats pointer is placed at the beginning of 'struct
nh_grp_entry', so that all the fields accessed for the data path reside
on the same cache line:
struct nh_grp_entry {
struct nexthop * nh; /* 0 8 */
struct nh_grp_entry_stats * stats; /* 8 8 */
u8 weight; /* 16 1 */
/* XXX 7 bytes hole, try to pack */
union {
struct {
atomic_t upper_bound; /* 24 4 */
} hthr; /* 24 4 */
struct {
struct list_head uw_nh_entry; /* 24 16 */
u16 count_buckets; /* 40 2 */
u16 wants_buckets; /* 42 2 */
} res; /* 24 24 */
}; /* 24 24 */
struct list_head nh_list; /* 48 16 */
/* --- cacheline 1 boundary (64 bytes) --- */
struct nexthop * nh_parent; /* 64 8 */
/* size: 72, cachelines: 2, members: 6 */
/* sum members: 65, holes: 1, sum holes: 7 */
/* last cacheline: 8 bytes */
};
Co-developed-by: Petr Machata <petrm@nvidia.com>
Signed-off-by: Petr Machata <petrm@nvidia.com>
Signed-off-by: Ido Schimmel <idosch@nvidia.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
In order to add per-nexthop statistics, but still not increase netlink
message size for consumers that do not care about them, there needs to be a
toggle through which the user indicates their desire to get the statistics.
To that end, add a new attribute, NHA_OP_FLAGS. The idea is to be able to
use the attribute for carrying of arbitrary operation-specific flags, i.e.
not make it specific for get / dump.
Add the new attribute to get and dump policies, but do not actually allow
any flags yet -- those will come later as the flags themselves are defined.
Add the necessary parsing code.
Signed-off-by: Petr Machata <petrm@nvidia.com>
Reviewed-by: David Ahern <dsahern@kernel.org>
Reviewed-by: Ido Schimmel <idosch@nvidia.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
A following patch will introduce a new attribute, op-specific flags to
adjust the behavior of an operation. Different operations will recognize
different flags.
- To make the differentiation possible, stop sharing the policies for get
and del operations.
- To allow querying for presence of the attribute, have all the attribute
arrays sized to NHA_MAX, regardless of what is permitted by policy, and
pass the corresponding value to nlmsg_parse() as well.
Signed-off-by: Petr Machata <petrm@nvidia.com>
Reviewed-by: David Ahern <dsahern@kernel.org>
Reviewed-by: Ido Schimmel <idosch@nvidia.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
This patch adds TC offload support for matching TCP flags
from TCP header.
Example usage:
tc qdisc add dev eth0 ingress
TC rule to drop the TCP SYN packets:
tc filter add dev eth0 ingress protocol ip flower ip_proto tcp tcp_flags
0x02/0x3f skip_sw action drop
Signed-off-by: Sai Krishna <saikrishnag@marvell.com>
Reviewed-by: Simon Horman <horms@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
It is useful to expose skb addr and sock addr to user in tracepoint
tcp_probe, so that we can get more information while monitoring
receiving of tcp data, by ebpf or other ways.
For example, we need to identify a packet by seq and end_seq when
calculate transmit latency between layer 2 and layer 4 by ebpf, but which is
not available in tcp_probe, so we can only use kprobe hooking
tcp_rcv_established to get them. But we can use tcp_probe directly if skb
addr and sock addr are available, which is more efficient.
Signed-off-by: fuyuanli <fuyuanli@didiglobal.com>
Reviewed-by: Jason Xing <kerneljasonxing@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
softnet_data->time_squeeze is sometimes used as a proxy for
host overload or indication of scheduling problems. In practice
this statistic is very noisy and has hard to grasp units -
e.g. is 10 squeezes a second to be expected, or high?
Delaying network (NAPI) processing leads to drops on NIC queues
but also RTT bloat, impacting pacing and CA decisions.
Stalls are a little hard to detect on the Rx side, because
there may simply have not been any packets received in given
period of time. Packet timestamps help a little bit, but
again we don't know if packets are stale because we're
not keeping up or because someone (*cough* cgroups)
disabled IRQs for a long time.
We can, however, use Tx as a proxy for Rx stalls. Most drivers
use combined Rx+Tx NAPIs so if Tx gets starved so will Rx.
On the Tx side we know exactly when packets get queued,
and completed, so there is no uncertainty.
This patch adds stall checks to BQL. Why BQL? Because
it's a convenient place to add such checks, already
called by most drivers, and it has copious free space
in its structures (this patch adds no extra cache
references or dirtying to the fast path).
The algorithm takes one parameter - max delay AKA stall
threshold and increments a counter whenever NAPI got delayed
for at least that amount of time. It also records the length
of the longest stall.
To be precise every time NAPI has not polled for at least
stall thrs we check if there were any Tx packets queued
between last NAPI run and now - stall_thrs/2.
Unlike the classic Tx watchdog this mechanism does not
ignore stalls caused by Tx being disabled, or loss of link.
I don't think the check is worth the complexity, and
stall is a stall, whether due to host overload, flow
control, link down... doesn't matter much to the application.
We have been running this detector in production at Meta
for 2 years, with the threshold of 8ms. It's the lowest
value where false positives become rare. There's still
a constant stream of reported stalls (especially without
the ksoftirqd deferral patches reverted), those who like
their stall metrics to be 0 may prefer higher value.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
The inlined helper function calc_tx_descs is not used and is redundant.
Remove it.
Cleans up clang scan build warning:
drivers/net/ethernet/chelsio/cxgb4/sge.c:814:28: warning: unused
function 'calc_tx_descs' [-Wunused-function]
Signed-off-by: Colin Ian King <colin.i.king@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The ethtool-nl family does a good job exposing various protocol
related and IEEE/IETF statistics which used to get dumped under
ethtool -S, with creative names. Queue stats don't have a netlink
API, yet, and remain a lion's share of ethtool -S output for new
drivers. Not only is that bad because the names differ driver to
driver but it's also bug-prone. Intuitively drivers try to report
only the stats for active queues, but querying ethtool stats
involves multiple system calls, and the number of stats is
read separately from the stats themselves. Worse still when user
space asks for values of the stats, it doesn't inform the kernel
how big the buffer is. If number of stats increases in the meantime
kernel will overflow user buffer.
Add a netlink API for dumping queue stats. Queue information is
exposed via the netdev-genl family, so add the stats there.
Support per-queue and sum-for-device dumps. Latter will be useful
when subsequent patches add more interesting common stats than
just bytes and packets.
The API does not currently distinguish between HW and SW stats.
The expectation is that the source of the stats will either not
matter much (good packets) or be obvious (skb alloc errors).
Acked-by: Stanislav Fomichev <sdf@google.com>
Reviewed-by: Amritha Nambiar <amritha.nambiar@intel.com>
Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240306195509.1502746-2-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Eric Dumazet says:
====================
net: group together hot data
While our recent structure reorganizations were focused
on increasing max throughput, there is still an
area where improvements are much needed.
In many cases, a cpu handles one packet at a time,
instead of a nice batch.
Hardware interrupt.
-> Software interrupt.
-> Network/Protocol stacks.
If the cpu was idle or busy in other layers,
it has to pull many cache lines.
This series adds a new net_hotdata structure, where
some critical (and read-mostly) data used in
rx and tx path is packed in a small number of cache lines.
Synthetic benchmarks will not see much difference,
but latency of single packet should improve.
net_hodata current size on 64bit is 416 bytes,
but might grow in the future.
Also move RPS definitions to a new include file.
====================
Link: https://lore.kernel.org/r/20240306160031.874438-1-edumazet@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
shellcheck recently helped to prevent issues. It is then good to fix the
other harmless issues in order to spot "real" ones later.
Here, two categories of warnings are now ignored:
- SC2317: Command appears to be unreachable. The cleanup() function is
invoked indirectly via the EXIT trap.
- SC2086: Double quote to prevent globbing and word splitting. This is
recommended, but the current usage is correct and there is no need to
do all these modifications to be compliant with this rule.
For the modifications:
- SC2034: ksft_skip appears unused.
- SC2004: $/${} is unnecessary on arithmetic variables.
Now this script is shellcheck (0.9.0) compliant. We can easily spot new
issues.
Reviewed-by: Mat Martineau <martineau@kernel.org>
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
Link: https://lore.kernel.org/r/20240306-upstream-net-next-20240304-selftests-mptcp-shared-code-shellcheck-v2-11-bc79e6e5e6a0@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
shellcheck recently helped to prevent issues. It is then good to fix the
other harmless issues in order to spot "real" ones later.
Here, two categories of warnings are now ignored:
- SC2317: Command appears to be unreachable. The cleanup() function is
invoked indirectly via the EXIT trap.
- SC2086: Double quote to prevent globbing and word splitting. This is
recommended, but the current usage is correct and there is no need to
do all these modifications to be compliant with this rule.
For the modifications:
- SC2034: ksft_skip appears unused.
- SC2154: optstring is referenced but not assigned.
- SC2006: Use $(...) notation instead of legacy backticks `...`.
Now this script is shellcheck (0.9.0) compliant. We can easily spot new
issues.
Reviewed-by: Mat Martineau <martineau@kernel.org>
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
Link: https://lore.kernel.org/r/20240306-upstream-net-next-20240304-selftests-mptcp-shared-code-shellcheck-v2-10-bc79e6e5e6a0@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
shellcheck recently helped to prevent issues. It is then good to fix the
other harmless issues in order to spot "real" ones later.
Here, two categories of warnings are now ignored:
- SC2317: Command appears to be unreachable. The cleanup() function is
invoked indirectly via the EXIT trap.
- SC2086: Double quote to prevent globbing and word splitting. This is
recommended, but the current usage is correct and there is no need to
do all these modifications to be compliant with this rule.
For the modifications:
- SC2034: ksft_skip appears unused.
- SC2006: Use $(...) notation instead of legacy backticks `...`.
- SC2145: Argument mixes string and array. Use * or separate argument.
Now this script is shellcheck (0.9.0) compliant. We can easily spot new
issues.
Reviewed-by: Mat Martineau <martineau@kernel.org>
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
Link: https://lore.kernel.org/r/20240306-upstream-net-next-20240304-selftests-mptcp-shared-code-shellcheck-v2-9-bc79e6e5e6a0@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
shellcheck recently helped to prevent issues. It is then good to fix the
other harmless issues in order to spot "real" ones later.
Here, two categories of warnings are now ignored:
- SC2317: Command appears to be unreachable. The cleanup() function is
invoked indirectly via the EXIT trap.
- SC2086: Double quote to prevent globbing and word splitting. This is
recommended, but the current usage is correct and there is no need to
do all these modifications to be compliant with this rule.
For the modifications:
- SC2034: ksft_skip appears unused.
- SC2181: Check exit code directly with e.g. 'if mycmd;', not
indirectly with $?.
- SC2004: $/${} is unnecessary on arithmetic variables.
- SC2155: Declare and assign separately to avoid masking return
values.
- SC2166: Prefer [ p ] && [ q ] as [ p -a q ] is not well defined.
- SC2059: Don't use variables in the printf format string. Use printf
'..%s..' "$foo".
Now this script is shellcheck (0.9.0) compliant. We can easily spot new
issues.
Reviewed-by: Mat Martineau <martineau@kernel.org>
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
Link: https://lore.kernel.org/r/20240306-upstream-net-next-20240304-selftests-mptcp-shared-code-shellcheck-v2-8-bc79e6e5e6a0@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
shellcheck recently helped to prevent issues. It is then good to fix the
other harmless issues in order to spot "real" ones later.
Here, two categories of warnings are now ignored:
- SC2317: Command appears to be unreachable. The cleanup() function is
invoked indirectly via the EXIT trap.
- SC2086: Double quote to prevent globbing and word splitting. This is
recommended, but the current usage is correct and there is no need to
do all these modifications to be compliant with this rule.
For the modifications:
- SC2034: ksft_skip appears unused.
- SC2046: Quote '$(get_msk_inuse)' to prevent word splitting.
- SC2006: Use $(...) notation instead of legacy backticks `...`.
Now this script is shellcheck (0.9.0) compliant. We can easily spot new
issues.
Reviewed-by: Mat Martineau <martineau@kernel.org>
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
Link: https://lore.kernel.org/r/20240306-upstream-net-next-20240304-selftests-mptcp-shared-code-shellcheck-v2-7-bc79e6e5e6a0@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
To avoid duplicated code in different MPTCP selftests, we can add and
use helpers defined in mptcp_lib.sh.
This patch unifies "pm_nl_ctl events" related code in userspace_pm.sh
and mptcp_join.sh into a helper mptcp_lib_events(). Define it in
mptcp_lib.sh and use it in both scripts.
Note that mptcp_lib_kill_wait is now call before starting 'events' for
mptcp_join.sh as well, but that's fine: each test is started from a new
netns, so there will not be any existing pid there, and nothing is done
when mptcp_lib_kill_wait is called with 0.
Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
Reviewed-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
Link: https://lore.kernel.org/r/20240306-upstream-net-next-20240304-selftests-mptcp-shared-code-shellcheck-v2-6-bc79e6e5e6a0@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
David Howells says:
====================
Here are some changes to AF_RXRPC:
(1) Cache the transmission serial number of ACK and DATA packets in the
rxrpc_txbuf struct and log this in the retransmit tracepoint.
(2) Don't use atomics on rxrpc_txbuf::flags[*] and cache the intended wire
header flags there too to avoid duplication.
(3) Cache the wire checksum in rxrpc_txbuf to make it easier to create
jumbo packets in future (which will require altering the wire header
to a jumbo header and restoring it back again for retransmission).
(4) Fix the protocol names in the wire ACK trailer struct.
(5) Strip all the barriers and atomics out of the call timer tracking[*].
(6) Remove atomic handling from call->tx_transmitted and
call->acks_prev_seq[*].
(7) Don't bother resetting the DF flag after UDP packet transmission. To
change it, we now call directly into UDP code, so it's quick just to
set it every time.
(8) Merge together the DF/non-DF branches of the DATA transmission to
reduce duplication in the code.
(9) Add a kvec array into rxrpc_txbuf and start moving things over to it.
This paves the way for using page frags.
(10) Split (sub)packet preparation and timestamping out of the DATA
transmission function. This helps pave the way for future jumbo
packet generation.
(11) In rxkad, don't pick values out of the wire header stored in
rxrpc_txbuf, buf rather find them elsewhere so we can remove the wire
header from there.
(12) Move rxrpc_send_ACK() to output.c so that it can be merged with
rxrpc_send_ack_packet().
(13) Use rxrpc_txbuf::kvec[0] to access the wire header for the packet
rather than directly accessing the copy in rxrpc_txbuf. This will
allow that to be removed to a page frag.
(14) Switch from keeping the transmission buffers in rxrpc_txbuf allocated
in the slab to allocating them using page fragment allocators. There
are separate allocators for DATA packets (which persist for a while)
and control packets (which are discarded immediately).
We can then turn on MSG_SPLICE_PAGES when transmitting DATA and ACK
packets.
We can also get rid of the RCU cleanup on rxrpc_txbufs, preferring
instead to release the page frags as soon as possible.
(15) Parse received packets before handling timeouts as the former may
reset the latter.
(16) Make sure we don't retransmit DATA packets after all the packets have
been ACK'd.
(17) Differentiate traces for PING ACK transmission.
(18) Switch to keeping timeouts as ktime_t rather than a number of jiffies
as the latter is too coarse a granularity. Only set the call timer at
the end of the call event function from the aggregate of all the
timeouts, thereby reducing the number of timer calls made. In future,
it might be possible to reduce the number of timers from one per call
to one per I/O thread and to use a high-precision timer.
(19) Record RTT probes after successful transmission rather than recording
it before and then cancelling it after if unsuccessful[*]. This
allows a number of calls to get the current time to be removed.
(20) Clean up the resend algorithm as there's now no need to walk the
transmission buffer under lock[*]. DATA packets can be retransmitted
as soon as they're found rather than being queued up and transmitted
when the locked is dropped.
(21) When initially parsing a received ACK packet, extract some of the
fields from the ack info to the skbuff private data. This makes it
easier to do path MTU discovery in the future when the call to which a
PING RESPONSE ACK refers has been deallocated.
[*] Possible with the move of almost all code from softirq context to the
I/O thread.
Link: https://lore.kernel.org/r/20240301163807.385573-1-dhowells@redhat.com/ # v1
Link: https://lore.kernel.org/r/20240304084322.705539-1-dhowells@redhat.com/ # v2
* tag 'rxrpc-iothread-20240305' of git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs: (21 commits)
rxrpc: Extract useful fields from a received ACK to skb priv data
rxrpc: Clean up the resend algorithm
rxrpc: Record probes after transmission and reduce number of time-gets
rxrpc: Use ktimes for call timeout tracking and set the timer lazily
rxrpc: Differentiate PING ACK transmission traces.
rxrpc: Don't permit resending after all Tx packets acked
rxrpc: Parse received packets before dealing with timeouts
rxrpc: Do zerocopy using MSG_SPLICE_PAGES and page frags
rxrpc: Use rxrpc_txbuf::kvec[0] instead of rxrpc_txbuf::wire
rxrpc: Move rxrpc_send_ACK() to output.c with rxrpc_send_ack_packet()
rxrpc: Don't pick values out of the wire header when setting up security
rxrpc: Split up the DATA packet transmission function
rxrpc: Add a kvec[] to the rxrpc_txbuf struct
rxrpc: Merge together DF/non-DF branches of data Tx function
rxrpc: Do lazy DF flag resetting
rxrpc: Remove atomic handling on some fields only used in I/O thread
rxrpc: Strip barriers and atomics off of timer tracking
rxrpc: Fix the names of the fields in the ACK trailer struct
rxrpc: Note cksum in txbuf
rxrpc: Convert rxrpc_txbuf::flags into a mask and don't use atomics
...
====================
Link: https://lore.kernel.org/r/
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Commit 3e2f544dd8 ("net: get stats64 if device if driver is
configured") moved the callback to dev_get_tstats64() to net core, so,
unless the driver is doing some custom stats collection, it does not
need to set .ndo_get_stats64.
Since this driver is now relying in NETDEV_PCPU_STAT_TSTATS, then, it
doesn't need to set the dev_get_tstats64() generic .ndo_get_stats64
function pointer.
Signed-off-by: Breno Leitao <leitao@debian.org>
Acked-by: Oliver Neukum <oneukum@suse.com>
Link: https://lore.kernel.org/r/20240306142643.2429409-2-leitao@debian.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
With commit 34d21de99c ("net: Move {l,t,d}stats allocation to core and
convert veth & vrf"), stats allocation could be done on net core
instead of in this driver.
With this new approach, the driver doesn't have to bother with error
handling (allocation failure checking, making sure free happens in the
right spot, etc). This is core responsibility now.
Remove the allocation in the usbnet driver and leverage the network
core allocation instead.
Signed-off-by: Breno Leitao <leitao@debian.org>
Acked-by: Oliver Neukum <oneukum@suse.com>
Link: https://lore.kernel.org/r/20240306142643.2429409-1-leitao@debian.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
