Commit de57a251 authored by Joe Damato's avatar Joe Damato Committed by David S. Miller
Browse files

eventpoll: Add per-epoll prefer busy poll option 



When using epoll-based busy poll, the prefer_busy_poll option is hardcoded
to false. Users may want to enable prefer_busy_poll to be used in
conjunction with gro_flush_timeout and defer_hard_irqs_count to keep device
IRQs masked.

Other busy poll methods allow enabling or disabling prefer busy poll via
SO_PREFER_BUSY_POLL, but epoll-based busy polling uses a hardcoded value.

Fix this edge case by adding support for a per-epoll context
prefer_busy_poll option. The default is false, as it was hardcoded before
this change.

Signed-off-by: default avatarJoe Damato <jdamato@fastly.com>
Acked-by: default avatarStanislav Fomichev <sdf@google.com>
Reviewed-by: default avatarJakub Kicinski <kuba@kernel.org>
Reviewed-by: default avatarEric Dumazet <edumazet@google.com>
Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
parent c6aa2a77

fs/eventpoll.c

+5 −2
Original line number Diff line number Diff line
@@ -231,6 +231,7 @@ struct eventpoll { 
	u32 busy_poll_usecs;

	/* busy poll packet budget */

	u16 busy_poll_budget;

	bool prefer_busy_poll;

#endif



#ifdef CONFIG_DEBUG_LOCK_ALLOC
@@ -438,13 +439,14 @@ static bool ep_busy_loop(struct eventpoll *ep, int nonblock) 
{

	unsigned int napi_id = READ_ONCE(ep->napi_id);

	u16 budget = READ_ONCE(ep->busy_poll_budget);

	bool prefer_busy_poll = READ_ONCE(ep->prefer_busy_poll);



	if (!budget)

		budget = BUSY_POLL_BUDGET;



	if (napi_id >= MIN_NAPI_ID && ep_busy_loop_on(ep)) {

		napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false,

			       budget);

		napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end,

			       ep, prefer_busy_poll, budget);

		if (ep_events_available(ep))

			return true;

		/*
@@ -2098,6 +2100,7 @@ static int do_epoll_create(int flags) 
#ifdef CONFIG_NET_RX_BUSY_POLL

	ep->busy_poll_usecs = 0;

	ep->busy_poll_budget = 0;

	ep->prefer_busy_poll = false;

#endif

	ep->file = file;

	fd_install(fd, file);