Commit 39806b96 authored by Alexander Lobakin's avatar Alexander Lobakin Committed by Jakub Kicinski
Browse files

page_pool: try direct bulk recycling 



Now that the checks for direct recycling possibility live inside the
Page Pool core, reuse them when performing bulk recycling.
page_pool_put_page_bulk() can be called from process context as well,
page_pool_napi_local() takes care of this at the very beginning.
Under high .ndo_xdp_xmit() traffic load, the win is 2-3% Pps assuming
the sending driver uses xdp_return_frame_bulk() on Tx completion.

Signed-off-by: default avatarAlexander Lobakin <aleksander.lobakin@intel.com>
Link: https://lore.kernel.org/r/20240329165507.3240110-3-aleksander.lobakin@intel.com


Signed-off-by: default avatarJakub Kicinski <kuba@kernel.org>
parent 4a96a4e8

net/core/page_pool.c

+5 −2
Original line number Diff line number Diff line
@@ -772,8 +772,11 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data, 
			     int count)

{

	int i, bulk_len = 0;

	bool allow_direct;

	bool in_softirq;



	allow_direct = page_pool_napi_local(pool);



	for (i = 0; i < count; i++) {

		struct page *page = virt_to_head_page(data[i]);
@@ -781,13 +784,13 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data, 
		if (!page_pool_is_last_ref(page))

			continue;



		page = __page_pool_put_page(pool, page, -1, false);

		page = __page_pool_put_page(pool, page, -1, allow_direct);

		/* Approved for bulk recycling in ptr_ring cache */

		if (page)

			data[bulk_len++] = page;

	}



	if (unlikely(!bulk_len))

	if (!bulk_len)

		return;



	/* Bulk producer into ptr_ring page_pool cache */