Alexei Starovoitov
af92793e52
slab: Introduce kmalloc_nolock() and kfree_nolock().
...
kmalloc_nolock() relies on ability of local_trylock_t to detect
the situation when per-cpu kmem_cache is locked.
In !PREEMPT_RT local_(try)lock_irqsave(&s->cpu_slab->lock, flags)
disables IRQs and marks s->cpu_slab->lock as acquired.
local_lock_is_locked(&s->cpu_slab->lock) returns true when
slab is in the middle of manipulating per-cpu cache
of that specific kmem_cache.
kmalloc_nolock() can be called from any context and can re-enter
into ___slab_alloc():
kmalloc() -> ___slab_alloc(cache_A) -> irqsave -> NMI -> bpf ->
kmalloc_nolock() -> ___slab_alloc(cache_B)
or
kmalloc() -> ___slab_alloc(cache_A) -> irqsave -> tracepoint/kprobe -> bpf ->
kmalloc_nolock() -> ___slab_alloc(cache_B)
Hence the caller of ___slab_alloc() checks if &s->cpu_slab->lock
can be acquired without a deadlock before invoking the function.
If that specific per-cpu kmem_cache is busy the kmalloc_nolock()
retries in a different kmalloc bucket. The second attempt will
likely succeed, since this cpu locked different kmem_cache.
Similarly, in PREEMPT_RT local_lock_is_locked() returns true when
per-cpu rt_spin_lock is locked by current _task_. In this case
re-entrance into the same kmalloc bucket is unsafe, and
kmalloc_nolock() tries a different bucket that is most likely is
not locked by the current task. Though it may be locked by a
different task it's safe to rt_spin_lock() and sleep on it.
Similar to alloc_pages_nolock() the kmalloc_nolock() returns NULL
immediately if called from hard irq or NMI in PREEMPT_RT.
kfree_nolock() defers freeing to irq_work when local_lock_is_locked()
and (in_nmi() or in PREEMPT_RT).
SLUB_TINY config doesn't use local_lock_is_locked() and relies on
spin_trylock_irqsave(&n->list_lock) to allocate,
while kfree_nolock() always defers to irq_work.
Note, kfree_nolock() must be called _only_ for objects allocated
with kmalloc_nolock(). Debug checks (like kmemleak and kfence)
were skipped on allocation, hence obj = kmalloc(); kfree_nolock(obj);
will miss kmemleak/kfence book keeping and will cause false positives.
large_kmalloc is not supported by either kmalloc_nolock()
or kfree_nolock().
Signed-off-by: Alexei Starovoitov <ast@kernel.org >
Reviewed-by: Harry Yoo <harry.yoo@oracle.com >
Signed-off-by: Vlastimil Babka <vbabka@suse.cz >
2025-09-29 09:42:36 +02:00
..
2025-06-09 09:56:18 -07:00
2025-07-15 16:03:24 -07:00
2025-06-10 09:34:32 +02:00
2025-07-29 11:42:31 -07:00
2025-06-30 17:57:15 -07:00
2025-08-07 10:07:06 +02:00
2025-07-31 01:33:20 +02:00
2025-07-13 15:36:25 +01:00
2025-07-27 01:41:21 -07:00
2025-07-18 12:34:56 -06:00
2025-07-29 12:27:40 -04:00
2025-07-31 11:50:25 -07:00
2025-07-23 01:38:56 -04:00
2025-07-18 14:14:24 +02:00
2025-07-31 13:43:02 -07:00
2025-06-18 14:20:32 -07:00
2025-07-24 09:38:29 -07:00
2025-07-25 18:36:42 +02:00
2025-07-08 17:58:49 +01:00
2025-06-15 21:19:08 +05:30
2025-06-18 13:26:36 +02:00
2025-07-31 13:34:06 -07:00
2025-06-24 10:57:43 +02:00
2025-06-30 09:07:26 +02:00
2025-07-09 22:57:54 -07:00
2025-06-27 00:29:07 +01:00
2025-08-03 03:28:52 +02:00
2025-07-31 16:29:46 -07:00
2025-08-02 12:07:09 -07:00
2025-07-21 10:43:21 +05:30
2025-07-31 13:43:02 -07:00
2025-08-09 07:20:44 +03:00
2025-07-31 13:16:09 -07:00
2025-07-29 20:21:54 -07:00
2025-06-30 17:57:21 -07:00
2025-05-21 11:34:06 +01:00
2025-07-09 22:42:15 -07:00
2025-06-23 09:50:51 -07:00
2025-06-09 11:24:43 +01:00
2025-07-21 18:18:51 +01:00
2025-06-17 18:42:44 -07:00
2025-06-16 17:00:06 -04:00
2025-07-13 16:38:32 -07:00
2025-07-02 13:09:54 +02:00
2025-07-02 13:09:54 +02:00
2025-05-20 08:44:33 +02:00
2025-06-13 06:19:34 -06:00
2025-07-31 11:28:03 -04:00
2025-07-31 11:28:03 -04:00
2025-06-30 15:50:32 -06:00
2025-07-28 16:14:18 -07:00
2025-06-30 15:50:32 -06:00
2025-07-01 10:24:19 -06:00
2025-08-21 07:11:11 -06:00
2025-07-24 16:55:11 -07:00
2025-07-03 19:31:30 -07:00
2025-06-09 16:17:11 -07:00
2025-07-31 11:30:05 -07:00
2025-07-31 11:30:05 -07:00
2025-07-09 19:32:31 -07:00
2025-07-07 08:25:07 -07:00
2025-07-16 14:48:18 +02:00
2025-06-13 06:20:17 -06:00
2025-07-22 19:10:17 -06:00
2025-07-31 18:23:53 -07:00
2025-07-31 18:23:53 -07:00
2025-06-17 10:01:18 -10:00
2025-05-27 20:59:53 -07:00
2025-08-01 15:47:06 -07:00
2025-06-19 18:25:48 -07:00
2025-07-09 22:42:06 -07:00
2025-07-29 17:19:35 -07:00
2025-07-29 17:19:35 -07:00
2025-08-04 14:15:14 +02:00
2025-08-18 14:48:16 +02:00
2025-05-21 13:41:03 +02:00
2025-05-21 13:39:45 +02:00
2025-07-16 13:56:23 +02:00
2025-07-31 14:57:54 -07:00
2025-05-14 11:56:17 +01:00
2025-07-29 17:42:52 -07:00
2025-08-20 22:23:44 +08:00
2025-07-31 16:52:32 -07:00
2025-05-12 23:50:33 -07:00
2025-05-21 10:48:21 -07:00
2025-05-21 10:48:21 -07:00
2025-07-19 19:08:23 -07:00
2025-05-13 20:37:16 -07:00
2025-06-30 09:31:57 -07:00
2025-06-30 09:31:57 -07:00
2025-06-30 09:31:57 -07:00
2025-06-30 09:31:57 -07:00
2025-07-07 12:24:50 +02:00
2025-06-11 11:06:06 +08:00
2025-07-19 18:59:57 -07:00
2025-07-31 14:57:54 -07:00
2025-06-11 13:39:52 -04:00
2025-07-09 13:30:29 +02:00
2025-07-14 20:22:59 +09:00
2025-07-09 22:42:19 -07:00
2025-07-29 12:15:39 -07:00
2025-05-28 15:52:42 -07:00
2025-06-13 08:26:49 +01:00
2025-05-27 20:09:06 -07:00
2025-06-26 15:18:04 -07:00
2025-07-01 13:55:37 -06:00
2025-06-27 16:38:02 -07:00
2025-05-23 13:24:53 -07:00
2025-08-09 18:10:01 +03:00
2025-05-13 14:34:18 +02:00
2025-07-31 10:20:04 -04:00
2025-06-21 08:10:25 -07:00
2025-07-08 11:56:40 -07:00
2025-08-02 12:06:12 -07:00
2025-08-11 16:16:36 +02:00
2025-06-23 12:17:33 +02:00
2025-07-22 15:57:02 +00:00
2025-06-23 12:45:13 +02:00
2025-05-19 22:46:34 +02:00
2025-07-04 16:14:39 +02:00
2025-06-10 13:16:19 +02:00
2025-07-30 09:58:50 -07:00
2025-07-31 11:28:03 -04:00
2025-08-11 22:22:33 +09:00
2025-05-21 14:34:38 +02:00
2025-07-29 17:19:29 -07:00
2025-07-02 09:45:09 +09:00
2025-06-18 11:20:19 +02:00
2025-06-23 12:17:33 +02:00
2025-07-08 10:25:19 +02:00
2025-07-31 14:57:54 -07:00
2025-05-21 14:35:34 +02:00
2025-08-04 16:27:21 -07:00
2025-07-16 14:28:21 +02:00
2025-06-19 12:29:17 +02:00
2025-05-29 10:34:26 -07:00
2025-07-28 18:14:38 +02:00
2025-07-22 20:15:56 -04:00
2025-07-11 16:02:00 +02:00
2025-07-19 19:08:28 -07:00
2025-09-29 09:42:35 +02:00
2025-07-24 16:24:49 +02:00
2025-07-01 10:24:11 -06:00
2025-05-21 14:21:34 +01:00
2025-07-15 17:28:27 +02:00
2025-07-09 22:42:11 -07:00
2025-07-19 18:59:51 -07:00
2025-06-23 17:00:27 +08:00
2025-07-24 19:12:35 -07:00
2025-07-24 19:12:32 -07:00
2025-07-19 19:08:26 -07:00
2025-06-03 08:39:20 -07:00
2025-06-24 10:45:32 -05:00
2025-05-22 11:07:05 +02:00
2025-05-19 22:23:57 +02:00
2025-05-20 22:49:39 +02:00
2025-07-11 11:42:38 -07:00
2025-06-25 15:23:06 -07:00
2025-07-02 15:29:46 -07:00
2025-06-17 18:06:39 -07:00
2025-07-25 15:29:58 -07:00
2025-05-27 08:25:42 +02:00
2025-07-29 17:19:35 -07:00
2025-06-23 13:14:50 -04:00
2025-07-03 11:09:32 +03:00
2025-07-03 11:09:35 +03:00
2025-07-03 11:09:43 +03:00
2025-06-27 21:43:40 +03:00
2025-07-28 16:30:12 -07:00
2025-08-02 12:06:10 -07:00
2025-05-20 15:04:13 +01:00
2025-07-14 10:51:33 +02:00
2025-07-31 12:43:08 -07:00
2025-07-18 17:33:08 -03:00
2025-07-31 15:00:41 -06:00
2025-08-11 17:26:38 +02:00
2025-08-19 16:35:56 -07:00
2025-07-25 13:06:19 -07:00
2025-08-01 09:46:24 -07:00
2025-05-27 08:00:46 -07:00
2025-06-20 13:52:41 -07:00
2025-07-30 17:14:01 -07:00
2025-07-24 10:57:26 +02:00
2025-05-20 10:31:12 -04:00
2025-07-09 22:57:49 -07:00
2025-09-29 09:42:36 +02:00
2025-08-13 17:18:43 +02:00
2025-07-22 20:15:56 -04:00
2025-05-12 23:50:40 -07:00
2025-08-02 12:01:38 -07:00
2025-06-11 11:57:14 -07:00
2025-07-09 22:42:13 -07:00
2025-06-25 15:55:03 -07:00
2025-07-23 11:52:47 +02:00
2025-07-09 22:42:13 -07:00
2025-07-21 21:40:39 -07:00
2025-06-20 13:39:42 -07:00
2025-06-25 09:51:33 -07:00
2025-06-23 09:50:53 -07:00
2025-06-19 10:37:06 +01:00
2025-07-02 17:18:01 +01:00
2025-07-31 12:56:21 +09:00
2025-07-01 11:07:43 -05:00
2025-05-14 13:16:24 +02:00
2025-07-17 07:41:56 -10:00
2025-09-29 09:42:35 +02:00
2025-09-29 09:42:35 +02:00
2025-07-14 21:57:20 -07:00
2025-07-17 06:01:16 -06:00
2025-07-04 16:14:39 +02:00
2025-05-26 16:23:39 -05:00
2025-07-13 16:38:24 -07:00
2025-07-24 09:20:39 +02:00
2025-06-12 18:12:37 -07:00
2025-07-24 16:55:11 -07:00
2025-09-29 09:42:36 +02:00
2025-07-09 22:42:13 -07:00
2025-07-13 16:38:17 -07:00
2025-07-09 22:42:06 -07:00
2025-07-13 16:38:17 -07:00
2025-05-21 09:55:15 -07:00
2025-05-13 10:23:24 +02:00
2025-08-19 16:35:57 -07:00
2025-06-24 16:48:02 +01:00
2025-05-22 14:55:37 -07:00
2025-07-31 14:57:54 -07:00
2025-08-05 16:02:07 +03:00
2025-07-09 22:42:13 -07:00
2025-08-02 12:06:11 -07:00
2025-07-09 22:41:56 -07:00
2025-05-31 22:46:13 -07:00
2025-07-19 18:59:47 -07:00
2025-06-06 12:45:35 -07:00
2025-08-06 07:32:52 +03:00
2025-08-06 07:32:52 +03:00
2025-06-29 19:03:29 -04:00
2025-06-17 18:18:46 -07:00
2025-05-16 17:53:48 -07:00
2025-07-30 17:14:01 -07:00
2025-08-01 15:47:06 -07:00
2025-05-26 08:02:43 -07:00
2025-05-22 15:32:00 +02:00
2025-05-28 15:24:36 -07:00
2025-05-23 13:57:12 +02:00
2025-08-12 14:43:05 +02:00
2025-07-25 18:40:01 +02:00
2025-08-15 15:56:49 +02:00
2025-05-19 16:06:42 -07:00
2025-07-25 13:59:41 -07:00
2025-05-28 17:17:13 -04:00
2025-07-22 08:10:41 -04:00
2025-07-14 15:20:02 -07:00
2025-08-19 11:16:02 -07:00
2025-07-14 15:20:02 -07:00
2025-05-28 17:17:14 -04:00
2025-07-04 13:17:30 +01:00
2025-07-13 16:38:15 -07:00
2025-07-08 19:11:57 -04:00
2025-05-22 14:55:36 -07:00
2025-07-17 13:38:07 +02:00
2025-05-21 14:28:27 +02:00
2025-07-08 18:35:52 +01:00
2025-06-01 08:41:11 -07:00
2025-06-25 15:30:00 -07:00
2025-06-18 17:46:27 +08:00
2025-07-19 18:59:57 -07:00
2025-05-12 23:50:49 -07:00
2025-08-02 12:06:08 -07:00
2025-07-13 16:38:18 -07:00
2025-07-13 16:38:17 -07:00
2025-07-31 14:57:54 -07:00
2025-07-09 22:42:07 -07:00
2025-07-30 16:23:12 -07:00
2025-07-29 11:45:39 -05:00
2025-06-16 11:50:34 -07:00
2025-05-30 12:21:57 -05:00
2025-07-24 16:51:17 -05:00
2025-05-28 16:47:56 -05:00
2025-07-24 16:51:41 -05:00
2025-06-23 13:09:01 -05:00
2025-07-23 01:27:32 -04:00
2025-07-31 16:11:43 -05:00
2025-05-21 16:46:37 +02:00
2025-07-09 22:42:15 -07:00
2025-05-25 00:53:48 -07:00
2025-06-05 14:37:53 +02:00
2025-07-09 22:42:19 -07:00
2025-08-02 12:06:10 -07:00
2025-05-20 18:17:43 -07:00
2025-07-09 19:32:30 -07:00
2025-06-19 14:28:24 +02:00
2025-06-19 14:28:24 +02:00
2025-07-29 11:42:31 -07:00
2025-07-22 18:01:15 +02:00
2025-07-22 18:07:11 +02:00
2025-06-27 20:13:12 +02:00
2025-07-31 21:39:01 -07:00
2025-07-01 12:29:29 +02:00
2025-06-13 08:47:18 +02:00
2025-07-09 15:27:00 -04:00
2025-07-31 14:57:54 -07:00
2025-07-10 09:39:18 +02:00
2025-07-01 12:31:37 +02:00
2025-07-09 13:40:21 +02:00
2025-06-25 07:27:27 -07:00
2025-07-03 15:36:04 +02:00
2025-07-07 08:39:36 +02:00
2025-06-23 12:17:33 +02:00
2025-06-19 17:02:04 -07:00
2025-07-25 13:44:33 +01:00
2025-07-14 22:27:46 -07:00
2025-07-09 22:57:52 -07:00
2025-05-16 12:10:20 +02:00
2025-06-16 21:06:12 +02:00
2025-08-03 15:03:04 -07:00
2025-08-02 12:06:11 -07:00
2025-09-29 09:42:35 +02:00
2025-07-28 16:47:34 -04:00
2025-08-03 16:23:09 -07:00
2025-08-07 06:30:17 -06:00
2025-06-11 22:42:35 -07:00
2025-08-17 05:57:47 -07:00
2025-07-28 18:20:32 -07:00
2025-06-17 13:42:34 +02:00
2025-06-19 13:24:16 +02:00
2025-07-09 22:41:57 -07:00
2025-05-13 11:12:48 +02:00
2025-08-01 14:40:53 -07:00
2025-09-29 09:42:36 +02:00
2025-05-23 08:49:16 +01:00
2025-07-29 13:00:20 -07:00
2025-08-04 10:54:36 -07:00
2025-07-19 10:17:56 +05:30
2025-07-16 09:51:08 +05:30
2025-07-16 09:51:08 +05:30
2025-06-18 14:30:53 -07:00
2025-07-17 12:10:56 -07:00
2025-07-17 16:39:49 -07:00
2025-07-30 19:26:49 -07:00
2025-07-24 19:12:28 -07:00
2025-07-19 19:08:24 -07:00
2025-07-02 17:05:17 +02:00
2025-07-23 11:56:02 +02:00
2025-06-27 13:01:11 +02:00
2025-07-21 17:48:32 -07:00
2025-05-13 16:38:03 +02:00
2025-06-04 11:27:11 +02:00
2025-07-01 15:38:25 +02:00
2025-07-18 13:45:33 +02:00
2025-07-03 14:44:15 +02:00
2025-06-08 09:07:37 +02:00
2025-06-25 15:12:17 -07:00
2025-06-03 07:39:23 -07:00
2025-07-28 18:18:16 -07:00
2025-07-22 20:15:51 -04:00
2025-05-14 11:19:32 -04:00
2025-06-24 15:32:56 +01:00
2025-06-26 09:44:45 -07:00
2025-06-02 16:00:26 -07:00
2025-07-31 10:20:10 -04:00
2025-07-31 10:20:11 -04:00
2025-07-29 14:46:07 -04:00
2025-07-29 14:46:07 -04:00
2025-07-21 17:40:02 +02:00
2025-07-24 19:12:29 -07:00
2025-05-31 19:12:53 -07:00
2025-08-05 15:41:14 -06:00
2025-08-05 15:41:14 -06:00
2025-07-08 18:04:58 +02:00
2025-07-08 18:04:58 +02:00
2025-07-08 18:05:47 +02:00
2025-07-08 18:05:07 +02:00
2025-08-01 09:11:09 -04:00
2025-08-01 14:17:48 -07:00
2025-07-09 21:07:56 -07:00
2025-05-28 14:55:35 -07:00
2025-07-19 18:59:46 -07:00
2025-06-24 16:45:35 +01:00
2025-08-01 09:11:08 -04:00
2025-06-23 09:50:58 -07:00
2025-07-17 11:26:56 +02:00
2025-06-23 08:07:06 -10:00
2025-07-09 22:41:58 -07:00
2025-08-02 12:01:37 -07:00
2025-07-13 16:38:29 -07:00