Commit 6fcc739a authored by Frederic Weisbecker's avatar Frederic Weisbecker
Browse files

Merge branch 'rcu/srcu' into next 

_ Properly handle SRCU readers within IRQ disabled sections in tiny SRCU

- Preparation to reimplement RCU Tasks Trace on top of SRCU fast:

    - Introduce API to expedite a grace period and test it through
      rcutorture.

    - Split srcu-fast in two flavours: SRCU-fast and SRCU-fast-updown.
      Both are still targeted toward faster readers (without full
      barriers on LOCK and UNLOCK) at the expense of heavier write side
      (using full RCU grace period ordering instead of simply full
      ordering) as compared to "traditional" non-fast SRCU. But those
      srcu-fast flavours are going to be optimized in two different
      ways:

         - SRCU-fast will become the reimplementation basis for
           RCU-TASK-TRACE for consolidation. Since RCU-TASK-TRACE must
           be NMI safe, SRCU-fast must be as well.

         - SRCU-fast-updown will be needed for uretprobes code in order
           to get rid of the read-side memory barriers while still
           allowing entering the reader at task level while exiting it
           in a timer handler. It is considered semaphore-like in that
           it can have different owners between LOCK and UNLOCK.
           However it is not NMI-safe.

      The actual optimizations are work in progress for the next cycle.
      Only the new interfaces are added for now, along with related
      torture and scalability test code.

- Create/document/debug/torture new proper initializers for RCU fast:
   DEFINE_SRCU_FAST() and init_srcu_struct_fast()

   This allows for using right away the proper ordering on the write
   side (either full ordering or full RCU grace period ordering) without
   waiting for the read side to tell which to use. Also this optimizes
   the read side altogether with moving flavour debug checks to debug
   config and with removing a costly RmW operation on their first call.

- Make some diagnostic functions tracing safe.
parents 3a866087 bfad3323

Documentation/RCU/Design/Requirements/Requirements.rst

+17 −16
Original line number Diff line number Diff line
@@ -2637,15 +2637,16 @@ synchronize_srcu() for some other domain ``ss1``, and if an 
that was held across as ``ss``-domain synchronize_srcu(), deadlock

would again be possible. Such a deadlock cycle could extend across an

arbitrarily large number of different SRCU domains. Again, with great

power comes great responsibility.

power comes great responsibility, though lockdep is now able to detect

this sort of deadlock.



Unlike the other RCU flavors, SRCU read-side critical sections can run

on idle and even offline CPUs. This ability requires that

srcu_read_lock() and srcu_read_unlock() contain memory barriers,

which means that SRCU readers will run a bit slower than would RCU

readers. It also motivates the smp_mb__after_srcu_read_unlock() API,

which, in combination with srcu_read_unlock(), guarantees a full

memory barrier.

Unlike the other RCU flavors, SRCU read-side critical sections can run on

idle and even offline CPUs, with the exception of srcu_read_lock_fast()

and friends.  This ability requires that srcu_read_lock() and

srcu_read_unlock() contain memory barriers, which means that SRCU

readers will run a bit slower than would RCU readers. It also motivates

the smp_mb__after_srcu_read_unlock() API, which, in combination with

srcu_read_unlock(), guarantees a full memory barrier.



Also unlike other RCU flavors, synchronize_srcu() may **not** be

invoked from CPU-hotplug notifiers, due to the fact that SRCU grace
@@ -2681,15 +2682,15 @@ run some tests first. SRCU just might need a few adjustment to deal with 
that sort of load. Of course, your mileage may vary based on the speed

of your CPUs and the size of your memory.



The `SRCU

API <https://lwn.net/Articles/609973/#RCU%20Per-Flavor%20API%20Table>`__

The `SRCU API

<https://lwn.net/Articles/609973/#RCU%20Per-Flavor%20API%20Table>`__

includes srcu_read_lock(), srcu_read_unlock(),

srcu_dereference(), srcu_dereference_check(),

synchronize_srcu(), synchronize_srcu_expedited(),

call_srcu(), srcu_barrier(), and srcu_read_lock_held(). It

also includes DEFINE_SRCU(), DEFINE_STATIC_SRCU(), and

init_srcu_struct() APIs for defining and initializing

``srcu_struct`` structures.

srcu_dereference(), srcu_dereference_check(), synchronize_srcu(),

synchronize_srcu_expedited(), call_srcu(), srcu_barrier(),

and srcu_read_lock_held(). It also includes DEFINE_SRCU(),

DEFINE_STATIC_SRCU(), DEFINE_SRCU_FAST(), DEFINE_STATIC_SRCU_FAST(),

init_srcu_struct(), and init_srcu_struct_fast() APIs for defining and

initializing ``srcu_struct`` structures.



More recently, the SRCU API has added polling interfaces:

Documentation/RCU/checklist.rst

+7 −5
Original line number Diff line number Diff line
@@ -417,11 +417,13 @@ over a rather long period of time, but improvements are always welcome! 
	you should be using RCU rather than SRCU, because RCU is almost

	always faster and easier to use than is SRCU.



	Also unlike other forms of RCU, explicit initialization and

	cleanup is required either at build time via DEFINE_SRCU()

	or DEFINE_STATIC_SRCU() or at runtime via init_srcu_struct()

	and cleanup_srcu_struct().  These last two are passed a

	"struct srcu_struct" that defines the scope of a given

	Also unlike other forms of RCU, explicit initialization

	and cleanup is required either at build time via

	DEFINE_SRCU(), DEFINE_STATIC_SRCU(), DEFINE_SRCU_FAST(),

	or DEFINE_STATIC_SRCU_FAST() or at runtime via either

	init_srcu_struct() or init_srcu_struct_fast() and

	cleanup_srcu_struct().	These last three are passed a

	`struct srcu_struct` that defines the scope of a given

	SRCU domain.  Once initialized, the srcu_struct is passed

	to srcu_read_lock(), srcu_read_unlock() synchronize_srcu(),

	synchronize_srcu_expedited(), and call_srcu().	A given

Documentation/RCU/whatisRCU.rst

+3 −0
Original line number Diff line number Diff line
@@ -1227,7 +1227,10 @@ SRCU: Initialization/cleanup/ordering:: 


	DEFINE_SRCU

	DEFINE_STATIC_SRCU

	DEFINE_SRCU_FAST        // for srcu_read_lock_fast() and friends

	DEFINE_STATIC_SRCU_FAST // for srcu_read_lock_fast() and friends

	init_srcu_struct

	init_srcu_struct_fast

	cleanup_srcu_struct

	smp_mb__after_srcu_read_unlock

include/linux/notifier.h

+1 −1
Original line number Diff line number Diff line
@@ -109,7 +109,7 @@ extern void srcu_init_notifier_head(struct srcu_notifier_head *nh); 
		.mutex = __MUTEX_INITIALIZER(name.mutex),	\

		.head = NULL,					\

		.srcuu = __SRCU_USAGE_INIT(name.srcuu),		\

		.srcu = __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu), \

		.srcu = __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu, 0), \

	}



#define ATOMIC_NOTIFIER_HEAD(name)				\

include/linux/srcu.h

+119 −28
Original line number Diff line number Diff line
@@ -25,8 +25,12 @@ struct srcu_struct; 


#ifdef CONFIG_DEBUG_LOCK_ALLOC



int __init_srcu_struct(struct srcu_struct *ssp, const char *name,

int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct lock_class_key *key);

#ifndef CONFIG_TINY_SRCU

int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, struct lock_class_key *key);

int __init_srcu_struct_fast_updown(struct srcu_struct *ssp, const char *name,

				   struct lock_class_key *key);

#endif // #ifndef CONFIG_TINY_SRCU



#define init_srcu_struct(ssp) \

({ \
@@ -35,10 +39,28 @@ int __init_srcu_struct(struct srcu_struct *ssp, const char *name, 
	__init_srcu_struct((ssp), #ssp, &__srcu_key); \

})



#define init_srcu_struct_fast(ssp) \

({ \

	static struct lock_class_key __srcu_key; \

	\

	__init_srcu_struct_fast((ssp), #ssp, &__srcu_key); \

})



#define init_srcu_struct_fast_updown(ssp) \

({ \

	static struct lock_class_key __srcu_key; \

	\

	__init_srcu_struct_fast_updown((ssp), #ssp, &__srcu_key); \

})



#define __SRCU_DEP_MAP_INIT(srcu_name)	.dep_map = { .name = #srcu_name },

#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */



int init_srcu_struct(struct srcu_struct *ssp);

#ifndef CONFIG_TINY_SRCU

int init_srcu_struct_fast(struct srcu_struct *ssp);

int init_srcu_struct_fast_updown(struct srcu_struct *ssp);

#endif // #ifndef CONFIG_TINY_SRCU



#define __SRCU_DEP_MAP_INIT(srcu_name)

#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
@@ -47,10 +69,12 @@ int init_srcu_struct(struct srcu_struct *ssp); 
#define SRCU_READ_FLAVOR_NORMAL		0x1		// srcu_read_lock().

#define SRCU_READ_FLAVOR_NMI		0x2		// srcu_read_lock_nmisafe().

//					0x4		// SRCU-lite is no longer with us.

#define SRCU_READ_FLAVOR_FAST	0x8		// srcu_read_lock_fast().

#define SRCU_READ_FLAVOR_FAST		0x4		// srcu_read_lock_fast().

#define SRCU_READ_FLAVOR_FAST_UPDOWN	0x8		// srcu_read_lock_fast().

#define SRCU_READ_FLAVOR_ALL		(SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_NMI | \

				SRCU_READ_FLAVOR_FAST) // All of the above.

#define SRCU_READ_FLAVOR_SLOWGP	SRCU_READ_FLAVOR_FAST

					 SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN)

						// All of the above.

#define SRCU_READ_FLAVOR_SLOWGP		(SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN)

						// Flavors requiring synchronize_rcu()

						// instead of smp_mb().

void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp);
@@ -259,29 +283,78 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) 
 * @ssp: srcu_struct in which to register the new reader.

 *

 * Enter an SRCU read-side critical section, but for a light-weight

 * smp_mb()-free reader.  See srcu_read_lock() for more information.

 *

 * If srcu_read_lock_fast() is ever used on an srcu_struct structure,

 * then none of the other flavors may be used, whether before, during,

 * or after.  Note that grace-period auto-expediting is disabled for _fast

 * srcu_struct structures because auto-expedited grace periods invoke

 * synchronize_rcu_expedited(), IPIs and all.

 *

 * Note that srcu_read_lock_fast() can be invoked only from those contexts

 * where RCU is watching, that is, from contexts where it would be legal

 * to invoke rcu_read_lock().  Otherwise, lockdep will complain.

 * smp_mb()-free reader.  See srcu_read_lock() for more information.  This

 * function is NMI-safe, in a manner similar to srcu_read_lock_nmisafe().

 *

 * For srcu_read_lock_fast() to be used on an srcu_struct structure,

 * that structure must have been defined using either DEFINE_SRCU_FAST()

 * or DEFINE_STATIC_SRCU_FAST() on the one hand or initialized with

 * init_srcu_struct_fast() on the other.  Such an srcu_struct structure

 * cannot be passed to any non-fast variant of srcu_read_{,un}lock() or

 * srcu_{down,up}_read().  In kernels built with CONFIG_PROVE_RCU=y,

 * __srcu_check_read_flavor() will complain bitterly if you ignore this

 * restriction.

 *

 * Grace-period auto-expediting is disabled for SRCU-fast srcu_struct

 * structures because SRCU-fast expedited grace periods invoke

 * synchronize_rcu_expedited(), IPIs and all.  If you need expedited

 * SRCU-fast grace periods, use synchronize_srcu_expedited().

 *

 * The srcu_read_lock_fast() function can be invoked only from those

 * contexts where RCU is watching, that is, from contexts where it would

 * be legal to invoke rcu_read_lock().  Otherwise, lockdep will complain.

 */

static inline struct srcu_ctr __percpu *srcu_read_lock_fast(struct srcu_struct *ssp) __acquires(ssp)

{

	struct srcu_ctr __percpu *retval;



	RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_fast().");

	srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST);

	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST);

	retval = __srcu_read_lock_fast(ssp);

	rcu_try_lock_acquire(&ssp->dep_map);

	return retval;

}



/**

 * srcu_read_lock_fast_updown - register a new reader for an SRCU-fast-updown structure.

 * @ssp: srcu_struct in which to register the new reader.

 *

 * Enter an SRCU read-side critical section, but for a light-weight

 * smp_mb()-free reader.  See srcu_read_lock() for more information.

 * This function is compatible with srcu_down_read_fast(), but is not

 * NMI-safe.

 *

 * For srcu_read_lock_fast_updown() to be used on an srcu_struct

 * structure, that structure must have been defined using either

 * DEFINE_SRCU_FAST_UPDOWN() or DEFINE_STATIC_SRCU_FAST_UPDOWN() on the one

 * hand or initialized with init_srcu_struct_fast_updown() on the other.

 * Such an srcu_struct structure cannot be passed to any non-fast-updown

 * variant of srcu_read_{,un}lock() or srcu_{down,up}_read().  In kernels

 * built with CONFIG_PROVE_RCU=y, __srcu_check_read_flavor() will complain

 * bitterly if you ignore this * restriction.

 *

 * Grace-period auto-expediting is disabled for SRCU-fast-updown

 * srcu_struct structures because SRCU-fast-updown expedited grace periods

 * invoke synchronize_rcu_expedited(), IPIs and all.  If you need expedited

 * SRCU-fast-updown grace periods, use synchronize_srcu_expedited().

 *

 * The srcu_read_lock_fast_updown() function can be invoked only from

 * those contexts where RCU is watching, that is, from contexts where

 * it would be legal to invoke rcu_read_lock().  Otherwise, lockdep will

 * complain.

 */

static inline struct srcu_ctr __percpu *srcu_read_lock_fast_updown(struct srcu_struct *ssp)

__acquires(ssp)

{

	struct srcu_ctr __percpu *retval;



	RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_fast_updown().");

	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN);

	retval = __srcu_read_lock_fast_updown(ssp);

	rcu_try_lock_acquire(&ssp->dep_map);

	return retval;

}



/*

 * Used by tracing, cannot be traced and cannot call lockdep.

 * See srcu_read_lock_fast() for more information.
@@ -291,7 +364,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_fast_notrace(struct srcu_ 
{

	struct srcu_ctr __percpu *retval;



	srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST);

	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST);

	retval = __srcu_read_lock_fast(ssp);

	return retval;

}
@@ -305,14 +378,15 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_fast_notrace(struct srcu_ 
 * srcu_down_read() for more information.

 *

 * The same srcu_struct may be used concurrently by srcu_down_read_fast()

 * and srcu_read_lock_fast().

 * and srcu_read_lock_fast().  However, the same definition/initialization

 * requirements called out for srcu_read_lock_safe() apply.

 */

static inline struct srcu_ctr __percpu *srcu_down_read_fast(struct srcu_struct *ssp) __acquires(ssp)

{

	WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi());

	RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_down_read_fast().");

	srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST);

	return __srcu_read_lock_fast(ssp);

	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN);

	return __srcu_read_lock_fast_updown(ssp);

}



/**
@@ -408,6 +482,23 @@ static inline void srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ct 
	RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_fast().");

}



/**

 * srcu_read_unlock_fast_updown - unregister a old reader from an SRCU-fast-updown structure.

 * @ssp: srcu_struct in which to unregister the old reader.

 * @scp: return value from corresponding srcu_read_lock_fast_updown().

 *

 * Exit an SRCU-fast-updown read-side critical section.

 */

static inline void

srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp) __releases(ssp)

{

	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN);

	srcu_lock_release(&ssp->dep_map);

	__srcu_read_unlock_fast_updown(ssp, scp);

	RCU_LOCKDEP_WARN(!rcu_is_watching(),

			 "RCU must be watching srcu_read_unlock_fast_updown().");

}



/*

 * Used by tracing, cannot be traced and cannot call lockdep.

 * See srcu_read_unlock_fast() for more information.
@@ -431,9 +522,9 @@ static inline void srcu_up_read_fast(struct srcu_struct *ssp, struct srcu_ctr __ 
	__releases(ssp)

{

	WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi());

	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST);

	__srcu_read_unlock_fast(ssp, scp);

	RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_up_read_fast().");

	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN);

	__srcu_read_unlock_fast_updown(ssp, scp);

	RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_up_read_fast_updown().");

}



/**