sched/core: Skip superfluous acquire barrier in ttwu

ttwu_remote() unconditionally locks the task's runqueue lock, which implies
a full barrier across the lock and unlock, so the acquire barrier after the
control dependency is only needed when the task isn't on the runqueue.

Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Change-Id: Ibb988fe341ba3109d381682a8725fbd2e7a648e3
fourteen
Sultan Alsawaf 3 years ago committed by Jenna
parent d1769a703f
commit 8aa7972223
  1. 29
      kernel/sched/core.c

@ -2217,16 +2217,17 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags,
* current.
*/
smp_rmb();
if (READ_ONCE(p->on_rq) && ttwu_remote(p, wake_flags))
if (READ_ONCE(p->on_rq)) {
if (ttwu_remote(p, wake_flags))
goto unlock;
} else {
#ifdef CONFIG_SMP
/*
* Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be
* possible to, falsely, observe p->on_cpu == 0.
* Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would
* be possible to, falsely, observe p->on_cpu == 0.
*
* One must be running (->on_cpu == 1) in order to remove oneself
* from the runqueue.
* One must be running (->on_cpu == 1) in order to remove
* oneself from the runqueue.
*
* [S] ->on_cpu = 1; [L] ->on_rq
* UNLOCK rq->lock
@ -2234,16 +2235,20 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags,
* LOCK rq->lock
* [S] ->on_rq = 0; [L] ->on_cpu
*
* Pairs with the full barrier implied in the UNLOCK+LOCK on rq->lock
* from the consecutive calls to schedule(); the first switching to our
* task, the second putting it to sleep.
* Pairs with the full barrier implied in the UNLOCK+LOCK on
* rq->lock from the consecutive calls to schedule(); the first
* switching to our task, the second putting it to sleep.
*
* Form a control-dep-acquire with p->on_rq == 0 above, to ensure
* schedule()'s deactivate_task() has 'happened' and p will no longer
* care about it's own p->state. See the comment in __schedule().
* Form a control-dep-acquire with p->on_rq == 0 above, to
* ensure schedule()'s deactivate_task() has 'happened' and p
* will no longer care about it's own p->state. See the comment
* in __schedule().
*/
smp_acquire__after_ctrl_dep();
#endif
}
#ifdef CONFIG_SMP
/*
* We're doing the wakeup (@success == 1), they did a dequeue (p->on_rq
* == 0), which means we need to do an enqueue, change p->state to

Loading…
Cancel
Save