Linux: sched/core: Call wakeup_preempt() for queued tasks during class change

In the Linux kernel, the following vulnerability has been resolved:

sched/core: Call wakeup_preempt() for queued tasks during class change

Paul reported recent RCU changes exposed a lost resched where CPU idles with a rcutorture thread waiting while being completely runnable.

Dumping the rq state at the time of idling with runnable tasks showed the following:

rq->next_class = &fair_sched_class rq->nr_running = 1 rq->cfs.h_nr_queued = 0 rq->rt.rt_nr_running = 1

Although the waiting task was an RT task, "rq->next_class" did not point to rt_sched_class and was stuck at the lower priority fair_sched_class.

Inspecting sched_change_end() revealed the following race with __sched_setscheduler() for queued, but non-running tasks:

CPU0 CPU1 ==== ====

__schedule()

/*

• rq->donor is a fair task that blocks.
• rq->next_class = &fair_sched_class

*/

rq_lock(rq0)

pick_next_task() /* nr_running = 0 */ pick_task_fair() sched_balance_newidle() raw_spin_rq_unlock(rq0) rq_lock(rq0)

activate_task(rq0, p) /* p is a fair task */ rq0->nr_running++; rq0->cfs.h_nr_queued++; wakeup_preempt(rq, p) rq->next_class = &fair_sched_class resched_curr()

rq_unlock(rq0)

__sched_setscheduler(p) /* p is moved to SCHED_RR */ task_rq_lock(p) sched_change_begin()

rq0->cfs.h_nr_queued--; rq0->rt.rt_nr_running++;

sched_change_end() /*

• p->on_rq = 1 (ctx->queued)
• but p is not rq->donor
• (!ctx->running) hence
• resched_curr() is and
• rq->next_class upgrade is
• skipped.

*/

task_rq_unlock()

raw_spin_rq_lock(rq0)

/*

• rq->nr_running = 1
• rq->cfs.h_nr_queued = 0
• rq->rt.rt_nr_running = 1
• rq->next_class = &fair_sched_class

*/

if (rq->cfs.h_nr_queued && ...) /* Above is false */

if (rq_modified_above(this_rq, &fair_sched_class)) /* Also false since rq->next_class is still fair */

/*

• sched_balance_newidle() is considered to
• have failed to pull any fair tasks and no
• other wakeups were detected.

*/ return NULL; /* pick_task_fair() */

pick_task_idle() return rq->idle;

next = rq->idle;

/* Preemption indicators are lost */ clear_tsk_need_resched(prev); clear_preempt_need_resched();

rq->curr = rq->idle;

!!! rq->idle switches in with rq->running == 1 but !need_resched() !!!

In simple words, enqueue, followed by a class change of the newly queued task goes unnoticed in rq->next_class indicator and sched_balance_newidle() sees no fair task, nor any higher priority task enqueued.

The above sequence will lead to CPU idling with a runnable task on the CPU.

Call wakeup_preempt() for ctx->queued cases in sched_change_end() to re-evaluate rq->next_class after class change (ENQUEUE_CLASS) and set the NEED_RESCHED indicator accordingly.

Newidle balance will correctly see an enqueue of a higher priority t ---truncated---