When task boost with special value 3 also have vruntime
bonus to run faster.
Change-Id: I7da2dd985a961671b27036a0d5b13cb79480c933
Signed-off-by: Maria Yu <aiquny@codeaurora.org>
Preempt/irqsoff tracer notes down preemption/irqs disabled timestamp
and compares it with enable timestamp to report prolonged disabling
callers at latency tracking. But they don't reset the time stamp at
re-enable time. The time stamp could be stale and possible to report
same latency multiple times or false positive. This could be possible
when callsites uses notrace API (__preempt_) for preemption disable
and tracable API (preempt_) for enable (ex: softirq).
Fix it by resetting the preempt/irqsoff disable timestamp at
re-enable time.
Change-Id: I6790227998881922a9339869ce9ee2dab6202567
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Threre is possible that time_delta is not initialized and caused
'Division by zero' exception, so move this trace into if condition
so that uninitialized data will not be used.
Change-Id: If7eb8df1c2cd7fe958b8a672d1f3b9c117caeed6
Signed-off-by: Biao long <blong@codeaurora.org>
Task T2 Task T3
trace_options_core_write() subsystem_open()
mutex_lock(trace_types_lock) mutex_lock(event_mutex)
set_tracer_flag()
trace_event_enable_tgid_record() mutex_lock(trace_types_lock)
mutex_lock(event_mutex)
This gives a circular dependency deadlock between trace_types_lock and
event_mutex. To fix this invert the usage of trace_types_lock and
event_mutex in trace_options_core_write(). This keeps the sequence of
lock usage consistent.
Change-Id: Ide261e25fb15f79391a6220eef246e274b0c6a69
Signed-off-by: Prateek Sood <prsood@codeaurora.org>
This change is for general scheduler improvements.
Change-Id: I3e0f7593e10d9d854f1953b74e420f9d07b3180f
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
This change is for general scheduler improvements.
Change-Id: Ib8505248b91fb33395fe53f2dfacc8ec68c0273c
Signed-off-by: Abhijeet Dharmapurikar <adharmap@codeaurora.org>
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I5fbadf248c0bfe27bc761686de7a925cec2e4163
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
If both wakee and waker are in top-app,don't place wakee on waker
cpu, because most of this case waker is not going to sleep,this
will cause waker preempted and lead high runnable time,so let
scheduler to select the best cpu in this case.
Change-Id: If6eb3c0acb202197e9dd90205c581398ac049649
Signed-off-by: Biao Long <blong@codeaurora.org>
This change is for general scheduler improvements.
Change-Id: I567213580a970abc374d95e3d1fdc0aa73eaf705
Signed-off-by: Abhijeet Dharmapurikar <adharmap@codeaurora.org>
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Reap tasks that are killed by ulmk only.
Change-Id: I4af044b3b8fb5fe1dcef6bac70c475cf949d0505
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
If this runqueue have iowait task and still force newly
idle load balance, will make iowait task performance
worse. Need to do the average idle time check if
there is iowait task on this runqueue to have upcoming
iowait task have enough cpu resource.
Change-Id: I8e4dd440edbbd14d5170fbcf358bdaf5cc5b26e8
Signed-off-by: Maria Yu <aiquny@codeaurora.org>
Signed-off-by: Biao Long <blong@codeaurora.org>
We often see traces that gold cpus goes to idle even when silver
has big tasks. This is because avg_idle checks does not allow
the gold cpus to enter idle balance. This is NOT okay when
silver CPUs have big tasks.
Change-Id: Ice00d43c70be105b5bc3ad09caa6047aa98d7402
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Runmin Wang <runminw@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I33e9ec890f8b54d673770d5d02dba489a8e08ce7
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
Lockdep stops book keeping of locks and its owners
after an error is detected. Keep lockdep tracking
on after an error is reported.
Change-Id: I0209a929a8da564ccda33d5af8b714594a62800c
Signed-off-by: Prateek Sood <prsood@codeaurora.org>
When lockdep reports a locking error, it disables global
debug_locks which results in other lock debugging features
getting disabled.
Disable only lockdep logging once a locking error is reported.
This will help in tracking lock owners and their stacks from
ramdumps even when lockdep is not reporting them on console.
Change-Id: Ie7b6aa57df84aaa4cf90fc912969d4ca49f77997
Signed-off-by: Prateek Sood <prsood@codeaurora.org>
Currently lockdep saves caller0 for locking API's in
task_struct. Save caller1 also for a task acquiring the
lock. This is helpful for locks like cpu_hotplug_lock
which is acquired/release inside a helper function like
cpus_read_lock(). Caller1 shall help in getting caller of
cpus_read_lock().
Change-Id: I3ea30990245f478778c2e58348a84ab40610056d
Signed-off-by: Prateek Sood <prsood@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I7e6cc760a6add9ebdab7d67a176559bc640f7790
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Got never update frequency if scheduled the irq
work on an offlined cpu and it will always pending.
Queue sugov irq work on any online cpu if current
cpu is offline.
Change-Id: I33fc691917b5866488b6aeb11ed902a2753130b2
Signed-off-by: Maria Yu <aiquny@codeaurora.org>
[clingutla@codeaurora.org: Resolved minor merge conflicts]
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
The current expedited RCU grace-period code expects that a task
requesting an expedited grace period cannot awaken until that grace
period has reached the wakeup phase. However, it is possible for a long
preemption to result in the waiting task never sleeping. For example,
consider the following sequence of events:
1. Task A starts an expedited grace period by invoking
synchronize_rcu_expedited(). It proceeds normally up to the
wait_event() near the end of that function, and is then preempted
(or interrupted or whatever).
2. The expedited grace period completes, and a kworker task starts
the awaken phase, having incremented the counter and acquired
the rcu_state structure's .exp_wake_mutex. This kworker task
is then preempted or interrupted or whatever.
3. Task A resumes and enters wait_event(), which notes that the
expedited grace period has completed, and thus doesn't sleep.
4. Task B starts an expedited grace period exactly as did Task A,
complete with the preemption (or whatever delay) just before
the call to wait_event().
5. The expedited grace period completes, and another kworker
task starts the awaken phase, having incremented the counter.
However, it blocks when attempting to acquire the rcu_state
structure's .exp_wake_mutex because step 2's kworker task has
not yet released it.
6. Steps 4 and 5 repeat, resulting in overflow of the rcu_node
structure's ->exp_wq[] array.
In theory, this is harmless. Tasks waiting on the various ->exp_wq[]
array will just be spuriously awakened, but they will just sleep again
on noting that the rcu_state structure's ->expedited_sequence value has
not advanced far enough.
In practice, this wastes CPU time and is an accident waiting to happen.
This commit therefore moves the rcu_exp_gp_seq_end() call that officially
ends the expedited grace period (along with associate tracing) until
after the ->exp_wake_mutex has been acquired. This prevents Task A from
awakening prematurely, thus preventing more than one expedited grace
period from being in flight during a previous expedited grace period's
wakeup phase.
Fixes: 3b5f668e71 ("rcu: Overlap wakeups with next expedited grace period")
Change-Id: I9bd12a8d639deec801b4e3546a1cf4729fe3f26c
Signed-off-by: Neeraj Upadhyay <neeraju@codeaurora.org>
[ paulmck: Added updated comment. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Git-commit: 3dcffb240ef452fd97aa30d9c88efd4f4b030a68
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
[prsood@codeaurora.org: improved comment style]
Signed-off-by: Prateek Sood <prsood@codeaurora.org>
Tasks waiting within exp_funnel_lock() for an expedited grace period to
elapse can be starved due to the following sequence of events:
1. Tasks A and B both attempt to start an expedited grace
period at about the same time. This grace period will have
completed when the lower four bits of the rcu_state structure's
->expedited_sequence field are 0b'0100', for example, when the
initial value of this counter is zero. Task A wins, and thus
does the actual work of starting the grace period, including
acquiring the rcu_state structure's .exp_mutex and sets the
counter to 0b'0001'.
2. Because task B lost the race to start the grace period, it
waits on ->expedited_sequence to reach 0b'0100' inside of
exp_funnel_lock(). This task therefore blocks on the rcu_node
structure's ->exp_wq[1] field, keeping in mind that the
end-of-grace-period value of ->expedited_sequence (0b'0100')
is shifted down two bits before indexing the ->exp_wq[] field.
3. Task C attempts to start another expedited grace period,
but blocks on ->exp_mutex, which is still held by Task A.
4. The aforementioned expedited grace period completes, so that
->expedited_sequence now has the value 0b'0100'. A kworker task
therefore acquires the rcu_state structure's ->exp_wake_mutex
and starts awakening any tasks waiting for this grace period.
5. One of the first tasks awakened happens to be Task A. Task A
therefore releases the rcu_state structure's ->exp_mutex,
which allows Task C to start the next expedited grace period,
which causes the lower four bits of the rcu_state structure's
->expedited_sequence field to become 0b'0101'.
6. Task C's expedited grace period completes, so that the lower four
bits of the rcu_state structure's ->expedited_sequence field now
become 0b'1000'.
7. The kworker task from step 4 above continues its wakeups.
Unfortunately, the wake_up_all() refetches the rcu_state
structure's .expedited_sequence field:
wake_up_all(&rnp->exp_wq[rcu_seq_ctr(rcu_state.expedited_sequence) & 0x3]);
This results in the wakeup being applied to the rcu_node
structure's ->exp_wq[2] field, which is unfortunate given that
Task B is instead waiting on ->exp_wq[1].
On a busy system, no harm is done (or at least no permanent harm is done).
Some later expedited grace period will redo the wakeup. But on a quiet
system, such as many embedded systems, it might be a good long time before
there was another expedited grace period. On such embedded systems,
this situation could therefore result in a system hang.
This issue manifested as DPM device timeout during suspend (which
usually qualifies as a quiet time) due to a SCSI device being stuck in
_synchronize_rcu_expedited(), with the following stack trace:
schedule()
synchronize_rcu_expedited()
synchronize_rcu()
scsi_device_quiesce()
scsi_bus_suspend()
dpm_run_callback()
__device_suspend()
This commit therefore prevents such delays, timeouts, and hangs by
making rcu_exp_wait_wake() use its "s" argument consistently instead of
refetching from rcu_state.expedited_sequence.
Fixes: 3b5f668e71 ("rcu: Overlap wakeups with next expedited grace period")
Change-Id: I6cf7fbdee075b0243254b78f3f21a9d2b745fad5
Signed-off-by: Neeraj Upadhyay <neeraju@codeaurora.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Git-commit: 97cb80092efe81f37fd4fee1ecbc64a167aa22f6
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
Signed-off-by: Prateek Sood <prsood@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I752b5518fd72f9c239d3f96e4c3e021a98a75684
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Timer expects the 'expires' value in jiffies. Pass it accordingly.
Change-Id: Ic2f9338bd7144c5de59cf12edf8789703068891c
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
Previously, walt sched clusters prepared with empty cpumask for
single core bootup, which causes panic. Now sched clusters preparation
avoids empty cpu mask, so the commit 'c94eae6f4f4b0c("sched: fix single
cpu running issue")' becomes stale now, and it is adding extra
overhead in scheduler hotpaths, so remove it.
Change-Id: Ie6167e7c148d35b94fb8101caea16282d6d133fc
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Currently, sched updown migration handler derives cluster topology
based on arch topology, the cluster information is already populated
in walt sched_cluster. So reuse it instead of deriving it again.
And move updown tunables support to under WALT.
Change-Id: Iddf4d18ddf75cc20637281d9889f671f42369513
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
If cluster topology parsing failed or system booted with single cpu,
then topology possible sibling mask could be empty. This leads to
initialization of sched_clusters with empty cpus and there are many
call sites, which use the cluster cpu mask without checking for empty mask.
They lead to accessing invalid cpus ( minus one) leading to crashes.
Fix it by resetting to an initial cluster structure that has all possible
cpus as cluster mask. Also, warn when an empty cluster is detected.
To continue using init_cluster on device tree failures:
- Use init_cluster as default entry in cluster_head, and once
cluster topology parsed successfully, cluster_head gets updated with
proper cpu cluster information.
- If cluster topology parsing failed, free allocated sched_clusters and
make sure rq points to init_cluster instead of invalid address.
- Remove unused cluster id bitmaps.
Change-Id: Ic65ba86ff90a271098fb593221d1679d634930b1
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: If4e1dfbdb1f71989a75d33c8fa995ba0397b3ba9
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I985879c5bf4951aeffaca1b43ea00f276e8c31d5
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
If walt causes BUG_ON, then dump all cpu's current walt status.
Change-Id: Ie4b0f19f9598714a5479dd6426d0cc769ec562b1
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: Ic6be9069f6119d52e3915ea032f72ea35d27ee3e
Signed-off-by: Jonathan Avila <avilaj@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I8459bcf7b412a5f301566054c28c910567548485
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I1aa4c378452c936a443a17e77834d1191b7b0563
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
