The UFS callback is the most time consuming callback in the dpm_resume
section of kernel resumes, taking around 30 ms. Making it async
improves resume latency by around 20 ms, and helps with decreasing
suspend times as well.
Bug: 134704391
Change-Id: I708c8a7bc8f2250d6b2365971ccc394c7fbf8896
Signed-off-by: Vincent Palomares <paillon@google.com>
Although try_to_freeze_tasks() stops when there's a wakeup, it doesn't
return an error when it successfully freezes everything it wants to freeze.
As a result, the suspend attempt can continue even after a wakeup is
issued. Although the wakeup will be eventually caught later in the suspend
process, kicking the can down the road is suboptimal; when there's a wakeup
detected, suspend should be immediately aborted by returning an error
instead. Make try_to_freeze_tasks() do just that, and also move the wakeup
check above the `todo` check so that we don't miss a wakeup from a process
that successfully froze.
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Change-Id: I6d0ff54b1e1e143df2679d3848019590725c6351
PM callbacks can be used as an indicator that the system is suspending, and
as such, a wakeup may be issued outside the notifier in order to cancel an
ongoing suspend. Clearing wakeups after running the PM callbacks can thus
cause wakeups to be missed. Fix it by simply clearing wakeups prior to
running PM callbacks.
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Change-Id: I49b6f900284ff42b245be77548f6ca756cf585ee
Calling s2idle_wake() once after pm_abort_suspend is incremented is all
that's needed to wake up from s2idle. Avoid multiple unnecessary attempts
to wake from s2idle by only doing the wakeup when pm_abort_suspend hits 1.
The s2idle machinery already provides the synchronization needed to make
this safe.
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Change-Id: I789857351353737f7199a8df5c5ba210050352e6
Freezing processes on Android usually takes less than 100 ms, and if it
takes longer than that to the point where the 20 second freeze timeout is
reached, it's because the remaining processes to be frozen are deadlocked
waiting for something from a process which is already frozen. There's no
point in burning power trying to freeze for that long, so reduce the freeze
timeout to a very generous 1 second for Android and don't let anything mess
with it.
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
When some CPUs cycle out of s2idle due to non-wakeup IRQs, it's possible
for them to run while the CPU responsible for jiffies updates remains idle.
This can delay the execution of timers indefinitely until the CPU managing
the jiffies updates finally wakes up, by which point everything could be
dead if enough time passes.
Fix it by handing off timekeeping duties when the timekeeping CPU enters
s2idle and freezes its tick. When all CPUs are in s2idle, the first one to
wake up for any reason (either from a wakeup IRQ or non-wakeup IRQ) will
assume responsibility for the timekeeping tick.
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Change-Id: Idbb2e825c489e174d5701e0c315b51a3149bfe49
Replace 'schedule(); try_to_freeze();' with a call to freezable_schedule().
Tasks calling freezable_schedule() set the PF_FREEZER_SKIP flag
before calling schedule(). Unlike tasks calling schedule();
try_to_freeze() tasks calling freezable_schedule() are not awaken by
try_to_freeze_tasks(). Instead they call try_to_freeze() when they
wake up if the freeze is still underway.
It is not a problem since sleeping tasks can't do anything which isn't
allowed for a frozen task while sleeping.
The result is a potential performance gain during freeze, since less
tasks have to be awaken.
For instance on a bare Debian vm running a 4.19 stable kernel, the
number of tasks skipped in freeze_task() went up from 12 without the
patch to 32 with the patch (out of 448), an increase of > x2.5.
Signed-off-by: Hugo Lefeuvre <hle@owl.eu.com>
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Cc: Joel Fernandes <joel@joelfernandes.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/20190207200352.GA27859@behemoth.owl.eu.com.local
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Change-Id: I1b90daf9ebc8cdfaffc5fe8fe8f1883b9588a6c6
* This is spurious and does not get destroyed, which keeps this
wakelock as pending. Remove this code to save power.
* To enable these wakelocks again, pass `-DIPA_WAKELOCKS` as a
cflag.
Signed-off-by: Vaisakh Murali <mvaisakh@statixos.com>
Change-Id: I4a424f7bddbfb793edbdeb3b6ac1b69d2f4a676e
The original dedup code does not handle collision from the observation
that it practically does not happen.
For additional peace of mind, use a bigger hash size for reducing the
possibility of collision even further.
Signed-off-by: Juhyung Park <qkrwngud825@gmail.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Change-Id: I4543ce22d69a21b2edcc72a72790d2c55a9e312c
Reference: 59e1a2f4bf83 ("ksm: replace jhash2 with xxhash")
Signed-off-by: Park Ju Hyung <qkrwngud825@gmail.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Change-Id: Icb6221a96d0d59e6bbc1c908ec1cd5e59d4a7117
xxhash's performance depends heavily on compiler optimizations including
inlines. Follow upstream's behavior and inline those helper functions.
Signed-off-by: Juhyung Park <qkrwngud825@gmail.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Change-Id: Ieee439865dc48f21a97032a8009941331c302227
These simply wraps memcpy().
Replace it with macros so that it is naturally inlined.
Signed-off-by: Juhyung Park <qkrwngud825@gmail.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Change-Id: I67887a706816aff46339d43b2f28e97813c1d4af
A measurably significant amount of CPU time is spent on logging events
for debugging purposes in lpm_cpuidle_enter. Kill the useless logging to
reduce overhead.
Signed-off-by: Danny Lin <danny@kdrag0n.dev>
Change-Id: I9d34ece7e910d92e6f1a55fa85b3db5828fbbb9e
Waking the GPU upon touch wastes power when the screen is being touched
in a way that does not induce animation or any actual need for GPU usage.
Instead of preemptively waking the GPU on touch input, wake it up upon
receiving a IOCTL_KGSL_GPU_COMMAND ioctl since it is a sign that the GPU
will soon be needed.
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Change-Id: I6387083562578b229ea0913b5b2fa6562d4a85e9
When the rotator is actually used (still an unsolved question in
computer science), these PM QoS requests block some CPUs in the LITTLE
cluster from entering deep idle because the driver assumes that display
rotating work occurs on a hardcoded set of CPUs, which is false. We
already have the IRQ PM QoS machinery for display rendering operations
that actually matter, so this cruft is unneeded.
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Change-Id: I30542fa3009dd46ff0c21058f14c8923d7fcb892
These are blocking some CPUs in the LITTLE cluster from entering deep
idle because the driver assumes that display rendering work occurs on a
hardcoded set of CPUs, which is false. The scope of this is also quite
large, which increases power consumption.
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Change-Id: I0ab629ac45d9d99b74f764030c1e1c2e6823fc24
Qualcomm's implementation used the value from device tree node
qcom,pm-qos-cpu-group-latency-us (67 microseconds for atoll). Set
it manually to match PM QoS requests in other drivers.
Change-Id: I7aa175e90293f369cf4b838c2ff7d6b99177f1a8
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Qualcomm's PM QoS solution suffers from a number of issues: applying
PM QoS to all CPUs, convoluted spaghetti code that wastes CPU cycles,
and keeping PM QoS applied for 10 ms after all requests finish
processing.
This implements a simple IRQ-affined PM QoS mechanism for each UFS
adapter which uses atomics to elide locking, and enqueues a worker to
apply PM QoS to the target CPU as soon as a command request is issued.
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Change-Id: I634873deb7baf0916208809c025eb0d3ccdb0fa3
This implementation is completely over the top and wastes lots of CPU
cycles. It's too convoluted to fix, so just scrap it to make way for a
simpler solution. This purges every PM QoS reference in the UFS drivers.
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Change-Id: I6b6a585aa975c10dcec5f1d6a0062cc6073ffdf4
UFS devices from a couple of vendors have a limitation with the
number of power cycles. They can withstand only a certain amount
of power cycles.
Disable turning off the link for those devices so that we can
minimize the number of power cycles.
Change-Id: I87030f63c9312eb216b292549238455054efb706
Signed-off-by: Veerabhadrarao Badiganti <vbadigan@codeaurora.org>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
The scsi_block_reqs_cnt increased in ufshcd_hold() is supposed to be
decreased back in ufshcd_ungate_work() in a paired way. However, if
specific ufshcd_hold/release sequences are met, it is possible that
scsi_block_reqs_cnt is increased twice but only one ungate work is
queued. To make sure scsi_block_reqs_cnt is handled by ufshcd_hold() and
ufshcd_ungate_work() in a paired way, increase it only if queue_work()
returns true.
Change-Id: I35f870e49cc832bb6f4b58763e3240f05b98ec13
Signed-off-by: Can Guo <cang@codeaurora.org>
Signed-off-by: Ziqi Chen <ziqichen@codeaurora.org>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Currently bits in hba->outstanding_tasks are cleared only after their
corresponding task management commands are successfully done by
__ufshcd_issue_tm_cmd().
If timeout happens in a task management command, its corresponding bit in
hba->outstanding_tasks will not be cleared until next task management
command with the same tag used successfully finishes.
This is wrong and can lead to some issues, like power issue. For example,
ufshcd_release() and ufshcd_gate_work() will do nothing if
hba->outstanding_tasks is not zero even if both UFS host and devices are
actually idle.
Solution is referred from error handling of device commands: bits in
hba->outstanding_tasks shall be cleared regardless of their execution
results.
Change-Id: I92269b69da38f65b23355aa928869ff2f98c9120
Signed-off-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Chun-Hung Wu <chun-hung.wu@mediatek.com>
Reviewed-by: Avri Altman <avri.altman@wdc.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Git-commit: b557217c8475f40bc765ee20ff6b3b9124c8a4fe
Git-repo: https://android.googlesource.com/kernel/common/
[nitirawa@codeaurora.org: Ported to 4.19 kernel]
Signed-off-by: Nitin Rawat <nitirawa@codeaurora.org>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Commit 53c12d0ef6 ("scsi: ufs: fix error recovery after the hibern8
exit failure") would leave hba->clk_gating.active_reqs++ and skip
subsequent actions in ufshcd_hold() if error handling is in progress.
It may cause next ufschcd_release() queue a new clk gate work into
hytimer even though the previous clk gate work has not yet got finish.
Under this corner case, ufshcd_gate_work() may change uic_link_state
to UIC_LINK_HIBERN8_STATE at the heels of setting clk state to CLK_ON
by ufshcd_hold() and then run into ufshcd_hold dead loop. To fix this
issue, we need to ensure there is no any pending and running clk gate
work before changing clk state to CLK_ON in ufshcd_hold().
Change-Id: I25fe35f2cad18f8a77fccf40755d856ee670594d
Signed-off-by: Ziqi Chen <ziqichen@codeaurora.org>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
In some rare scenario, inconsistent state b/w
clk_gating.state and uic link state is observed
when back to back clk_gate_work gets scheduled.
Sequence of events:
1. hibern8 enter fails as a part of ufshc_gate_work.
2. Link recovery marks eh_in_progress and schedules
the error handler to recovery the failure.
3. Error handler finished its work, it
clears eh_in_progress and returns, thus link recovery
succeeds
4. THe next retry of hibern8 succeed and sets the
clk_gating.state to CLK_OFF.
5. B/w step 2 and 4 when eh_in_progress is already set,
if any async thread says IOCTL calls ufshcd_hold, only
clk_gating.active_requests increases and it bails out.
6. After async thread finishes its job and calls ufshcd_release.
if ufshcd_release() is called after eh_in_progress is
cleared, it would schedule the
clk_gate_work instead of bailing out.
7. when 2nd gate_work runs after 1st clk gate work is over,
since 1st gate work would have set the clk_gating.state
as CLKS_OFF, the current gate work sets
clk_gating.state to CLKS_ON and bails out.
This leaves the clk_gating.state as CLKS_ON and link as
hibern8 state which causes inconsistency and causes i/o block
and deadloop when new i/o request calls ufshcd_hold functions
with sync flag as false and true successfully.
Fix this by checking clk_gating.state, if it is already CLKS_OFF,
bail from clk_gate_work().
Change-Id: Ia60a4c872c39636c1e5144584025f8938e712202
Signed-off-by: Can Guo <cang@codeaurora.org>
Signed-off-by: Nitin Rawat <nitirawa@codeaurora.org>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
The initial value of pos is zero.
This causes UFS driver to access err_hist->reg[-1].
Bug: 174649668
Signed-off-by: Randall Huang <huangrandall@google.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Change-Id: I4f0bc416ae8f52bf4e4f88ba84a8db98513e165b
[1] tried to fix wrong lrbp=NULL after clear_bit_unlock, which caused a race
condition. But, it caused a power regression since __ufshcd_release was called
before lrb_in_use is released.
Actually, we should have followed the sequence:
1. lrbp->cmd = NULL
2. clear_bit_unlock()
3. __ufshcd_release()
4. __ufshcd_hibern8_release()
Let's add right fix.
[1] f0e7e5baba0a (scsi: ufs: Avoid potential lrb race caused by early release of lrb_in_use")
Bug: 157450639
Fixes: 639e1063a57e ("Revert "scsi: ufs: Avoid potential lrb race caused by early release of lrb_in_use"")
Fixes: f0e7e5baba0a (scsi: ufs: Avoid potential lrb race caused by early release of lrb_in_use")
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Change-Id: If738dc3e8e9caa7eee28c729dea3a47a3ed56b97
[dereference23: Apply to msm-4.14]
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
The descriptor access function has a potential issue. It makes a
buffer overflow bug and trigger a kernel panic. Fix the boundary check
and return EINVAL when it has an invalid input.
log:
Kernel panic - not syncing: stack-protector: Kernel stack is corrupted
in: ufs_sysfs_read_desc_param+0x1a4/0x1a4
Call trace:
dump_backtrace+0x0/0x1a0
dump_stack+0xbc/0xf8
panic+0x150/0x2d4
clear_warn_once_fops_open+0x0/0x30
lun_write_protect_show+0x0/0x74
Bug: 153344835
Test: adb shell cat /sys/devices/platform/soc/1d84000.ufshc/*_descriptor*/*
Change-Id: Ie57cfacc6f7b32f68e1b54bb1cf059d60e6d17c6
Signed-off-by: Leo Liou <leoliou@google.com>
[dereference23: Apply to msm-4.14]
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Matched with Pixel-4.19, we can disable clocks and links if autohibern8 is on.
The patch fixes power regression, 10~15mW, caused by the below two patches.
Bug: 151181812
Fixes: 92967c0c2ecb "scsi: ufs: set autohibern8 timer regardless of hibern8_on_idle"
Fixes: 5a57ae46dd6f "scsi: ufs: disable hibern8_on_idle"
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Change-Id: I00fe1add360922559d7e227ef630f34d04a42958
We observed several failures when probing hba.
Since there is no error recovery design, the device is
not boot into Android.
Log as shown below:
pwr ctrl cmd 0x2 failed, host upmcrs:0x4
...
ufshcd_change_power_mode: power mode change failed 4
ufshcd_probe_hba: Failed setting power mode, err = 4
Because we do not support removable UFS card,
we can introduce a retry design in async_scan().
The UFS can be initlized after re-probing hba.
Log as shown below:
ufshcd_async_scan failed. Err = 4. Retry 3
qcom_ice 1d90000.ufsice: QC ICE 3.1.79 device found @0xffffff800d9a0000
ufshcd_print_pwr_info:[RX, TX]: gear=[1, 1], lane[1, 1], pwr[SLOWAUTO_MODE, SLOWAUTO_MODE], rate = 0
ufshcd_print_pwr_info:[RX, TX]: gear=[3, 3], lane[2, 2], pwr[FAST MODE, FAST MODE], rate = 2
Bug: 151080047
Test: inject failure and boot to Android
Change-Id: I76af5c89f91c72aecb1b5824fcada66dadbc5c4a
Signed-off-by: Randall Huang <huangrandall@google.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
This reverts commit 28652f3b71d2eeb780d61085907c8a04d5ced116.
Change log from v1:
- add is_binary_power_count to toggle the power_count under phy mutex
- no funcitonal changes on other components
= v1 description:
This can contribute a race condition between is_phy_pwr_on and phy_power_on/off.
Instead of it, we must rely on mutex with power_count in phy_power_on/off more
precisely. And, in order to fix the origina issue caused by multiple power on/
off, we must use power_count as a flag.
So, the final approach is same as moving "is_phy_pwr_on" under mutex.
Bug: 139262967
Change-Id: Ia2ca0a1187b97e0f5b9e46d8e8465ac0b28d2862
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Must have WQ_MEM_RECLAIM
``WQ_MEM_RECLAIM``
All wq which might be used in the memory reclaim paths **MUST**
have this flag set. The wq is guaranteed to have at least one
execution context regardless of memory pressure.
Bug: 158050260
Bug: 155410470
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Link: https://lore.kernel.org/linux-scsi/20200915204532.1672300-3-jaegeuk@kernel.org/T/#u
Change-Id: I65f2608650fa3436503581a60ac539f85273a21e
(cherry picked from commit ceed5e02518f5cedc799f2f37bd30211243a7959)
Signed-off-by: Adam W. Willis <return.of.octobot@gmail.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
When enabling/disabling hibern8, we will call flush_work
to wait for the task finished. Using async operation
to avoid long waiting time.
Bug: 138085490
Test: Boot, cat /sys/kernel/debug/ufshcd0/show_hba and
check hibern8_exit_cnt
Signed-off-by: Martin Liu <liumartin@google.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Change-Id: I427cd8e40cbc6674a145142babbd92a1062dc511
This merged the following fix:
6a317b49c98c ("scsi: ufs: revise commit ecd2676bd513 ("disallow SECURITY_PROTOCOL_IN without _OUT")")
If we allow this, Hynix will give timeout due to spec violation.
The latest Hynix controller gives error instead of timeout.
Bug: 113580864
Bug: 79898356
Bug: 109850759
Bug: 117682499
Bug: 112560467
Change-Id: Ie7820a9604e4c7bc4cc530acf41bb5bb72f33d5b
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Signed-off-by: Randall Huang <huangrandall@google.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
We should call up_write() in the error case.
Bug: 77551464
Change-Id: I2e959a17f2dcafd2a2b8ff78dc4f8fa971929ac1
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
ufshcd_read_desc_param() will allocate full-size buffer and give it required
data back. Otherwise, it can cause memory buffer corruption.
Bug: 77551464
Change-Id: I84ed0cd1470c0dcc12f0aff9d631433fe16244e4
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
ufshcd_probe_hba() puts pm_runtime all the time. So, during the reset flow,
we need to get one.
pm_runtime_get_sync (1)
ufshcd_async_scan
pm_runtime_get_sync (2)
ufshcd_hold_all
ufshcd_probe_hba
- pm_runtime_put_sync (1)
- ufshcd_reset_and_restore
- ufshcd_detect_device
- ufshcd_host_reset_and_restore
- ufshcd_probe_hba
- pm_runtime_put_sync (0)
ufshcd_release_all
pm_runtime_put_sync (-1)
Bug: 157744625
Bug: 153043714
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Change-Id: I2d5696d6143842790fa25218beda12b71cfcc1d6
A non-zero return value could come from ufshcd_hba_enable() when the
host fails at reinit flow. But ufshcd_probe_hba() didn't take the error
value of this case. The patch is to fix this issue.
Bug: 160290772
Change-Id: Iedceeaeb056fe9fb519370ec42424e0542a73ed2
Signed-off-by: Leo Liou <leoliou@google.com>
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Bug: 134949663
Bug: 137150088
Bug: 149155051
Test: test with powerhint feature
Change-Id: Ie5002107a69e7d56a889138eec0e593de1bf6a61
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Signed-off-by: Leo Liou <leoliou@google.com>
[dereference23: Apply to msm-4.14]
Signed-off-by: Alexander Winkowski <dereference23@outlook.com>
Vincent Palomares