xa_cmpxchg() was one of the largest functions in the xarray
implementation. By turning it into a wrapper and having the callers
take the lock (like several other functions), we save 160 bytes on a
tinyconfig build and reduce the duplication in xarray.c.
Change-Id: I31c97a82dae4eaab5ddaada6901092a94d392cb0
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The xa_reserve() function was a little unusual in that it attempted to
be callable for all kinds of locking scenarios. Make it look like the
other APIs with __xa_reserve, xa_reserve_bh and xa_reserve_irq variants.
Change-Id: I292670b8f2610bd87f51cd201aa5616e718f7bfd
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This version of xa_store_range() really only supports load and store.
Our only user only needs basic load and store functionality, so there's
no need to do the extra work to support marking and overlapping stores
correctly yet.
Change-Id: I240de2bbfd3113758f810b152dfebff110a2c35f
Signed-off-by: Matthew Wilcox <willy@infradead.org>
We construct an XA_STATE and use it to delete the node with
xas_store() rather than adding a special function for this unique
use case. Includes a test that simulates this usage for the
test suite.
Change-Id: Ie0562e2e12a140c8d83054665850156caf880b4a
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Rename the function from page_cache_tree_delete_batch to just
page_cache_delete_batch.
Change-Id: I717f92bd99e9802fec02c2b9b2c7d1f7e46ff2f1
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Use the XArray APIs to add and replace pages in the page cache. This
removes two uses of the radix tree preload API and is significantly
shorter code. It also removes the last user of __radix_tree_create()
outside radix-tree.c itself, so make it static.
Change-Id: Ic234ad7ff50538314e9ec448fcb2334238480d21
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Add the optional ability to track which entries in an XArray are free
and provide xa_alloc() to replace most of the functionality of the IDR.
Change-Id: I1e51411a11f6007bb5c290a452872567aa7a5c02
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This function reserves a slot in the XArray for users which need
to acquire multiple locks before storing their entry in the tree and
so cannot use a plain xa_store().
Change-Id: Ic8073c1f9fde56d5e239ebed66fe3c92b4ed26f6
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This hopefully temporary function is useful for users who have not yet
been converted to multi-index entries.
Change-Id: I2edc893d5e723c21eaf9efbf88deba8831af6055
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This iterator iterates over each entry that is stored in the index or
indices specified by the xa_state. This is intended for use for a
conditional store of a multiindex entry, or to allow entries which are
about to be removed from the xarray to be disposed of properly.
Change-Id: I711639d3ce472e596503a985e551999b4ffa0d52
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The xas_next and xas_prev functions move the xas index by one position,
and adjust the rest of the iterator state to match it. This is more
efficient than calling xas_set() as it keeps the iterator at the leaves
of the tree instead of walking the iterator from the root each time.
Change-Id: I2db0ff13439561c45debd089a68dfe8637e581b5
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This function frees all the internal memory allocated to the xarray
and reinitialises it to be empty.
Change-Id: I3981880850e53bee90ae8472c1b3d6f7fcbe2ba8
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The xa_extract function combines the functionality of
radix_tree_gang_lookup() and radix_tree_gang_lookup_tagged().
It extracts entries matching the specified filter into a normal array.
Change-Id: I87805acc6e0f33b8a640dfd019f3cebe50bf0d2f
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The xa_for_each iterator allows the user to efficiently walk a range
of the array, executing the loop body once for each entry in that
range that matches the filter. This commit also includes xa_find()
and xa_find_after() which are helper functions for xa_for_each() but
may also be useful in their own right.
In the xas family of functions, we have xas_for_each(), xas_find(),
xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(),
xas_next_tagged() and xas_pause().
Change-Id: I8a8bf73a071c637a8834096f37ec58bb9e947bb2
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Like cmpxchg(), xa_cmpxchg will only store to the index if the current
entry matches the old entry. It returns the current entry, which is
usually more useful than the errno returned by radix_tree_insert().
For the users who really only want the errno, the xa_insert() wrapper
provides a more convenient calling convention.
Change-Id: Ic1b6b7fa94a4f7c5c3fbd95872fcd8e16f8327d4
Signed-off-by: Matthew Wilcox <willy@infradead.org>
xa_store() differs from radix_tree_insert() in that it will overwrite an
existing element in the array rather than returning an error. This is
the behaviour which most users want, and those that want more complex
behaviour generally want to use the xas family of routines anyway.
For memory allocation, xa_store() will first attempt to request memory
from the slab allocator; if memory is not immediately available, it will
drop the xa_lock and allocate memory, keeping a pointer in the xa_state.
It does not use the per-CPU cache, although those will continue to exist
until all radix tree users are converted to the xarray.
This patch also includes xa_erase() and __xa_erase() for a streamlined
way to store NULL. Since there is no need to allocate memory in order
to store a NULL in the XArray, we do not need to trouble the user with
deciding what memory allocation flags to use.
Change-Id: Ice8910b2746c054c47944c79f7ba115b2cebcea2
Signed-off-by: Matthew Wilcox <willy@infradead.org>
XArray marks are like the radix tree tags, only slightly more strongly
typed. They are renamed in order to distinguish them from tagged
pointers. This commit adds the basic get/set/clear operations.
Change-Id: Ibd405ae12bbd0de3609e52a555d485d8e2bec5d9
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The xa_load function brings with it a lot of infrastructure; xa_empty(),
xa_is_err(), and large chunks of the XArray advanced API that are used
to implement xa_load.
As the test-suite demonstrates, it is possible to use the XArray functions
on a radix tree. The radix tree functions depend on the GFP flags being
stored in the root of the tree, so it's not possible to use the radix
tree functions on an XArray.
Change-Id: I63189fb05efb31166da1da4e2df6ed417f6600fa
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This is a direct replacement for struct radix_tree_node. A couple of
struct members have changed name, so convert those. Use a #define so
that radix tree users continue to work without change.
Change-Id: I387df706412c9e6d07bb87f73d9784eff87a42c5
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Reviewed-by: Josef Bacik <jbacik@fb.com>
This is a direct replacement for struct radix_tree_root. Some of the
struct members have changed name; convert those, and use a #define so
that radix_tree users continue to work without change.
Change-Id: I5a2ed4f4b39af3c0801ba1131c063dd796174af5
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Reviewed-by: Josef Bacik <jbacik@fb.com>
Instead of storing a pointer to the slot containing the canonical entry,
store the offset of the slot. Produces slightly more efficient code
(~300 bytes) and simplifies the implementation.
Change-Id: Ifc93b83c7ad88c25a7e34cdd152c43fb26c06a6b
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Reviewed-by: Josef Bacik <jbacik@fb.com>
An upcoming change to the encoding of internal entries will set the bottom
two bits to 0b10. Unfortunately, m68k only aligns some data structures
to 2 bytes, so the IDR will interpret them as internal entries and things
will go badly wrong.
Change the radix tree so that it stops either when the node indicates
that it's the bottom of the tree (shift == 0) or when the entry is not an
internal entry. This means we cannot insert an arbitrary kernel pointer
as a multiorder entry, but the IDR does not permit multiorder entries.
Annoyingly, this means the IDR can no longer take advantage of the radix
tree's ability to store a single entry at offset 0 without allocating
memory. A pointer which is 2-byte aligned cannot be stored directly in
the root as it would be indistinguishable from a node, so we must allocate
a node in order to store a 2-byte pointer at index 0. The idr_replace()
function does not take a GFP flags argument, so cannot allocate memory.
If a user inserts a 4-byte aligned pointer at index 0 and then replaces
it with a 2-byte aligned pointer, we must be able to store it.
Arbitrary pointer values are still not permitted; pointers of the
form 2 + (i * 4) for values of i between 0 and 1023 are reserved for
the implementation. These are not valid kernel pointers as they would
point into the zero page.
This change does cause a runtime memory consumption regression for
the IDA. I will recover that later.
Change-Id: I9b4ffce4ba4767287e30112e6b3d1704207ed901
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Introduce xarray value entries and tagged pointers to replace radix
tree exceptional entries. This is a slight change in encoding to allow
the use of an extra bit (we can now store BITS_PER_LONG - 1 bits in a
value entry). It is also a change in emphasis; exceptional entries are
intimidating and different. As the comment explains, you can choose
to store values or pointers in the xarray and they are both first-class
citizens.
Change-Id: I24a3019e30f4cfe176e5d750b3466317d62e60bc
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Reviewed-by: Josef Bacik <jbacik@fb.com>
Until VM_DONTEXPAND was added in commit 1c1914d6e8c6 ("dma-buf: heaps:
Don't track CMA dma-buf pages under RssFile") it was possible to obtain
a mapping larger than the buffer size via mremap and bypass the overflow
check in dma_buf_mmap_internal. When using such a mapping to attempt to
fault past the end of the buffer, the CMA heap fault handler also checks
the fault offset against the buffer size, but gets the boundary wrong by
1. Fix the boundary check so that we don't read off the end of the pages
array and insert an arbitrary page in the mapping.
Bug: 254441685
Reported-by: Xingyu Jin <xingyuj@google.com>
Fixes: a5d2d29e24be ("dma-buf: heaps: Move heap-helper logic into the cma_heap implementation")
Cc: stable@vger.kernel.org # Applicable >= 5.10. Needs adjustments only for 5.10.
Signed-off-by: T.J. Mercier <tjmercier@google.com>
Acked-by: John Stultz <jstultz@google.com>
Signed-off-by: Sumit Semwal <sumit.semwal@linaro.org>
Link: https://patchwork.freedesktop.org/patch/msgid/20240830192627.2546033-1-tjmercier@google.com
(cherry picked from commit ea5ff5d351b520524019f7ff7f9ce418de2dad87)
Signed-off-by: Lee Jones <joneslee@google.com>
Change-Id: I741caf611c97af8aff28025c202bba5ec672e518
When dma_buf_stats_setup() fails, it closes the dmabuf file which
results into the calling of dma_buf_file_release() where it does
list_del(&dmabuf->list_node) with out first adding it to the proper
list. This is resulting into panic in the below path:
__list_del_entry_valid+0x38/0xac
dma_buf_file_release+0x74/0x158
__fput+0xf4/0x428
____fput+0x14/0x24
task_work_run+0x178/0x24c
do_notify_resume+0x194/0x264
work_pending+0xc/0x5f0
Fix it by moving the dma_buf_stats_setup() after dmabuf is added to the
list.
Bug: 254441685
Fixes: bdb8d06dfefd ("dmabuf: Add the capability to expose DMA-BUF stats in sysfs")
Signed-off-by: Charan Teja Reddy <quic_charante@quicinc.com>
Tested-by: T.J. Mercier <tjmercier@google.com>
Acked-by: T.J. Mercier <tjmercier@google.com>
Cc: <stable@vger.kernel.org> # 5.15.x+
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/1652125797-2043-1-git-send-email-quic_charante@quicinc.com
(cherry picked from commit ef3a6b70507a2add2cd2e01f5eb9b54d561bacb9)
Signed-off-by: Lee Jones <joneslee@google.com>
Change-Id: I8e4d70e803b2a2dfbeddc45062c83a7038873e24
It appears like nr could be a Spectre v1 gadget as it's supplied by a
user and used as an array index. Prevent the contents
of kernel memory from being leaked to userspace via speculative
execution by using array_index_nospec.
Bug: 254441685
Signed-off-by: Jordy Zomer <jordy@pwning.systems>
Fixes: c02a81fba74f ("dma-buf: Add dma-buf heaps framework")
Cc: <stable@vger.kernel.org> # v5.6+
Acked-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Sumit Semwal <sumit.semwal@linaro.org>
[sumits: added fixes and cc: stable tags]
Link: https://patchwork.freedesktop.org/patch/msgid/20220129150604.3461652-1-jordy@pwning.systems
(cherry picked from commit 92c4cfaee6872038563c5b6f2e8e613f9d84d47d)
Signed-off-by: Lee Jones <joneslee@google.com>
Change-Id: Ibfa678ae63b9d95b8be7f7ddc44bf78ab887b328
DMA-BUF sysfs statistics are an option of DMA-BUF. It does not make
much sense to bother the user with a question about DMA-BUF sysfs
statistics if DMA-BUF itself is not enabled. Worse, enabling the
statistics enables the feature.
Bug: 254441685
Fixes: bdb8d06dfefd666d ("dmabuf: Add the capability to expose DMA-BUF stats in sysfs")
Signed-off-by: Geert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: Sumit Semwal <sumit.semwal@linaro.org>
Link: https://patchwork.freedesktop.org/patch/msgid/20210902124913.2698760-4-geert@linux-m68k.org
(cherry picked from commit 87fd9ef47597b2fcee3264eb5f288410b9f376d5)
Signed-off-by: Lee Jones <joneslee@google.com>
Change-Id: I43784118104af3308e7ab770ef7ef088c3347c83
[ Upstream commit 05b252cccb2e5c3f56119d25de684b4f810ba40a ]
Check vm_fault->pgoff before using it. When we removed the warning, we
also removed the check.
Fixes: 7b26e4e2119d ("udmabuf: drop WARN_ON() check.")
Reported-by: zdi-disclosures@trendmicro.com
Suggested-by: Linus Torvalds <torvalds@linuxfoundation.org>
Change-Id: Iacced848b0766fd44cc6b2d41065460b8616785d
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
[ Upstream commit 2b6dd600dd72573c23ea180b5b0b2f1813405882 ]
Syzbot has reported GPF in sg_alloc_append_table_from_pages(). The
problem was in ubuf->pages == ZERO_PTR.
ubuf->pagecount is calculated from arguments passed from user-space. If
user creates udmabuf with list.size == 0 then ubuf->pagecount will be
also equal to zero; it causes kmalloc_array() to return ZERO_PTR.
Fix it by validating ubuf->pagecount before passing it to
kmalloc_array().
Fixes: fbb0de795078 ("Add udmabuf misc device")
Reported-and-tested-by: syzbot+2c56b725ec547fa9cb29@syzkaller.appspotmail.com
Change-Id: I04cd865c4b681fe791e09f842a5cc1c38b74d98a
Signed-off-by: Pavel Skripkin <paskripkin@gmail.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20211230142649.23022-1-paskripkin@gmail.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
For previous version, it uses 'sg_table.nent's to traverse sg_table in pages
free flow.
However, 'sg_table.nents' is reassigned in 'dma_map_sg', it means the number of
created entries in the DMA adderess space.
So, use 'sg_table.nents' in pages free flow will case some pages can't be freed.
Here we should use sg_table.orig_nents to free pages memory, but use the
sgtable helper 'for each_sgtable_sg'(, instead of the previous rather common
helper 'for_each_sg' which maybe cause memory leak) is much better.
Fixes: d963ab0f15fb0 ("dma-buf: system_heap: Allocate higher order pages if available")
Signed-off-by: Guangming <Guangming.Cao@mediatek.com>
Reviewed-by: Robin Murphy <robin.murphy@arm.com>
Cc: <stable@vger.kernel.org> # 5.11.*
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Sumit Semwal <sumit.semwal@linaro.org>
Link: https://patchwork.freedesktop.org/patch/msgid/20211126074904.88388-1-guangming.cao@mediatek.com
(cherry picked from commit 679d94cd7d900871e5bc9cf780bd5b73af35ab42)
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Change-Id: I17beb50fe37a3471f1831ba41e4bf54adada05de
[ Upstream commit f492283b157053e9555787262f058ae33096f568 ]
It is expected from the clients to follow the below steps on an imported
dmabuf fd:
a) dmabuf = dma_buf_get(fd) // Get the dmabuf from fd
b) dma_buf_attach(dmabuf); // Clients attach to the dmabuf
o Here the kernel does some slab allocations, say for
dma_buf_attachment and may be some other slab allocation in the
dmabuf->ops->attach().
c) Client may need to do dma_buf_map_attachment().
d) Accordingly dma_buf_unmap_attachment() should be called.
e) dma_buf_detach () // Clients detach to the dmabuf.
o Here the slab allocations made in b) are freed.
f) dma_buf_put(dmabuf) // Can free the dmabuf if it is the last
reference.
Now say an erroneous client failed at step c) above thus it directly
called dma_buf_put(), step f) above. Considering that it may be the last
reference to the dmabuf, buffer will be freed with pending attachments
left to the dmabuf which can show up as the 'memory leak'. This should
at least be reported as the WARN().
Change-Id: Ic682f9412953e6059f7c3e2a004723077b7123ff
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/1627043468-16381-1-git-send-email-charante@codeaurora.org
Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
dma_buf_attach() does not release his lock when returning in his normal
path. It will cause detaching thread blocked before llkd crashes kernel.
Bug: 194737348
Fixes: e618a51f562f (ANDROID: Delete the DMA-BUF attachment sysfs
statistics)
Change-Id: I41f20195888d35007f28c4e9f481add84958e728
Reported-by: Haidong.Yao <haidong.yao@unisoc.com>
Cc: Ping Wang <ping.wang@unisoc.com>
Cc: Ziwei Dai <ziwei.dai@unisoc.com>
Cc: Liangcai Fan <liangcai.fan@unisoc.com>
Signed-off-by: Orson Zhai <orson.zhai@unisoc.com>
The overhead of sysfs directory creation/teardown during
dma_buf_attach()/dma_buf_detach() is causing perf regressions for
certain drivers.
Bug: 192621117
Change-Id: I908aa3b2717bf2e183628be3446e0069ce24c68a
Fixes: 90935b28546d (BACKPORT: FROMLIST: dmabuf: Add the capability to
expose DMA-BUF stats in sysfs)
Signed-off-by: Hridya Valsaraju <hridya@google.com>
This reverts commit 18e91c017978f0c91832a18a20066a797bfac550.
Reason for revert: mmap_count is no longer used for reporting dma-bufs and introduces subtle bugs related to changing the vm_ops
Bug: 192459295
Signed-off-by: Kalesh Singh <kaleshsingh@google.com>
Change-Id: I7b2ba232a96833fdcadbed4b3d16b956b864afe4
Daniel Mentz relayed to me that Janghyuck Kim noted that
dma_buf_page_pool_create() returns NULL on error, so the
IS_ERR() check on the return value will never be true.
This patch fixes up the checking so we look for NULL instead and
properly return ENOMEM in that case.
Cc: Daniel Mentz <danielmentz@google.com>
Reported-by: Janghyuck Kim <janghyuck.kim@samsung.com>
Signed-off-by: John Stultz <john.stultz@linaro.org>
Fixes: 23762f02e172 ("ANDROID: dma-buf: system_heap: Add pagepool support to system heap")
Change-Id: Ia428e262cda5cc6bd39a3fbcddade7877473a563
dmabuf_page_pool_init_shrinker needs to be static to prevent a warning
when compiling with -Wmissing-prototypes. Change it to be static.
Fixes: bd4db96bfc24 ("ANDROID: dma-buf: heaps: Add a shrinker controlled page pool")
Bug: 168742043
Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: I64184cf4062e33c14a60b9c3d505db922f2b9c0b
mmap_count can be used to efficiently calculate the
PSS for any DMA buffer that the process has mapped.
Bug: 167141117
Change-Id: I8296981f0135a5532969aca3a8baa5da61dfe39f
Signed-off-by: Kalesh Singh <kaleshsingh@google.com>
Upstream has no modules that use it, but in common
we system/cma/etc are modular, so we need to export
the new dma_heap_get_name() symbol
Signed-off-by: John Stultz <john.stultz@linaro.org>
Bug: 167709539
Change-Id: I25d77467742ea9c309c909874e0f1e9adf8a64af
