Christoph Lameter demonstrated very poor scalability on the SGI 512-way, with
a many-threaded application which concurrently initializes different parts of
a large anonymous area.
This patch corrects that, by using a separate spinlock per page table page, to
guard the page table entries in that page, instead of using the mm's single
page_table_lock. (But even then, page_table_lock is still used to guard page
table allocation, and anon_vma allocation.)
In this implementation, the spinlock is tucked inside the struct page of the
page table page: with a BUILD_BUG_ON in case it overflows - which it would in
the case of 32-bit PA-RISC with spinlock debugging enabled.
Splitting the lock is not quite for free: another cacheline access. Ideally,
I suppose we would use split ptlock only for multi-threaded processes on
multi-cpu machines; but deciding that dynamically would have its own costs.
So for now enable it by config, at some number of cpus - since the Kconfig
language doesn't support inequalities, let preprocessor compare that with
NR_CPUS. But I don't think it's worth being user-configurable: for good
testing of both split and unsplit configs, split now at 4 cpus, and perhaps
change that to 8 later.
There is a benefit even for singly threaded processes: kswapd can be attacking
one part of the mm while another part is busy faulting.
Signed-off-by: Hugh Dickins <hugh@veritas.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Prepare arm for the split page_table_lock: three issues.
Signal handling's preserve and restore of iwmmxt context currently involves
reading and writing that context to and from user space, while holding
page_table_lock to secure the user page(s) against kswapd. If we split the
lock, then the structure might span two pages, secured by to read into and
write from a kernel stack buffer, copying that out and in without locking (the
structure is 160 bytes in size, and here we're near the top of the kernel
stack). Or would the overhead be noticeable?
arm_syscall's cmpxchg emulation use pte_offset_map_lock, instead of
pte_offset_map and mm-wide page_table_lock; and strictly, it should now also
take mmap_sem before descending to pmd, to guard against another thread
munmapping, and the page table pulled out beneath this thread.
Updated two comments in fault-armv.c. adjust_pte is interesting, since its
modification of a pte in one part of the mm depends on the lock held when
calling update_mmu_cache for a pte in some other part of that mm. This can't
be done with a split page_table_lock (and we've already taken the lowest lock
in the hierarchy here): so we'll have to disable split on arm, unless
CONFIG_CPU_CACHE_VIPT to ensures adjust_pte never used.
Signed-off-by: Hugh Dickins <hugh@veritas.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Convert those few architectures which are calling pud_alloc, pmd_alloc,
pte_alloc_map on a user mm, not to take the page_table_lock first, nor drop it
after. Each of these can continue to use pte_alloc_map, no need to change
over to pte_alloc_map_lock, they're neither racy nor swappable.
In the sparc64 io_remap_pfn_range, flush_tlb_range then falls outside of the
page_table_lock: that's okay, on sparc64 it's like flush_tlb_mm, and that has
always been called from outside of page_table_lock in dup_mmap.
Signed-off-by: Hugh Dickins <hugh@veritas.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
First step in pushing down the page_table_lock. init_mm.page_table_lock has
been used throughout the architectures (usually for ioremap): not to serialize
kernel address space allocation (that's usually vmlist_lock), but because
pud_alloc,pmd_alloc,pte_alloc_kernel expect caller holds it.
Reverse that: don't lock or unlock init_mm.page_table_lock in any of the
architectures; instead rely on pud_alloc,pmd_alloc,pte_alloc_kernel to take
and drop it when allocating a new one, to check lest a racing task already
did. Similarly no page_table_lock in vmalloc's map_vm_area.
Some temporary ugliness in __pud_alloc and __pmd_alloc: since they also handle
user mms, which are converted only by a later patch, for now they have to lock
differently according to whether or not it's init_mm.
If sources get muddled, there's a danger that an arch source taking
init_mm.page_table_lock will be mixed with common source also taking it (or
neither take it). So break the rules and make another change, which should
break the build for such a mismatch: remove the redundant mm arg from
pte_alloc_kernel (ppc64 scrapped its distinct ioremap_mm in 2.6.13).
Exceptions: arm26 used pte_alloc_kernel on user mm, now pte_alloc_map; ia64
used pte_alloc_map on init_mm, now pte_alloc_kernel; parisc had bad args to
pmd_alloc and pte_alloc_kernel in unused USE_HPPA_IOREMAP code; ppc64
map_io_page forgot to unlock on failure; ppc mmu_mapin_ram and ppc64 im_free
took page_table_lock for no good reason.
Signed-off-by: Hugh Dickins <hugh@veritas.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Patch from Nicolas Pitre
Fix XIP support after recent bootmem code refactoring.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Deepak Saxena
This patch adds support for 36-bit static mapped I/O. While there
are no platforms in the tree ATM that use it, it has been tested
tested on the IXP2350 NPU and I would like to get the support for
that chipset upstream one piece at a time. There are also other
Intel chipset ports in development that are waiting on this to go
upstream.
The patch replaces the print formats for physical addresses with
%016llx which will create a bit extraneous output on 32-bit systems,
but I think that is cleaner than having #ifdefs, specially since
users will only see the output in error cases.
Depends on 3016/1.
Signed-off-by: Deepak Saxena <dsaxena@plexity.net>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Deepak Saxena
Convert map_desc.physical to map_desc.pfn. This allows us to add
support for 36-bit addressed physical devices in the static maps
without having to resort to u64 variables.
Signed-off-by: Deepak Saxena <dsaxena@plexity.net>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Make ARM independent of the way bootmem operates internally. We
now map each node as we initialise it, and place the bootmem bitmap
inside each node, rather than all in the first node.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Fix sparse warnings in arch/arm/kernel/module.c,
arch/arm/mm/consistent.c, drivers/pcmcia/sa1111_generic.c,
and platform support files.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Tony Lindgren
Machine restart calls cpu_proc_fin() to clean and disable
cache, and turn off interrupts. This patch adds proper
cpu_v6_proc_fin.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from George G. Davis
Fix leading, trailing and other miscellaneous whitespace issues
in arch/arm/kernel/alignment.c.
Signed-off-by: George G. Davis <gdavis@mvista.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from George G. Davis
Add test for invalid LDRD/STRD Rd cases in ARM alignment handler
and restore SWP printk KERN_ERR.
Signed-off-by: Steve Longerbeam <slongerbeam@mvista.com>
Signed-off-by: George G. Davis <gdavis@mvista.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Catalin Marinas
There is no reason to not allow these config options. They are useful when
the hardware has problems.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Catalin Marinas
Data abort caused by ldrex/strex can leave the exclusive monitor in an
unpredictable state. It is recommended that a clrex/strex is performed to
clear this state.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Gen FUKATSU
Invalidate BTB entry instruction flushes two instruction
at a time. Therefore this instruction should be done four
times after invalidate instruction cache line.
Signed-off-by: Gen Fukatsu
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Catalin Marinas
When CONFIG_CPU_CACHE_VIPT is defined, the flush_pfn_alias() function is
implicitely declared and it later conflicts with its actual definition.
This patch moves the function definition to the beginning of the file.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
As per x86, we may deadlock while trying to get the mmap semaphore.
Implement the same fix, which allows (eg) recursive faults to cause
an oops instead of deadlocking.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Ben Dooks
The `make buildcheck` is erroneously reporting that the .proc.info
list is referencing items in the .init section as it is not itself
postfixed with .init
Signed-off-by: Ben Dooks <ben-linux@fluff.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
This adds the necessary changes to ensure that we flush the
caches correctly with aliasing VIPT caches.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Timothy Baldwin
All data aborts are treated as read accesses. The existing code updates the wrong bit of r1, also the comments are wrong in that the sense of the L bit is inverted.
Signed-off-by: Timothy E. Baldwin <T.E.Baldwin99@members.leeds.ac.uk>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
We weren't explicitly setting the page table bits we desired
in user_prot in the protection table, which resulted in the
user mappings for v6 CPUs being marked global.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
No point checking what CPU architecture level we have each time
within the loop, so precompute the base PMD flags outside the
loop.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Steve Longerbeam
Adds an implementation of unaligned LDRD and STRD fixups.
Also fixes a bug where do_alignment() would misinterpret and
fixup an unaligned LDRD/STRD as LDRH/STRH, causing memory
corruption.
This is the same as Patch #2867/1, but with minor whitespace
and comments changes, plus a check for arch-level >= v5TE
before printing ai_dword count in proc_alignment_read().
Signed-off-by: Steve Longerbeam <stevel@mwwireless.net>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Deepak Saxena
Working on adding support for 36-bit static mappings for ARMv6 and
Intel's XSC3 core and noticed that alloc_init_supersection currently
increments the phys addr by 1MB on each of the 16 iterations and then
forces alignment to supersection size (16MB). This is really uneeded
b/c we have already forced the phys address to be 16MB aligned in
create_mapping(). Furthermore, this breaks 36-bit addressing b/c bits
[23:20] of the PMD contain bits [35:32] of the physical address and
the masking causes us to loose those bits thus ending up with an
incorrect virt -> phys translation. The other option is to have an
alloc_init_supersection36.
Tested on Intel IXP2350 CPU with 36-bit static I/O mappings.
Signed-off-by: Deepak Saxena <dsaxena@plexity.net>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Sean Lee
In the arch/arm/mm/Kconfig file, the CPU_DCACHE_WRITETHROUGH
option is depend on the CPU_DISABLE_DCACHE, but the "Disable
D-Cache" option is configured as CPU_DCACHE_DISABLE.
The CPU_DISABLE_DCACHE should be CPU_DCACHE_DISABLE
Signed-off-by: Sean Lee <beginner2arm@eyou.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Unfortunately, we can't use the "user" bit in the page tables to
control whether a page table entry is "global" or "asid" specific,
since the vector page is mapped as "user" accessible but is not
process specific.
Therefore, give direct control of the ARMv6 "nG" (not global)
bit to the mm layers.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
The ARM fault handler is optimised to make the fast path, err, fast.
The renumbering of the VM_FAULT_* codes broke this because numbers
were used instead of the definitions. Fix this.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Deepak Saxena
The XScale locking code is not something that has been validated
on 2.6 and needs to be replaced with a more generic API to use
with other ARMs that support locking features.
Signed-off-by: Deepak Saxena <dsaxena@plexity.net>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
ARMv6 introduces memory types into the page tables. Mark devices
mappings with the "shared device" memory type.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Tony Lindgren
This patch by Paul Mundt and other OMAP developers modifies
ARM specific Kconfig to allow sharing code between OMAP1 and
OMAP2 architectures.
In order to share code between OMAP1 and OMAP2, all OMAP1
specific code is moved into mach-omap1 directory in the
following patch. A new mach-omap2 directory will be added
later on.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Deepak Saxena
The code in mm-armv.c checks for the condition (cpu_architecture()<= ARMv5)
in a few places but should be checking for ARMv5TEJ as the MMU is shared
across all v5 variations.
Signed-off-by: Deepak Saxena <dsaxena@plexity.net>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Catalin Marinas
The VFP instructions trigger undefined exceptions because the access to
CP11 is disabled (only CP10 is currently enabled by the kernel). The patch
fixes this problem.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Catalin Marinas
The range for the ARMv6 block cache operations is inclusive but the
kernel doesn't re-calculate the end address, causing a page fault when
used (this only happens with support for cache aliasing, otherwise the
blk_flush_kern_dcache_page() is not called). This patch subtracts
L1_CACHE_BYTES from the end address.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Catalin Marinas
This patch fixes the V bit setting for the ARM1020x processors. At
reset, this bit is automatically set to the value of the HIVECSINIT
input signal which just happened to be 1 but it is not mandatory.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Catalin Marinas
This patch fixes a broken comment in the proc-arm1020.S file which
prevents the file compilation
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
If we receive an unrecognised abort during boot, don't try to
send a signal to pid0, but instead report the current state.
This leads to less confusing debug reports.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
It doesn't make sense for this to be in mm-armv.c now that 26-bit
ARM support is no longer integrated into arch/arm.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
It doesn't make sense to have the PGD kernel pointers initialisation
separate from the PGD user pointers, especially when we clean the
data cache over the whole range.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Patch from Catalin Marinas
cpu_v6_set_pte() sets the kernel access rights to r/o for user
pages (L_PTE_USER) when neither L_PTE_WRITE nor L_PTE_DIRTY are
set. This causes a kernel data abort when writing the TLS value
in the 0xffff0000 page. This patch enables the kernel r/w access.
Signed-off-by: Catalin Marinas
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Since meminfo.bank[] array contains page-aligned start/size, we
no longer need to explicitly round up/down the addresses when
converting to PFNs.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Ingo recently introduced a great speedup for allocating new mmaps using the
free_area_cache pointer which boosts the specweb SSL benchmark by 4-5% and
causes huge performance increases in thread creation.
The downside of this patch is that it does lead to fragmentation in the
mmap-ed areas (visible via /proc/self/maps), such that some applications
that work fine under 2.4 kernels quickly run out of memory on any 2.6
kernel.
The problem is twofold:
1) the free_area_cache is used to continue a search for memory where
the last search ended. Before the change new areas were always
searched from the base address on.
So now new small areas are cluttering holes of all sizes
throughout the whole mmap-able region whereas before small holes
tended to close holes near the base leaving holes far from the base
large and available for larger requests.
2) the free_area_cache also is set to the location of the last
munmap-ed area so in scenarios where we allocate e.g. five regions of
1K each, then free regions 4 2 3 in this order the next request for 1K
will be placed in the position of the old region 3, whereas before we
appended it to the still active region 1, placing it at the location
of the old region 2. Before we had 1 free region of 2K, now we only
get two free regions of 1K -> fragmentation.
The patch addresses thes issues by introducing yet another cache descriptor
cached_hole_size that contains the largest known hole size below the
current free_area_cache. If a new request comes in the size is compared
against the cached_hole_size and if the request can be filled with a hole
below free_area_cache the search is started from the base instead.
The results look promising: Whereas 2.6.12-rc4 fragments quickly and my
(earlier posted) leakme.c test program terminates after 50000+ iterations
with 96 distinct and fragmented maps in /proc/self/maps it performs nicely
(as expected) with thread creation, Ingo's test_str02 with 20000 threads
requires 0.7s system time.
Taking out Ingo's patch (un-patch available per request) by basically
deleting all mentions of free_area_cache from the kernel and starting the
search for new memory always at the respective bases we observe: leakme
terminates successfully with 11 distinctive hardly fragmented areas in
/proc/self/maps but thread creating is gringdingly slow: 30+s(!) system
time for Ingo's test_str02 with 20000 threads.
Now - drumroll ;-) the appended patch works fine with leakme: it ends with
only 7 distinct areas in /proc/self/maps and also thread creation seems
sufficiently fast with 0.71s for 20000 threads.
Signed-off-by: Wolfgang Wander <wwc@rentec.com>
Credit-to: "Richard Purdie" <rpurdie@rpsys.net>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu> (partly)
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Patch from Bellido Nicolas
Core support for AAEC-2000 based platforms.
This is an updated version of the previous patch, and takes
into account Russell's comments.
AAED-2000 default configuration will follow as soon
as some problems with the bootloader are sorted out...
Signed-off-by: Nicolas Bellido
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Hugh Dickins