We have architectures where the size of page_to_pfn and pfn_to_page are
significant enough to overall image size that they wish to push them out of
line. However, in the process we have grown a second copy of the
implementation of each of these routines for each memory model. Share the
implmentation exposing it either inline or out-of-line as required.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
The buddy allocator has a requirement that boundaries between contigious
zones occur aligned with the the MAX_ORDER ranges. Where they do not we
will incorrectly merge pages cross zone boundaries. This can lead to pages
from the wrong zone being handed out.
Originally the buddy allocator would check that buddies were in the same
zone by referencing the zone start and end page frame numbers. This was
removed as it became very expensive and the buddy allocator already made
the assumption that zones boundaries were aligned.
It is clear that not all configurations and architectures are honouring
this alignment requirement. Therefore it seems safest to reintroduce
support for non-aligned zone boundaries. This patch introduces a new check
when considering a page a buddy it compares the zone_table index for the
two pages and refuses to merge the pages where they do not match. The
zone_table index is unique for each node/zone combination when
FLATMEM/DISCONTIGMEM is enabled and for each section/zone combination when
SPARSEMEM is enabled (a SPARSEMEM section is at least a MAX_ORDER size).
Signed-off-by: Andy Whitcroft <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Yasunori Goto <[email protected]>
Cc: KAMEZAWA Hiroyuki <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
When we fail to mount from a valid root device list out the filesystems we
have tried to mount it with. This gives the user vital diagnostics as to
what is missing from their kernel.
For example in the fragment below the kernel does not have CRAMFS compiled
into the kernel and yet appears to recognise it at the RAMDISK detect
stage. Later the mount fails as we don't have the filesystem.
RAMDISK: cramfs filesystem found at block 0
RAMDISK: Loading 1604KiB [1 disk] into ram disk... done.
XFS: bad magic number
XFS: SB validate failed
No filesystem could mount root, tried: reiserfs ext3 ext2 msdos vfat
iso9660 jfs xfs
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Seems we are trying to init the node_mem_map when we don't need to, for
example when SPARSEMEM is enabled. This causes the error below during
compilation. Use CONFIG_FLAT_NODE_MEM_MAP to gate allocation and init.
arch/x86_64/mm/numa.c: In function `setup_node_zones':
arch/x86_64/mm/numa.c:191: error: structure has no member
named `node_mem_map'
Signed-off-by: Andy Whitcroft <[email protected]>
Acked-by: Andi Kleen <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Add some documentation regarding the utilisation of the flags field in
struct page. This field is overloaded for per page bits and to hold node,
zone and SPARSEMEM information. Make it clear which areas are used for
what and how many bits are in each area.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
GFP_ZONETYPES calculate from GFP_ZONEMASK
GFP_ZONETYPES's value is directly related to the value of GFP_ZONEMASK. It
takes one of two forms depending whether the top bit of GFP_ZONEMASK is a
'loner'. Supply both forms, enabling the loner.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
GFP_ZONETYPES define using GFP_ZONEMASK and add commentry
Add commentry explaining the optimisation that we can apply to GFP_ZONETYPES
when the leftmost bit is a 'loaner', it can only be set in isolation.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
In 2.6.15-git6 a change was commited in the oprofile support in
the powerpc architecture. It introduced the powerpc_oprofile_type
which contains the define G4. This causes a name clash with the
existing wacom usb tablet driver.
CC [M] drivers/usb/input/wacom.o
drivers/usb/input/wacom.c:98: error: conflicting types for `G4'
include/asm/cputable.h:37: error: previous declaration of `G4'
CC [M] drivers/usb/mon/mon_text.o
make[3]: *** [drivers/usb/input/wacom.o] Error 1
make[2]: *** [drivers/usb/input] Error 2
The elements of an enum declared in global scope are effectivly
global identifiers themselves. As such we need to ensure the names
are unique. This patch updates the later oprofile support to use
unique names.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Paul Mackerras <[email protected]>
When we select ppc32 under the powerpc architecture we get the
error below. This relates to defining distribute_irqs when this
configuratiom option is undefined.
CC arch/powerpc/sysdev/mpic.o
.../arch/powerpc/sysdev/mpic.c: In function `mpic_setup_this_cpu':
.../arch/powerpc/sysdev/mpic.c:788: error: `CONFIG_IRQ_ALL_CPUS'
undeclared (first use in this function)
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Paul Mackerras <[email protected]>
Clean up the currently available memory models for ppc32 under
the powerpc architecture. We need FLATMEM for ppc32: enable it.
SPARSEMEM is not parameterised for ppc32 so disable that. Take this
opportunity to clean up white space for FLATMEM_ENABLE.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Paul Mackerras <[email protected]>
When compiling without BOOTX_TEXT the following warning is emitted.
Fix up the definition to only be made when required.
CC arch/powerpc/platforms/powermac/udbg_adb.o
.../arch/powerpc/platforms/powermac/udbg_adb.c:41: warning:
`udbg_adb_use_btext' defined but not used
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Paul Mackerras <[email protected]>
udbg_adb_init() has become dependent on btext_drawchar, even when
BOOTX_TEXT support is not selected. This leads to the error below.
Make the check dependant on BOOTX_TEXT.
LD .tmp_vmlinux1
arch/powerpc/platforms/built-in.o(.toc1+0xa40): undefined
reference to `btext_drawchar'
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Paul Mackerras <[email protected]>
Allow SPARSEMEM to be enabled on non-numa x86 systems. This is made
dependant on EXPERIMENTAL also being set. When an in-tree user (such as
simulated numa) exists it should be made dependant on that.
The plan is to have no options and no selector as normal when
!EXPERIMENTAL. When EXPERIMENTAL we enable the FLATMEM and SPARSEMEM
options for X86_PC whilst maintaining DISCONTIGMEM and SPARSEMEM for NUMA.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Before SPARSEMEM is initialised we cannot provide an efficient pfn_to_nid()
implmentation; before initialisation is complete we use early_pfn_to_nid()
to provide location information. Until recently there was no non-init user
of this functionality. Provide a post init pfn_to_nid() implementation.
Note that this implmentation assumes that the pfn passed has been validated
with pfn_valid(). The current single user of this function already has
this check.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
There are three places we define pfn_to_nid(). Two in linux/mmzone.h and one
in asm/mmzone.h. These in essence represent the three memory models. The
definition in linux/mmzone.h under !NEED_MULTIPLE_NODES is both the FLATMEM
definition and the optimisation for single NUMA nodes; the one under SPARSEMEM
is the NUMA sparsemem one; the one in asm/mmzone.h under DISCONTIGMEM is the
discontigmem one. This is not in the least bit obvious, particularly the
connection between the non-NUMA optimisations and the memory models.
Two patches:
flatmem-split-out-memory-model: simplifies the selection of pfn_to_nid()
implementations. The selection is based primarily off the memory model
selected. Optimisations for non-NUMA are applied where needed.
sparse-provide-pfn_to_nid: implement pfn_to_nid() for SPARSEMEM
This patch:
pfn_to_nid is memory model specific
The pfn_to_nid() call is memory model specific. It represents the locality
identifier for the memory passed. Classically this would be a NUMA node,
but not a chunk of memory under DISCONTIGMEM.
The SPARSEMEM and FLATMEM memory model non-NUMA versions of pfn_to_nid()
are folded together under NEED_MULTIPLE_NODES, while DISCONTIGMEM has its
own optimisation. This is all very confusing.
This patch splits out each implementation of pfn_to_nid() so that we can
see them and the optimisations to each.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
pfn_to_pgdat() isn't used in common code. Remove definition.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
kvaddr_to_nid() isn't used in common code nor in i386 code. Remove these
definitions.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
register_memory is global and declared so in linux/memory.h. Update the
HOTPLUG specific definition to match. This fixes a compile warning when
HOTPLUG is enabled.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Both register_memory_notifer and unregister_memory_notifier are global and
declared so in linux/memory.h. Update the HOTPLUG specific definitions to
match. This fixes a compile warning when HOTPLUG is enabled.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
__add_section defines an unused pointer to the zones pgdat. Remove this
definition. This fixes a compile warning.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
With the new powerpc architecture we don't seem to be able to disable huge
pages anymore.
mm/built-in.o(.toc1+0xae0): undefined reference to `HPAGE_SHIFT'
make: *** [.tmp_vmlinux1] Error 1
We seem to need to define HPAGE_SHIFT to something when HUGETLB_PAGE isn't
defined. This patch defines it to PAGE_SHIFT when we have no support.
Signed-off-by: Andy Whitcroft <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Benjamin Herrenschmidt <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Currently when we first select memory model (FLAT, DISCONTIG, SPARSE) then
select whether the machine is NUMA. However NUMA systems may not be FLAT.
This constraint it not honoured and we may configure a NUMA/FLAT system.
Reorder the configuration such that we choose NUMA first which allows us to
only list the memory models which are valid. We now default NUMA for known
NUMA systems. Note that this new order also matches that used in x86.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Joel Schopp <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Paul Mackerras <[email protected]>
Originally __free_pages_bulk used the relative page number within a zone to
define its buddies. This meant that to maintain the "maximally aligned"
requirements (that an allocation of size N will be aligned at least to N
physically) zones had to also be aligned to 1<<MAX_ORDER pages. When
__free_pages_bulk was updated to use the relative page frame numbers of the
free'd pages to pair buddies this released the alignment constraint on the
'left' edge of the zone. This allows _either_ edge of the zone to contain
partial MAX_ORDER sized buddies. These simply never will have matching
buddies and thus will never make it to the 'top' of the pyramid.
The patch below removes a now redundant check ensuring that the mem_map was
aligned to MAX_ORDER.
Signed-off-by: Andy Whitcroft <[email protected]>
Cc: Christoph Lameter <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
It seems that X86 architectures in general need the setup-bus.o
not just those with HOTPLUG. This avoids the following error on
X86_NUMAQ and x86_64:
arch/i386/pci/built-in.o(.init.text+0x15a6): In function `pcibios_init':
: undefined reference to `pci_assign_unassigned_resources'
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Greg Kroah-Hartman <[email protected]>
Provide hooks for PPC64 to allow memory models to be informed of installed
memory areas. This allows SPARSEMEM to instantiate mem_map for the populated
areas.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Dave Hansen <[email protected]>
Signed-off-by: Martin Bligh <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Provide an implementation of early_pfn_to_nid for PPC64. This is used by
memory models to determine the node from which to take allocations before the
memory allocators are fully initialised.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Dave Hansen <[email protected]>
Signed-off-by: Martin Bligh <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Make sparse's initalization be accessible at runtime. This allows sparse
mappings to be created after boot in a hotplug situation.
This patch is separated from the previous one just to give an indication how
much of the sparse infrastructure is *just* for hotplug memory.
The section_mem_map doesn't really store a pointer. It stores something that
is convenient to do some math against to get a pointer. It isn't valid to
just do *section_mem_map, so I don't think it should be stored as a pointer.
There are a couple of things I'd like to store about a section. First of all,
the fact that it is !NULL does not mean that it is present. There could be
such a combination where section_mem_map *is* NULL, but the math gets you
properly to a real mem_map. So, I don't think that check is safe.
Since we're storing 32-bit-aligned structures, we have a few bits in the
bottom of the pointer to play with. Use one bit to encode whether there's
really a mem_map there, and the other one to tell whether there's a valid
section there. We need to distinguish between the two because sometimes
there's a gap between when a section is discovered to be present and when we
can get the mem_map for it.
Signed-off-by: Dave Hansen <[email protected]>
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Jack Steiner <[email protected]>
Signed-off-by: Bob Picco <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
The part of the sparsemem patch which modifies memmap_init_zone() has recently
become a problem. It changes behavior so that there is a call to
pfn_to_page() for each individual page inside of a node's range:
node_start_pfn through node_end_pfn. It used to simply do this once, at the
beginning of the node, but having sparsemem's non-contiguous mem_map[]s inside
of a node made it necessary to change.
Mike Kravetz recently wrote a patch which made the NUMA code accept some new
kinds of layouts. The system's memory was laid out like this, with node 0's
memory in two pieces: one before and one after node 1's memory:
Node 0: +++++ +++++
Node 1: +++++
Previous behavior before Mike's patch was to assign nodes like this:
Node 0: 00000 XXXXX
Node 1: 11111
Where the 'X' areas were simply thrown away. The new behavior was to make the
pg_data_t span node 0 across all of its areas, including areas that are really
node 1's: Node 0: 000000000000000 Node 1: 11111
This wastes a little bit of mem_map space, but ends up being OK, and more
fully utilizes the system's memory. memmap_init_zone() initializes all of the
"struct page"s for node 0, even for the "hole", but those never get used,
because there is no pfn_to_page() that resolves to those pages. However, only
calling pfn_to_page() once, memmap_init_zone() always uses the pages that were
allocated for node0->node_mem_map because:
struct page *start = pfn_to_page(start_pfn);
// effectively start = &node->node_mem_map[0]
for (page = start; page < (start + size); page++) {
init_page_here();...
page++;
}
Slow, and wasteful, but generally harmless.
But, modify that to call pfn_to_page() for each loop iteration (like sparsemem
does):
for (pfn = start_pfn; pfn < < (start_pfn + size); pfn++++) {
page = pfn_to_page(pfn);
}
And you end up trying to initialize node 1's pages too early, along with bogus
data from node 0. This patch checks for those weird layouts and declines to
touch the pages, making the more frequent pfn_to_page() calls OK to do.
Signed-off-by: Dave Hansen <[email protected]>
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Dave Hansen <[email protected]>
Signed-off-by: Martin Bligh <[email protected]>
Signed-off-by: Adrian Bunk <[email protected]>
Signed-off-by: Yasunori Goto <[email protected]>
Signed-off-by: Bob Picco <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Allow architectures to indicate that they will be providing hooks to indice
installed memory areas, memory_present(). Provide prototypes for the i386
implementation.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Dave Hansen <[email protected]>
Signed-off-by: Martin Bligh <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Provide a default implementation for early_pfn_to_nid returning node 0. Allow
architectures to override this with their own implementation out of
asm/mmzone.h.
Signed-off-by: Andy Whitcroft <[email protected]>
Signed-off-by: Dave Hansen <[email protected]>
Signed-off-by: Martin Bligh <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Andy Whitcroft