We currently set x27 in compat_sys_sigreturn_wrapper and
compat_sys_rt_sigreturn_wrapper, similarly to what we do with r8/why on
32-bit ARM, in an attempt to prevent sigreturns from being restarted.
However, on arm64 we have always used pt_regs::syscallno for syscall
restarting (for both native and compat tasks), and x27 is never
inspected again before being overwritten in kernel_exit.
This patch removes the pointless register assignments.
Signed-off-by: Mark Rutland <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
The dts for the CoreTile Express A15x2 A7x3 (TC2) only describes the
PMUs of the Cortex-A15 CPUs, and not the Cortex-A7 CPUs.
Now that we have a mechanism for describing disparate PMUs and their
interrupts in device tree, this patch makes use of these to describe the
PMUs for all CPUs in the system. For consistency, the existing A15 PMU
interrupt-affinity property is reflowed across two lines.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Will Deacon <[email protected]>
Acked-by: Sudeep Holla <[email protected]>
Cc: Liviu Dudau <[email protected]>
Cc: Lorenzo Pieralisi <[email protected]>
Signed-off-by: Kevin Hilman <[email protected]>
Currently we enable debug exceptions before reading ESR_EL1 in both
el0_inv and el1_inv. If a debug exception is taken before we read
ESR_EL1, the value will have been corrupted.
As el*_inv is typically fatal, an intervening debug exception results in
misleading debug information being logged to the console, but is not
otherwise harmful.
As with the other entry paths, we can use the ESR_EL1 value stashed
earlier in the exception entry (in x25 for el0_sync{,_compat}, and x1
for el1_sync), giving us better error reporting in this case.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
Commit 6c81fe7925 ("arm64: enable context tracking") did not
update el0_sp_pc to use ct_user_exit, but this appears to have been
unintentional. In commit 6ab6463aeb ("arm64: adjust el0_sync so
that a function can be called") we made x0 available, and in the return
to userspace we call ct_user_enter in the kernel_exit macro.
Due to this, we currently don't correctly inform RCU of the user->kernel
transition, and may erroneously account for time spent in the kernel as
if we were in an extended quiescent state when CONFIG_CONTEXT_TRACKING
is enabled.
As we do record the kernel->user transition, a userspace application
making accesses from an unaligned stack pointer can demonstrate the
imbalance, provoking the following warning:
------------[ cut here ]------------
WARNING: CPU: 2 PID: 3660 at kernel/context_tracking.c:75 context_tracking_enter+0xd8/0xe4()
Modules linked in:
CPU: 2 PID: 3660 Comm: a.out Not tainted 4.1.0-rc7+ #8
Hardware name: ARM Juno development board (r0) (DT)
Call trace:
[<ffffffc000089914>] dump_backtrace+0x0/0x124
[<ffffffc000089a48>] show_stack+0x10/0x1c
[<ffffffc0005b3cbc>] dump_stack+0x84/0xc8
[<ffffffc0000b3214>] warn_slowpath_common+0x98/0xd0
[<ffffffc0000b330c>] warn_slowpath_null+0x14/0x20
[<ffffffc00013ada4>] context_tracking_enter+0xd4/0xe4
[<ffffffc0005b534c>] preempt_schedule_irq+0xd4/0x114
[<ffffffc00008561c>] el1_preempt+0x4/0x28
[<ffffffc0001b8040>] exit_files+0x38/0x4c
[<ffffffc0000b5b94>] do_exit+0x430/0x978
[<ffffffc0000b614c>] do_group_exit+0x40/0xd4
[<ffffffc0000c0208>] get_signal+0x23c/0x4f4
[<ffffffc0000890b4>] do_signal+0x1ac/0x518
[<ffffffc000089650>] do_notify_resume+0x5c/0x68
---[ end trace 963c192600337066 ]---
This patch adds the missing ct_user_exit to the el0_sp_pc entry path,
correcting the context tracking for this case.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Will Deacon <[email protected]>
Fixes: 6c81fe7925 ("arm64: enable context tracking")
Cc: <[email protected]> # v3.17+
Signed-off-by: Catalin Marinas <[email protected]>
The kernel memtest utility is incredibly useful for detecting memory
problems, but sadly isn't in defconfig.
The memtest itself is only run when the user has explicitly passed a
memtest option on the kernel command line, so simply enabling the option
should not have a negative impact.
Signed-off-by: Mark Rutland <[email protected]>
Cc: Vladimir Murzin <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
Now that the arm_pmu framework is only used for CPU PMUs, there's no
reason to keep the pseudo-generic and CPU-specific framework portions
separate.
This patch folds the two into perf_event.c.
Signed-off-by: Mark Rutland <[email protected]>
[will: fixed up irq cfg to match upstream]
Signed-off-by: Will Deacon <[email protected]>
Now that the core arm perf code maintains no global state and all
microarchitecture-specific PMU data can be fed in through the shared
probe function, it's possible to use it as a library and get rid of the
C file includes we have currently.
This patch factors out the ARMv7-specific portions out into the ARMv7
driver. For the moment this is always built if perf event support is
enabled, but the preprocessor guards will leave behind an empty file.
Now that perf_event_cpu.c contains no microarchitecture-specific data,
the associated probing code is removed, completing its relegation to a
library file. The vestigal "arm-pmu" platform device ID is removed in
this patch, as it has been unused since platform files were updated to
specify a more specific PMU variant.
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Now that the core arm perf code maintains no global state and all
microarchitecture-specific PMU data can be fed in through the shared
probe function, it's possible to use it as a library and get rid of the
C file includes we have currently.
This patch factors out the ARMv6-specific portions out into the ARMv6
driver. For the moment this is always built if perf event support is
enabled, but the preprocessor guards will leave behind an empty file.
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Now that the core arm perf code maintains no global state and all
microarchitecture-specific PMU data can be fed in through the shared
probe function, it's possible to use it as a library and get rid of the
C file includes we have currently.
This patch factors out the xscale-specific portions out into the xscale
driver. For the moment this is always built if perf event support is
enabled, but the preprocessor guards will leave behind an empty file.
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Enable the probe function to be shared with other drivers, which will
inject the appropriate of_device_id and pmu_probe_info tables.
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Currently the arm perf code has platdata callbacks for runtime PM and
irq handling, but no platform implements the hooks for the former. Kill
these off.
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Now that we can describe PMUs in heterogeneous systems, the only item in
the way of perf support for big.LITTLE is the singleton cpu_pmu variable
used for OProfile compatibility.
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
In heterogeneous systems, the number of counters may differ across
clusters. To find the number of counters for a cluster, we must probe
the PMU from a CPU in that cluster.
Reviewed-by: Will Deacon <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Different CPU microarchitectures implement different PMU events, and
thus events which can be scheduled on one microarchitecture cannot be
scheduled on another, and vice-versa. Some archicted events behave
differently across microarchitectures, and thus cannot be meaningfully
summed. Due to this, we reject the scheduling of an event on a CPU of a
different microarchitecture to that the event targets.
When the core perf code is scheduling events and encounters an event
which cannot be scheduled, it stops attempting to schedule events. As
the perf core periodically rotates the list of events, for some
proportion of the time events which are unschedulable will block events
which are schedulable, resulting in low utilisation of the hardware
counters.
This patch implements a pmu::filter_match callback such that we can
detect and skip such events while scheduling early, before they can
block the schedulable events. This prevents the low HW counter
utilisation issue.
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
In multi-cluster systems, the PMUs can be different across clusters, and
so our logical PMU may not be able to schedule events on all CPUs.
This patch adds a cpumask to encode which CPUs a PMU driver supports
controlling events for, and limits the driver to scheduling events on
those CPUs, and enabling and disabling the physical PMUs on those CPUs.
The cpumask is built based on the interrupt-affinity property, and in
the absence of such a property a homogenous system is assumed.
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
To support multiple PMUs we'll need to pass the arm_pmu instance around.
Update of_pmu_irq_cfg to take an arm_pmu, and acquire the platform
device from this.
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
In certain circumstances it may not be possible to schedule particular
events due to constraints other than a lack of hardware counters (e.g.
on big.LITTLE systems where CPUs support different events). The core
perf event code does not distinguish these cases and pessimistically
assumes that any failure to schedule an event means that it is not worth
attempting to schedule later events, even if some hardware counters are
still unused.
When an event a pmu cannot schedule exists in a flexible group list it
can unnecessarily prevent event groups following it in the list from
being scheduled (until it is rotated to the end of the list). This means
some events are scheduled for only a portion of the time they could be,
and for short running programs no events may be scheduled if the list is
initially sorted in an unfortunate order.
This patch adds a new (optional) filter_match function pointer to struct
pmu which a pmu driver can use to tell perf core when an event matches
pmu-specific scheduling requirements. This plugs into the existing
event_filter_match logic, and makes it possible to avoid the scheduling
problem described above. When no filter is provided by the PMU, the
existing behaviour is retained.
Cc: Paul Mackerras <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Acked-by: Will Deacon <[email protected]>
Acked-by: Peter Zijlstra <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
The 32-bit ARM port doesn't have ACPI headers, and conditionally
including them is going to look horrendous. In preparation for sharing
the PSCI invocation code with 32-bit, move the acpi_psci_* function
declarations and definitions such that the PSCI client code need not
include ACPI headers.
While it would seem like we could simply hide the ACPI includes in
psci.h, the ACPI headers have hilarious circular dependencies which make
this infeasible without reorganising most of ACPICA. So rather than
doing that, move the acpi_psci_* prototypes into psci.h.
The psci_acpi_init function is made dependent on CONFIG_ACPI (with a
stub implementation in asm/psci.h) such that it need not be built for
32-bit ARM or kernels without ACPI support. The currently missing __init
annotations are added to the prototypes in the header.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Acked-by: Hanjun Guo <[email protected]>
Reviewed-by: Al Stone <[email protected]>
Reviewed-by: Ashwin Chaugule <[email protected]>
Tested-by: Hanjun Guo <[email protected]>
Cc: Lorenzo Pieralisi <[email protected]>
Cc: Will Deacon <[email protected]>
A PSCI 1.0 implementation may choose to use the new extended StateID
format, the presence of which may be queried via the PSCI_FEATURES call.
The layout of this new StateID format is incompatible with the existing
format, and so to handle both we must abstract attempts to parse the
fields.
In preparation for PSCI 1.0 support, this patch introduces
psci_power_state_loses_context and psci_power_state_is_valid functions
to query information from a PSCI power state, which is no longer
decomposed (and hence the pack/unpack functions are removed). As it is
no longer decomposed, it is now passed round as an opaque u32 token.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Acked-by: Lorenzo Pieralisi <[email protected]>
Tested-by: Hanjun Guo <[email protected]>
Cc: Will Deacon <[email protected]>
Software resident in the secure world (a "Trusted OS") may cause CPU_OFF
calls for the CPU it is resident on to be denied. Such a denial would be
fatal for the kernel, and so we must detect when this can happen before
the point of no return.
This patch implements Trusted OS detection for PSCI 0.2+ systems, using
MIGRATE_INFO_TYPE and MIGRATE_INFO_UP_CPU. When a trusted OS is detected
as resident on a particular CPU, attempts to hot unplug that CPU will be
denied early, before they can prove fatal.
Trusted OS migration is not implemented by this patch. Implementation of
migratable UP trusted OSs seems unlikely, and the right policy for
migration is unclear (and will likely differ across implementations). As
such, it is likely that migration will require cooperation with Trusted
OS drivers.
PSCI implementations prior to 0.1 do not provide the facility to detect
the presence of a Trusted OS, nor the CPU any such OS is resident on, so
without additional information it is not possible to handle Trusted OSs
with PSCI 0.1.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Tested-by: Hanjun Guo <[email protected]>
Cc: Lorenzo Pieralisi <[email protected]>
Cc: Will Deacon <[email protected]>
PSCI_VERSION and MIGRATE_INFO_TYPE_UP_CPU return unsigned values, with
the latter returning a 64-bit value. However, the PSCI invocation
functions have prototypes returning int.
This patch upgrades the invocation functions to return unsigned long,
with a new typedef to keep things legible. As PSCI_VERSION cannot return
a negative value, the erroneous check against PSCI_RET_NOT_SUPPORTED is
also removed. The unrelated psci_initcall_t typedef is moved closer to
its first user, to avoid confusion with the invocation functions.
In preparation for sharing the code with ARM, unsigned long is used in
preference of u64. In the SMC32 calling convention, the relevant fields
will be 32 bits wide.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Hanjun Guo <[email protected]>
Tested-by: Hanjun Guo <[email protected]>
Cc: Lorenzo Pieralisi <[email protected]>
Cc: Will Deacon <[email protected]>
PSCI 0.1 did not define canonical IDs for CPU_ON, CPU_OFF, CPU_SUSPEND,
or MIGRATE, and so these need to be provided when using firmware
compliant to PSCI 0.1.
However, functions introduced in 0.2 or later have canonical IDs, and
these cannot be provided via DT. There's no need to indirect the IDs via
a table; they can be used directly at callsites (and already are for
SYSTEM_OFF and SYSTEM_RESET).
This patch removes the unnecessary function ID indirection for
AFFINITY_INFO and MIGRATE_INFO_TYPE.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Acked-by: Lorenzo Pieralisi <[email protected]>
Reviewed-by: Hanjun Guo <[email protected]>
Tested-by: Hanjun Guo <[email protected]>
Cc: Will Deacon <[email protected]>
cpu_kill currently returns one for success and zero for failure, which
is unlike all the other cpu_operations, which return zero for success
and an error code upon failure. This difference is unnecessarily
confusing.
Make cpu_kill consistent with the other cpu_operations.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Acked-by: Lorenzo Pieralisi <[email protected]>
Reviewed-by: Hanjun Guo <[email protected]>
Tested-by: Hanjun Guo <[email protected]>
Cc: Will Deacon <[email protected]>
The PSCI MIGRATE_INFO_UP_CPU call returns a physical ID, which we will
need to map back to a Linux logical ID.
Implement a reusable get_logical_index to map from a physical ID to a
logical ID.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Acked-by: Lorenzo Pieralisi <[email protected]>
Reviewed-by: Hanjun Guo <[email protected]>
Tested-by: Hanjun Guo <[email protected]>
Cc: Will Deacon <[email protected]>
We make use of the PSCI function IDs, but don't explicitly include the
header which defines them. Relying on transitive header includes is
fragile and will be broken as headers are refactored.
This patch includes the relevant header file directly so as to avoid
future breakage.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Christoffer Dall <[email protected]>
Reviewed-by: Hanjun Guo <[email protected]>
Tested-by: Hanjun Guo <[email protected]>
Cc: Marc Zyngier <[email protected]>
The documented semantics of flush_cache_all are not possible to provide
for arm64 (short of flushing the entire physical address space by VA),
and there are currently no users; KVM uses VA maintenance exclusively,
cpu_reset is never called, and the only two users outside of arch code
cannot be built for arm64.
While cpu_soft_reset and related functions (which call flush_cache_all)
were thought to be useful for kexec, their current implementations only
serve to mask bugs. For correctness kexec will need to perform
maintenance by VA anyway to account for system caches, line migration,
and other subtleties of the cache architecture. As the extent of this
cache maintenance will be kexec-specific, it should probably live in the
kexec code.
This patch removes flush_cache_all, and related unused components,
preventing further abuse.
Signed-off-by: Mark Rutland <[email protected]>
Cc: AKASHI Takahiro <[email protected]>
Cc: Geoff Levand <[email protected]>
Acked-by: Ard Biesheuvel <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Acked-by: Lorenzo Pieralisi <[email protected]>
Acked-by: Marc Zyngier <[email protected]>
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
At boot time we round the memblock limit down to section size in an
attempt to ensure that we will have mapped this RAM with section
mappings prior to allocating from it. When mapping RAM we iterate over
PMD-sized chunks, creating these section mappings.
Section mappings are only created when the end of a chunk is aligned to
section size. Unfortunately, with classic page tables (where PMD_SIZE is
2 * SECTION_SIZE) this means that if a chunk is between 1M and 2M in
size the first 1M will not be mapped despite having been accounted for
in the memblock limit. This has been observed to result in page tables
being allocated from unmapped memory, causing boot-time hangs.
This patch modifies the memblock limit rounding to always round down to
PMD_SIZE instead of SECTION_SIZE. For classic MMU this means that we
will round the memblock limit down to a 2M boundary, matching the limits
on section mappings, and preventing allocations from unmapped memory.
For LPAE there should be no change as PMD_SIZE == SECTION_SIZE.
Signed-off-by: Mark Rutland <[email protected]>
Reported-by: Stefan Agner <[email protected]>
Tested-by: Stefan Agner <[email protected]>
Acked-by: Laura Abbott <[email protected]>
Tested-by: Hans de Goede <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Steve Capper <[email protected]>
Cc: [email protected]
Signed-off-by: Russell King <[email protected]>
While skeleton.dtsi was initially conceived as a simple way to bootstrap
writing a dts, it has proven to be problematic:
* The #address-cells and #size-cells values used in skeleton.dtsi may
not match what a user wants (e.g. when they need to describe a range
larger than 4GB).
* For dts files where memory nodes have unit-addresses, it adds a
redundant /memory node, for which the reg entry may not be
appropriately sized (e.g. where #size-cells has been overridden).
* For dts files which assume that a bootloader will fill in the memory
node(s), no node is present in the dts (and hence there is no attached
comment), making it hard to distinguish these cases from bad dts
files, and masking any warnings dtc may produce w.r.t. missing nodes.
* The default empty /chosen and /aliases are somewhat useless, and it
would be preferable for dts to fill these in (e.g. for
/aliases/serial0 and /chosen/stdout-path).
This patch removes skeleton.dtsi from arm64. There are currently no
users, so we can remove it before any appear.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Rob Herring <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Olof Johansson <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Arnd Bergmann <[email protected]>
The ARM Generic Timer (AKA the architected timer, arm_arch_timer)
features a CPU register (CNTFRQ) which firmware is intended to
initialize, and non-secure software can read to determine the frequency
of the timer. On CPUs with secure state, this register cannot be written
from non-secure states.
The firmware of early SoCs featuring the timer did not correctly
initialize CNTFRQ correctly on all CPUs, requiring the frequency to be
described in DT as a workaround. This workaround is not complete however
as it is exposed to all software in a privileged non-secure mode
(including guests running under a hypervisor). The firmware and DTs for
recent SoCs have followed the example set by these early SoCs.
This patch updates the arch timer binding documentation to make it
clearer that the use of the clock-frequency property is a poor
work-around. The MMIO generic timer binding is similarly updated, though
this is less of a concern as there is generally no need to expose the
MMIO timers to guest OSs.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Acked-by: Marc Zyngier <[email protected]>
Acked-by: Olof Johansson <[email protected]>
Acked-by: Stephen Boyd <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Rob Herring <[email protected]>
arm64 builds with GCC 5 have caused the __asmeq assertions in the PSCI
calling code to fire, so move the ARM PSCI calls out of line into their
own assembly file for consistency and to safeguard against the same
issue occuring with the 32-bit toolchain.
[will: brought into line with arm64 implementation]
Reported-by: Andy Whitcroft <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Signed-off-by: Russell King <[email protected]>
Commit 9aabf810a6 ("mm/slub: optimize alloc/free fastpath by removing
preemption on/off") introduced an occasional hang for kernels built with
CONFIG_PREEMPT && !CONFIG_SMP.
The problem is the following loop the patch introduced to
slab_alloc_node and slab_free:
do {
tid = this_cpu_read(s->cpu_slab->tid);
c = raw_cpu_ptr(s->cpu_slab);
} while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid));
GCC 4.9 has been observed to hoist the load of c and c->tid above the
loop for !SMP kernels (as in this case raw_cpu_ptr(x) is compile-time
constant and does not force a reload). On arm64 the generated assembly
looks like:
ldr x4, [x0,#8]
loop:
ldr x1, [x0,#8]
cmp x1, x4
b.ne loop
If the thread is preempted between the load of c->tid (into x1) and tid
(into x4), and an allocation or free occurs in another thread (bumping
the cpu_slab's tid), the thread will be stuck in the loop until
s->cpu_slab->tid wraps, which may be forever in the absence of
allocations/frees on the same CPU.
This patch changes the loop condition to access c->tid with READ_ONCE.
This ensures that the value is reloaded even when the compiler would
otherwise assume it could cache the value, and also ensures that the
load will not be torn.
Signed-off-by: Mark Rutland <[email protected]>
Cc: Catalin Marinas <[email protected]>
Acked-by: Christoph Lameter <[email protected]>
Cc: David Rientjes <[email protected]>
Cc: Jesper Dangaard Brouer <[email protected]>
Cc: Joonsoo Kim <[email protected]>
Cc: Pekka Enberg <[email protected]>
Cc: Steve Capper <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
We write idmap_t0sz with SCTLR_EL1.{C,M} clear, but we only have the
guarnatee that the kernel Image is clean, not invalid in the caches, and
therefore we might read a stale value once the MMU is enabled.
This patch ensures we invalidate the corresponding cacheline after the
write as we do for all other data written before we set SCTLR_EL1.{C.M},
guaranteeing that the value will be visible later. We rely on the DSBs
in __create_page_tables to complete the maintenance.
Signed-off-by: Mark Rutland <[email protected]>
CC: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
After writing the page tables, we use __inval_cache_range to invalidate
any stale cache entries. Strongly Ordered memory accesses are not
ordered w.r.t. cache maintenance instructions, and hence explicit memory
barriers are required to provide this ordering. However,
__inval_cache_range was written to be used on Normal Cacheable memory
once the MMU and caches are on, and does not have any barriers prior to
the DC instructions.
This patch adds a DMB between the page tables being written and the
corresponding cachelines being invalidated, ensuring that the
invalidation makes the new data visible to subsequent cacheable
accesses. A barrier is not required before the prior invalidate as we do
not access the page table memory area prior to this, and earlier
barriers in preserve_boot_args and set_cpu_boot_mode_flag ensures
ordering w.r.t. any stores performed prior to entering Linux.
Signed-off-by: Mark Rutland <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Fixes: c218bca74e ("arm64: Relax the kernel cache requirements for boot")
Signed-off-by: Will Deacon <[email protected]>
Fixmap indices are in the interval (FIX_HOLE, __end_of_fixed_addresses),
but in __set_fixmap we only check idx <= __end_of_fixed_addresses, and
therefore indices <= FIX_HOLE are erroneously accepted. If called with
such an idx, __set_fixmap may corrupt page tables outside of the fixmap
region.
This patch ensures that we validate the idx against both endpoints of
the interval.
Cc: Catalin Marinas <[email protected]>
Cc: Kees Cook <[email protected]>
Acked-by: Ard Biesheuvel <[email protected]>
Acked-by: Laura Abbott <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
The FIX_TEXT_POKE0 is currently at the end of the temporary fixmap
slots, despite the fact that it can be used at any point during runtime
(e.g. for poking the text of loaded modules), and thus should be a
permanent fixmap slot (as is the case on arm and x86).
This patch moves FIX_TEXT_POKE0 into the set of permanent fixmap slots.
Cc: Catalin Marinas <[email protected]>
Cc: Kees Cook <[email protected]>
Acked-by: Ard Biesheuvel <[email protected]>
Acked-by: Laura Abbott <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
We currently don't log the boot mode for arm64 as we do for arm, and
without KVM the user is provided with no indication as to which mode(s)
CPUs were booted in, which can seriously hinder debugging in some cases.
Add logging to the boot path once all CPUs are up. Where CPUs are
mismatched in violation of the boot protocol, WARN and set a taint (as
we do for CPU other CPU feature mismatches) given that the
firmware/bootloader is buggy and should be fixed.
Cc: Catalin Marinas <[email protected]>
Cc: Marc Zyngier <[email protected]>
Tested-by: Ard Biesheuvel <[email protected]>
Reviewed-by: Ard Biesheuvel <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Commit 828e9834e9 ("arm64: head: create a new function for setting
the boot_cpu_mode flag") added BOOT_CPU_MODE_EL1, a nonzero value
replacing uses of zero. However it failed to update __boot_cpu_mode
appropriately.
A CPU booted at EL2 writes BOOT_CPU_MODE_EL2 to __boot_cpu_mode[0], and
a CPU booted at EL1 writes BOOT_CPU_MODE_EL1 to __boot_cpu_mode[1].
Later is_hyp_mode_mismatched() determines there to be a mismatch if
__boot_cpu_mode[0] != __boot_cpu_mode[1].
If all CPUs are booted at EL1, __boot_cpu_mode[0] will be set to
BOOT_CPU_MODE_EL1, but __boot_cpu_mode[1] will retain its initial value
of zero, and is_hyp_mode_mismatched will erroneously determine that the
boot modes are mismatched. This hasn't been a problem so far, but later
patches which will make use of is_hyp_mode_mismatched() expect it to
work correctly.
This patch initialises __boot_cpu_mode[1] to BOOT_CPU_MODE_EL1, fixing
the erroneous mismatch detection when all CPUs are booted at EL1.
Cc: Catalin Marinas <[email protected]>
Cc: Marc Zyngier <[email protected]>
Tested-by: Ard Biesheuvel <[email protected]>
Reviewed-by: Ard Biesheuvel <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Currently we only perform alternative patching for kernels built with
CONFIG_SMP, as we call apply_alternatives_all() in smp.c, which is only
built for CONFIG_SMP. Thus !SMP kernels may not have necessary
alternatives patched in.
This patch ensures that we call apply_alternatives_all() once all CPUs
are booted, even for !SMP kernels, by having the smp_init_cpus() stub
call this for !SMP kernels via up_late_init. A new wrapper,
do_post_cpus_up_work, is added so we can hook other calls here later
(e.g. boot mode logging).
Cc: Andre Przywara <[email protected]>
Cc: Catalin Marinas <[email protected]>
Fixes: e039ee4ee3 ("arm64: add alternative runtime patching")
Tested-by: Ard Biesheuvel <[email protected]>
Reviewed-by: Ard Biesheuvel <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Several dts only list "arm,cortex-a7-gic" or "arm,gic-400" in their GIC
compatible list, and while this is correct (and supported by the GIC
driver), KVM will fail to detect that it can support these cases.
This patch adds the missing strings to the VGIC code. The of_device_id
entries are padded to keep the probe function data aligned.
Signed-off-by: Mark Rutland <[email protected]>
Cc: Andre Przywara <[email protected]>
Cc: Christoffer Dall <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: Michal Simek <[email protected]>
Acked-by: Marc Zyngier <[email protected]>
Signed-off-by: Christoffer Dall <[email protected]>
With certain restrictions it is possible for a wakeup device to share
an IRQ with an IRQF_NO_SUSPEND user, and the warnings introduced by
commit cab303be91 are spurious. The new
IRQF_COND_SUSPEND flag allows drivers to tell the core when these
restrictions are met, allowing spurious warnings to be silenced.
This patch documents how IRQF_COND_SUSPEND is expected to be used,
updating some of the text now made invalid by its addition.
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Rafael J. Wysocki <[email protected]>
The IRQF_NO_SUSPEND flag is intended to be used for interrupts required
to be enabled during the suspend-resume cycle. This mostly consists of
IPIs and timer interrupts, potentially including chained irqchip
interrupts if these are necessary to handle timers or IPIs. If an
interrupt does not fall into one of the aforementioned categories,
requesting it with IRQF_NO_SUSPEND is likely incorrect.
Using IRQF_NO_SUSPEND does not guarantee that the interrupt can wake the
system from a suspended state. For an interrupt to be able to trigger a
wakeup, it may be necessary to program various components of the system.
In these cases it is necessary to use {enable,disabled}_irq_wake.
Unfortunately, several drivers assume that IRQF_NO_SUSPEND ensures that
an IRQ can wake up the system, and the documentation can be read
ambiguously w.r.t. this property.
This patch updates the documentation regarding IRQF_NO_SUSPEND to make
this caveat explicit, hopefully making future misuse rarer. Cleanup of
existing misuse will occur as part of later patch series.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Peter Zijlstra (Intel) <[email protected]>
Signed-off-by: Rafael J. Wysocki <[email protected]>
Now that we can specify which PMU variant we're likely to deal with, do
so in the omap board code. This will allow us to split the ARMv6, ARMv7,
and XScale PMU drivers.
The unnecessary include of asm/pmu.h is also removed.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Tony Lindgren <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Olof Johansson <[email protected]>
Signed-off-by: Olof Johansson <[email protected]>
Now that we can specify which PMU variant we're likely to deal with, do
so in the iop board code. This will allow us to split the ARMv6, ARMv7,
and XScale PMU drivers.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Linus Walleij <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Olof Johansson <[email protected]>
Cc: Russell King <[email protected]>
Signed-off-by: Olof Johansson <[email protected]>
Now that we can specify which PMU variant we're likely to deal with, do
so in the realview board code. This will allow us to split the ARMv6,
ARMv7, and XScale PMU drivers.
The Realview EB may be used with ARMv6 or ARMv7 CPUs, but luckily
there's only a single ARMv7 CPU, so we can match that explicitly to
determine whether or not we have an ARMv7 PMU.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Linus Walleij <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Olof Johansson <[email protected]>
Cc: Russell King <[email protected]>
Signed-off-by: Olof Johansson <[email protected]>
Currently the adjusments made as part of perf_event_task_tick() use the
percpu rotation lists to iterate over any active PMU contexts, but these
are not used by the context rotation code, having been replaced by
separate (per-context) hrtimer callbacks. However, some manipulation of
the rotation lists (i.e. removal of contexts) has remained in
perf_rotate_context(). This leads to the following issues:
* Contexts are not always removed from the rotation lists. Removal of
PMUs which have been placed in rotation lists, but have not been
removed by a hrtimer callback can result in corruption of the rotation
lists (when memory backing the context is freed).
This has been observed to result in hangs when PMU drivers built as
modules are inserted and removed around the creation of events for
said PMUs.
* Contexts which do not require rotation may be removed from the
rotation lists as a result of a hrtimer, and will not be considered by
the unthrottling code in perf_event_task_tick.
This patch fixes the issue by updating the rotation ist when events are
scheduled in/out, ensuring that each rotation list stays in sync with
the HW state. As each event holds a refcount on the module of its PMU,
this ensures that when a PMU module is unloaded none of its CPU contexts
can be in a rotation list. By maintaining a list of perf_event_contexts
rather than perf_event_cpu_contexts, we don't need separate paths to
handle the cpu and task contexts, which also makes the code a little
simpler.
As the rotation_list variables are not used for rotation, these are
renamed to active_ctx_list, which better matches their current function.
perf_pmu_rotate_{start,stop} are renamed to
perf_pmu_ctx_{activate,deactivate}.
Reported-by: Johannes Jensen <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Fengguang Wu <[email protected]>
Cc: Linus Torvalds <[email protected]>
Link: http://lkml.kernel.org/r/20150129134511.GR17721@leverpostej
Signed-off-by: Ingo Molnar <[email protected]>
When initialising an event, perf_init_event will call try_module_get() to
ensure that the PMU's module cannot be removed for the lifetime of the
event, with __free_event() dropping the reference when the event is
finally destroyed. If something fails after the event has been
initialised, but before the event is installed, perf_event_alloc will
drop the reference on the module.
However, if we fail to initialise an event for some reason (e.g. we ask
an uncore PMU to perform sampling, and it refuses to initialise the
event), we do not drop the refcount. If we try to open such a bogus
event without a precise IDR type, we will loop over each PMU in the pmus
list, incrementing each of their refcounts without decrementing them.
This patch adds a module_put when pmu->event_init(event) fails, ensuring
that the refcounts are balanced in failure cases. As the innards of the
precise and search based initialisation look very similar, this logic is
hoisted out into a new helper function. While the early return for the
failed try_module_get is removed from the search case, this is handled
by the remaining return when ret is not -ENOENT.
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Linus Torvalds <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>
The {pgd,pud,pmd}_bad family of macros have slightly fuzzy
cross-architecture semantics, and seem to imply a populated entry that
is not a next-level table, rather than a particular type of entry (e.g.
a section map).
In arm64 code, for those cases where we care about whether an entry is a
section mapping, we can instead use the {pud,pmd}_sect macros to
explicitly check for this case. This helps to document precisely what we
care about, making the code easier to read, and allows for future
relaxation of the *_bad macros to check for other "bad" entries.
To that end this patch updates the table dumping and initial table setup
to check for section mappings with {pud,pmd}_sect, and adds/restores
BUG_ON(*_bad((*p)) checks after we've handled the *_sect and *_none
cases so as to catch remaining "bad" cases.
In the fault handling code, show_pte is left with *_bad checks as it
only cares about whether it can walk the next level table, and this path
is used for both kernel and userspace fault handling. The former case
will be followed by a die() where we'll report the address that
triggered the fault, which can be useful context for debugging.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Steve Capper <[email protected]>
Cc: Ard Biesheuvel <[email protected]>
Cc: Kees Cook <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
In paging_init, we call flush_cache_all, but this is backed by Set/Way
operations which may not achieve anything in the presence of cache line
migration and/or system caches. If the caches are already in an
inconsistent state at this point, there is nothing we can do (short of
flushing the entire physical address space by VA) to empty architected
and system caches. As such, flush_cache_all only serves to mask other
potential bugs. Hence, this patch removes the boot-time call to
flush_cache_all.
Immediately after the cache maintenance we flush the TLBs, but this is
also unnecessary. Before enabling the MMU, the TLBs are invalidated, and
thus are initially clean. When changing the contents of active tables
(e.g. in fixup_executable() for DEBUG_RODATA) we perform the required
TLB maintenance following the update, and therefore no additional
maintenance is required to ensure the new table entries are in effect.
Since activating the MMU we will not have modified system register
fields permitted to be cached in a TLB, and therefore do not need
maintenance for any cached system register fields. Hence, the TLB flush
is unnecessary.
Shortly after the unnecessary TLB flush, we update TTBR0 to point to an
empty zero page rather than the idmap, and flush the TLBs. This
maintenance is necessary to remove the global idmap entries from the
TLBs (as they would conflict with userspace mappings), and is retained.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Marc Zyngier <[email protected]>
Acked-by: Steve Capper <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
As with x86, mark the sys_call_table const such that it will be placed
in the .rodata section. This will cause attempts to modify the table
(accidental or deliberate) to fail when strict page permissions are in
place. In the absence of strict page permissions, there should be no
functional change.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
The arm64 dump code is currently relying on some definitions which are
pulled in via transitive dependencies. It seems we have implicit
dependencies on the following definitions:
* MODULES_VADDR (asm/memory.h)
* MODULES_END (asm/memory.h)
* PAGE_OFFSET (asm/memory.h)
* PTE_* (asm/pgtable-hwdef.h)
* ENOMEM (linux/errno.h)
* device_initcall (linux/init.h)
This patch ensures we explicitly include the relevant headers for the
above items, fixing the observed build issue and hopefully preventing
future issues as headers are refactored.
Signed-off-by: Mark Rutland <[email protected]>
Reported-by: Mark Brown <[email protected]>
Acked-by: Steve Capper <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
PCI IO space was intended to be 16MiB, at 32MiB below MODULES_VADDR, but
commit d1e6dc91b5 ("arm64: Add architectural support for PCI")
extended this to cover the full 32MiB. The final 8KiB of this 32MiB is
also allocated for the fixmap, allowing for potential clashes between
the two.
This change was masked by assumptions in mem_init and the page table
dumping code, which assumed the I/O space to be 16MiB long through
seaparte hard-coded definitions.
This patch changes the definition of the PCI I/O space allocation to
live in asm/memory.h, along with the other VA space allocations. As the
fixmap allocation depends on the number of fixmap entries, this is moved
below the PCI I/O space allocation. Both the fixmap and PCI I/O space
are guarded with 2MB of padding. Sites assuming the I/O space was 16MiB
are moved over use new PCI_IO_{START,END} definitions, which will keep
in sync with the size of the IO space (now restored to 16MiB).
As a useful side effect, the use of the new PCI_IO_{START,END}
definitions prevents a build issue in the dumping code due to a (now
redundant) missing include of io.h for PCI_IOBASE.
Signed-off-by: Mark Rutland <[email protected]>
Cc: Kees Cook <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Liviu Dudau <[email protected]>
Cc: Steve Capper <[email protected]>
Cc: Will Deacon <[email protected]>
[[email protected]: reorder FIXADDR and PCI_IO address_markers_idx enum]
Signed-off-by: Catalin Marinas <[email protected]>
When booting with EFI, we acquire the EFI memory map after parsing the
early params. This unfortuantely renders the option useless as we call
memblock_enforce_memory_limit (which uses memblock_remove_range behind
the scenes) before we've added any memblocks. We end up removing
nothing, then adding all of memory later when efi_init calls
reserve_regions.
Instead, we can log the limit and apply this later when we do the rest
of the memblock work in memblock_init, which should work regardless of
the presence of EFI. At the same time we may as well move the early
parameter into arm64's mm/init.c, close to arm64_memblock_init.
Any memory which must be mapped (e.g. for use by EFI runtime services)
must be mapped explicitly reather than relying on the linear mapping,
which may be truncated as a result of a mem= option passed on the kernel
command line.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Acked-by: Ard Biesheuvel <[email protected]>
Tested-by: Ard Biesheuvel <[email protected]>
Cc: Leif Lindholm <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
To aid the developer when something triggers an unexpected exception,
decode the ESR_ELx.EC field when logging an ESR_ELx value using the
newly introduced esr_get_class_string. This doesn't tell the developer
the specifics of the exception encoded in the remaining IL and ISS bits,
but it can be helpful to distinguish between exception classes (e.g.
SError and a data abort) without having to manually decode the field,
which can be tiresome.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Christoffer Dall <[email protected]>
Reviewed-by: Christoffer Dall <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: Peter Maydell <[email protected]>
Cc: Will Deacon <[email protected]>
Now that all users have been moved over to the common ESR_ELx_* macros,
remove the redundant ESR_EL2 macros. To maintain compatibility with the
fault handling code shared with 32-bit, the FSC_{FAULT,PERM} macros are
retained as aliases for the common ESR_ELx_FSC_{FAULT,PERM} definitions.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Christoffer Dall <[email protected]>
Reviewed-by: Christoffer Dall <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: Peter Maydell <[email protected]>
Cc: Will Deacon <[email protected]>
Now that all users have been moved over to the common ESR_ELx_* macros,
remove the redundant ESR_EL1 macros.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Christoffer Dall <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: Peter Maydell <[email protected]>
Cc: Will Deacon <[email protected]>
Now that we have common ESR_ELx macros, make use of them in the arm64
KVM code. The addition of <asm/esr.h> to the include path highlighted
badly ordered (i.e. not alphabetical) include lists; these are changed
to alphabetical order.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Christoffer Dall <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: Peter Maydell <[email protected]>
Cc: Will Deacon <[email protected]>
To aid the developer when something triggers an unexpected exception,
decode the ESR_ELx.EC field when logging an ESR_ELx value. This doesn't
tell the developer the specifics of the exception encoded in the
remaining IL and ISS bits, but it can be helpful to distinguish between
exception classes (e.g. SError and a data abort) without having to
manually decode the field, which can be tiresome.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Christoffer Dall <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: Peter Maydell <[email protected]>
Cc: Will Deacon <[email protected]>
Currently we have separate ESR_EL{1,2}_* macros, despite the fact that
the encodings are common. While encodings are architected to refer to
the current EL or a lower EL, the macros refer to particular ELs (e.g.
ESR_ELx_EC_DABT_EL0). Having these duplicate definitions is redundant,
and their naming is misleading.
This patch introduces common ESR_ELx_* macros that can be used in all
cases, in preparation for later patches which will migrate existing
users over. Some additional cleanups are made in the process:
* Suffixes for particular exception levelts (e.g. _EL0, _EL1) are
replaced with more general _LOW and _CUR suffixes, matching the
architectural intent.
* ESR_ELx_EC_WFx, rather than ESR_ELx_EC_WFI is introduced, as this
EC encoding covers traps from both WFE and WFI. Similarly,
ESR_ELx_WFx_ISS_WFE rather than ESR_ELx_EC_WFI_ISS_WFE is introduced.
* Multi-bit fields are given consistently named _SHIFT and _MASK macros.
* UL() is used for compatiblity with assembly files.
* Comments are added for currently unallocated ESR_ELx.EC encodings.
For fields other than ESR_ELx.EC, macros are only implemented for fields
for which there is already an ESR_EL{1,2}_* macro.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Christoffer Dall <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: Peter Maydell <[email protected]>
Cc: Will Deacon <[email protected]>
The cachepolicy kernel parameter was intended to aid in the debugging of
coherency issues, but it is fundamentally broken for several reasons:
* On SMP platforms, only the boot CPU's tcr_el1 is altered. Secondary
CPUs may therefore use differ w.r.t. the attributes they apply to
MT_NORMAL memory, resulting in a loss of coherency.
* The cache maintenance using flush_dcache_all (based on Set/Way
operations) is not guaranteed to empty a given CPU's cache hierarchy
while said CPU has caches enabled, it cannot empty the caches of
other coherent PEs, nor is it guaranteed to flush data to the PoC
even when caches are disabled.
* The TLBs are not invalidated around the modification of MAIR_EL1 and
TCR_EL1, as required by the architecture (as both are permitted to be
cached in a TLB). This may result in CPUs using attributes other than
those expected for some memory accesses, resulting in a loss of
coherency.
* Exclusive accesses are not architecturally guaranteed to function as
expected on memory marked as Write-Through or Non-Cacheable. Thus
changing the attributes of MT_NORMAL away from the (architecurally
safe) defaults may cause uses of these instructions (e.g. atomics) to
behave erratically.
Given this, the cachepolicy code cannot be used for debugging purposes
as it alone is likely to cause coherency issues. This patch removes the
broken cachepolicy code.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
Currently the arm page table dumping code starts dumping page tables
from USER_PGTABLES_CEILING. This is unnecessary for skipping any entries
related to userspace as the swapper_pg_dir does not contain such
entries, and results in a couple of unfortuante side effects.
Firstly, any kernel mappings which might exist below
USER_PGTABLES_CEILING will not be accounted in the dump output. This
masks any entries erroneously created below this address.
Secondly, if the final page table entry walked is part of a valid
mapping the page table dumping code will not log the region this entry
is part of, as the final note_page call in walk_pgd will trigger an
early return when 0 < USER_PGTABLES_CEILING. Luckily this isn't seen on
contemporary systems as they typically don't have enough RAM to extend
the linear mapping right to the end of the address space.
Due to the way addr is constructed in the walk_* functions, it can never
be less than USER_PGTABLES_CEILING when walking the page tables, so it
is not necessary to avoid dereferencing invalid table addresses. The
existing checks for st->current_prot and st->marker[1].start_address are
sufficient to ensure we will not print and/or dereference garbage when
trying to log information.
This patch removes both problematic uses of USER_PGTABLES_CEILING from
the arm page table dumping code, preventing both of these issues. We
will now report any low mappings, and the final note_page call will not
return early, ensuring all regions are logged.
Signed-off-by: Mark Rutland <[email protected]>
Cc: Steve Capper <[email protected]>
Cc: Kees Cook <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Russell King <[email protected]>
We don't currently check a number of registers exposed to AArch32 guests
(MVFR{0,1,2}_EL1 and ID_DFR0_EL1), despite the fact these describe
AArch32 feature support exposed to userspace and KVM guests similarly to
AArch64 registers which we do check. We do not expect these registers to
vary across a set of CPUs.
This patch adds said registers to the cpuinfo framework and sanity
checks. No sanity check failures have been observed on a current ARMv8
big.LITTLE platform (Juno).
Cc: Catalin Marinas <[email protected]>
Reported-by: Suzuki K. Poulose <[email protected]>
Signed-off-by: Suzuki K. Poulose <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Commit 97b56be103 (arm64: compat: Enable bpf syscall) made the
usual mistake of forgetting to update __NR_compat_syscalls. Due to this,
when el0_sync_compat calls el0_svc_naked, the test against sc_nr
(__NR_compat_syscalls) will fail, and we'll call ni_sys, returning
-ENOSYS to userspace.
This patch bumps __NR_compat_syscalls appropriately, enabling the use of
the bpf syscall from compat tasks.
Due to the reorganisation of unistd{,32}.h as part of commit
f3e5c847ec (arm64: Add __NR_* definitions for compat syscalls) it
is not currently possible to include both headers and sanity-check the
value of __NR_compat_syscalls at build-time to prevent this from
happening again. Additional rework is required to make such niceties a
possibility.
Cc: Will Deacon <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
If the final page table entry we walk is a valid mapping, the page table
dumping code will not log the region this entry is part of, as the final
note_page call in ptdump_show will trigger an early return. Luckily this
isn't seen on contemporary systems as they typically don't have enough
RAM to extend the linear mapping right to the end of the address space.
In note_page, we log a region when we reach its end (i.e. we hit an
entry immediately afterwards which has different prot bits or is
invalid). The final entry has no subsequent entry, so we will not log
this immediately. We try to cater for this with a subsequent call to
note_page in ptdump_show, but this returns early as 0 < LOWEST_ADDR, and
hence we will skip a valid mapping if it spans to the final entry we
note.
Unlike 32-bit ARM, the pgd with the kernel mapping is never shared with
user mappings, so we do not need the check to ensure we don't log user
page tables. Due to the way addr is constructed in the walk_* functions,
it can never be less than LOWEST_ADDR when walking the page tables, so
it is not necessary to avoid dereferencing invalid table addresses. The
existing checks for st->current_prot and st->marker[1].start_address are
sufficient to ensure we will not print and/or dereference garbage when
trying to log information.
This patch removes the unnecessary check against LOWEST_ADDR, ensuring
we log all regions in the kernel page table, including those which span
right to the end of the address space.
Cc: Kees Cook <[email protected]>
Acked-by: Laura Abbott <[email protected]>
Acked-by: Steve Capper <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
When building with 48-bit VAs, it's possible to get the following
warning when building the arm64 page table dumping code:
arch/arm64/mm/dump.c: In function ‘walk_pgd’:
arch/arm64/mm/dump.c:266:2: warning: right shift count >= width of type
pgd_t *pgd = pgd_offset(mm, 0);
^
As pgd_offset is a macro and the second argument is not cast to any
particular type, the zero will be given integer type by the compiler.
As pgd_offset passes the pargument to pgd_index, we then try to shift
the 32-bit integer by at least 39 bits (for 4k pages).
Elsewhere the pgd_offset is passed a second argument of unsigned long
type, so let's do the same here by passing '0UL' rather than '0'.
Cc: Kees Cook <[email protected]>
Acked-by: Laura Abbott <[email protected]>
Acked-by: Steve Capper <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
In the arm64 arch_static_branch implementation we place an A64 NOP into
the instruction stream and log relevant details to a jump_entry in a
__jump_table section. Later this may be replaced with an immediate
branch without link to the code for the unlikely case.
At init time, the core calls arch_jump_label_transform_static to
initialise the NOPs. On x86 this involves inserting the optimal NOP for
a given microarchitecture, but on arm64 we only use the architectural
NOP, and hence replace each NOP with the exact same NOP. This is
somewhat pointless.
Additionally, at module load time we don't call jump_label_apply_nops to
patch the optimal NOPs in, unlike other architectures, but get away with
this because we only use the architectural NOP anyway. A later notifier
will patch NOPs with branches as required.
Similarly to x86 commit 11570da1c5 (x86/jump-label: Do not bother
updating NOPs if they are correct), we can avoid patching NOPs with
identical NOPs. Given that we only use a single NOP encoding, this means
we can NOP-out the body of arch_jump_label_transform_static entirely. As
the default __weak arch_jump_label_transform_static implementation
performs a patch, we must use an empty function to achieve this.
Cc: Catalin Marinas <[email protected]>
Cc: Jiang Liu <[email protected]>
Cc: Laura Abbott <[email protected]>
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
While we currently expect self-hosted debug support to be identical
across CPUs, we don't currently sanity check this.
This patch adds logging of the ID_AA64DFR{0,1}_EL1 values and associated
sanity checking code.
It's not clear to me whether we need to check PMUVer, TraceVer, and
DebugVer, as we don't currently rely on these fields at all.
Signed-off-by: Mark Rutland <[email protected]>
Cc: Catalin Marinas <[email protected]>
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
A missing newline in the WARN_TAINT_ONCE string results in ugly and
somewhat difficult to read output in the case of a sanity check failure,
as the next print does not appear on a new line:
Unsupported CPU feature variation.Modules linked in:
This patch adds the missing newline, fixing the output formatting.
Signed-off-by: Mark Rutland <[email protected]>
Cc: Catalin Marinas <[email protected]>
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
It seems that Cortex-A53 r0p4 added support for AIFSR and ADFSR, and
ID_MMFR0.AuxReg has been updated accordingly to report this fact. As
Cortex-A53 could be paired with CPUs which do not implement these
registers (e.g. all current revisions of Cortex-A57), this may trigger a
sanity check failure at boot.
The AuxReg value describes the availability of the ACTLR, AIFSR, and
ADFSR registers, which are only of use to 32-bit guest OSs, and have
IMPLEMENTATION DEFINED contents. Given the nature of these registers it
is likely that KVM will need to trap accesses regardless of whether the
CPUs are heterogeneous.
This patch masks out the ID_MMFR0.AuxReg value from the sanity checks,
preventing spurious warnings at boot time.
Signed-off-by: Mark Rutland <[email protected]>
Reported-by: Andre Przywara <[email protected]>
Cc: Catalin Marinas <[email protected]>
Acked-by: Will Deacon <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: Peter Maydell <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
We currently allocate different levels of page tables with a variety of
differing flags, and the PGALLOC_GFP flags, intended for use when
allocating any level of page table, are only used for ptes in
pte_alloc_one. On x86, PGALLOC_GFP is used for all page table
allocations.
Currently the major differences are:
* __GFP_NOTRACK -- Needed to ensure page tables are always accessible in
the presence of kmemcheck to prevent recursive faults. Currently
kmemcheck cannot be selected for arm64.
* __GFP_REPEAT -- Causes the allocator to try to reclaim pages and retry
upon a failure to allocate.
* __GFP_ZERO -- Sometimes passed explicitly, sometimes zalloc variants
are used.
While we've no encountered issues so far, it would be preferable to be
consistent. This patch ensures all levels of table are allocated in the
same manner, with PGALLOC_GFP.
Cc: Steve Capper <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
When a CPU hotplugged out, we call perf_remove_from_context() (via
perf_event_exit_cpu()) to rip each CPU-bound event out of its PMU's cpu
context, but leave siblings grouped together. Freeing of these events is
left to the mercy of the usual refcounting.
When a CPU-bound event's refcount drops to zero we cross-call to
__perf_remove_from_context() to clean it up, detaching grouped siblings.
This works when the relevant CPU is online, but will fail if the CPU is
currently offline, and we won't detach the event from its siblings
before freeing the event, leaving the sibling list corrupt. If the
sibling list is later walked (e.g. because the CPU cam online again
before a remaining sibling's refcount drops to zero), we will walk the
now corrupted siblings list, potentially dereferencing garbage values.
Given that the events should never be scheduled again (as we removed
them from their context), we can simply detatch siblings when the CPU
goes down in the first place. If the CPU comes back online, the
redundant call to __perf_remove_from_context() is safe.
Reported-by: Drew Richardson <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Cc: [email protected]
Cc: Vince Weaver <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Linus Torvalds <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>
Commit d7a49086f2 (arm64: cpuinfo: print info for all CPUs)
attempted to clean up /proc/cpuinfo, but due to concerns regarding
further changes was reverted in commit 5e39977edf (Revert "arm64:
cpuinfo: print info for all CPUs").
There are two major issues with the arm64 /proc/cpuinfo format
currently:
* The "Features" line describes (only) the 64-bit hwcaps, which is
problematic for some 32-bit applications which attempt to parse it. As
the same names are used for analogous ISA features (e.g. aes) despite
these generally being architecturally unrelated, it is not possible to
simply append the 64-bit and 32-bit hwcaps in a manner that might not
be misleading to some applications.
Various potential solutions have appeared in vendor kernels. Typically
the format of the Features line varies depending on whether the task
is 32-bit.
* Information is only printed regarding a single CPU. This does not
match the ARM format, and does not provide sufficient information in
big.LITTLE systems where CPUs are heterogeneous. The CPU information
printed is queried from the current CPU's registers, which is racy
w.r.t. cross-cpu migration.
This patch attempts to solve these issues. The following changes are
made:
* When a task with a LINUX32 personality attempts to read /proc/cpuinfo,
the "Features" line contains the decoded 32-bit hwcaps, as with the
arm port. Otherwise, the decoded 64-bit hwcaps are shown. This aligns
with the behaviour of COMPAT_UTS_MACHINE and COMPAT_ELF_PLATFORM. In
the absense of compat support, the Features line is empty.
The set of hwcaps injected into a task's auxval are unaffected.
* Properties are printed per-cpu, as with the ARM port. The per-cpu
information is queried from pre-recorded cpu information (as used by
the sanity checks).
* As with the previous attempt at fixing up /proc/cpuinfo, the hardware
field is removed. The only users so far are 32-bit applications tied
to particular boards, so no portable applications should be affected,
and this should prevent future tying to particular boards.
The following differences remain:
* No model_name is printed, as this cannot be queried from the hardware
and cannot be provided in a stable fashion. Use of the CPU
{implementor,variant,part,revision} fields is sufficient to identify a
CPU and is portable across arm and arm64.
* The following system-wide properties are not provided, as they are not
possible to provide generally. Programs relying on these are already
tied to particular (32-bit only) boards:
- Hardware
- Revision
- Serial
No software has yet been identified for which these remaining
differences are problematic.
Cc: Greg Hackmann <[email protected]>
Cc: Ian Campbell <[email protected]>
Cc: Serban Constantinescu <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Acked-by: Catalin Marinas <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
While efi-entry.S mentions that efi_entry() will have relocated the
kernel image, it actually means that efi_entry will have placed a copy
of the kernel in the appropriate location, and until this is branched to
at the end of efi_entry.S, all instructions are executed from the
original image.
Thus while the flush in efi_entry.S does ensure that the copy is visible
to noncacheable accesses, it does not guarantee that this is true for
the image instructions are being executed from. This could have
disasterous effects when the MMU and caches are disabled if the image
has not been naturally evicted to the PoC.
Additionally, due to a missing dsb following the ic ialluis, the new
kernel image is not necessarily clean in the I-cache when it is branched
to, with similar potentially disasterous effects.
This patch adds additional flushing to ensure that the currently
executing stub text is flushed to the PoC and is thus visible to
noncacheable accesses. As it is placed after the instructions cache
maintenance for the new image and __flush_dcache_area already contains a
dsb, we do not need to add a separate barrier to ensure completion of
the icache maintenance.
Comments are updated to clarify the situation with regard to the two
images and the maintenance required for both.
Fixes: 3c7f255039
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Joel Schopp <[email protected]>
Reviewed-by: Roy Franz <[email protected]>
Tested-by: Tom Lendacky <[email protected]>
Cc: Ard Biesheuvel <[email protected]>
Cc: Ian Campbell <[email protected]>
Cc: Leif Lindholm <[email protected]>
Cc: Mark Salter <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: [email protected]
Signed-off-by: Catalin Marinas <[email protected]>
In certain debugging scenarios it's useful to know the physical ID (i.e.
the MPIDR_EL1.Aff* fields) of the boot CPU, but we don't currently log
this as we do for 32-bit ARM kernels.
This patch makes the kernel log the physical ID of the boot CPU early in
the boot process. The CPU logical map initialisation is folded in to
smp_setup_processor_id (which contrary to its name is also called by UP
kernels). This is called before setup_arch, so should not adversely
affect existing cpu_logical_map users.
Acked-by: Sudeep Holla <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Acked-by: Lorenzo Pieralisis <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
In the absence of a DTB configuration table, the EFI stub will happily
continue attempting to boot a kernel, despite the fact that this kernel
may not function without a description of the hardware. In this case, as
with a typo'd "dtb=" option (e.g. "dbt=") or many other possible
failures, the only output seen by the user will be the rather terse
output from the EFI stub:
EFI stub: Booting Linux Kernel...
To aid those attempting to debug such failures, this patch adds a notice
when no DTB is found, making the output more helpful:
EFI stub: Booting Linux Kernel...
EFI stub: Generating empty DTB
Additionally, a positive acknowledgement is added when a user-specified
DTB is in use:
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from command line
Similarly, a positive acknowledgement is added when a DTB from a
configuration table is in use:
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Leif Lindholm <[email protected]>
Acked-by: Ard Biesheuvel <[email protected]>
Acked-by: Roy Franz <[email protected]>
Acked-by: Matt Fleming <[email protected]>
Signed-off-by: Ard Biesheuvel <[email protected]>
Handling multiple PMUs using a single hotplug notifier requires a list
of PMUs to be maintained, with synchronisation in the probe, remove, and
notify paths. This is error-prone and makes the code much harder to
maintain.
Instead of using a single notifier, we can dynamically allocate a
notifier block per-PMU. The end result is the same, but the list of PMUs
is implicit in the hotplug notifier list rather than within a perf-local
data structure, which makes the code far easier to handle.
Signed-off-by: Mark Rutland <mark.rutland at arm.com>
Reviewed-by: Stephen Boyd <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
To support multiple PMUs, each PMU will need its own accounting data.
As we don't know how (in general) many PMUs we'll have to support at
compile-time, we must allocate the data at runtime dynamically
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Stephen Boyd <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Currently the percpu_pmu pointers used as percpu_irq dev_id values are
defined separately from the other per-cpu accounting data, which make
dynamically allocating the data (as will be required for systems with
heterogeneous CPUs) difficult.
This patch moves the percpu_pmu pointers into pmu_hw_events (which is
itself allocated per cpu), which will allow for easier dynamic
allocation. Both percpu and regular irqs are requested using percpu_pmu
pointers as tokens, freeing us from having to know whether an irq is
percpu within the handler, and thus avoiding a radix tree lookup on the
handler path.
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Reviewed-by: Stephen Boyd <[email protected]>
Tested-by: Stephen Boyd <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Now that the arm pmu code is limited to CPU PMUs the get_hw_events()
function is superfluous, as we'll always have a set of per-cpu
pmu_hw_events structures.
This patch removes the get_hw_events() function, replacing it with
a percpu hw_events pointer. Uses of get_hw_events are updated to use
this_cpu_ptr.
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Reviewed-by: Stephen Boyd <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Commit 3fc2c83087 (ARM: perf: remove event limit from pmu_hw_events) got
rid of the upper limit on the number of events an arm_pmu could handle,
but introduced additional complexity and places a burden on each PMU
driver to allocate accounting data somehow. So far this has not
generally been useful as the only users of arm_pmu are the CPU backend
and the CCI driver.
Now that the CCI driver plugs into the perf subsystem directly, we can
remove some of the complexities that get in the way of supporting
heterogeneous CPU PMUs.
This patch restores the original limits on pmu_hw_events fields such
that the pmu_hw_events data can be allocated as a contiguous block. This
will simplify dynamic pmu_hw_events allocation in later patches.
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Reviewed-by: Stephen Boyd <[email protected]>
Tested-by: Stephen Boyd <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
For systems with heterogeneous CPUs (e.g. big.LITTLE systems) the PMUs
can be different in each cluster, and not all events can be migrated
between clusters. To allow userspace to deal with this, it must be
possible to address each PMU independently.
This patch changes PMUs to be registered with dynamic (IDR) types,
allowing them to be targeted individually. Each PMU's type can be found
in ${SYSFS_ROOT}/bus/event_source/devices/${PMU_NAME}/type.
From userspace, raw events can be targeted at a specific PMU:
$ perf stat -e ${PMU_NAME}/config=V,config1=V1,.../
Doing this does not break existing tools which use existing perf types:
when perf core can't find a PMU of matching type (in perf_init_event)
it'll iterate over the set of all PMUs. If a compatible PMU exists,
it'll be found eventually. If more than one compatible PMU exists, the
event will be handled by whichever PMU happens to be earlier in the pmus
list (which currently will be the last compatible PMU registered).
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
The current PMU probing logic consists of a single switch statement,
which means that the core arm_pmu core in perf_event_cpu.c needs to know
about every CPU PMU variant supported by a driver using the arm_pmu
framework. This makes it rather difficult to decouple the drivers from
the (otherwise generic) probing code.
The patch refactors that switch statement to a table-driven lookup,
separating the logic and knowledge (in the form of the table). Later
patches will split the table across the relevant PMU drivers, which can
pass their tables to the generic probing function.
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Reviewed-by: Stephen Boyd <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Most of the pr_info format strings in perf_event_cpu.c are missing
newlines. Currently we get away with this as the format strings for
subsequent calls to printk (including all pr_* calls) begin with a log
prefix, and the printk core adds the omitted newline for this case.
While generates the output we expect, we probably should not rely on the
format of successive printk calls in order to get legible output.
This patch adds the missing newlines to pr_info format strings in
perf_event_cpu.c, making them consistent with the format strings for
other pr_info, warn, and pr_err calls, and preventing potentially
illegible output if the next printk/pr_* format string doesn't begin
with a log prefix.
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Stephen Boyd <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
The ARM callchain handling code is currently bundled with the ARM PMU
management code, despite the two having no dependency on each other.
This bundling has the unfortunate property of making callchain handling
depend on CONFIG_HW_PERF_EVENTS, even though the callchain handling
could be applied to software events in the absence of PMU hardware
support.
This patch separates the two, placing the callchain handling in
perf_callchain.c and making it depend on CONFIG_PERF_EVENTS rather than
CONFIG_HW_PERF_EVENTS, enabling callchain recording on kernels built
without hardware perf event support.
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
The ARM CPU PMUs and the ARM CCI PMU are using the same framework
despite being substantially different in programming model, which makes
it difficult to handle either particularly well.
This patch migrates the ARM CCI PMU driver away from the arm_pmu
framework, matching the style of the CCN PMU driver and other 'uncore'
PMU drivers. This will enable refactoring of the arm_pmu framework to
better support CPU PMUs. Event context migration on hotplug is not yet
added due to a race on event->ctx in the core perf code.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Punit Agrawal <[email protected]>
Cc: Pawel Moll <[email protected]>
Cc: Will Deacon <[email protected]>
[will: fix whitespace issues]
Signed-off-by: Will Deacon <[email protected]>
Commit ac490f4dca (Documentation: this_cpu_ops.txt: Update description
of this_cpu_ops) added lists of {__,}this_cpu operations, but these have
duplicate, parameter-less entries for {__,}this_cpu_add which don't
correspond to any implementation. No other operations have such
duplicate entries.
Given both are also listed with their full complement of arguments, the
empty forms are redundant and can be removed. This patch performs said
removal.
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Pranith Kumar <[email protected]>
Acked-by: Christoph Lameter <[email protected]>
Signed-off-by: Randy Dunlap <[email protected]>
Signed-off-by: Jiri Kosina <[email protected]>
In certain cases the cpu-release-addr of a CPU may not fall in the
linear mapping (e.g. when the kernel is loaded above this address due to
the presence of other images in memory). This is problematic for the
spin-table code as it assumes that it can trivially convert a
cpu-release-addr to a valid VA in the linear map.
This patch modifies the spin-table code to use a temporary cached
mapping to write to a given cpu-release-addr, enabling us to support
addresses regardless of whether they are covered by the linear mapping.
Acked-by: Leif Lindholm <[email protected]>
Tested-by: Leif Lindholm <[email protected]>
Tested-by: Mark Salter <[email protected]>
Signed-off-by: Mark Rutland <[email protected]>
[ardb: added (__force void *) cast]
Signed-off-by: Ard Biesheuvel <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
On revisions of Cortex-A15 prior to r3p3, a CLREX instruction at PL1 may
falsely trigger a watchpoint exception, leading to potential data aborts
during exception return and/or livelock.
This patch resolves the issue in the following ways:
- Replacing our uses of CLREX with a dummy STREX sequence instead (as
we did for v6 CPUs).
- Removing the clrex code from v7_exit_coherency_flush and derivatives,
since this only exists as a minor performance improvement when
non-cached exclusives are in use (Linux doesn't use these).
Benchmarking on a variety of ARM cores revealed no measurable
performance difference with this change applied, so the change is
performed unconditionally and no new Kconfig entry is added.
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Cc: [email protected]
Signed-off-by: Russell King <[email protected]>
The ARMv6 and ARMv7 early abort handlers clear the exclusive monitors
upon entry to the kernel, but this is redundant:
- We clear the monitors on every exception return since commit
200b812d00 ("Clear the exclusive monitor when returning from an
exception"), so this is not necessary to ensure the monitors are
cleared before returning from a fault handler.
- Any dummy STREX will target a temporary scratch area in memory, and
may succeed or fail without corrupting useful data. Its status value
will not be used.
- Any other STREX in the kernel must be preceded by an LDREX, which
will initialise the monitors consistently and will not depend on the
earlier state of the monitors.
Therefore we have no reason to care about the initial state of the
exclusive monitors when a data abort is taken, and clearing the monitors
prior to exception return (as we already do) is sufficient.
This patch removes the redundant clearing of the exclusive monitors from
the early abort handlers.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Will Deacon <[email protected]>
Cc: [email protected]
Signed-off-by: Russell King <[email protected]>
Due to a missing newline in the I-cache policy detection log output,
it's possible to get some ratehr unfortunate output at boot time:
CPU1: Booted secondary processor
Detected VIPT I-cache on CPU1CPU2: Booted secondary processor
Detected VIPT I-cache on CPU2CPU3: Booted secondary processor
Detected VIPT I-cache on CPU3CPU4: Booted secondary processor
Detected PIPT I-cache on CPU4CPU5: Booted secondary processor
Detected PIPT I-cache on CPU5Brought up 6 CPUs
SMP: Total of 6 processors activated.
This patch adds the missing newline to the format string, cleaning up
the output.
Fixes: 59ccc0d41b ("arm64: cachetype: report weakest cache policy")
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Currently reading /proc/cpuinfo will result in information being read
out of the MIDR_EL1 of the current CPU, and the information is not
associated with any particular logical CPU number.
This is problematic for systems with heterogeneous CPUs (i.e.
big.LITTLE) where MIDR fields will vary across CPUs, and the output will
differ depending on the executing CPU.
This patch reorganises the code responsible for /proc/cpuinfo to print
information per-cpu. In the process, we perform several cleanups:
* Property names are coerced to lower-case (to match "processor" as per
glibc's expectations).
* Property names are simplified and made to match the MIDR field names.
* Revision is changed to hex as with every other field.
* The meaningless Architecture property is removed.
* The ripe-for-abuse Machine field is removed.
The features field (a human-readable representation of the hwcaps)
remains printed once, as this is expected to remain in use as the
globally support CPU features. To enable the possibility of the addition
of per-cpu HW feature information later, this is printed before any
CPU-specific information.
Comments are added to guide userspace developers in the right direction
(using the hwcaps provided in auxval). Hopefully where userspace
applications parse /proc/cpuinfo rather than using the readily available
hwcaps, they limit themselves to reading said first line.
If CPU features differ from each other, the previously installed sanity
checks will give us some advance notice with warnings and
TAINT_CPU_OUT_OF_SPEC. If we are lucky, we will never see such systems.
Rework will be required in many places to support such systems anyway.
Signed-off-by: Mark Rutland <[email protected]>
Cc: Ard Biesheuvel <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Marcus Shawcroft <[email protected]>
Cc: Peter Maydell <[email protected]>
Acked-by: Will Deacon <[email protected]>
[[email protected]: remove machine_name as it is no longer reported]
Signed-off-by: Catalin Marinas <[email protected]>
Unexpected variation in certain system register values across CPUs is an
indicator of potential problems with a system. The kernel expects CPUs
to be mostly identical in terms of supported features, even in systems
with heterogeneous CPUs, with uniform instruction set support being
critical for the correct operation of userspace.
To help detect issues early where hardware violates the expectations of
the kernel, this patch adds simple runtime sanity checks on important ID
registers in the bring up path of each CPU.
Where CPUs are fundamentally mismatched, set TAINT_CPU_OUT_OF_SPEC.
Given that the kernel assumes CPUs are identical feature wise, let's not
pretend that we expect such configurations to work. Supporting such
configurations would require massive rework, and hopefully they will
never exist.
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Reviewed-by: Catalin Marinas <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
In big.LITTLE systems, the I-cache policy may differ across CPUs, and
thus we must always meet the most stringent maintenance requirements of
any I-cache in the system when performing maintenance to ensure
correctness. Unfortunately this requirement is not met as we always look
at the current CPU's cache type register to determine the maintenance
requirements.
This patch causes the I-cache policy of all CPUs to be taken into
account for icache_is_aliasing and icache_is_aivivt. If any I-cache in
the system is aliasing or AIVIVT, the respective function will return
true. At boot each CPU may set flags to identify that at least one
I-cache in the system is aliasing and/or AIVIVT.
The now unused and potentially misleading icache_policy function is
removed.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Will Deacon <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Reviewed-by: Catalin Marinas <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
Several kernel subsystems need to know details about CPU system register
values, sometimes for CPUs other than that they are executing on. Rather
than hard-coding system register accesses and cross-calls for these
cases, this patch adds logic to record various system register values at
boot-time. This may be used for feature reporting, firmware bug
detection, etc.
Separate hooks are added for the boot and hotplug paths to enable
one-time intialisation and cold/warm boot value mismatch detection in
later patches.
Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Reviewed-by: Catalin Marinas <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
The MIDR_EL1 register is composed of a number of bitfields, and uses of
the fields has so far involved open-coding of the shifts and masks
required.
This patch adds shifts and masks for each of the MIDR_EL1 subfields, and
also provides accessors built atop of these. Existing uses within
cputype.h are updated to use these accessors.
The read_cpuid_part_number macro is modified to return the extracted
bitfield rather than returning the value in-place with all other fields
(including revision) masked out, to better match the other accessors.
As the value is only used in comparison with the *_CPU_PART_* macros
which are similarly updated, and these values are never exposed to
userspace, this change should not affect any functionality.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Will Deacon <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Reviewed-by: Catalin Marinas <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
Commit 78d7530ac3 ("ARM: Clean up linker script using new linker script
macros.") modified the arm kernel linker script to use the STABS_DEBUG
macro, but left a .comment section definition. As STABS_DEBUG defines
the .comment section in an identical way, the second section definition
is redundant and can be removed.
This patch removes the redundant .comment section definition.
Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Russell King <[email protected]>
The arm64 Image header contains a text_offset field which bootloaders
are supposed to read to determine the offset (from a 2MB aligned "start
of memory" per booting.txt) at which to load the kernel. The offset is
not well respected by bootloaders at present, and due to the lack of
variation there is little incentive to support it. This is unfortunate
for the sake of future kernels where we may wish to vary the text offset
(even zeroing it).
This patch adds options to arm64 to enable fuzz-testing of text_offset.
CONFIG_ARM64_RANDOMIZE_TEXT_OFFSET forces the text offset to a random
16-byte aligned value value in the range [0..2MB) upon a build of the
kernel. It is recommended that distribution kernels enable randomization
to test bootloaders such that any compliance issues can be fixed early.
Signed-off-by: Mark Rutland <[email protected]>
Acked-by: Tom Rini <[email protected]>
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
Mark Rutland