2018-01-30 01:23:49 +08:00
|
|
|
/*
|
|
|
|
|
* Copyright (c) 2018 Intel corporation
|
|
|
|
|
*
|
|
|
|
|
* SPDX-License-Identifier: Apache-2.0
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include <kernel.h>
|
|
|
|
|
#include <kernel_structs.h>
|
|
|
|
|
#include <spinlock.h>
|
2018-01-18 03:34:50 +08:00
|
|
|
#include <kswap.h>
|
2018-02-09 01:10:46 +08:00
|
|
|
#include <kernel_internal.h>
|
2018-01-30 01:23:49 +08:00
|
|
|
|
kernel: Rework SMP irq_lock() compatibility layer
This was wrong in two ways, one subtle and one awful.
The subtle problem was that the IRQ lock isn't actually globally
recursive, it gets reset when you context switch (i.e. a _Swap()
implicitly releases and reacquires it). So the recursive count I was
keeping needs to be per-thread or else we risk deadlock any time we
swap away from a thread holding the lock.
And because part of my brain apparently knew this, there was an
"optimization" in the code that tested the current count vs. zero
outside the lock, on the argument that if it was non-zero we must
already hold the lock. Which would be true of a per-thread counter,
but NOT a global one: the other CPU may be holding that lock, and this
test will tell you *you* do. The upshot is that a recursive
irq_lock() would almost always SUCCEED INCORRECTLY when there was lock
contention. That this didn't break more things is amazing to me.
The rework is actually simpler than the original, thankfully. Though
there are some further subtleties:
* The lock state implied by irq_lock() allows the lock to be
implicitly released on context switch (i.e. you can _Swap() with the
lock held at a recursion level higher than 1, which needs to allow
other processes to run). So return paths into threads from _Swap()
and interrupt/exception exit need to check and restore the global
lock state, spinning as needed.
* The idle loop design specifies a k_cpu_idle() function that is on
common architectures expected to enable interrupts (for obvious
reasons), but there is no place to put non-arch code to wire it into
the global lock accounting. So on SMP, even CPU0 needs to use the
"dumb" spinning idle loop.
Finally this patch contains a simple bugfix too, found by inspection:
the interrupt return code used when CONFIG_SWITCH is enabled wasn't
correctly setting the active flag on the threads, opening up the
potential for a race that might result in a thread being scheduled on
two CPUs simultaneously.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-04-13 03:50:05 +08:00
|
|
|
static atomic_t global_lock;
|
2019-02-15 13:04:19 +08:00
|
|
|
static atomic_t start_flag;
|
2018-01-30 01:23:49 +08:00
|
|
|
|
2019-03-09 05:19:05 +08:00
|
|
|
unsigned int z_smp_global_lock(void)
|
2018-01-30 01:23:49 +08:00
|
|
|
{
|
2019-11-08 04:43:29 +08:00
|
|
|
unsigned int key = arch_irq_lock();
|
2018-01-30 01:23:49 +08:00
|
|
|
|
kernel: Rework SMP irq_lock() compatibility layer
This was wrong in two ways, one subtle and one awful.
The subtle problem was that the IRQ lock isn't actually globally
recursive, it gets reset when you context switch (i.e. a _Swap()
implicitly releases and reacquires it). So the recursive count I was
keeping needs to be per-thread or else we risk deadlock any time we
swap away from a thread holding the lock.
And because part of my brain apparently knew this, there was an
"optimization" in the code that tested the current count vs. zero
outside the lock, on the argument that if it was non-zero we must
already hold the lock. Which would be true of a per-thread counter,
but NOT a global one: the other CPU may be holding that lock, and this
test will tell you *you* do. The upshot is that a recursive
irq_lock() would almost always SUCCEED INCORRECTLY when there was lock
contention. That this didn't break more things is amazing to me.
The rework is actually simpler than the original, thankfully. Though
there are some further subtleties:
* The lock state implied by irq_lock() allows the lock to be
implicitly released on context switch (i.e. you can _Swap() with the
lock held at a recursion level higher than 1, which needs to allow
other processes to run). So return paths into threads from _Swap()
and interrupt/exception exit need to check and restore the global
lock state, spinning as needed.
* The idle loop design specifies a k_cpu_idle() function that is on
common architectures expected to enable interrupts (for obvious
reasons), but there is no place to put non-arch code to wire it into
the global lock accounting. So on SMP, even CPU0 needs to use the
"dumb" spinning idle loop.
Finally this patch contains a simple bugfix too, found by inspection:
the interrupt return code used when CONFIG_SWITCH is enabled wasn't
correctly setting the active flag on the threads, opening up the
potential for a race that might result in a thread being scheduled on
two CPUs simultaneously.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-04-13 03:50:05 +08:00
|
|
|
if (!_current->base.global_lock_count) {
|
|
|
|
|
while (!atomic_cas(&global_lock, 0, 1)) {
|
|
|
|
|
}
|
2018-01-30 01:23:49 +08:00
|
|
|
}
|
|
|
|
|
|
kernel: Rework SMP irq_lock() compatibility layer
This was wrong in two ways, one subtle and one awful.
The subtle problem was that the IRQ lock isn't actually globally
recursive, it gets reset when you context switch (i.e. a _Swap()
implicitly releases and reacquires it). So the recursive count I was
keeping needs to be per-thread or else we risk deadlock any time we
swap away from a thread holding the lock.
And because part of my brain apparently knew this, there was an
"optimization" in the code that tested the current count vs. zero
outside the lock, on the argument that if it was non-zero we must
already hold the lock. Which would be true of a per-thread counter,
but NOT a global one: the other CPU may be holding that lock, and this
test will tell you *you* do. The upshot is that a recursive
irq_lock() would almost always SUCCEED INCORRECTLY when there was lock
contention. That this didn't break more things is amazing to me.
The rework is actually simpler than the original, thankfully. Though
there are some further subtleties:
* The lock state implied by irq_lock() allows the lock to be
implicitly released on context switch (i.e. you can _Swap() with the
lock held at a recursion level higher than 1, which needs to allow
other processes to run). So return paths into threads from _Swap()
and interrupt/exception exit need to check and restore the global
lock state, spinning as needed.
* The idle loop design specifies a k_cpu_idle() function that is on
common architectures expected to enable interrupts (for obvious
reasons), but there is no place to put non-arch code to wire it into
the global lock accounting. So on SMP, even CPU0 needs to use the
"dumb" spinning idle loop.
Finally this patch contains a simple bugfix too, found by inspection:
the interrupt return code used when CONFIG_SWITCH is enabled wasn't
correctly setting the active flag on the threads, opening up the
potential for a race that might result in a thread being scheduled on
two CPUs simultaneously.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-04-13 03:50:05 +08:00
|
|
|
_current->base.global_lock_count++;
|
2018-01-30 01:23:49 +08:00
|
|
|
|
kernel: Rework SMP irq_lock() compatibility layer
This was wrong in two ways, one subtle and one awful.
The subtle problem was that the IRQ lock isn't actually globally
recursive, it gets reset when you context switch (i.e. a _Swap()
implicitly releases and reacquires it). So the recursive count I was
keeping needs to be per-thread or else we risk deadlock any time we
swap away from a thread holding the lock.
And because part of my brain apparently knew this, there was an
"optimization" in the code that tested the current count vs. zero
outside the lock, on the argument that if it was non-zero we must
already hold the lock. Which would be true of a per-thread counter,
but NOT a global one: the other CPU may be holding that lock, and this
test will tell you *you* do. The upshot is that a recursive
irq_lock() would almost always SUCCEED INCORRECTLY when there was lock
contention. That this didn't break more things is amazing to me.
The rework is actually simpler than the original, thankfully. Though
there are some further subtleties:
* The lock state implied by irq_lock() allows the lock to be
implicitly released on context switch (i.e. you can _Swap() with the
lock held at a recursion level higher than 1, which needs to allow
other processes to run). So return paths into threads from _Swap()
and interrupt/exception exit need to check and restore the global
lock state, spinning as needed.
* The idle loop design specifies a k_cpu_idle() function that is on
common architectures expected to enable interrupts (for obvious
reasons), but there is no place to put non-arch code to wire it into
the global lock accounting. So on SMP, even CPU0 needs to use the
"dumb" spinning idle loop.
Finally this patch contains a simple bugfix too, found by inspection:
the interrupt return code used when CONFIG_SWITCH is enabled wasn't
correctly setting the active flag on the threads, opening up the
potential for a race that might result in a thread being scheduled on
two CPUs simultaneously.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-04-13 03:50:05 +08:00
|
|
|
return key;
|
2018-01-30 01:23:49 +08:00
|
|
|
}
|
|
|
|
|
|
2019-03-09 05:19:05 +08:00
|
|
|
void z_smp_global_unlock(unsigned int key)
|
2018-01-30 01:23:49 +08:00
|
|
|
{
|
kernel: Rework SMP irq_lock() compatibility layer
This was wrong in two ways, one subtle and one awful.
The subtle problem was that the IRQ lock isn't actually globally
recursive, it gets reset when you context switch (i.e. a _Swap()
implicitly releases and reacquires it). So the recursive count I was
keeping needs to be per-thread or else we risk deadlock any time we
swap away from a thread holding the lock.
And because part of my brain apparently knew this, there was an
"optimization" in the code that tested the current count vs. zero
outside the lock, on the argument that if it was non-zero we must
already hold the lock. Which would be true of a per-thread counter,
but NOT a global one: the other CPU may be holding that lock, and this
test will tell you *you* do. The upshot is that a recursive
irq_lock() would almost always SUCCEED INCORRECTLY when there was lock
contention. That this didn't break more things is amazing to me.
The rework is actually simpler than the original, thankfully. Though
there are some further subtleties:
* The lock state implied by irq_lock() allows the lock to be
implicitly released on context switch (i.e. you can _Swap() with the
lock held at a recursion level higher than 1, which needs to allow
other processes to run). So return paths into threads from _Swap()
and interrupt/exception exit need to check and restore the global
lock state, spinning as needed.
* The idle loop design specifies a k_cpu_idle() function that is on
common architectures expected to enable interrupts (for obvious
reasons), but there is no place to put non-arch code to wire it into
the global lock accounting. So on SMP, even CPU0 needs to use the
"dumb" spinning idle loop.
Finally this patch contains a simple bugfix too, found by inspection:
the interrupt return code used when CONFIG_SWITCH is enabled wasn't
correctly setting the active flag on the threads, opening up the
potential for a race that might result in a thread being scheduled on
two CPUs simultaneously.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-04-13 03:50:05 +08:00
|
|
|
if (_current->base.global_lock_count) {
|
|
|
|
|
_current->base.global_lock_count--;
|
|
|
|
|
|
|
|
|
|
if (!_current->base.global_lock_count) {
|
|
|
|
|
atomic_clear(&global_lock);
|
|
|
|
|
}
|
2018-01-30 01:23:49 +08:00
|
|
|
}
|
|
|
|
|
|
2019-11-08 04:43:29 +08:00
|
|
|
arch_irq_unlock(key);
|
kernel: Rework SMP irq_lock() compatibility layer
This was wrong in two ways, one subtle and one awful.
The subtle problem was that the IRQ lock isn't actually globally
recursive, it gets reset when you context switch (i.e. a _Swap()
implicitly releases and reacquires it). So the recursive count I was
keeping needs to be per-thread or else we risk deadlock any time we
swap away from a thread holding the lock.
And because part of my brain apparently knew this, there was an
"optimization" in the code that tested the current count vs. zero
outside the lock, on the argument that if it was non-zero we must
already hold the lock. Which would be true of a per-thread counter,
but NOT a global one: the other CPU may be holding that lock, and this
test will tell you *you* do. The upshot is that a recursive
irq_lock() would almost always SUCCEED INCORRECTLY when there was lock
contention. That this didn't break more things is amazing to me.
The rework is actually simpler than the original, thankfully. Though
there are some further subtleties:
* The lock state implied by irq_lock() allows the lock to be
implicitly released on context switch (i.e. you can _Swap() with the
lock held at a recursion level higher than 1, which needs to allow
other processes to run). So return paths into threads from _Swap()
and interrupt/exception exit need to check and restore the global
lock state, spinning as needed.
* The idle loop design specifies a k_cpu_idle() function that is on
common architectures expected to enable interrupts (for obvious
reasons), but there is no place to put non-arch code to wire it into
the global lock accounting. So on SMP, even CPU0 needs to use the
"dumb" spinning idle loop.
Finally this patch contains a simple bugfix too, found by inspection:
the interrupt return code used when CONFIG_SWITCH is enabled wasn't
correctly setting the active flag on the threads, opening up the
potential for a race that might result in a thread being scheduled on
two CPUs simultaneously.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-04-13 03:50:05 +08:00
|
|
|
}
|
2018-01-30 01:23:49 +08:00
|
|
|
|
2019-03-09 05:19:05 +08:00
|
|
|
/* Called from within z_swap(), so assumes lock already held */
|
|
|
|
|
void z_smp_release_global_lock(struct k_thread *thread)
|
kernel: Rework SMP irq_lock() compatibility layer
This was wrong in two ways, one subtle and one awful.
The subtle problem was that the IRQ lock isn't actually globally
recursive, it gets reset when you context switch (i.e. a _Swap()
implicitly releases and reacquires it). So the recursive count I was
keeping needs to be per-thread or else we risk deadlock any time we
swap away from a thread holding the lock.
And because part of my brain apparently knew this, there was an
"optimization" in the code that tested the current count vs. zero
outside the lock, on the argument that if it was non-zero we must
already hold the lock. Which would be true of a per-thread counter,
but NOT a global one: the other CPU may be holding that lock, and this
test will tell you *you* do. The upshot is that a recursive
irq_lock() would almost always SUCCEED INCORRECTLY when there was lock
contention. That this didn't break more things is amazing to me.
The rework is actually simpler than the original, thankfully. Though
there are some further subtleties:
* The lock state implied by irq_lock() allows the lock to be
implicitly released on context switch (i.e. you can _Swap() with the
lock held at a recursion level higher than 1, which needs to allow
other processes to run). So return paths into threads from _Swap()
and interrupt/exception exit need to check and restore the global
lock state, spinning as needed.
* The idle loop design specifies a k_cpu_idle() function that is on
common architectures expected to enable interrupts (for obvious
reasons), but there is no place to put non-arch code to wire it into
the global lock accounting. So on SMP, even CPU0 needs to use the
"dumb" spinning idle loop.
Finally this patch contains a simple bugfix too, found by inspection:
the interrupt return code used when CONFIG_SWITCH is enabled wasn't
correctly setting the active flag on the threads, opening up the
potential for a race that might result in a thread being scheduled on
two CPUs simultaneously.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-04-13 03:50:05 +08:00
|
|
|
{
|
|
|
|
|
if (!thread->base.global_lock_count) {
|
|
|
|
|
atomic_clear(&global_lock);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2019-02-20 08:03:39 +08:00
|
|
|
#if CONFIG_MP_NUM_CPUS > 1
|
2021-04-01 19:46:57 +08:00
|
|
|
|
|
|
|
|
void z_smp_thread_init(void *arg, struct k_thread *thread)
|
2018-01-18 03:34:50 +08:00
|
|
|
{
|
2020-07-24 05:33:31 +08:00
|
|
|
atomic_t *cpu_start_flag = arg;
|
2018-01-18 03:34:50 +08:00
|
|
|
|
|
|
|
|
/* Wait for the signal to begin scheduling */
|
2020-07-24 05:33:31 +08:00
|
|
|
while (!atomic_get(cpu_start_flag)) {
|
2019-02-15 13:04:19 +08:00
|
|
|
}
|
2018-01-18 03:34:50 +08:00
|
|
|
|
2021-04-01 19:46:57 +08:00
|
|
|
z_dummy_thread_init(thread);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void z_smp_thread_swap(void)
|
|
|
|
|
{
|
|
|
|
|
z_swap_unlocked();
|
|
|
|
|
}
|
|
|
|
|
|
2021-08-18 21:28:11 +08:00
|
|
|
static inline FUNC_NORETURN void smp_init_top(void *arg)
|
2021-04-01 19:46:57 +08:00
|
|
|
{
|
|
|
|
|
struct k_thread dummy_thread;
|
|
|
|
|
|
|
|
|
|
z_smp_thread_init(arg, &dummy_thread);
|
2018-01-27 04:30:21 +08:00
|
|
|
smp_timer_init();
|
2021-04-01 19:46:57 +08:00
|
|
|
|
2019-03-09 05:19:05 +08:00
|
|
|
z_swap_unlocked();
|
2018-01-18 03:34:50 +08:00
|
|
|
|
2021-01-15 17:09:58 +08:00
|
|
|
CODE_UNREACHABLE; /* LCOV_EXCL_LINE */
|
2018-01-18 03:34:50 +08:00
|
|
|
}
|
2021-08-18 21:28:11 +08:00
|
|
|
|
|
|
|
|
void z_smp_start_cpu(int id)
|
|
|
|
|
{
|
|
|
|
|
(void)atomic_clear(&start_flag);
|
|
|
|
|
arch_start_cpu(id, z_interrupt_stacks[id], CONFIG_ISR_STACK_SIZE,
|
|
|
|
|
smp_init_top, &start_flag);
|
|
|
|
|
(void)atomic_set(&start_flag, 1);
|
|
|
|
|
}
|
|
|
|
|
|
2021-04-01 19:46:57 +08:00
|
|
|
#endif
|
2018-01-18 03:34:50 +08:00
|
|
|
|
2019-06-05 23:58:42 +08:00
|
|
|
void z_smp_init(void)
|
2018-01-18 03:34:50 +08:00
|
|
|
{
|
2018-09-14 02:24:30 +08:00
|
|
|
(void)atomic_clear(&start_flag);
|
2018-01-18 03:34:50 +08:00
|
|
|
|
2021-04-01 19:46:57 +08:00
|
|
|
#if CONFIG_MP_NUM_CPUS > 1 && !defined(CONFIG_SMP_BOOT_DELAY)
|
2020-03-13 06:37:29 +08:00
|
|
|
for (int i = 1; i < CONFIG_MP_NUM_CPUS; i++) {
|
|
|
|
|
arch_start_cpu(i, z_interrupt_stacks[i], CONFIG_ISR_STACK_SIZE,
|
|
|
|
|
smp_init_top, &start_flag);
|
|
|
|
|
}
|
2018-01-18 03:34:50 +08:00
|
|
|
#endif
|
|
|
|
|
|
2018-09-14 02:24:30 +08:00
|
|
|
(void)atomic_set(&start_flag, 1);
|
2018-01-18 03:34:50 +08:00
|
|
|
}
|
2019-02-20 08:03:39 +08:00
|
|
|
|
2020-02-07 05:39:52 +08:00
|
|
|
bool z_smp_cpu_mobile(void)
|
|
|
|
|
{
|
|
|
|
|
unsigned int k = arch_irq_lock();
|
|
|
|
|
bool pinned = arch_is_in_isr() || !arch_irq_unlocked(k);
|
|
|
|
|
|
|
|
|
|
arch_irq_unlock(k);
|
|
|
|
|
return !pinned;
|
|
|
|
|
}
|