OrgZephyr/zephyr
OrgZephyr
/
zephyr
mirror of https://github.com/zephyrproject-rtos/zephyr.git
1
0
Fork 0
Code Issues Releases Wiki Activity
zephyr/kernel/include/kswap.h

96 lines
2.2 KiB
C
Raw Normal View History

kernel: Add kswap.h header to unbreak cycles The xtensa-asm2 work included a patch that added nano_internal.h includes in lots of places that needed to have _Swap defined, because it had to break a cycle and this no longer got pulled in from the arch headers. Unfortunately those new includes created new and more amusing cycles elsewhere which led to breakage on other platforms. Break out the _Swap definition (only) into a separate header and use that instead. Cleaner. Seems not to have any more hidden gotchas. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-01-26 07:24:15 +08:00
/*
* Copyright (c) 2018 Intel Corporation
*
* SPDX-License-Identifier: Apache-2.0
*/
#ifndef _KSWAP_H
#define _KSWAP_H
#include <ksched.h>
#include <kernel_arch_func.h>
#ifdef CONFIG_TIMESLICING
extern void _update_time_slice_before_swap(void);
#else
#define _update_time_slice_before_swap() /**/
#endif
#ifdef CONFIG_STACK_SENTINEL
extern void _check_stack_sentinel(void);
#else
#define _check_stack_sentinel() /**/
#endif
kernel/kswap.h: Hook event logger from switch-based _Swap The new generic _Swap() forgot the event logger hook Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-02-10 06:34:05 +08:00
extern void _sys_k_event_logger_context_switch(void);
kernel: Rework SMP irq_lock() compatibility layer This was wrong in two ways, one subtle and one awful. The subtle problem was that the IRQ lock isn't actually globally recursive, it gets reset when you context switch (i.e. a _Swap() implicitly releases and reacquires it). So the recursive count I was keeping needs to be per-thread or else we risk deadlock any time we swap away from a thread holding the lock. And because part of my brain apparently knew this, there was an "optimization" in the code that tested the current count vs. zero outside the lock, on the argument that if it was non-zero we must already hold the lock. Which would be true of a per-thread counter, but NOT a global one: the other CPU may be holding that lock, and this test will tell you *you* do. The upshot is that a recursive irq_lock() would almost always SUCCEED INCORRECTLY when there was lock contention. That this didn't break more things is amazing to me. The rework is actually simpler than the original, thankfully. Though there are some further subtleties: * The lock state implied by irq_lock() allows the lock to be implicitly released on context switch (i.e. you can _Swap() with the lock held at a recursion level higher than 1, which needs to allow other processes to run). So return paths into threads from _Swap() and interrupt/exception exit need to check and restore the global lock state, spinning as needed. * The idle loop design specifies a k_cpu_idle() function that is on common architectures expected to enable interrupts (for obvious reasons), but there is no place to put non-arch code to wire it into the global lock accounting. So on SMP, even CPU0 needs to use the "dumb" spinning idle loop. Finally this patch contains a simple bugfix too, found by inspection: the interrupt return code used when CONFIG_SWITCH is enabled wasn't correctly setting the active flag on the threads, opening up the potential for a race that might result in a thread being scheduled on two CPUs simultaneously. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-04-13 03:50:05 +08:00
/* In SMP, the irq_lock() is a spinlock which is implicitly released
* and reacquired on context switch to preserve the existing
* semantics. This means that whenever we are about to return to a
* thread (via either _Swap() or interrupt/exception return!) we need
* to restore the lock state to whatever the thread's counter
* expects.
*/
void _smp_reacquire_global_lock(struct k_thread *thread);
void _smp_release_global_lock(struct k_thread *thread);
kernel: Add kswap.h header to unbreak cycles The xtensa-asm2 work included a patch that added nano_internal.h includes in lots of places that needed to have _Swap defined, because it had to break a cycle and this no longer got pulled in from the arch headers. Unfortunately those new includes created new and more amusing cycles elsewhere which led to breakage on other platforms. Break out the _Swap definition (only) into a separate header and use that instead. Cleaner. Seems not to have any more hidden gotchas. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-01-26 07:24:15 +08:00
/* context switching and scheduling-related routines */
#ifdef CONFIG_USE_SWITCH
/* New style context switching. _arch_switch() is a lower level
* primitive that doesn't know about the scheduler or return value.
* Needed for SMP, where the scheduler requires spinlocking that we
* don't want to have to do in per-architecture assembly.
*/
static inline unsigned int _Swap(unsigned int key)
{
struct k_thread *new_thread, *old_thread;
kernel: Don't _arch_switch() to yourself The SMP testing missed the case where _Swap() decides to return back into the _current. Obviously there is no valid switch handle for the running thread into which we can restore, and everything blows up. (What happened is that the new scheduler code opened up a spot where k_thread_priority_set() does a _reschedule() unconditionally and doens't check to see whether or not it's needed like the old code). But that isn't incorrect! It's entirely possible that _Swap() may find that no thread is runnable except _current (due, for example, to another CPU racing the other thread you expected off to sleep or something). Don't blow up, check and return a noop. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-05-15 05:46:27 +08:00
int ret = 0;
kernel: Add kswap.h header to unbreak cycles The xtensa-asm2 work included a patch that added nano_internal.h includes in lots of places that needed to have _Swap defined, because it had to break a cycle and this no longer got pulled in from the arch headers. Unfortunately those new includes created new and more amusing cycles elsewhere which led to breakage on other platforms. Break out the _Swap definition (only) into a separate header and use that instead. Cleaner. Seems not to have any more hidden gotchas. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-01-26 07:24:15 +08:00
kernel: Move per-cpu _kernel_t fields into separate struct When in SMP mode, the nested/irq_stack/current fields are specific to the current CPU and not to the kernel as a whole, so we need an array of these. Place them in a _cpu_t struct and implement a _arch_curr_cpu() function to retrieve the pointer. When not in SMP mode, the first CPU's fields are defined as a unioned with the first _cpu_t record. This permits compatibility with legacy assembly on other platforms. Long term, all users, including uniprocessor architectures, should be updated to use the new scheme. Fundamentally this is just renaming: the structure layout and runtime code do not change on any existing platforms and won't until someone defines a second CPU. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-01-26 08:39:35 +08:00
old_thread = _current;
kernel: Add kswap.h header to unbreak cycles The xtensa-asm2 work included a patch that added nano_internal.h includes in lots of places that needed to have _Swap defined, because it had to break a cycle and this no longer got pulled in from the arch headers. Unfortunately those new includes created new and more amusing cycles elsewhere which led to breakage on other platforms. Break out the _Swap definition (only) into a separate header and use that instead. Cleaner. Seems not to have any more hidden gotchas. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-01-26 07:24:15 +08:00
_check_stack_sentinel();
_update_time_slice_before_swap();
kernel/kswap.h: Hook event logger from switch-based _Swap The new generic _Swap() forgot the event logger hook Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-02-10 06:34:05 +08:00
#ifdef CONFIG_KERNEL_EVENT_LOGGER_CONTEXT_SWITCH
_sys_k_event_logger_context_switch();
#endif
kernel: Add kswap.h header to unbreak cycles The xtensa-asm2 work included a patch that added nano_internal.h includes in lots of places that needed to have _Swap defined, because it had to break a cycle and this no longer got pulled in from the arch headers. Unfortunately those new includes created new and more amusing cycles elsewhere which led to breakage on other platforms. Break out the _Swap definition (only) into a separate header and use that instead. Cleaner. Seems not to have any more hidden gotchas. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-01-26 07:24:15 +08:00
new_thread = _get_next_ready_thread();
kernel: Don't _arch_switch() to yourself The SMP testing missed the case where _Swap() decides to return back into the _current. Obviously there is no valid switch handle for the running thread into which we can restore, and everything blows up. (What happened is that the new scheduler code opened up a spot where k_thread_priority_set() does a _reschedule() unconditionally and doens't check to see whether or not it's needed like the old code). But that isn't incorrect! It's entirely possible that _Swap() may find that no thread is runnable except _current (due, for example, to another CPU racing the other thread you expected off to sleep or something). Don't blow up, check and return a noop. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-05-15 05:46:27 +08:00
if (new_thread != old_thread) {
kernel: Scheduler rewrite This replaces the existing scheduler (but not priority handling) implementation with a somewhat simpler one. Behavior as to thread selection does not change. New features: + Unifies SMP and uniprocessing selection code (with the sole exception of the "cache" trick not being possible in SMP). + The old static multi-queue implementation is gone and has been replaced with a build-time choice of either a "dumb" list implementation (faster and significantly smaller for apps with only a few threads) or a balanced tree queue which scales well to arbitrary numbers of threads and priority levels. This is controlled via the CONFIG_SCHED_DUMB kconfig variable. + The balanced tree implementation is usable symmetrically for the wait_q abstraction, fixing a scalability glitch Zephyr had when many threads were waiting on a single object. This can be selected via CONFIG_WAITQ_FAST. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-05-04 05:51:49 +08:00
kernel: Don't _arch_switch() to yourself The SMP testing missed the case where _Swap() decides to return back into the _current. Obviously there is no valid switch handle for the running thread into which we can restore, and everything blows up. (What happened is that the new scheduler code opened up a spot where k_thread_priority_set() does a _reschedule() unconditionally and doens't check to see whether or not it's needed like the old code). But that isn't incorrect! It's entirely possible that _Swap() may find that no thread is runnable except _current (due, for example, to another CPU racing the other thread you expected off to sleep or something). Don't blow up, check and return a noop. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-05-15 05:46:27 +08:00
old_thread->swap_retval = -EAGAIN;
kernel: Add kswap.h header to unbreak cycles The xtensa-asm2 work included a patch that added nano_internal.h includes in lots of places that needed to have _Swap defined, because it had to break a cycle and this no longer got pulled in from the arch headers. Unfortunately those new includes created new and more amusing cycles elsewhere which led to breakage on other platforms. Break out the _Swap definition (only) into a separate header and use that instead. Cleaner. Seems not to have any more hidden gotchas. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-01-26 07:24:15 +08:00
kernel: SMP-aware scheduler The scheduler needs a few tweaks to work in SMP mode: 1. The "cache" field just doesn't work. With more than one CPU, caching the highest priority thread isn't useful as you may need N of them at any given time before another thread is returned to the scheduler. You could recalculate it at every change, but that provides no performance benefit. Remove. 2. The "bitmask" designed to prevent the need to individually check priorities is likewise dropped. This could work, but in fact on our only current SMP system and with current K_NUM_PRIOPRITIES values it provides no real benefit. 3. The individual threads now have a "current cpu" and "active" flag so that the choice of the next thread to run can correctly skip threads that are active on other CPUs. The upshot is that a decent amount of code gets #if'd out, and the new SMP implementations for _get_highest_ready_prio() and _get_next_ready_thread() are simpler and smaller, at the expense of having to drop older optimizations. Note that scheduler synchronization is unchanged: all scheduler APIs used to require that an irq_lock() be held, which means that they now require the global spinlock via the same API. This should be a very early candidate for lock granularity attention! Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-01-30 06:55:20 +08:00
#ifdef CONFIG_SMP
kernel: Don't _arch_switch() to yourself The SMP testing missed the case where _Swap() decides to return back into the _current. Obviously there is no valid switch handle for the running thread into which we can restore, and everything blows up. (What happened is that the new scheduler code opened up a spot where k_thread_priority_set() does a _reschedule() unconditionally and doens't check to see whether or not it's needed like the old code). But that isn't incorrect! It's entirely possible that _Swap() may find that no thread is runnable except _current (due, for example, to another CPU racing the other thread you expected off to sleep or something). Don't blow up, check and return a noop. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-05-15 05:46:27 +08:00
new_thread->base.cpu = _arch_curr_cpu()->id;
kernel: Rework SMP irq_lock() compatibility layer This was wrong in two ways, one subtle and one awful. The subtle problem was that the IRQ lock isn't actually globally recursive, it gets reset when you context switch (i.e. a _Swap() implicitly releases and reacquires it). So the recursive count I was keeping needs to be per-thread or else we risk deadlock any time we swap away from a thread holding the lock. And because part of my brain apparently knew this, there was an "optimization" in the code that tested the current count vs. zero outside the lock, on the argument that if it was non-zero we must already hold the lock. Which would be true of a per-thread counter, but NOT a global one: the other CPU may be holding that lock, and this test will tell you *you* do. The upshot is that a recursive irq_lock() would almost always SUCCEED INCORRECTLY when there was lock contention. That this didn't break more things is amazing to me. The rework is actually simpler than the original, thankfully. Though there are some further subtleties: * The lock state implied by irq_lock() allows the lock to be implicitly released on context switch (i.e. you can _Swap() with the lock held at a recursion level higher than 1, which needs to allow other processes to run). So return paths into threads from _Swap() and interrupt/exception exit need to check and restore the global lock state, spinning as needed. * The idle loop design specifies a k_cpu_idle() function that is on common architectures expected to enable interrupts (for obvious reasons), but there is no place to put non-arch code to wire it into the global lock accounting. So on SMP, even CPU0 needs to use the "dumb" spinning idle loop. Finally this patch contains a simple bugfix too, found by inspection: the interrupt return code used when CONFIG_SWITCH is enabled wasn't correctly setting the active flag on the threads, opening up the potential for a race that might result in a thread being scheduled on two CPUs simultaneously. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-04-13 03:50:05 +08:00
kernel: Don't _arch_switch() to yourself The SMP testing missed the case where _Swap() decides to return back into the _current. Obviously there is no valid switch handle for the running thread into which we can restore, and everything blows up. (What happened is that the new scheduler code opened up a spot where k_thread_priority_set() does a _reschedule() unconditionally and doens't check to see whether or not it's needed like the old code). But that isn't incorrect! It's entirely possible that _Swap() may find that no thread is runnable except _current (due, for example, to another CPU racing the other thread you expected off to sleep or something). Don't blow up, check and return a noop. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-05-15 05:46:27 +08:00
_smp_release_global_lock(new_thread);
kernel: SMP-aware scheduler The scheduler needs a few tweaks to work in SMP mode: 1. The "cache" field just doesn't work. With more than one CPU, caching the highest priority thread isn't useful as you may need N of them at any given time before another thread is returned to the scheduler. You could recalculate it at every change, but that provides no performance benefit. Remove. 2. The "bitmask" designed to prevent the need to individually check priorities is likewise dropped. This could work, but in fact on our only current SMP system and with current K_NUM_PRIOPRITIES values it provides no real benefit. 3. The individual threads now have a "current cpu" and "active" flag so that the choice of the next thread to run can correctly skip threads that are active on other CPUs. The upshot is that a decent amount of code gets #if'd out, and the new SMP implementations for _get_highest_ready_prio() and _get_next_ready_thread() are simpler and smaller, at the expense of having to drop older optimizations. Note that scheduler synchronization is unchanged: all scheduler APIs used to require that an irq_lock() be held, which means that they now require the global spinlock via the same API. This should be a very early candidate for lock granularity attention! Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-01-30 06:55:20 +08:00
#endif
kernel: Don't _arch_switch() to yourself The SMP testing missed the case where _Swap() decides to return back into the _current. Obviously there is no valid switch handle for the running thread into which we can restore, and everything blows up. (What happened is that the new scheduler code opened up a spot where k_thread_priority_set() does a _reschedule() unconditionally and doens't check to see whether or not it's needed like the old code). But that isn't incorrect! It's entirely possible that _Swap() may find that no thread is runnable except _current (due, for example, to another CPU racing the other thread you expected off to sleep or something). Don't blow up, check and return a noop. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-05-15 05:46:27 +08:00
_current = new_thread;
_arch_switch(new_thread->switch_handle,
&old_thread->switch_handle);
kernel: Add kswap.h header to unbreak cycles The xtensa-asm2 work included a patch that added nano_internal.h includes in lots of places that needed to have _Swap defined, because it had to break a cycle and this no longer got pulled in from the arch headers. Unfortunately those new includes created new and more amusing cycles elsewhere which led to breakage on other platforms. Break out the _Swap definition (only) into a separate header and use that instead. Cleaner. Seems not to have any more hidden gotchas. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-01-26 07:24:15 +08:00
kernel: Don't _arch_switch() to yourself The SMP testing missed the case where _Swap() decides to return back into the _current. Obviously there is no valid switch handle for the running thread into which we can restore, and everything blows up. (What happened is that the new scheduler code opened up a spot where k_thread_priority_set() does a _reschedule() unconditionally and doens't check to see whether or not it's needed like the old code). But that isn't incorrect! It's entirely possible that _Swap() may find that no thread is runnable except _current (due, for example, to another CPU racing the other thread you expected off to sleep or something). Don't blow up, check and return a noop. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-05-15 05:46:27 +08:00
ret = _current->swap_retval;
}
kernel: Add kswap.h header to unbreak cycles The xtensa-asm2 work included a patch that added nano_internal.h includes in lots of places that needed to have _Swap defined, because it had to break a cycle and this no longer got pulled in from the arch headers. Unfortunately those new includes created new and more amusing cycles elsewhere which led to breakage on other platforms. Break out the _Swap definition (only) into a separate header and use that instead. Cleaner. Seems not to have any more hidden gotchas. Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2018-01-26 07:24:15 +08:00
irq_unlock(key);
return ret;
}
#else /* !CONFIG_USE_SWITCH */
extern unsigned int __swap(unsigned int key);
static inline unsigned int _Swap(unsigned int key)
{
_check_stack_sentinel();
_update_time_slice_before_swap();
return __swap(key);
}
#endif
#endif /* _KSWAP_H */