Troubleshooting guide

This means that the kernel crashes before the console is enabled. You should enable the CONFIG_EARLY_PRINTK option. For some architectures (blackfin, x86, arm), enabling this option also requires passing the earlyprintk parameter on the kernel command line. See Documentation/kernel-parameters.txt for possible values.

For the ARM architecture, you have to enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_LL in order to be able to enable CONFIG_EARLY_PRINTK.

Please make sure that you have followed the "Kernel configuration" section. Then, try capturing the oops text (using a serial console or netconsole) post the oops to the xenomai mailing list, with the kernel configuration you used to compile the failing kernel.

Your distribution may be configured to pass the quiet option on the kernel command line. In this case, the kernel does not print all the log messages, however, they are still available using the dmesg command.

You selected a CPU which as a TSC ("Pentium classic", and above), but the CPU on which you run the kernel has no TSC. This issue was resolved in the I-pipe core for Linux 3.4, but for prior versions, you need to select a CPU without a TSC when configuring the kernel, and recompile the kernel.

This happens when running in emulators. In such a case, Xenomai can not run as it needs to know the clock frequency.

This may also happen when using I-pipe core patch for Linux 3.2, compiled for a CPU with a TSC, but running on a CPU without a TSC, as in the "Xenomai: compiled for TSC case.

See code -19.

See code -19.

First you should run the latency test under some load and see if you experience any pathological latency ("pathological" meaning more than, say, 100 micro-seconds). If you do not observe any such latency, then this warning is harmless, and if you find it annoying, you may add the parameter "xeno_hal.smi=-1" on the kernel command line. You can skip the rest of this section.

If you observe any high latency then you have a problem with SMI, and this warning was intended for you. But the Xenomai patched kernel parameters allow you to enable two workarounds which may help you. These workarounds are enabled by adding the parameter "xeno_hal.smi=1" on the kernel command line.

The first workaround which you should try is to disable all SMI sources. In order to do this, simply boot with the parameter "xeno_hal.smi=1" on the kernel command line. This option is the most reliable workaround, because when enabled, no SMI can interfere with hardware interrupt management behind your back and cause high latencies. Once this workaround enabled, you should run the latency test again, verify that your high latency disappeared but most importantly, verify that every peripheral you intend to use with Xenomai is working properly.

If everything is working properly, then try and stress-test the system, and check for overheating. If the motherboard sensors do not allow to check temperature, and you get an unexplained reboot in the middle of the stress-test, chances are that you have an overheating issue, otherwise, you are done with the SMI workaround.

If some peripheral is not working properly, then it probably needs SMI, in which case you can not simply disable SMI globally, you will need to disable all SMI sources on your system except the SMI needed by your peripheral. The same goes if the system overheat, you have to find a way to keep the SMI source which is controlling the thermal control enabled. This is a much less reliable choice, since you have to know all SMI sources to disable them, one by one. In order to choose this second workaround, check in the documentation for the Intel chipset you use, for the documentation of the SMI_EN register. Then set the "xeno_hal.smi_mask" parameter on the kernel command line with a value where all bits set to 0 will be disabled when Xenomai starts.

You should then run the latency test again and verify that you do not observe any high latency and that all your peripherals are functioning correctly. If when running the latency test again, your peripheral is working properly and you still observe high latencies, then you are out of luck, the peripheral you want is likely to be the cause of such latencies.

The most probable reason is that Xenomai could not find a timer.

Check that you have not enabled one of the options in the "Kernel configuration" section.

On the ppc64 platform, check whether CONFIG_PPC_64K_PAGES is defined in your kernel configuration. If so, then you likely need to raise all Xenomai parameters defining the size of internal heaps, such as CONFIG_XENO_OPT_SYS_HEAPSZ, CONFIG_XENO_OPT_GLOBAL_SEM_HEAPSZ, CONFIG_XENO_OPT_SEM_HEAPSZ and CONFIG_XENO_OPT_SYS_STACKPOOLSZ, so that (size / 64k) > 2. The default values for these parameters are currently based on the assumption that PAGE_SIZE = 4k.

This error message means that you are trying to run the latency test as a non-root user. Using Xenomai services requires root privileges (more precisely CAP_SYS_NICE). However, you can allow a specific group to access Xenomai services, by following the instructions on this page.

On the x86 architecture, the configure script option --enable-x86-sep allows Xenomai to use the SYSENTER/SYSEXIT mechanism for issuing system calls.

However, this mechanism requires support from the libc. Currently, we know the glibc with NPTL has this support, other libraries will cause Xenomai applications to fail with this error message.

You have launched latency -t 1 or latency -t 2 which both require the kernel to have been compiled with the CONFIG_XENO_DRIVERS_TIMERBENCH option enabled.

See the "ARM tsc emulation issues" section.

Each Xenomai branch (2.1, 2.2, 2.3, 2.4, 2.5, 2.6,…) defines a kernel/user ABI, so that it is possible to mix kernels and user-space supports of different versions in the same branch. So, for instance, after having build a system with a kernel and user-space support using Xenomai 2.6.0, it is possible to update the user-space support to Xenomai 2.6.1 without changing the kernel.

However, it is not possible to mix kernel and user-space supports of different branches.

A common reason for this error is when you run a kernel compiled with Xenomai 2.6.1 support on a system where you have a user-space installed by your Debian based Linux distribution (notably Ubuntu) from the 2.5 branch, this can not work, the two branches use different ABIs. See README.INSTALL for details on how to compile a user-space support, or to build a new xenomai-runtime Debian package.

If you compiled and installed the correct Xenomai user-space support, there are probably files on your system remaining from a previous installation.

Since kernel-space support and user-space support are compiled separately, each Xenomai application checks, at startup, whether the kernel and user-space supports have been configured with compatible options. If you see this message, it means they have not. See README.INSTALL for further details. The following sections detail the most frequent reasons for this message.

See the "ARM tsc emulation issues" section.

Possible reasons for this error are:

On the ARM platform this message happens when there is a mismatch between kernel and user for the EABI setting: for instance you compiled the user-space support with a toolchain generating OABI code, and are trying to run the result on a kernel with CONFIG_AEABI but without CONFIG_OABI_COMPAT. Or vice versa, when running user-space compiled with an EABI toolchain on a kernel without CONFIG_AEABI.

Xenomai watchdog has stopped the latency test because it was using all the CPU in primary mode. This is likely due to a too short period, re-run the latency test passing a longer period using the -p option.

See the "ARM tsc emulation issues" section.

The most common reason for this issues is a too short period passed with the -p option, try increasing the period. If you enable the watchdog (option CONFIG_XENO_OPT_WATCHDOG, in your kernel configuration), you should see the "Xenomai: watchdog triggered (period too short?)" message.

The latency test runs, but you are seeing high latencies.

With olders versions of X-window, keep the existing driver, but add the following line to the Device section:

In order to allow applications to measure short durations with as little overhead as possible, Xenomai uses a 64 bits high resolution counter. On x86, the counter used for this purpose is the time-stamp counter, aka "tsc".

ARM processors generally do not have a 64 bits high resolution counter available in user-space, so this counter is emulated by reading whatever high resolution counter is available on the processor, and used as clock source in kernel-space, and extend it to 64 bits by using data shared with the kernel. If Xenomai libraries are compiled without emulated tsc support, system calls are used, which have a much higher overhead than the emulated tsc code.

In recent versions of the I-pipe patch, SOCs generally select the CONFIG_IPIPE_ARM_KUSER_TSC option, which means that the code for reading this counter is provided by the kernel at a predetermined address (in the vector page, a page which is mapped at the same address in every process) and is the code used if you do not pass the --enable-arm-tsc or --disable-arm-tsc option to configure, or pass --enable-arm-tsc=kuser.

This default should be fine with recent patches and most ARM SOCs.

However, if you see the following message:

It means that you are either using an old patch, or that the SOC you are using does not select the CONFIG_IPIPE_ARM_KUSER_TSC option.

So you should resort to what Xenomai did before branch 2.6: select the tsc emulation code when compiling Xenomai user-space support by using the --enable-arm-tsc option. The parameter passed to this option is the name of the SOC or SOC family for which you are compiling Xenomai. Typing:

will return the list of valid values for this option.

If after having enabled this option and recompiled, you see the following message when starting the latency test:

or

It means that you selected the wrong SOC or SOC family, reconfigure Xenomai user-space support by passing the right parameter to --enable-arm-tsc and recompile.

The following message:

means that the kernel-space support for the SOC you are using does not provide support for tsc emulation in user-space. In that case, you should recompile Xenomai user-space support passing the --disable-arm-tsc option.

2.6.2 (like any previous Xenomai release) does not handle the extended processor state (xsave/xrstor) yet.

2.6.2 automatically disables this CPU feature at boot when the host kernel detects it, so no action is to be taken by the user. However, this feature shall be disabled manually for older Xenomai releases, by passing the "noxsave" parameter on the kernel command line (see Documentation/kernel-parameters.txt).

Failing to do so, running with extended processor state support enabled on these Xenomai releases beget random execution errors in userland, typically when the switchtest program runs in the background, due to incorrect FPU management in real-time mode.

Where <service> is a thread creation service, one of:

Starting with Xenomai 3, the skins will not export their interface to kernel modules anymore, at the notable exception of the RTDM device driver API, which by essence must be used from kernel space for writing real-time device drivers. Those warnings are there to remind you that application code should run in user-space context instead.

The reason for this is fully explained in the project Roadmap document, see "What Will Change With Xenomai 3".

You may switch those warnings off by enabling the CONFIG_XENO_OPT_NOWARN_DEPRECATED option in your kernel configuration, but nevertheless, you have been WARNED.

In order to avoid unwanted transitions to secondary domain, an application using Xenomai services should call, before any Xenomai service:

Even if your system has no swap, the Linux kernel may "swap out" some pages, for instance the program pages which are known to exist on flash or on disk, causing page faults (and a secondary mode switch for xenomai threads running in primary mode) when the program tries to access this page.

So, Xenomai libraries abort when an application which has not called mlockall is detected.

Note that some skins allow mlockall to be called automatically by Xenomai libraries startup, this is enabled when configuring Xenomai user-space support with the configure script.

See configure --help.

Such transition requires to wake up the Linux task underlying your real-time thread when running in secondary mode, since the latter needs to leave the Xenomai domain for executing under the control of the regular Linux scheduler. Therefore, it all depends on the Linux kernel granularity, i.e. its ability to reach the next rescheduling point as soon as such wakeup has been requested. Additionally, the task wakeup request is performed from a virtual interrupt handler which has to be run from the Linux domain upon request from the Xenomai domain, so the time required to handle and dispatch this interrupt outside of any critical kernel section also needs to be accounted for. Even if the kernel granularity improves at each new release, there are still a few catches:

Possible reasons for this error are:

Your user-space application unexpectedly reserves a lot of virtual memory, as reported by "top" or /proc/<pid>/maps. Sometimes OOM situations even appear during runtime on systems with limited memory.

The Xenomai tasks are underlaid by native POSIX threads, for which a huge default amount of stack space memory is reserved by the native POSIX support, usually 8MiB per thread, so the overall allocated space is about 8MiB * +nr_threads+, which are likely to be locked using the mlockall() service, which in turn even commits such space to RAM.

Unfortunately, this behaviour cannot be controlled by the "stacksize" parameter passed to the various thread creation routines, i.e. the latter is about limiting the addressable stack space on a per-thread basis, but does not affect the amount of stack memory initially reserved by the POSIX library. A work-around consists of setting a lower user-limit for initial stack allocation, like calling:

in your parent shell before running your application (defaults to 8192).