123doku.com

Embedded Linux Conference 2014SMP bring up onARM SoCsGregory CLEMENTFree [email protected] Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com1/31

Gregory CLEMENTIEmbedded Linux engineer and trainer at Free Electrons since2010IIIEmbedded Linux development: kernel and driverdevelopment, system integration, boot time and powerconsumption optimization, consulting, etc.Embedded Linux training, Linux driver development trainingand Android system development training, with materialsfreely available under a Creative Commons license.http://free-electrons.comIContributing to kernel support for the Armada 370, 375,38x and Armada XP ARM SoCs from Marvell.ICo-maintainer of mvebu sub-architecture (SoCs from MarvellEngineering Business Unit)ILiving near Lyon, FranceFree Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com2/31

Motivation and OverviewIMotivationIIIIIAll the new ARM CPUs are now SMP capable.Most of the new ARM SoCs actually uses multiple CPUs.Did SMP bring up for two kinds of ARM SoCs during past year.Documentation I would have like to find.OverviewIIISMP operations and functions.Implementation of these functions.Issues encountered during SMP bring up.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com3/31

SMP on ARM SoC?IA symmetric multiprocessor system (SMP) is a multiprocessorsystem with centralized shared memory called main memory(MM) operating under a single operating system with two ormore homogeneous processors. (Wikipedia)IMost of the SMP code is not architecture dependent (inkernel directory).IMost of the hardware support is related to the ARMspecification.ISoC level: all that is beyond the CPU, mainly initializationand power down the CPUs.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com4/31

AssumptionsISoC support is working on one CPU core.ISupport is following the last requirements(see “Your new ARM SoC Linux support check-list” from Thomas s/2013/elc/arm-socchecklist/)IDatasheet is available.INo hardware bug (or at least they are documented).Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com5/31

SMP operationsIFew SMP functions related to the SoC:struct smp operations {void (*smp init cpus)(void);void (*smp prepare cpus)(unsigned int max cpus);void (*smp secondary init)(unsigned int cpu);int (*smp boot secondary)(unsigned int cpu,struct task struct *idle);[.]};IOnly smp boot secondary() is mandatory, others areoptional and depend on the need of the SoCs.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com6/31

Role of each SMP operations 1/2Ismp init cpus():IIIISetup the set of possible CPUs (via cpu possible()).Can be removed if the CPU topology is up to date in thedevice tree.Called very early during the boot process (fromsetup arch()).smp prepare cpus():IIIIEnables coherency.Initializes cpu possible map.Prepares the resources (power, ram, clock.).Called early during the boot process (before the initcalls butafter setup arch()).Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com7/31

Role of each SMP operations 2/2Ismp secondary init():IIIISupposed to “perform platform specific initialization of thespecified CPU”.Mostly dealing with pen release stuff which seems to be wrong.Russell King: The pen release stuff is only there forplatforms where there’s no proper way of controlling thesecondary CPUs except by using a software method.Called from secondary start kernel() on the CPU whichhas just been started.smp boot secondary():IIActually boots a secondary CPU identified by the CPU numbergiven in parameter.Called from cpu up() on the booting CPU.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com8/31

Booting sequence of the SMP operationsFree Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com9/31

Hotplug SMP operationsIWhen CONFIG HOTPLUG CPU is selected three more functionsare added to smp operations:struct smp operations {[.]int (*cpu kill)(unsigned int cpu);void (*cpu die)(unsigned int cpu);int (*cpu disable)(unsigned int cpu);};ITo support the hotplug features cpu kill() and cpu die()must be provided.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com10/31

Role of each SMP hotplug operations 1/2Icpu kill():IIIIISupposed to do the powering off and/or cutting off clocks tothe dying CPU.Need to be synchronized with cpu die().Needed for kexec (only user ofplatform can cpu hotplug(), which checks the presence ofthis function).Called on the thread which is asking for a CPU to beshutdown.cpu die():IIIISupposed to ensure death of the CPU.Not supposed to return from this function, returns only if theSoC can’t power down the CPU.Either power down the CPU or at least call cpu do idle().Called from the idle thread for the CPU which has beenshutdown.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com11/31

Role of each SMP hotplug operations 2/2Icpu disable():IIICommon ARM code takes care of the shutdown of a CPU: nomore interrupts can be handled by the kernel after the routinereturns, including local timers which are shutdown.By default, the common code allows disabling all CPUs exceptthe first one (the boot CPU). This function can be used tochange this behavior.Runs on the CPU to be shutdown.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com12/31

SMP related features: nice to haveNot mandatory features, but they are usually added in the sametime that SMP support.I IRQ affinity: allows to dedicate one or several CPUs to aninterrupt.IIIPart of the IRQ chip driver.Implemented in the following handler:int (*irq set affinity)(struct irq data *data,const struct cpumask *dest, bool force)Local timer: timer dedicated to a CPU, thanks to this, nomore need to broadcast a tick to all the CPUs.IINo more a specific API to register it since 3.12.Still needs to use the percpu API and the notifiers toimplement it.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com13/31

Use of the PSCI for SMPPSCI stands for Power State Coordination Interface.INeeded for virtualization.IUsed to coordinate OSes and hypervisors.IOnly solution to deal with secure mode.IProvides functions used for SMP bring-up such as CPU ON orCPU OFF.IFunction provided by the firmware, the SMP operations are nomore used.IID of these functions provided through the device tree.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com14/31

Implementing the SMP operations 1/2Ismp init cpus()IIIITypically if an SCU unit is available the number of CPUs isretrieved using scu get core count().And then the CPU possible map is builtusing set cpu possible().However as the cpu topology is supposed to be described inthe device tree, the whole map is already built in the functionarm dt init cpu maps.smp prepare cpus()IIIIf SCU is present then the coherency is enabled by usingscu enable().Most of the time resources are allocated and the registers aremapped using ioremap() and its friends.The way to enable (if needed) the power and the clocks ishighly platform specific and does not involve common code.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com15/31

Implementing the SMP operations 2/2Ismp secondary init()IINo common code used here but a lot of copy/paste of thepen release mechanism.smp boot secondary()IIIReleases the CPU from reset by writing on a powermanagement register.Some time writes “magic” pattern read by the bootrom whichwill release the CPU from reset.Uses the pen release if the secondary CPU has already beenreleased from reset.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com16/31

Implementing the SMP hotplug operations 1/2Icpu kill()IIIIMost of the platforms check that the CPU is going to die, byreading a register of the SoC wrote by the dying CPU.Some platforms really check that the dying CPU went in thereset state.Then it actually powers down the CPU if the dying CPU hasnot done it by itself.cpu die()IIIOn most of the platforms, the dying CPU writes a register ofthe SoC, usually by setting to 0 the jump address of the CPU.Some platforms switch in low power mode.The others just call cpu do idle().Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com17/31

Implementing the SMP hotplug operations 2/2Icpu disable()IIOnly implemented if the behavior of the platforms is differentof the default (ie all CPUs except the first one can be killed).Currently only used by shmobile where any CPU can be killed.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com18/31

Registering the SMP operationsIUsed to be part of the machine description. Thesmp operations structure of the SoC was associated to the.smp field using the smp ops() helper.IFrom 3.16, they should be directly registered using theCPU METHOD OF DECLARE() helper.IThen the operation will be fetched from the device tree.Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com19/31

Implementing a local timerIShould be part of a clock source driver indrivers/clocksource/.IEach local timer will be called during the CPU bring up(CPU STARTING), so the setup of the timer must beassociated to a notifier using register cpu notifier().IThe timer must be stopped when the CPU is shutting down(CPU DYING), so this function will be also associated to thenotifier.IThe interrupt has to be registered withrequest percpu irq().IAnd the clock event device has to be allocated withalloc percpu().Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://free-electrons.com20/31