Thread suspension is a technique commonly used by Application Monitors such as:
- debuggers
- stop-the-world garbage collectors
- security tools
Besides it is useful from technical point of view, having possibility to stop and resume nothing suspecting threads brings me joy. Maybe I’m becoming control freak. Or just spending too much time in front of my computer.
Anyway, let’s cut the crap out and see what possibilities of thread suspension do we have on different operating systems:
- Windows: WinApi functions SuspendThread / ResumeThread
- Unix: pthread_suspend / pthread_resume (name depends on particular system)
- Linux: ptrhread_suspend_np / ptrhread_resume_np
_np postfix in Linux functions doesn’t look good – it means that they are non-portable and according to documentation – available only on RtLinux systems. If we want to have possibility of suspending threads on other Linuxes we have to do it by ourselves. And we want to have this possibility.
General idea
To implement thread suspension from user-mode we will have to break somehow into thread’s execution context. Fortunately this is exactly what signal mechanism provides us. General schema looks like this:
- Send signal to victim thread
- In signal handler place blocking operation
- During resume, notify blocking mechanism and exit signal handler which will resume thread’s execution
To make this mechanism safe we will have to carefully choose what operation we are performing during signal handler and what signal we are blocking:
- Signal used to resume and suspend will be SIGUSR1 which is left free to programmer
- Before signal handler is executed, signal mask of victim thread has to be changed to block SIGUSR1. If we won’t do this then multiple concurrent requests to suspend/resume can cause deadlocks and races. It can be done by specifying mask in sigaction function.
- suspend function has to wait on signal delivery to victim thread. If it will exit asynchronously before signal is delivered it will be worthless as synchronization mechanism and could trick user to think that thread is already blocked (but in fact can still executes). Signal delivery synchronization can be done by spinlock shared between signal handler and issuing thread.
- blocking operation in signal handler has to be reentrant (it means it has to be on the list of async-safe functions provided in man 7 signal). It also has to atomically change signal mask when entering (to not block SIGUSR1 anymore) and restore it when exiting. Fortunately such function exists and it is sigsuspend. It will block until specified signal is delivered and will temporarily replace current signal mask.
Implmentation
Code is available on github: https://github.com/bit-envoy/threadmgmt
#pragma once #include <pthread.h> #define USED_SIG SIGUSR1 int thread_mgmt_init(void); int thread_mgmt_release(void); int thread_mgmt_suspend(pthread_t t); int thread_mgmt_resume(pthread_t t);
#include "threadmgmt.h"
#include <signal.h>
#include <string.h>
#include <pthread.h>
#define CPU_RELAX() asm("pause")
#define SMP_WB()
#define SMP_RB()
typedef struct thread_mgmt_op
{
int op;
volatile int done;
volatile int res;
}thread_mgmt_op_t;
struct sigaction old_sigusr1;
static __thread volatile int thread_state = 1;
static void thread_mgmt_handler(int, siginfo_t*, void*);
static int thread_mgmt_send_op(pthread_t, int);
int thread_mgmt_init(void)
{
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sa.sa_sigaction = (void*)thread_mgmt_handler;
sa.sa_flags = SA_SIGINFO;
sigfillset(&sa.sa_mask);
//register signal handler which will full signal mask
return sigaction(USED_SIG, &sa, NULL);
}
int thread_mgmt_release()
{
//restore previous signal handler
return sigaction(USED_SIG, &old_sigusr1, NULL);
}
int thread_mgmt_suspend(pthread_t t)
{
return thread_mgmt_send_op(t, 0);
}
int thread_mgmt_resume(pthread_t t)
{
return thread_mgmt_send_op(t, 1);
}
static int thread_mgmt_send_op(pthread_t t, int opnum)
{
thread_mgmt_op_t op = {.op = opnum, .done = 0, .res = 0};
sigval_t val = {.sival_ptr = &op};
if(pthread_sigqueue(t, USED_SIG, val))
return -1;
//spin wait till signal is delivered
while(!op.done)
CPU_RELAX();
SMP_RB();
return op.res;
}
static void thread_mgmt_handler(int signum, siginfo_t* info, void* ctx)
{
thread_mgmt_op_t *op = (thread_mgmt_op_t*)(info->si_value.sival_ptr);
if(op->op == 0 && thread_state == 1)
{
//suspend
thread_state = 0;
op->res = 0;
SMP_WB();
op->done = 1;
sigset_t mask;
sigfillset(&mask);
sigdelset(&mask, USED_SIG);
//wait till SIGUSR1
sigsuspend(&mask);
}
else if(op->op == 1 && thread_state == 0)
{
//resume
thread_state = 1;
op->res = 0;
SMP_WB();
op->done = 1;
}
else
{
//resume resumed thread or
//suspend suspended thread
op->res = -1;
SMP_WB();
op->done = 1;
}
}
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include "threadmgmt.h"
void* function(void* arg)
{
int i = 0;
while(1) printf("thread(%d) %p\n", arg, i++);
return 0;
}
int main( void )
{
if(thread_mgmt_init())
return -1;
pthread_t t1, t2;
pthread_create(&t1, NULL, function, (void*)1);
pthread_create(&t2, NULL, function, (void*)2);
sleep(1);
if(thread_mgmt_suspend(t1))
return -2;
if(thread_mgmt_suspend(t2))
return -2;
sleep(2);
if(thread_mgmt_resume(t1))
return -3;
if(thread_mgmt_resume(t2))
return -3;
sleep(4);
thread_mgmt_release();
return 0;
}
Additional notes
- On processors that can reorder writes and reads to different addresses (e.g. some ARMs) macro SMP_WB and SMP_RB should be defined with proper memory barrier instructions. On x86 and x64 macros are empty because those processor does not perform reordering in described case.
- Disadvantage of using signals to implement suspending is that signals will interrupt blocking operations like sleep(). So threads that will be suspended should be prepared for it. If you know better way how to implement thread suspension on Linux feel free to let me know. Sad true about thread suspension is it should be implemented in kernel (like in Windows and some Unixes) not hacked in user mode.
Final word
Suspending threads is handy mechanism in many situations but it also can be dangerous. If thread is suspended during holding some lock, and then issuing thread will try acquire the same lock it will deadlock. This effect can be observed on test application – when thread is suspended during printf (which acquire locks) and other thread tries to printf something it will hang on the same lock. You have know what you are doing – use it in monitoring / instrumentation scenarios – where you don’t have full control over monitored thread code.
LAST SECTOR 