Thread Stood Still

Thread suspension is a technique commonly used by Application Monitors such as:

  • debuggers
  • stop-the-world garbage collectors
  • security tools

Besides it is useful from technical point of view, having possibility to stop and resume nothing suspecting threads brings me joy. Maybe I’m becoming control freak. Or just spending too much time in front of my computer.

Anyway, let’s cut the crap out and see what possibilities of thread suspension do we have on different operating systems:

  • Windows: WinApi functions SuspendThread / ResumeThread
  • Unix: pthread_suspend / pthread_resume (name depends on particular system)
  • Linux: ptrhread_suspend_np / ptrhread_resume_np

_np postfix in Linux functions doesn’t look good – it means that they are non-portable and according to documentation – available only on RtLinux systems. If we want to have possibility of suspending threads on other Linuxes we have to do it by ourselves. And we want to have this possibility.

General idea
To implement thread suspension from user-mode we will have to break somehow into thread’s execution context. Fortunately this is exactly what signal mechanism provides us. General schema looks like this:

  1. Send signal to victim thread
  2. In signal handler place blocking operation
  3. During resume, notify blocking mechanism and exit signal handler which will resume thread’s execution

To make this mechanism safe we will have to carefully choose what operation we are performing during signal handler and what signal we are blocking:

  • Signal used to resume and suspend will be SIGUSR1 which is left free to programmer
  • Before signal handler is executed, signal mask of victim thread has to be changed to block SIGUSR1. If we won’t do this then multiple concurrent requests to suspend/resume can cause deadlocks and races. It can be done by specifying mask in sigaction function.
  • suspend function has to wait on signal delivery to victim thread. If it will exit asynchronously before signal is delivered it will be worthless as synchronization mechanism and could trick user to think that thread is already blocked (but in fact can still executes). Signal delivery synchronization can be done by spinlock shared between signal handler and issuing thread.
  • blocking operation in signal handler has to be reentrant (it means it has to be on the list of async-safe functions provided in man 7 signal). It also has to atomically change signal mask when entering (to not block SIGUSR1 anymore) and restore it when exiting. Fortunately such function exists and it is sigsuspend. It will block until specified signal is delivered and will temporarily replace current signal mask.

Implmentation
Code is available on github: https://github.com/bit-envoy/threadmgmt

#pragma once
#include <pthread.h>

#define USED_SIG SIGUSR1

int thread_mgmt_init(void);

int thread_mgmt_release(void);

int thread_mgmt_suspend(pthread_t t);

int thread_mgmt_resume(pthread_t t);
#include "threadmgmt.h"
#include <signal.h>
#include <string.h>
#include <pthread.h>

#define CPU_RELAX() asm("pause")
#define SMP_WB()
#define SMP_RB()

typedef struct thread_mgmt_op
{
    int op;
    volatile int done;
    volatile int res;
}thread_mgmt_op_t;

struct sigaction old_sigusr1;
static __thread volatile int thread_state = 1;

static void thread_mgmt_handler(int, siginfo_t*, void*);
static int thread_mgmt_send_op(pthread_t, int);


int thread_mgmt_init(void)
{
  struct sigaction sa;
  memset(&sa, 0, sizeof(sa));
  sa.sa_sigaction = (void*)thread_mgmt_handler;
  sa.sa_flags = SA_SIGINFO;      
  sigfillset(&sa.sa_mask);
  //register signal handler which will full signal mask
  return sigaction(USED_SIG, &sa, NULL);
}

int thread_mgmt_release()
{
    //restore previous signal handler
    return sigaction(USED_SIG, &old_sigusr1, NULL);
}

int thread_mgmt_suspend(pthread_t t)
{
    return thread_mgmt_send_op(t, 0);
}

int thread_mgmt_resume(pthread_t t)
{    
    return thread_mgmt_send_op(t, 1);
}

static int thread_mgmt_send_op(pthread_t t, int opnum)
{
    thread_mgmt_op_t op = {.op = opnum, .done = 0, .res = 0};
    sigval_t val = {.sival_ptr = &op};
    if(pthread_sigqueue(t, USED_SIG, val))
        return -1;
    
    //spin wait till signal is delivered
    while(!op.done) 
        CPU_RELAX();

    SMP_RB();
    return op.res;
}

static void thread_mgmt_handler(int signum, siginfo_t* info, void* ctx)
{
    thread_mgmt_op_t *op = (thread_mgmt_op_t*)(info->si_value.sival_ptr);
    if(op->op == 0 && thread_state == 1)
    {
        //suspend
        thread_state = 0;
        op->res = 0;
        SMP_WB();
        op->done = 1;
        
        sigset_t mask;
        sigfillset(&mask);
        sigdelset(&mask, USED_SIG);

        //wait till SIGUSR1
        sigsuspend(&mask);
    }
    else if(op->op == 1 && thread_state == 0)
    {
        //resume
        thread_state = 1;
        op->res = 0;
        SMP_WB();
        op->done = 1;
    }
    else
    {
        //resume resumed thread or
        //suspend suspended thread
        op->res = -1;
        SMP_WB();
        op->done = 1;
    }
}
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>

#include "threadmgmt.h"

void*  function(void*  arg)
{
    int i = 0;
    while(1) printf("thread(%d) %p\n", arg, i++);
    return 0;
}

int main( void )
{
    if(thread_mgmt_init())
        return -1;
    
   pthread_t t1, t2;
   pthread_create(&amp;t1, NULL, function, (void*)1);
   pthread_create(&amp;t2, NULL, function, (void*)2);

   
   sleep(1);
   if(thread_mgmt_suspend(t1))
       return -2;

   if(thread_mgmt_suspend(t2))
       return -2;
   
   sleep(2);
   if(thread_mgmt_resume(t1))
       return -3;

   if(thread_mgmt_resume(t2))
       return -3;
   
   sleep(4);
   thread_mgmt_release();
   
   return 0;
}

Additional notes

  • On processors that can reorder writes and reads to different addresses (e.g. some ARMs) macro SMP_WB and SMP_RB should be defined with proper memory barrier instructions. On x86 and x64 macros are empty because those processor does not perform reordering in described case.
  • Disadvantage of using signals to implement suspending is that signals will interrupt blocking operations like sleep(). So threads that will be suspended should be prepared for it. If you know better way how to implement thread suspension on Linux feel free to let me know. Sad true about thread suspension is it should be implemented in kernel (like in Windows and some Unixes) not hacked in user mode.

Final word
Suspending threads is handy mechanism in many situations but it also can be dangerous. If thread is suspended during holding some lock, and then issuing thread will try acquire the same lock it will deadlock. This effect can be observed on test application – when thread is suspended during printf (which acquire locks) and other thread tries to printf something it will hang on the same lock. You have know what you are doing – use it in monitoring / instrumentation scenarios – where you don’t have full control over monitored thread code.

Advertisements

Complication overload

Every kid knows that string processing in C is mundane job and it pretty easy to make a mistake. That’s why in C++ we have std::string class, right? Now we can easily concatenate strings, do other stuff, concatenate strings… Concatenate strings. Let’s do some concatenation.

#include <string>
#include <cstdio>
int main(int argc, char** argv)
{
 std::string text = "current app is: \"";
 text = text + argv[0] + '"';
 puts(text.c_str());
 return 0;
}

Output

current app is: "./prog"

So far everything is ok, but there is small inefficiency – instead text=text+… we can replace it with construction text+=… This will save additional reallocation. Besides, concatenation with std::string is easy – what can possibly go wrong?

#include <string>
#include <cstdio>
int main(int argc, char** argv)
{
 std::string text = "current app is: \"";
 //text = text + argv[0] + '"';
 text += argv[0] + '"';
 puts(text.c_str());
 return 0;
}

Output

current app is: "r/local/bin:/usr/bin:/bin

What just happened? My application for sure does not have this fancy name…
Let’s take a closer look at line:

text += argv[0] + '"';

It looks quite intuitively and harmless, and it should be the same as text=text+argv[0]+'”‘, which worked in previous case. Well, it actually causes a problem.

Expression ‘”‘ is treated as integer with value 0x22 (ascii code of ). At first right side is executed so we will get expression  argv[0]+'”‘ which is substitute for argv[0]+0x22. This expression will be finally concatenated to text. Due to pointer arithmetic argv[0]+0x22 is just a pointer that points outside the argv[0] so we will reference some random memory.
Expression text=text+argv[0]+'”‘ is different because at first text+argv[0] is executed which results in std::string. Then operator+(‘”‘) is executed on string which give us correct result.

As you can see expressions a=a+b and a+=b will not necessary give always same results, not only because of performance.