how to prevent output from getting interleaved

If an application only has one process and one thread, that is fine, there will be no competition to print information, which goes to stdout.
But what if there is one process but multiple threads? It is fine to use a single printf to print whatever you want, and that will not get interleaved with information printed by other threads. Below is an example.

void ThreadFn() {
    while (true) {
       printf("This is first info\nThis is second info\n");
       printf("This is third info\n");
   }
}

For the example above, you can always see a first info is followed by a second info, but may not followed by a third info. As far as we use a single printf, we are safe to guarantee that it is in integrity. However, information printed by more that one printf is vulnerable to get interleaved. The reason is in the implementation of printf, when it want to write to stdout, there is a line buffer(of course you can set it to unbuffered mode, but that is unusual). Multiple threads will use the same buffer for the same file, like stdout. To make sure the access to the buffer is thread-safe, it is considered by libraries to be a critical section which is protected by a mutex. Only up to one thread can access the buffer at a time. So the content written by a single printf is copied to the buffer in a single access to the buffer. But for two printf, the thread should apply for the mutex and access buffer more than once, there may be other thread accessing the buffer between two printf.

Then what about multiple processes? There is no synchronization mechanism between multiple processes for printing. When a process calls printf, the content goes to a buffer, when the buffer is full, it is dumped to the terminal/file by write/writev system call. There is no guarantee of the buffer size or how the buffer content will be dumped. To dump the buffer, some libraries may dump constant size at a time, others may dump all at once. So to guarantee integrity of information printed, we'd better call write/writev system call directly when multiple processes is getting involved. Below is an example.

void ProccessFn() {
   ...
   int nwrite = strlen(buf);
   char* p = buf;
   while (nwrite > 0) {
      int written = write(fileno(stdout), p, nwrite);
      if (written == -1) {
          if (errno != EINTR) {
             perror("write");
             break;
          }
      } else {
          nwrite -= written;
          p += written;
      }
   }
}

use popen to implement single direction communication between two processes

popen() is a posix api to implement single direction communication between two processes. It is suitable for the following conditions: 1. create a subprocess and read its standard output. 2. create a subprocess and write to its standard input.
I think it is something between system() and pipe() plus fork/exec. With system(), you can only start a child process and wait for the termination of child process. With pipe/fork/exec, as it is a little complex, I will show the typical usage below:

MainProcessFn() {
  int pipe_fd[2];
  pipe(pipe_fd);
  int pid = fork();
  if (pid == 0) {
     close(pipe_fd[0]);  // close read side
     dup2(pipe_fd[1], 1); // dup to stdout
     close(pipe_fd[1]);
     exec("some command");
  } else {
     close(pipe_fd[1]) // close write side
     FILE *fp = fdopen(pipe_fd[0], "r");
     // read from child process's standard output.
  }
}

As we can see, we can do bidirectional communication between two processes with pipe. We can only do unidirectional communication with popen, while the benefit is simpler code.

MainProcessFn() {
  FILE* fp = popen("some command", "r");
  // read from child process's standard output.
  pclose(fp);
}
With pclose, we can wait for the termination of child process, and get the exit status. In fact, popen/pclose is just a library api implemented based on system calls like pipe/fork/exec/waitpid, but just more convenient to use.

One way to realize parallelism

When we want to take benefits from multiprocessors, or just multitasking operating systems, we have to write multi-threads or multi-processes system. I met with such a condition before.
What I need to do is to spawn many processes in one process, each process does a small piece of job, gives feedback to the main process. There are about 1000 processes I need to spawn, so I may not spawn them at once as it makes heavy burden in system.
The first solution I can think up of is to use multi-threads to spawn processes. For example, I first create 10 threads in the main process, and in each thread, it spawns a process and waits for its finish, then it spawns and waits another process. This progress goes on until all jobs are finished. And I can make sure at any time there will not be more than 11 processes and 10 more threads. But the problem is I use pthread to create thread, and use fork to create process. It is very dangerous to use thread and fork at the same time. When you call fork in one thread, the new process will only have one thread, other threads are just stopped running, but the resources (like memory/lock) they occupy are not freed for reuse. For example, when one thread calls fork, another is using printf to print something to stdout. It seems fine, but in printf implementation, it may uses mutex lock to prevent different threads accessing the print buffer at the same time (that is the reason why we have clean output in multi-threads). If the thread using printf has just got the mutex lock when fork happens, the mutex lock will not be freed, so the result is the new forked process can not print anything to stdout (Because it will get the mutex lock for stdout which will never be freed).
After taught a lesson by pthread plus fork, I should find another way to realize parallelism. The idea is simple that if I can't spawn 1000 processes at once, what about 10 at a time? I can keep an array of spawned processes, once one child process is finished, I can spawn another one. By using an array of 10, I can make up to 10 spawned processes running at the same time. The work process is like below:

   while (not all work finished) {
      spawn processes until the running process array is full.
      wait for some process finish, collect the result.
   }

It is even simpler than the multi-thread one, but it works better.  So I think it deserves to record it :-)