Multi-Process Debugging

Index of All Documentation » Wing Pro Reference Manual » Debugger »

Wing Pro's debugger can debug multiple processes at once, either processes launched separately from the IDE, or (optionally) sub-processes spawned by a parent process.

When multiple processes are running at once, Wing adds a process selector to the stack selection area of the various debugging tools. This selector displays all the connected debug processes, arranged into an indented tree that indicates which processes are children of others. The selector annotates each process entry to show its process ID and whether or not it is paused or running.

Multi-process debugging is on by default but can be disabled with the Debugger > Processes > Enabled Multi-Process Debugging preference. When disabled, only one debug process can connect at a time or be created from the IDE.

Debugging Child Processes

Sub-processes started with the Python multiprocessing module or with os.fork() can optionally be debugged automatically, so that each child process appears as a separate debug process in Wing. This is disabled by default but can be enabled with the Debugger > Processes > Debug Child Processes preference or by setting Debug/Execute > Debug Child Processes in Project Properties.

Sub-processes started with os.system(), CreateProcess (on Windows), os.exec() (on Posix), or similar calls will not be debugged automatically because the OS completely replaces the parent process context and there is no way to keep a debug connection intact. However, it is still possible to debug processes launched in this way by manually initiating debug in the sub-process as described in Debugging Externally Launched Code.

Notice that processes started by os.fork() followed by os.exec() will be debugged for the (usually brief) period of time between the os.fork() and os.exec() calls.

Process Control

When multi-process debugging is enabled, Wing will allow creation of multiple processes from the Debug > Processes sub-menu. This menu also provides a way to continue, pause, restart, or terminate all debug processes at once.

Pressing the Alt key while clicking on the Continue, Terminate, or Restart toolbar icons also causes the operation to be applied to all applicable debug processes at once.

By default when a new process connects and reaches a breakpoint or exception, it is made into the current debug process only if there is no previously current and paused debug process, or if it is the first process in the launched process group that has stopped. In other cases, Wing displays a message at the bottom of the IDE window indicating that a debug process has stopped but does not make it the current process.

This behavior can be changed using the Debug > Processes > Switch to Stopped Processes preference. Setting this preference to Always Switch can be confusing if many processes are reaching a stopping point at once.

Wing also lets you control the maximum number of debug processes that may be attached to the IDE at once using the Debugger > Processes > Maximum Process Count preference.

Terminating Processes

When a debug process is terminated from Wing, the IDE will by default also terminate all other processes in the process group. This is appropriate behavior in many but not all cases. The Debugger > Processes > Termination Model preference provides several options for managing termination of debug processes in a multi-processing environment:

Leave Other Processes Running -- This kills only the selected current process and leaves all other processes running.

Kill Child Processes with Parent -- This also kills all children, grand-children, and other processes spawned by the parent or its children. However, any parent or grand-parent processes and their children are left running.

Kill Entire Process Group -- This kills all processes in the group, including all parents, grand-parents, children, grand-children, etc. This is the default termination model.

Prompt for Action When a Process is Killed -- This displays a dialog listing processes associated with the debug process that was terminated and offers to kill selected processes, all children, or the entire process group.

Note that when not all processes in a group are killed, those remaining processes that expect to interact with one of the terminated processes may raise "broken pipe" or similar errors.

Notes on Debugging Child Processes Created with sys.executable

By default when debugging sub-processes is enabled, Wing replaces sys.executable to cover some of the common ways in which sub-processes may be launched, particularly on Windows. This can be disabled with the Debugger > Processes > Replace sys.executable preference.

On Windows this option should be disabled if the parent process launches children with a command line that contains a Handle created specifically for its child process, for example by setting hTargetProcessHandle in a call to DuplicateHandle. In this case, the handle will be invalid in the child because replacing sys.executable creates an intervening process and the child runs as the grand-child instead.

If a Handle is instead set to be inheritable for all child processes, for example by setting bInheritHandle in a call to DuplicateHandle, then replacing sys.executable will work without any problems.

Because the multiprocessing standard library module uses sys.executable to launch its children on Windows, this option must be enabled there in order to debug children created by that module.

Wing replaces sys.executable at startup only. As a result, user code that alters the value (other than by calling multiprocessing.forking.set_executable) will break debugging of child processes that are launched with a command line that contains sys.executable.

One way to work around cases where sys.executable replacement does not work is to manually initiating debug in the sub-process as described in Debugging Externally Launched Code.

Other Notes and Limitations

When debugging child processes created with the multiprocessing module, Wing will stop on exceptions raised in child processes. Continuing debug from that point will pack up and return the exception to the parent process, as in normal operation. Exceptions in children can be ignored with the Ignore this exception location checkbox in the Exceptions tool.

When child process debugging is enabled, Wing sets sys.executable so it bootstrap debugging in child processes. User code that invokes sys.executable to start a child process must also provide the environment variables starting with WINGDB_ to the child process. Otherwise, the debugger cannot determine which Python to run or how to connect to the IDE and the child process will fail to start.

If child processes are created with sys.executable the code that starts the child processes will need to correctly handle spaces in the path within sys.executable. Otherwise, child processes will fail to launch if Wing is installed into a directory path that has spaces in it and child process debugging is enabled.

Overriding the _bootstrap method of multiprocessing.process.Process (or multiprocessing.process.BaseProcess in Python 3.4+) in a custom process class will prevent Wing from stopping on exceptions in child processes unless the exception is propagated to the inherited method. A work-around for this would be to call logging.exception with any exception before sending it out to the parent process.

Some approaches to spawning child processes may result in the creation of intermediate processes that appear in Wing's process tree display. For example, using the shell=True option in subprocess.Popen will do this on Linux. When setting shell=False you may need to change the command passed to Popen to a list rather than a string.

Debug overhead may reveal timing bugs not seen outside of the debugger. For example, if a parent process may attempt to interact with a child process too quickly, causing problems only under the debugger. This is particularly likely on Windows, where there is an intermediate process created between the parent and child process.