Bug Hunting

In times of almost infinite computing power, it makes no sense to rely on hard copy and a magnifying glass when you're debugging a script. I'll look at the extensive Eclipse integrated development environment (IDE) that offers a number of very convenient debugging functions. To begin, you should download the Eclipse IDE for Java Developers [1] with the Eclipse installer – the steps that follow are based on the Oxygen 2 version from December 2017.

Next, get the shell extension from the Eclipse Marketplace [2] and drag and drop the Install button to start the shell extension installation. Make sure you do not place the button on the Welcome screen; use an empty editor page instead (Figure 1). The Marketplace client integrated into new versions of Eclipse then automatically starts installing the plugin.

Setting Up Eclipse

Annoyingly, the Eclipse plugin uses the Dash shell by default, which cannot compare with Bash in terms of ease of use. To change this, click Window | Preferences | Shell Script | Interpreters and check bash (Figure 2).

The next step is to create a new shell script project under File | New | Other. This container converts shell scripts to a format that the Eclipse IDE understands. The newly created project has no content at the beginning, so right-click it and select the New | Other option to add a new script.

Although you can edit shell scripts using the syntax highlighting feature of Eclipse and run them locally, you cannot set breakpoints at the moment. To do so, you first need to download and extract the BashEclipse ZIP archive from SourceForge [3]; while the IDE is shut down, move the two JAR files to the /home/<home>/java-oxygen/eclipse/dropins directory.

Note that Eclipse tends to exhibit several bugs when using OpenJDK, so make sure you run Eclipse in an Oracle JDK environment to avoid them.

Changing Paths

Eclipse and BashEclipse communicate via _DEBUG.sh, which contains:

exec 33<>/dev/tcp/localhost/33333
function _________DEBUG_TRAP ()
{
    local _________DEBUG_COMMAND
    read -u 33 _________DEBUG_COMMAND
    eval $_________DEBUG_COMMAND
}
set -o functrace
trap _________DEBUG_TRAP DEBUG
This code uses the operating system's <C>trap<C> feature to set up a function that takes incoming commands from the remote debugger and parses them in the open session. Although this setup is not necessarily secure, as long as you're working on a secure network, it should not be too risky.

After copying the file into the project directory, create the script you want to try out. In the following steps, the test object is a group of Bash-specific language constructs:

. _DEBUG.sh
echo $SHELL
a=1234567890
echo ${a}
echo ${a:3}

Of particular importance here is the inclusion of the debug script, which establishes a connection between the shell and the development environment.

In the next step, you need to create an execution configuration by clicking Run | Debug Configurations and create a new configuration based on the Bash script template. The execution configuration generator known from the normal Eclipse workflow then shows you a setup wizard, which you parameterize (Figure 3). By the way, the debugger port is always 33333 – at least as long as you don't customize _DEBUG.sh.

Integrating the Last Eclipse Extension

In the next step, click on Debug in Eclipse to run the program. Note that the IDE itself is not able to launch the small script. Instead, switch to the Debugger perspective by clicking the icon in the upper right corner of the Eclipse workspace; then, open a new shell window in which you request the execution of the script by entering bash ./<script_name>:

tamhan@TAMHAN14:~/workspace$ bash ./firsttest.sh
/bin/bash
1234567890
4567890

In the present execution configuration, Eclipse reacts by displaying the error message Unable to open editor, unknown editor ID, because the Bash debugger requires a specific editor path on the Eclipse side.

The solution to this problem is to visit the discontinued SourceForge website of the ShellEd project [4]. Ignore the hints that the information contained there is outdated and download the net.sourceforge.shelled-site-2.0.3.zip file.

In the next step, open Eclipse and choose Help | Install new software. In the dialog that appears, you need to press the Add button next to the address bar to open the window for adding local repositories. Use the Location field to add a link to the archive.

After confirming, the installation process starts, which requires a restart of the IDE and outputs some warnings about obsolete certificates. From this point on, you can debug scripts in the same way you would debug Java or C code.

Troubleshooting at the Command Line

Working in Eclipse is convenient but not always possible in practice (e.g., on a server without a graphical environment). Even though the Eclipse debugger is able to contact shell scripts running on external servers, it is not a very convenient approach. Bash Debugger is a better alternative; you can install this on your workstation by typing:

sudo apt-get install bashdb

After successfully completing the installation, you can instrument the execution of scripts by prefixing them with the bashdb command. To launch the script created above, which resides in the Eclipse workspace, enter:

tamhan@TAMHAN14:~/workspace$ bashdb ./firsttest.sh
bash debugger, bashdb, release 4.2-0.8
...

The classic bashdb step-by-step debugger comes with more than two dozen commands. A complete list can be found on the corresponding SourceForge page [5]. However, the command line of the program is a bit confusing; for example, look at the following extract:

:
3: a=1234567890
bashdb<2>

For each line, the debugger informs you which script you are currently in and which statements the line contains. Below this is the bashdb prompt for the input of commands.

The value in the angle brackets, normally only a consecutive number, is bracketed multiple times if the program flow is nested deeper in stack frames.

If you want to walk through the code step by step, you can do so by entering next several times (Figure 4). Remember that bashdb does not terminate after program execution; you need to press q and Enter to exit the debugger mask.

Step-by-Step with Exit Option

Developers brought up on the GNU Debugger know that step-by-step debuggers implement a comparatively complex command line, from which the user can also obtain information about the content of the currently loaded variable. The current content of a variable can be output, for example, with the examine command if you know the variable name:

bashdb<6> examine a
declare -- a="1234567890"

Bash scripts can be nested through functions and loops. The script below executes a wildcard calculation within a function:

function worker {
for i in `seq 1 10`;
do
echo "worker says: " $i
done
}
echo "go!"
worker
echo "end!"

If you run this program in bashdb, you will notice that you only have to enter next three times. The entire body of worker is processed in one step. Control only returns in the echo "end!" line.

Special features of bashdb are the alternative step and finish commands. For lines like for i in `seq 1 10`; that comprise multiple statements, the step command tells the debugger to go one step deeper into your program.

Each line of the program is only accessed once, so for the loop, you toggle between the for and echo statements. If you get bored with this procedure at some point, you can tell bashdb to leave the current execution context by typing finish bashdb. However, this must be a file loaded into the script or a called function – at the moment, it is not possible to break out of a "classic" loop in this way.

Working with Breakpoints

Linear code execution will tend to become annoying if your shell script contains several hundred or even several thousand lines. Breakpoints let you work around this problem. Their behavior in bashdb is pretty much what you will be used to with debuggers for other programming languages.

Developers have two options at this point: You can create the desired breakpoints after starting the bashdb session, or you can use bashdb-trace. In the first option, the debugger recognizes various breakpoint declarations that let you address line numbers, files, and function names. This first example sets a breakpoint in the worker function:

9: echo "go!"
bashdb<0> break worker
Breakpoint 1 set in file /home/tamhan/workspace/functest.sh, line 1.

The following three statements set breakpoints variously by line number:

break 28
Breakpoint 2 set in file parm.sh, line 28.
break parm.sh:29
Breakpoint 3 set in file parm.sh, line 29.
break 28 if x==5
Breakpoint 4 set in file parm.sh, line 28.

The first statement sets a breakpoint in the specified line of the currently opened file. The second statement demonstrates how to address "remote" files – unfortunately, it is not possible to specify function names here. The third statement shows a conditional breakpoint, which is only activated when its condition is met. In all cases, a program with breakpoints is run by entering cont:

bashdb<1> cont

After this command, the code runs until bashdb detects the activation of a breakpoint. At that point, you can either take action by typing next or start another pass by entering cont. If you are interested in specific changes to variables, you will want to use watchpoints instead, which you can set up with watch. The procedure is the same as for classic breakpoints.

Much Code, Much Work

Permanently setting new breakpoints is too much work. The save command for Python databases, gdb, and the like are not available in bashdb. To work around the problem, the debugger statements go in the shell script, which additionally makes sense because executing a script in the debugger slows things down significantly. Although this might be irrelevant for a small script, anyone who has analyzed a complete compile process will be familiar with the waits, even on fast machines.

In the first step, you need to load the bashdb-trace file. The path printed here is correct for my workstation, but may be different on yours:

source /usr/share/bashdb/bashdb-trace
function worker {
. . .
}
echo "go!"
worker
_Dbg_debugger ; : ; :
echo "end!"

The debugger is then launched by _Dbg_ debugger ; : ; :. According to the documentation, the need for the two "dead" semicolons is not quite clear, probably even to the bashdb authors.

Note in both bashdb and Eclipse that Bash scripts while running are shell scripts and remain so. If, for example, you delete a file during debugging, it cannot be recovered afterward.

Conclusions

Manual searches for problems in Bash scripts is a waste of precious time in the age of eight-core workstations. If you work with shell scripts, you will definitely want to invest some time studying the tools. With the widespread distribution of Bash, however, it is not possible to provide a complete overview of debugging in the context of a short article. For further information, you can search phrases such as shell script debugger (e.g., in Stack Overflow [6]) to find a number of discussions on various methods.

As another alternative, Microsoft's JavaScript-based Visual Studio Code development environment has gained massive support in the area of shell script programming. The plugin provided in Visual Studio Marketplace [7] supports integration with bashdb, which saves time and nerves in the presence of a GUI stack.