Eyes on the Network

Administrators are well advised to monitor the availability and quality of their networks continuously. The open source ntopng tool has been tried and tested for years. In this article, I investigate how to commission the latest Enterprise version and explore its feature set.

Ntopng was originally developed by Luca Deri, a scientist at the University of Pisa, under the name ntop [1], which explains why the business still operates under the name "ntop di Deri Luca." The name ntop is derived from the Unix top program, which lets network administrators view system information related to CPU and memory usage and the currently running processes of a Unix system.

In this vein, ntopng is a network top program that lets admins display all the relevant parameters for the connected networks. Ntopng is a passive network monitoring tool that supports statistical evaluation of traffic data on the connected networks; it does not actively intervene in the network traffic (but see the "Layer 7 Manipulation" box). Ntopng is therefore ideally suited as a tool for administrators wanting to answer, among others, the following questions:

• What devices are currently on the network?
• How much traffic do the various devices cause on the network?
• Which devices are communicating or exchanging data with others (internally and externally)?
• What kind of bandwidth is used by each device, or which device is currently hogging the Internet connection?
• What protocols exist on the network, and how is network traffic distributed among them?
• Is any suspicious data traffic on the network caused by, for example, viruses or Trojans?

Ntopng is ideally suited for monitoring small and medium-sized Class C networks at gigabit speeds but can also be used for monitoring larger networks, given appropriate hardware.

Open Source Editions

The Community edition already contains ntopng's most important features. Armed with the free version, you can analyze network traffic on up to 32 network interface cards in real time; identify application protocols such as BitTorrent, Facebook, Dropbox, and YouTube; and generate alerts (e.g., if a system is using too much bandwidth).

The commercial editions (see the "Versions and Licensing" box) offer five days of installation support, support for up to 128 network interface cards (Enterprise), and, above all, the possibility of permanently storing analysis data with the additional n2disk module, which is the only way to evaluate historical data. The ability to connect to third-party systems such as Nagios, Icinga, and Suricata or integrate with LDAP (for single sign-on authentication at the web interface) is reserved for the commercial versions. By the way, universities, educational and scientific research institutions, and nonprofit organizations can obtain licenses for all ntop products free of charge. Details of the requirements and registration can be found online [3]; also see the "Versions and Licensing" box.

Network Architecture

To give ntopng a wide view of network traffic, it makes sense to connect the system to the mirror port on the core switch by way of an (additional) network interface card. Otherwise, ntopng only sees the communication of its own computer and its communication partners.

The recommendation is to install ntopng on a computer with two network interface cards, one of which is used to collect network data and the other to manage the system itself. Usually you will not mirror all the ports of a switch to the mirror port – only the uplink to the Internet – so it is usually fine to monitor only the port to which the firewall is connected.

System Requirements

Ntopng is available for x64 Linux and Windows systems; the 32-bit architecture is no longer supported. An ARM version for the Raspberry Pi and Raspbian operating system is also available. Unfortunately, the Windows version lacks some important features for filtering network traffic, such as time and traffic quotas. More importantly, Windows does not provide for permanent storage of the analysis data with n2disk.

Ntopng focuses on two important Linux distribution branches: Debian/Ubuntu and Red Hat/CentOS. The packages required for ntopng are easily installed after setting up the operating system with the distributions' built-in tools (apt/deb or yum/rpm). A preconfigured distribution or appliance is not available.

Installation and Commissioning

For the Linux distributions, including Raspbian, ntop provides a separate repository each for stable and nightly builds [6]. The nightly builds are more suitable for admins who like to try out the latest features. For production use, the stable versions are the more obvious choice. On Ubuntu 18.04 LTS, the installation comprises the few simple commands shown in Listing 1.

sudo apt install software-properties-common wget
sudo add-apt-repository universe
wget http://apt-stable.ntop.org/18.04/all/apt-ntop-stable.deb
sudo apt install ./apt-ntop-stable.deb
 **
sudo apt clean all
sudo apt update
sudo apt install pfring nprobe ntopng ntopng-data n2disk cento nbox

The "Supernode Remote Access" box describes how ntopng sets up remote support.

First Launch and Web Interface

Ntopng launches automatically after installation; you can access the web interface directly over http://<IP address>:3000. Unfortunately, the web server launches with an unencrypted connection; this setup is not a good idea for security reasons because ntop is delivered with self-signed SSL certificates that throw warning messages in the browser. If you prefer to run ntopng under an official domain name, you can secure the SSL/TLS connection with an official certificate.

The free Let's Encrypt CA is a good way to do this. A step-by-step guide can be found in a blog post online [7]. For internal use, self-signed certificates are certainly justifiable: Disable the unencrypted connection and enable the SSL-protected web server by adding the following lines to the /etc/ntopng/ntopng.conf file,

--http-port=0
--https-port=443

and restarting ntopng by typing:

sudo systemctl restart ntopng

To avoid conflicts with other web servers, you will want to make sure first that another web server is not already using TCP port 443 on the ntop server. If this cannot be avoided, you can move the ntopng HTTP server to another port above 1024 (e.g., use --https-port=4443 and then log in with the URL https://<IP address>:4443).

The first time you log in with the admin username and password, you will be prompted to set a new password. Because the admin user cannot be disabled, you will want to set a very secure password for this user and define a second user with admin rights for administration work later on. Click on the gear (Settings) icon top right on the dashboard (Figure 1) to enter user management and create a new user. From the dashboard, you can assign to each user individually the network interfaces or networks they are permitted to evaluate.

On the Command Line

You can also start ntopng with the desired parameters directly at the command line. For an overview of the available options, type ntopng -h. For example, if you want ntopng to monitor only certain network interfaces (by default, all network interface cards are monitored), simply specify them with the -i parameter:

ntopng -i en01 -i enp3s0

You can just as easily disable DNS resolution completely, prevent automatic logout from the web interface, output a list of the application protocols recognized by ntopng as follows

ntopng -n 3
ntopng -q
ntopng --print-ndpi-protocols

and much, much more.

Working with ntopng

Ntopng offers an enormous range of functions. For an easier start, I first take a look at the basic concepts, each of which is represented by its own menu item in the web interface. Under the Interfaces menu, ntopng lists all network interfaces built into the hardware, including not only physical interfaces like LAN and WiFi network interface controllers (NICs), but also logical interfaces like the loopback adapter.

In Hosts | Host Pools, ntopng monitors the traffic on the selected network interface cards and identifies systems that are sending and receiving packets on the network. Each IPv4 and IPv6 address that ntopng identifies is therefore interpreted as a separate host. The IP address tells ntopng whether it is a local or remote host. For local hosts only, additional information (e.g., the web pages visited, DNS requests, and the Layer 7 protocols used) are recorded.

The hosts can be grouped in host pools (Figure 2) so the cumulated traffic can be evaluated across a group of hosts. The filters in the Host menu let you further restrict the information displayed (Figure 3). In addition to hosts and host pools, MAC Addresses, Countries, and Operating Systems are also available as filters.

A "flow" in ntopng-speak is a bidirectional communication channel between two hosts. The Flows menu displays the data traffic between two hosts in real time (Figure 4). Moreover, ntopng can react to certain events and sound an alarm if previously defined threshold values are exceeded. For example, ntopng generates an alert as soon as a new device appears on the network for the first time, a status change on a switch port is detected by SNMP, or a system contacts a malware host.

Additionally, all activities of users logged on to the web interface are also registered as alarms. In this way, failed logon attempts, for example, can be traced. Alerts are displayed on the Alerts dashboard (reached from the Alerts icon – the warning sign – at the top) but can also be processed further – more about this later.

Traffic

Now that you are familiarized with the basic functions of ntopng, I'll take a look at the system in practical use. When you log in to the web interface, ntopng loads the traffic dashboard by default (Figure 1). All the information visible in the web interface always relates to the current state of the network. In the table on the left side of the screen you can see the "top talkers" (local and remote), which are the systems currently causing the most traffic.

The two upper graphs on the right side show the current load for the selected network interfaces and break down the traffic by applications (e.g., SMB, TLS, POP). The two lower graphs contain the same data, but for the last 24 hours. You can access the data from previous periods by clicking on the Traffic icon and selecting the Traffic Report option. When you get there, you can either specify the exact time period you want to analyze or use shortcuts to select, say, an hour, day, or month as the period.

Traffic Profiles

With the help of traffic profiles, network traffic can be evaluated in a very specific way at both the network and application level. For example, if you want to know how much traffic an individual (or all) host generates specifically through the use of YouTube, Facebook, or Wikipedia, traffic profiles are the right tool for this. You can also check how much traffic a particular MAC address is causing or what device on the network is still communicating with the outdated and insecure SMBv1 protocol.

The syntax for traffic profiles is based on the Berkeley Packet Filter (BPF) format, which tcpdump also uses. You will find the traffic profiles in the Settings menu; some examples are also listed here. The application protocols detected by ntopng can be determined by entering the command:

ntopng --print-ndpi-protocols

The desired application protocols are referenced in traffic profiles by means of the corresponding index number. A traffic profile for identifying the data stream to the Wikipedia website can be created with the l7proto 176 command, but the network traffic between the IP addresses 192.168.0.9 and 192.168.0.224 can be aggregated as follows:

ip host 192.168.0.9 and 192.168.0.224

Ntopng applies the traffic profiles to the flows; accordingly, the evaluation feature is found in the Flows menu.

Settings and Interfaces

The Interfaces menu lets you change the settings for a network interface. The interface page shows status information on the current traffic, and you can return to this view at any time by clicking the Home icon. To make sure that ntopng saves network traffic permanently, press the hard disk button (fifth from the right) and check the Continuous Traffic Recording box.

After successfully starting recording, a blue hard disk icon appears in the Status tab. You can specify here how many gigabytes of data you want ntopng to store at the most. Directly below this field you will also find details of the volume of data ntopng has already collected or how much disk space has already been used. Ntopng relies on an independent module named n2disk, for which you need a separate license, to record the data traffic. For information on the license, look under Settings | Traffic Recording | License in the main menu. N2disk is only available for Linux installations.

Another important point in the network interface settings is the threshold values for generating alerts on the Alerts dashboard. In the interface settings, click on the icon with the warning label and set the desired thresholds for which ntopng should generate an alert. You can set your own thresholds for each time interval (every minute, every five minutes, hourly, daily). Don't forget to save the changes afterward by pressing the Save Configuration button.

Hosts and Host Pools

The Hosts menu lets you access all devices that generate traffic on the network from a list. The filter settings above the column headers of the table let you restrict the view to local hosts and IPv4 hosts to improve clarity. Clicking on the column headers sorts in ascending or descending order; clicking on a host's IP address calls up a detailed view of the system in question. As in the settings for an interface, the icon with the warning sign lets you define thresholds for generating alerts and, of course, retrieve information on network traffic and the protocols used.

One interesting option is the ability to group hosts in host pools. From the default Home view of the Host Details page, click the gear icon next to the Host Pool label and then click Edit Host Pools. After you have defined groups such as Servers and Clients, you can assign individual hosts to these groups. Under Hosts | Host Pools, you can then display the volume of network traffic a group has generated in total. By the way, the Autonomous Systems item in the Hosts menu are remote hosts that ntopng has assigned to a specific network provider such as Google, Hetzner, and the like according to their IP addresses.

Flows

Flows – active connections between clients and servers – are displayed in a table, as in the Host view. As before, you can further hone the list of flows with the filters on the right to improve clarity. In addition to known filters such as local or remote Hosts and IP Version, the Flow view provides additional filters for the application protocols currently in use. Applications shows all the protocols currently being used in flows. The list becomes even clearer if you filter for network applications with Categories. Ntopng then shows the categories used by the devices, such as Cloud, Chat, and Web.

Alerts

As mentioned earlier, ntopng can generate alerts for certain events and for cases when thresholds are exceeded or not reached. Although supported events (e.g., adding a new device or contacting a malware host) are hard-coded in the program, you can set threshold alerts individually.

All alerts are displayed on the Alerts dashboard (Figure 5), where they can be filtered by time period and whether local or remote for a better overview. A click on the respective host IP address takes you to detailed information on the respective alert. Ntopng can also display all alerts in tabular form with Detected Alerts.

Furthermore, ntopng can both display the alarms in the web interface and forward them to third-party applications. The currently supported crop of applications is email, Slack, Syslog, Nagios, and web hooks. The web hook option provides a universal HTTP interface for encoding alerts as JSON messages and passing them to an HTTP endpoint. Depending on certain alerts, the web hook can be used to control systems with a RESTful API (e.g., firewall or network management systems) and trigger certain responses.

Conclusions

Ntopng helps administrators monitor their networks and provides detailed information on bandwidth usage and the protocols and applications used, as well as deep insights into network traffic. The tool is indispensable for troubleshooting network problems.

The cost of Professional and Enterprise versions are very manageable, and both offer considerable advantages over the free Community edition. Unfortunately, the Windows version contains significantly fewer features than the Linux version.