Techniques

Monitoring tools

Monitoring can detect problems before end users notice them, help plan for growth and make troubleshooting easier when an unexpected problem does occur.

Use systems like Ganglia and Cacti to gather statistics from all servers and network devices. Building up graphs of these statistics allows you to see normal usage patterns and spot any spikes more quickly. These graphs also allow you to visualize what conditions were like during an incident that occurred temporarily.

Ensure log messages are being aggregated by Syslog. Using a tool like LogAnalyzer to visualize the log messages in colour makes it much easier to spot unusual activity.

Use an alerting tool such as Nagios or Icinga to monitor the graphs (see Ganglia-Nagios-Bridge) and logs (see Syslog-Nagios-Bridge)and also monitor the availability of services such as the SIP and TURN processes and the expiry dates of TLS certificates.

Appendix C, Configuring Nagios to monitor SIP, XMPP and TURN contains specific instructions to configure Nagios/Icinga to monitor SIP, XMPP and TURN servers.

Check the logs

Check /var/log/syslog and /var/log/daemon.log.

Many of the RTC processes can also create their own logfiles in other locations. Refer to the configuration files to see which type of logging each process uses.

The repro daemon web interface has a control on the Settings page where the log level can be changed at runtime without restarting the daemon.

Check the web interface

For those processes that have a web interface, this can be a useful way to see runtime activity at a glance.

The repro daemon web interface has a Registrations page where all known SIP registrations can be seen.

Operating system utilities

Example 5.2, “Inspecting DNS entries with dig demonstrates how to use the dig utility to verify that the DNS entries exist.

Use the netstat or lsof commands to verify that each process is listening on the correct IP addresses and ports.

If a process is failing to start and the reason is not clear in the log file or console output, use the strace utility to determine whether any syscall is failing.

the section called “Testing with s_client demonstrates how to use the OpenSSL s_client utility to make a test connection to the SIP proxy on the TLS and WebSocket over TLS ports.

Packet sniffers

Use tcpdump to capture the packets to a file on the server. Copy the capture file to a workstation and inspect it with Wireshark.

For SIP over UDP or TCP, without encryption, the ngrep utility can be useful for identifying packets containing a particular string.

Debugging mode

Run the process in debug mode, in the foreground on a terminal, to see what it is doing. Running the repro daemon this way will allow you to see the SIP messages in the console.

WebRTC and WebSockets

Both major browsers have a built-in troubleshooting system for WebRTC. In the Mozilla Firefox browser, enter the URL about:webrtc. In the Google Chrome browser, the URL is chrome://webrtc-internals.