TRS-0000

The packet-forwarder dies and is not restarted

Symptoms

After some time of running, there is no more packet received on the server side. The network is configured correctly and the server is running.
After investigation, it appears the packet-forwarder is not running anymore.

The LORIOT agent is not concerned by this problem

Explication

This is most likely due to the fact the Semtech packet-forwarder and its derived have not been originally created for long and stable production capable setup.
The main intention was to provide a tool to help developers creating their own LoRa server solution. The idea being to report errors and not to run for a long time, the packet-forwarder simply terminates on most errors even minor.

The LORIX One supports the LORIOT agent which is more adapted for professional usage. If well monitored, the Semtech UDP packet-forwarder works also fine, security put aside (no encryption).

Solution 1

Since the packet-forwarder needs monitoring to work over a long period of time, we added a process monitoring tool in the LORIX OS which also allows you to read the forwarder's logs in real time.
Among being very convenient for installation troubleshooting, it also ensures the forwarder you may chose is always running and restarts it in case of unexpected termination.

The best option is then simply to update your LORIX One to LORIX OS following this documentation.

Solution 2

If you can't update to the LORIX OS, Monit can manage the UDP packet-forwarder and restart it in case of unexpected termination.

To configure this, you will have to:

1. Stop and disable the packet-forwarder with the clouds-manager
2. Install monit
3. Confgure monit to start and monitor the packet forwarder

Stop and disable the packet-forwarder

To do so, run the clouds-manager with the following commands:

sudo /etc/init.d/clouds-manager.sh stop
sudo /etc/init.d/clouds-manager.sh configure

==========================================
| LORIX One clouds manager configuration |
==========================================

Actual configuration:
  autostart=true
      cloud=packet-forwarder

Do you want to enable autostart at boot time?
[Yes|No]
 > no

Which cloud app. do you want to use ?
[loriot|packet-forwarder|ttn|manual]
 > packet-forwarder

New configuration:
  autostart=false
      cloud=packet-forwarder

At the configuration stage, disable auto the autostart answering no, the cloud app. selection doesn't matter since we will not manage it anymore from this script.

Since the forwarder is not managed anymore by the clouds-manager.sh script, all the commands /etc/init.d/clouds-manager.sh like status, stop and start should not be used to avoid any conflict with Monit.

Install monit

Please follow the instruction as explained here.

Configure monit

Create the Monit script /etc/monit.d/forwarder.monit and add the following text inside:

check process forwarder with pidfile /var/run/<forwarder ID>.pid
    start program = "/etc/init.d/<forwarder ID> start"
    stop program = "/etc/init.d/<forwarder ID> stop"

Replace the <forwarder ID> text by:

All is in place now to reload and start monitoring the forwarder using Monit, run the following command to make Monit aware about the new script:

$ sudo monit reload
Reinitializing monit daemon

That's it! The forwarder is now monitored. You can consult its status:

$ sudo monit status
Monit 5.25.2 uptime: 3h 5m

Process 'forwarder'
  status                       OK
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  pid                          2145
  parent pid                   1
  uid                          0
  effective uid                0
  gid                          0
  uptime                       1m
  threads                      1
  children                     0
  cpu                          0.0%
  cpu total                    0.0%
  memory                       2.2% [2.6 MB]
  memory total                 2.2% [2.6 MB]
  security attribute           -
  data collected               Mon, 15 Jun 2020 11:12:00

System 'sama5d4-lorix-one-512'
  status                       OK
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  load average                 [0.00] [0.00] [0.00]
  cpu                          0.1%us 0.1%sy 0.0%wa
  memory usage                 12.9 MB [10.7%]
  swap usage                   0 B [0.0%]
  uptime                       3h 9m
  boot time                    Mon, 15 Jun 2020 08:02:58
  data collected               Mon, 15 Jun 2020 11:12:00

You can also start or stop this Monit service using the commands sudo monit start forwarder and respectively sudo monit stop forwarder. 
In all case, it will autostart on boot and restart the process each time it dies.