A Tale of UDP tracking: Hytale server auto stop and start with systemd

This is a quick guide on how to auto-stop a Hytale server if no activity is seen after 120 seconds to pause the world time.

This article will be a bit different since I’ll first provide you the solution, and if you want, the next heading will have a walk through on the specifics of why I have created this solution

## Setup

Requirements:

This tutorial assumes you already have a server set and authenticated like explained at the Hytale server and you can start it with the java -jar command so we can focus on automating it’s start and stop.

Other requirements for this tutorial.

  • A working copy of the server-side software inside your $HOME. In this case, I’m hosting it at /home/nwildner/hytale so, you should adequate all data provided here to your working paths.
  • A Server/ directory inside this path containing the HytaleServer.jar and HytaleServer.aot files.

Step-by-Step:

All the steps here are executed as your normal unprivileged user (in my case, nwildner). You should only assume I’m using the root user if I explicitly tell you so.

  1. Create a systemd .socket unit which will be responsibe for tracking the first datagrams arriving on port 5520/udp: Command: systemctl edit --user --force --full hytale.socket
[Unit]
Description=Hytale UDP Server Socket

[Socket]
ListenDatagram=5520
Accept=no

[Install]
WantedBy=sockets.target
  1. Create a systemd .service that will be automatically started by the socket: Command: systemctl edit --user --force --full hytale.service
[Unit]
Description=Hytale Server
After=network.target

[Service]
Environment=HOME=/home/nwildner
WorkingDirectory=/home/nwildner/hytale
ExecStart=/usr/bin/java -XX:AOTCache=Server/HytaleServer.aot -jar Server/HytaleServer.jar --assets Assets.zip
TimeoutStartSec=300
Restart=no

[Install]
WantedBy=multi-user.target
  1. Create the watchdog.sh script which will be reponsible for tracking our user’s activity because UDP is not connection oriented. More details will be provided on the next topic, just trust the process here: vim ~/hytale/watchdog.sh.

Do not forget to add execute permissions after editing this script with chmod +x ~/hytale/watchdog.sh

#!/bin/bash
SERVICE="hytale.service"
SOCKET="hytale.socket"
TIMEOUT=120

# Get hytale.service PID
get_pid() {
  systemctl --user show --property=MainPID --value "$SERVICE"
}

# Return current UDP packet count (InDatagrams)
get_udp_packets() {
  awk '/^Udp:/ {getline; print $2}' /proc/net/snmp
}

# If there is no pid, do nothing
pid=$(get_pid)
[ "$pid" -eq 0 ] && exit 0

# Initialize date and the current amount of UPD
# datagrams transited
last_packets=$(get_udp_packets)
last_activity=$(date +%s)

while kill -0 "$pid" 2>/dev/null; do
  packets=$(get_udp_packets)

  # Compare current snapshot with last snapshot
  if [ "$packets" -gt "$last_packets" ]; then
    last_packets="$packets"
    last_activity=$(date +%s)
  fi

  # Calculate current time minus time last active
  current_time=$(date +%s)
  delta_time=$(( current_time - last_activity ))  

  # Stop service if timeout exceeded
  if [ "$delta_time" -gt "$TIMEOUT" ]; then
    echo "Idle timeout reached, stopping $SERVICE"
    systemctl stop --user "$SOCKET"
    sleep 10
    systemctl stop --user "$SERVICE"
    sleep 10
    systemctl start --user "$SOCKET"
    exit 0
  fi

  sleep 5

  # Refresh PID in case service restarted
  pid=$(get_pid)
done

You can also customize the TIMEOUT=120 variable to reflect how many seconds of inactivity will be needed for the server to be shutdown

  1. Create a systemd .service unit for our watchdog script: Command: systemctl edit --user --force --full hytale-watchdog.service
[Unit]
Description=Hytale Idle Watchdog

[Service]
Type=simple
ExecStart=/home/nwildner/hytale/watchdog.sh
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
  1. Last but not least, enable hytale.socket and hytale-watchguard.service systemd units:
$ systemctl --user enable --now hytale.socket
$ systemctl --user enable --now hytale-watchdog.service

Done. Your server is ready to start as soon as the first UDP Datagram starts on that port and to shutdown after 120 seconds of inactivity.

## Explanation - FAQ

  1. Why did you came up with this solution in the first place? Answer: This solution was crafted because currently the hytale server software lacks a mechanism to pause in-game time when no user is connected to the server. So, the solution was to shutdown the server gracefully.

  2. Why are you not using knockd? Answer: I have tried but I got some nasty errors where multiple servers were being launched since this is a UDP(quic) server. Also, it is more elegant to manage this as a service.

  3. Why are you tracking /proc/net/snmp instead of /proc/net/udp, and what are the drawbacks on that? Answer: fair enough. I’m tracking /proc/net/snmp because this is the only visible interface I have on Debian 13 to track how many udp datagrams transited this host without having to escalate to root user. /proc/net/udp is always showing zero for some reason and it is not available to the common user. The drawback is that we’re counting UDP datagrams globally and if you are hosting other softwares that use UDP on this same server (like a DNS server), your hytale server might not stop automatically since we’re not tracking “datagrams per service”.

  4. Why not use ss -u or tcpdump on even conntrack for this datagram count? Answer: ss is not able to show connections since it is UDP(connectionless) and only shows that the port 5520/udp is on LISTEN state by the java process. Also, using tcpdump or conntrack as a common user would require me to provide extra permissions to an unprivileged user, and that is something I was avoiding to do.

  5. Why not manage everything with systemd? Answer: while systemd is able to launch a service through a .socket with the ListenDatagram directive, it will not “track” that port anymore and his service is done afterwards.

  6. Why not use the watchdogh.sh script as a ExecStartPost= parameter of hytale.service? Answer: that’s because the script has a kill -0 loop that will be constantly running, and using this script as a ExecStartPost= parameter for the main service will keep this service in an activating state and not effectivelly starting. This was making the service to be restarted frequently by systemd internal algorithm to ensure the service reached the active state for obvious reasons.

  7. Why not user the Type=Forking on hytale.service with ExecStartPost= as explained on item 5? Answer: The script was being ignored when the service type was set to Forking.

  8. Why the watchguard.sh service has those nasty sleep entries between each service stop and why those exist? Answer: Those are needed because if you stop the service way to fast and almost simultaneously with the socket unit, the service will be brought up again by systemd internals. That’s why I’ve added that dramatic pause there.

  9. Why it always fail with “no server available” on the client side when I first try to connect and afterwards, I’m able to connect using my password? Answer: That’s because the hytale server takes about 15 seconds to fully initialize and the hytale client will timeout after 10 seconds of trying. I’ve already opened a feedaback on that matter inside the game.

That’s all folks. I hope you enjoy this solution.