Linux SimpleHelp Server tuning sharing experience

Server Administration
1

Hello,

Running a SimpleHelp server with about 700 permenant hosts, I’ve had a lot of trouble with performance.
Here are some of the work I’ve done to improve speed:

  1. The bad proxy
    Most of my clients are behind Watchguard firewalls, which tend to have http/https proxy rules.
    I’ve managed to create rules for SimpleHelp (http/tcp, http/udp, https/tcp and https/udp) that bypass the http(s) proxy of the UTM firewalls, which resolved some of the speed issues I had.

  2. The server itself
    My server has a public IP, and the only firewall between simplehelp and the internet is the linux firewall that comes with CentOS 7.

I still tried to sqeeze the most out of the server by modifying the following :slight_smile:

2.1. General tuning
There’s a system tuning tool in CentOS called tuned which has a network-latency preset so I gave that a try:

yum install tuned
systemctl enable --now tuned
tuned-adm profile network-latency

The network-latency presets can be found in /usr/lib/tuned/network-latency/tuned.conf

2.2 kernel
CentOS 7 comes with kernel 3.10, whereas some major tcp performance enhancements came with 4.9 series, so I upgraded the stock kernel using the elrepo provided kernels.

yum install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install kernel-ml
reboot

2.3. Tcp scheduler
In order to achieve better performance, I switched the tcp scheduler from cubic to bbr (Bottleneck Bandwidth and RTT)

Added the following two lines in /etc/sysctl.conf
net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr

Applied by running the following command:
sysctl --system

2.4. Other tcp settings
Providing that my simplehelp clients may have various MTU settings, I decided to set another batch of tcp stack optimizations in /etc/sysctl.conf
net.ipv4.tcp_mtu_probing=1
net.ipv4.tcp_slow_start_after_idle=0

Applied by running the following command:
sysctl --system

Now here I am with a (hopefully) well tuned server for running SimpleHelp.
This setup may still be widely improved regarding IRQs, hardware polling, cpu sticky processes, changing firewalld backend to nftables…

I’d love to hear about your experiences and performance tips for running a smooth SimpleHelp server too :slightly_smiling_face:

Best regards.

2

Is the low performance noticeable while accessing over the internal network?
How much memory do you have allocated under the “Server Health” tab that you can find under the Administration tab? Please check it as it can be on the “red”
Can you give us more info about the server specs?
We have an in-house server running ubuntu server and the latest simplehelp business edition… and with hundreds of clients we still didn’t noticed any degradation.

3

Our experience: We’ve been running SH on a linux (ubuntu server) box on digital ocean for about 4 or 5 years now. The box has 2GB ram (although it sits at about 50% utilization). We have anywhere from 700 - 800 online hosts at a time with a total of around 1500 in all. We often have 5 simultaneous connections going, we have several alert policies in place also. We have not had any issues with performance or slowness.

4

Sorry if it wasn’t clear in my first message, I was talking about network latency only.
My server has 4 vCPU 4GB of RAM, 1000Mbits enterprise grade network and no IOPS problems.
Simplehelp is granted 2GB of memory, which is really doesn’t consume.

I usually have between 10-15 simultaneous connections.
My technicians tell me it’s sometimes really slow operating remote connections, even though the client network has optical fiber links and QoS set for SimpleHelp, and my network monitoring does not show any incidents.

I’m searching all solutions to tune the environment my SimpleHelp instance is running in.

No one noticed some regular lags on their remote connections ?

5

We only use 5 simultaneous connections, so can’t comment on that. You’ve done quite a lot of tuning, and I’m a Debian guy, so I don’t have much to say on that issue. Have you tried having the techs that are experiencing lag switch to a direct udp connection to see if dropping the server out of the middle fixes the issue?
https://simple-help.com/technician-guide#Chat-and-Tools

6

I’m still investigating (having outruled a lot of stuff like my BGP routers, my internal network, my hypervisors that have the SimpleHelp VM).
I’m currently doing a lot of SimpleHelp RDP tunnel tests since the RDP session goes black on even the tiniest lost packet.
I’ll report back once I have outruled everything I can think of.