I’m out of time and luck. Bad guys got into my simple-help 5.1.8 and then onto multiple servers. First business I’m looking at they changed my domain admin password and encrypted drives. I’m screwed.
Do I just purchase the current version of SH and install on top of the existing (offline) install?
Just wanna say, ouch, sorry to hear that.
I’ve found that upgrading is pretty much exactly what you said… But I’d be very protective of how I bring that whole server online, and I’d utilize IP allow-listing as much as possible for a bit. If they got in to the point that they could grab the servers keys, they could potentially poison the DNS of your clients if they had that much access and still have the ability to use your clients persistent installs.
I’d probably be trying to wipe out any traces of Simplehelp at my clients and starting over myself. But maybe I’m overly cautious.
Of course, if they encrypted drives, and changed admin, … sorry for your weekend.
I’d also suggest restarting from scratch, you can’t trust that server anymore.
If you selfhost, consider hosting on a security hardened machine, ie GitHub - netinvent/el_scripts: Script collection to enhance Linux security / initial configuration & automate install process for example which hardens RHEL clones and Debian.
I’d also suggest you restrict technician & admin tasks to certain IPs (or VPN into your host server if you need access from everywhere).
Have to ask, but I assume they managed to logon to SimpleHelp technician and then onto servers via the console? Did you have 2FA enabled and if so, then all of us should be worried.
Restricting login by IP would eventually require me to drive to physical locations when my IP changes. I’ve got 17 businesses, the majority without VPN’s setup because I’d be the only one using it.
They used the SimpleHelp vulnerability to get into my technician console. Trying to figure out how they reset admin password showed me Terminal Mode didn’t require logging into the server and Net User made it trivial for them to get in. I’m flabbergasted Terminal didn’t require credentials. 2FA for Windows server is something I’d never heard suggested and was not using.
All Friday/Saturday and half of Sunday to restore, cleanup and reinstall servers and workstations. More than anything was the embarrassment of the situation. Thankfully I run UrBackup on my servers and nothing important was lost. I’m just glad MY server isn’t on SimpleHelp or I’d be out of business.
From what I understand they get into your server. Once on the server though, if they wanted to, they could have probably changed your client install configurations (Added additional servers to your persistent clients, etc.), so that is something to look out for. Having access to the server also means that they could have pulled down your server key, which means that if the client reaches another server with that key it would communicate with it. This would either be done by adding another server to the client or by poisoning the DNS used by the client. I’ve not heard of any cases of this, just saying what I think are possible attack consequences.
99% of my persistent client installs are also protected by a second password which I think would have protected me from any of the client concerns, if I had seen any signs of breach, which I have not on my system. That second password is required for connecting OR changing client configurations, and I’m sitting happy right now that I took that extra step. Nothing I’ve seen indicates that an attacker could have gotten past that.
In addition to keeping my clients protected with a second password on the persistent client, I also tend to keep my server offline / blocked at the firewall when I’m not using it. It’s extra work, but… yeah…
Yes, this is my concern as well. Embarrassing at best, lost clients/income at worse and even worse, sued. I’m very disappointed that SimpleHelp didn’t reach out to us directly via email or that the technician client didn’t inform me that there was a new version. I usually act on the later.
It’s equally very embarrassing for SimpleHelp but IMO the best approach is to be open. The wording on the new version isn’t strong enough either:
Security Improvements
- This release includes critical security fixes and is strongly recommended for all users. Find out more.
I would say it should be more than strongly recommended especially as there’s now evidence of the flaw being exploited.
It’s totally wrong that this has only been brought to my attention because you, an end user, posted here and I happen to have notifications setup. So thank you but less so to SimpleHelp
I have signed up multiple times to get informed on new releases, somehow it doesn’t work.
I check the site regularly for updates, although I was informed about this particular issue and updated right away. Perhaps we should consider one of those services that notify us if a particular page changes.
I have enabled 2FA, it’s been enabled for a long time.
I signed up with a free service called smtp2go to get notified if during login the password is in incorrect.
My main server is my home pc so when I don’t need it it’s off. I did, for a period, played with the firewall rule to turn it on and off as needed but it’s not practical. I do have a secondary server that is always on, I should probably remote in that server with a third party service, start the SH server…etc. but that would get kk
Besides turning off the SH when it’s not needed not sure what else can we do.
I don’t see how 2FA would have prevented this. File transfer mode, no logon needed. Command line access, reset admin password, encrypt files on servers and workstations without needing to log on.
net user administrator *
arp -a
net use x: “\ip\c$” administrator
(new password)
copy the bad file to workstation, run it
do it again
Same here but the screen you get to now looks different/new so I’m hopeful.
I thought of the VPN access for admin & technician access only (restrict technician / admin connections to private VPN IP). Clients will not need vpn and still connect via pubic IPs.
Basically, I have tech/admin restricted to IP’s. I am rarely at the office and mobile all the time so I use tailscale and have a couple linux based VPS boxes in the cloud to create my own private VPN and I have the admin/tech restricted to those IP’s. It works well for me.
Another suggestion I would recommend to anyone is to set up notifications for most everything (under Alerts tab → Events) - especially ‘server configuration changes’ and tech logins/logouts and access/support connections (including failed logins and failed MFA).
Yes, you may be inundated with emails while going about your own business (just delete them), but hopefully you will at least get some type of notice that something is going on, if you know you (or whoever else you might allow access) are not doing anything with the server or clients. That might give you time to at least remote in to your server to kill it or shut it down until you can physically get to it, get it offline, and take further action.