tl;dr: Somebody social engineered their way into my hosting provider, failed to steal funds, and then deleted everything (including backups).
medium_of_exchange was hosted in the cloud, on a bare metal server at Vultr (vultr.com). About three weeks ago, I was working on the node and suddenly the API began returning errors that the wallet wasn't unlocked. Immediately I became on high alert, and soon it was obvious that the node had been compromised. The attacker quickly locked me out of the server.
I tried to log into Vultr, but the credentials had been changed. The email was also changed, so it was impossible to regain control of the account. I reached out to support and indicated that it was a security emergency, but they nevertheless took about 16 hours to respond.
Meanwhile, once the attacker realized that he was unable to take funds from the lightning node (because the wallet was locked), he decided to delete everything and hold my data for ransom. He emailed me the next day to discuss getting my data back for a fee.
When Vultr support finally got back to me, they realized they had made a mistake and restored access to my account. Of course by then it was way too late. The hacker had already deleted all the servers, including the backup server which had my static channel backups.
While the issue is still under investigation with law enforcement and the full story is forthcoming, it appears that what happened was a high degree of sophisticated social engineering. There's no evidence that any software systems were compromised or any 0-days used. However, through the clever use of multiple identities, the attacker was able to get enough information about me and my vultr account (directly from vultr) to be able to convince vultr that they were in fact me, and I had lost access to my original email and password and 2fa.
Of course, the natural question is, why didn't vultr at least email me to check whether the story they were getting from this third party was legitimate? This simple action seems like it would have gone a long way towards mitigating this attack. In fact, it seems there is virtually nothing that I could have done to be securely hosted at Vultr. I had 2 factor authentication enabled and my email was secure. Once somebody targeted me, Vultr was helpless to stop them from taking over my account because they don't have secure policies in place. My only conclusion is that it must be considered an insecure place to host a server. I will never host another server there.
I did not pay any ransom to the attacker. I'm pleased to say that through publically available tools (specifically chantools by guggero), I have been able to recover the vast majority (95%+) of funds I had in channels. For the handful of channels that remain open, if you own the other side, please force close or contact me.
It was perhaps an obvious mistake to host a large amount of funds on a platform like Vultr that doesn't take security too seriously. Hopefully anyone that reads this will take care to evaluate all aspects of their security setup and further similar attacks can be reduced in the future.
I won't be deterred by this. I will be back! Though you may not know which node is me.
questions/concerns? I can be reached at