OVH Support and useless firewall against other internal servers
This post is about being able to jump the OVH Firewall simply using a server inside OVH itself.
Lessons learned in short:
1) OVH does not have any kind of internal network isolation, so every other OVH customer can contact your servers directly (!!!) by simply poiting your public IP. You’ll not find this small “detail” reported anywhere.
2) TO install and maintain a firewall on your operating system (windows or linux doesn’t care) it’s absolutely mandatory: never, never, never rely exclusively on OVH firewall. Yes, it’s powerful enough to stop DDoS attacks, trigger mitigations and BGP reroute, but it’s totally transparent (maybe it’s not involved in the routing ?) when the attacking IP comes from inside OVH. So it’s useless even to stop a silly RDP “wannacry-like” attack spread from some other OVH customer.
3) The OVH support does not necessary know how the things work on their side. They might come up to you with a random answer. It’s up to you to find out how the things really work, despite what they tell you.
4) The “Abuse Team” is not able to STOP other servers from attacking you, expect, I guess, when the attack involves a huge network traffic that makes a big and red button flash somewhere. At least I hope. What the Abuse team is able to do, for sure, is simply contact the IP reported in the “abuse form” to ask the offending IP to kindly “stop” any action, even WITHOUT VERIFYING if there’s really something wrong happening (!!!) between who reported the supposed abuse.
Here below you can find the complete history of the support tickets… from the beginning to the last one…
29/06/17 18:40 From me to OVH Support:
Hello, could you please tell me if OVH has a schedule that opens 3-4 simultaneous RDP connection attempts, trying to log on all my virtual servers under my private cloud, using ADMINISTRATOR as username and some wrong password, every single minute, 24h/day ? Thank you.
29/06/17 23:01 from OVH:
Hello,
Apologies for the delayed response. I’m not aware of anything off the top of my head that might be doing this. Can you please send me some logs that show
this is happening and I’ll see what I can to do to find out if it is us and ifwe can resolve the resolve the issue.
Let me know and I’ll aim to get this resolved as quickly as possible.
Thank you,
Craig
30/06/17 06:43 from me:
Dear Craig,
I was able to packet capture some requests received from one of my servers, last ones came from IP 37.187.48.9 and IP 51.254.79.9 and they tried several logons on RDP trying to connect to my virtual server with IP XX.XX.XX.XX and using “ADMINISTRATOR” as username with a wrong password.
I don’t know how they are able to reach my servers, considering the OVH firewall rules are set to deny any kind of traffic and allow RDP only from my home IP address.
I like to understand what’s happening.
Best Regards,
Luis.
30/06/17 10:39 from OVH:
Hello,
Thank you for that. I was in contact with our PCC team and they said with confidence that this not related to us in any way.
Please contact abuse@ovh.net and they will assist you in stopping this.
Best Regards,
Craig
01/07/2017 18:15 from me:
Dear Craig,
I sent an email to them as per your request, waiting for their answer. In the meanwhile, could you please confirm me that every other customer inside OVH network can reach my virtual servers network “jumping” the external OVH firewall rules and doing network tcp and port scans ? Because I need to take proper actions to improve my security and I really wonder how that IP’s are reaching my servers. This is a very important security detail I would need to know. Thanks.
03/07/2017 10:10 from OVH:
Hello,
They should not be able to do that. People on OVH would have to go through the public firewall
if they are trying to access your server. If they can, it would be a bug on our end.
The abuse team will be able to provide you with more information on your problem and be able to resolve it for you.
Thank you for choosing OVH,
Craig
14/07/2017 12:25 from me:
Dear support,
Kindly notice that:
– As per your suggestion, I sent an email to abuse@ovh.net regarding the two IP’s I was receiving RDP connection attempts from, but nobody
answered still at today, after 14 days – however, after a few hours I sent the emai, without doing anything, all at a sudden there were no
more RDP connection attempts to all of my servers, at the same time.
– Even if you say that “should not be able to do that” (and I agree with you), I inform you that it is unfortunally currently possible to do
that… and I think this is a huge security issue in OVH because **it is currently possible** to reach every OVH customer server, jumping the OVH
firewall, simply pointing the public IP of the machines, without doing anything special. I just tried again right now with a friend that has a
server in OVH Spain, and he can start an RDP session to all of my servers, even if RDP port is closed on OVH firewall (!!!!). It’s very
easy to test. Just try it… Please tell me if this is considered “normal behaviour” for OVH, so I can take appropriate actions on my side
to better protect my servers.
Awaiting some news,
Luis
Il 03/07/2017 11:10, OVH Customer Support Service ha scritto:
> Hello,
> They should not be able to do that. People on OVH would have to go through the public firewall if they are trying to access your server.
> If they can, it would be a bug on our end.
> The abuse team will be able to provide you with more information on your problem and be able to resolve it for you.
>
> Thank you for choosing OVH,
>
> Craig
14/07/17 15:07 from OVH:
Hello,
Can you please send me the email that you sent to the abuse team and I’ll contact them to get it resolved more quickly.
In regards to the RDP Requests getting through the firewall. That shouldn’t be happening. Can you please send me some proof of this and I’ll do what I can to
get this resolved.
Best Regards,
Craig
14/07/17 15:08 from me:
Dear Craig,
Please find below the email I sent on 30 June to abuse.
Regarding the RDP connection:
– Screenshot of current OVH firewall settings for IP of my server XX.XXX.XX.XXX
https://www.dropbox.com/s/XXXXXX
– Screenshot of telnet from a computer OUTSIDE OVH network trying to
connect to TCP RDP port (connection gets stuck, this is OK, OVH firewall
is blocking correctly):
https://www.dropbox.com/s/XXXXXXX
– Screenshot of telnet from a computer INSIDE OVH network that gets a
response to RDP (this is not fine, OVH firewall is being jumped) – this
works also from IP’s of different OVH countries, like when my friend
tried from Spain.
https://www.dropbox.com/s/XXXXXXXX
– Packet capture where you can see RDP connections reaching my server
(search for IP 37.187.48.9 and IP 51.254.79.9 for example)
https://www.dropbox.com/s/XXXXXXXX
Luis.
——– Messaggio Inoltrato ——–
Oggetto: security issue trough firewall
Data: Fri, 30 Jun 2017 16:59:10 +0200
Mittente: Luis Dragotto <XXXXX@XXXX.XX>
A: abuse@ovh.net
Hello,
As per your collegue request trough the ticket I opened from my panel, I’m forwarding this information to you.
Kindly notice that I’m receiving to almost all my virtual servers hosted in my private cloud on OVH (for example, one of my server’s IP is
XX.XX.XX.XX) several RDP connection attempts with username “ADMINISTRATOR” and a wrong password at a rate of nearly 4 request per
minute from IP 37.187.48.9 and IP 51.254.79.9, which are unknown to me, and both belonging to OVH.
Awaiting your news,
Luis.
Luis.Il 14/07/2017 16:07, OVH Customer Support Service ha scritto:
> Hello,
>
> Can you please send me the email that you sent to the abuse team and
> I’ll contact them to get it resolved more quickly.
>
> In regards to the RDP Requests getting through the firewall. That
> shouldn’t be happening. Can you please send me some proof of this and
> I’ll do what I can to get this resolved.
>
>
> Best Regards,
>
> Craig
19/07/17 15:20 from OVH:
Hello,
Apologies for the delayed response. I’m am still waiting for a reply from our abuse team in regards to saying this issue is resolved.
Are you still receiving the connection attempts? If so, can you send me the IP addresses that are making the request and I will investigate it further.
Best Regards,
Craig
19/07/17 15:35 from me:
Dear Craig,
Thank you for the update, I hope they’ll answer soon as this seems a very bad security issue for OVH.
I turned on a windows firewall to block the requests so now I’m fine, but the moment I turn it off, in a couple of hours the connection attempts with “administrator” starts again, nearly 4 at a time togheter, every minute.
To be able to catch the IP addresses I should start a packet capture and filter among all the other traffic, because windows doesn’t log the IP automatically (unfortunally) on RDP failures… if you need some more source ip’s I can do some packet capture and analize the results during the weekend.
Let me know,
Luis.
19/07/17 15:54 from OVH:
Hi Luis,
If you have the time to do a packet capture that would be perfect. Once I know where the requests are coming from and if it’s in our network,
I can do something about it. Please send them in whenever you can.
Thank you,
Craig
19/07/17 16:11 from me:
Dear Craig,
I just created a firewall log of discarded packets, it’s updated once every minute more or less, when the firewall buffers out. You can find it here:
http://XX.XX.com/fwdrops.txt
There you can already see some ip’s who are reaching my server, jumping the OVH firewall (you should not consider the UDP traffic attempte to 224.0.0.252 which is “normal” for windows).
For example, just in a few minutes I’m already receiving connection attempts to TCP port 445 from IP:
46.105.42.91
37.187.71.195
158.69.151.73
178.33.108.220
147.135.228.100
Also notice one month ago I had a server hit by a “NSA randsomware”, I never understood from where they came from, now I know. In that case I had a backup and I was able to recover the server and install any updates to prevent future problems.
Best,
Luis.
19/07/17 17:10 from OVH:
Hi Luis,
Yeah, it looks like it’s coming in through the network alright. That is a very strange issue.
I contact the abuse team and they told me that you should fill out the following form:
https://www.ovh.ie/abuse/
And paste in the logs of each IP. Once that is done, message me and I’ll get it fixed.
Thanks,
Craig
19/07/17 17:32 from me:
Ok, done. Abuse report reference #PVRKZWNFXB
Thanks,
Luis.
20/07/17 09:43 from OVH:
Hi Luis,
Thank you for that. I have spoken with the abuse team and it will be investigated and repaired with high priority.
Kind Regards,
Craig
25/07/17 16:08 from me:
Hi Craig,
Have you got some news about this issue ? I’m still receiving more then 650 connection attempts daily from different IP’s all coming from OVH
network…
Kind Regards,
Luis.
25/07/17 16:36 from OVH:
Hi Luis,
I would have assumed it was resolved by now. Let me contact the abuse team and see if I can find out more.
Hopefully I’ll get it sorted quickly. I’ll inform you.
Kind Regards,
Craig
25/07/17 16:39 from me:
Thank you, in case you need it I moved the firewall logs here, always updated every minute.
http://www.XXXXX.it/fwdrops.txt
Kid Regards,
Luis.
Il 25/07/2017 17:36, OVH Customer Support Service ha scritto:
> Hi Luis,
>
> I would have assumed it was resolved by now. Let me contact the abuse
> team and see if I can find out more.
>
> Hopefully I’ll get it sorted quickly. I’ll inform you.
>
>
> Kind Regards,
>
> Craig
26/07/17 13:51 from OVH
Hi Luis,
Thanks for that. I have been in contact with the abuse team and they said they messaged you.
If you’ve need any help from me, let me know and I’ll do what I can to assist you.
Kind Regards,
Craig
26/07/17 14:02 from me:
Dear Craig,
I confirm the abuse team contacted me … asking ME to investigate and fix the intrusion behaviour I’m doing (!!!!) in violation of OVH terms of service, and including as a proof … the same reports I sent to them to proof I’m receiving attacks from a list of IP’s.
Honestly, I don’t know if laugh or cry.
Kind Regards,
Luis.
26/07/17 16:34 from OVH:
Hi Luis,
I’m sorry about that. It is indeed crazy. I was talking to one of our network
engineers about the issue and he was saying that the because the IPs are
internal, they’re not blocked by the OVH Firewall as that protects against
outside traffic. The best course of action to protect against these
connections would be to disable RDP on these machines. If RDP is necessary,
only whitelist the necessary IPs. This will resolve the issue.
We’re also going to look into the ips attempting to connect to your server and
get them removed. But I really would consider turning off Remote Desktop if it
is not necessary.
Kind Regards,
Craig
26/07/17 16:48
Dear Craig, thanks for the answer. It’s crazy that everybody who has a server inside OVH can do TCP port scan to every other OVH server and try to attack every server, and most of all it’s incredible that the most effort the abuse team can do is … to forward the email to the alleged attacker asking him to kindly stop it 🙂
I tought internal OVH network was much more controlled, secured and isolated… I’ll take note of it and rise up adeguate security rules on my servers.
Thank you and best regards,
Luis.
27/07/17 15:40
Hi Luis,
Our internal network blocks out malicious attacks. The reason why this wasn’t
the most threatening email sent from the abuse team that we have to account
for the risk that the customer is legitimate and just typed in an incorrect
IP. If we detect anything that poses a serious threat to our services or
customers, we stop it immediately and close the customers account/services. I
think in this case, because it RDP attempts with no password attempts, I think
they were less malicious.
Anyway, I’m really sorry that it went on for too long. I was told to pass it
to the abuse team and I’ll know better than to do that in the future. If
you’re having any further problems or issues in the future and let me know and
I’ll get it resolved much much quicker than this.
Kind Regards,
Craig
Related
Comments
Copyright @ 2018 - RAMBO. Designed by Webriti
Andrés | May 18,2020
Any solution for this? I use Ubuntu. They attack me through TCP.
Luis Dragotto | Jul 20,2020
yes, you have to mount your own firewall… under Ubuntu, I suggest ufw (apt-get install -y ufw)
Andy | Jan 19,2021
Hey Luis,
I don’t know whether to shout at you for being such a patient dude or to simply loose my faith in humanity!
Surely you have taken your custom elsewhere right? Tell me you have, please!
By saying that “we only send soft warning to alleged offenders is completely missing the point – HELLO! Offending or not, there IP’s should NOT have access to your machines.
Don’t get me started on the customer “service” provided by OVH, I’ve yet to find anyone that has any good thing to say about them.
Let me go and urge my client to liquidate on OVH and shift to AWS (yes they own the world but at least it’s a secure world!)
Luis Dragotto | Jan 19,2021
Hello Andy. At the end, I moved some of my services outside, but still have some others in OVH. They’re the cheaper ones with the highest DDoS protection in the world, and unfortunately they have no competitors still in 2021. AWS is very good, but it can get very complicated to configure (at least initially), and they are one of the more expensive out there – but they have really really damn good engineers on the customer support side. OVH improved in the meanwhile their support packages too, thank god, and they now have standard, premium, business and enterprise, with the business (for 360 usd/month) having a real 24/7 incident management (finally). In my case, with AWS I should spend around 4x what I’m currently paying to OVH, to have exactly the same things, so I’m forced to stay with OVH and cross my fingers that everything will run smooth. A good additional DDoS protection, but just for some kind of configurations (not my case), is now possible with CloudFlare, and at an affordable price (around 200 usd/month)… so it’s easy to combine CloudFlare with some other provider to add that DDoS protection extra layer that normally costs around 3K usd/month just “to start talking”. Regarding the problem of customer support not being able to provide a decent support level to customers has become, sadly very common since several years now, and it’s getting worse year after year. I’m just trying to get used to it: my only way to fight it is to provide the best support I can to my customers while waiting to see improvements somewhere else. But, honestly, I’m still waiting… 🙁
Othman | Apr 11,2021
@Luis Thank you very much for writing this post and for your steeled patience.
Unfortunately we are experiencing a similar issue but ours is from external (at least currently). Our OVH server keeps getting hit by malicious attacks on port 445 (SMB), 3389 (RDP), and 5985 (RDP via HTTP) from all sorts of public IPs (I suspect the source IPs are spoofed though) constantly throughout the day every minute (or several times per minute). OVH have offered very potato support to resolve this issue, and have even sent as an email stating that we are violating conditions because we are port scanning others (We’ve opened at least 5 tickets with them asking them to help us investigate the large number of port scans against our server!).
There is some evidence also to suggest that OVH has been shutting down our server when a large DDoS attack against it happens… It is partially understandable since they would want to protect their infrastructure from using up too much resources, but at the same time it is rather problematic, considering they don’t notify us officially of these hard resets.
All in all, after a lot of packet captures, firewall log analysis, etc, we have disabled RDP and opted to use Teamviewer to login remotely, and blocked the three ports mentioned above. We will also be moving some of our firewall rules from our OVH server to the OVH firewall, in hopes that OVH will less likely reboot our server (a forum somewhere implied that OVH shutsdown servers to maintain its ‘integrity’ if it is being heavily attacked by DDoS).
Luis Dragotto | Apr 11,2021
Dear @Othman , thanks for your comment.
It’s almost 8 years I’m with OVH, I can assure you I’ve been gone trough several DDoS attacks, and I never had any server shut down by them. But I was using a PCC (private cloud), which recently I had to switch with single VM’s inside a public cloud project. At the end, the best thing to do is to use a combination of both OS Firewall (windows firewall, or ufw on linux) together with their OVH Firewall, and then setup a custom VPN Server (I made one with ubuntu) to access your internal private network. Under Windows you need to play a little to to separate “Private” and “Public” networks, so you can restrict RDP/SMB only on the internal/private network, and block it on the “public” interface. This way you can stop OVH customers trying to access your servers, and get the DDoS protection from the OVH Firewall from the “outsiders”, which you can also keep it “always on” when you see several attacks coming trough (otherwise, you’ll need to wait for the triggers to realize, which sometimes could not run immediately). The email you received from OVH stating that you were violating conditions because of port scanning seems the same I received back in 2017 when I wrote this post. Looks they haven’t improved at all, under this point of view.