OVH Support and useless firewall against other internal servers

This post is about being able to jump the OVH Firewall simply using a server inside OVH itself.

Lessons learned in short:
1) OVH does not have any kind of internal network isolation, so every other OVH customer can contact your servers directly (!!!) by simply poiting your public IP. You’ll not find this small “detail” reported anywhere.

2) TO install and maintain a firewall on your operating system (windows or linux doesn’t care) it’s absolutely mandatory: never,  never, never rely exclusively on OVH firewall. Yes, it’s powerful enough to stop DDoS attacks, trigger mitigations and BGP reroute, but it’s totally transparent (maybe it’s not involved in the routing ?) when the attacking IP comes from inside OVH. So it’s useless even to stop a silly RDP “wannacry-like” attack spread from some other OVH customer.

3) The OVH support does not necessary know how the things work on their side. They might come up to you with a random answer. It’s up to you to find out how the things really work, despite what they tell you.

4) The “Abuse Team” is not able to STOP other servers from attacking you, expect, I guess, when the attack involves a huge network traffic that makes  a big and red button flash somewhere. At least I hope. What the Abuse team is able to do, for sure, is simply contact the IP reported in the “abuse form” to ask the offending IP to kindly “stop” any action, even WITHOUT VERIFYING if there’s really something wrong happening (!!!) between who reported the supposed abuse.

Here below you can find the complete history of the support tickets… from the beginning to the last one…

29/06/17 18:40 From me to OVH Support:

Hello, could you please tell me if OVH has a schedule that opens 3-4 simultaneous RDP connection attempts, trying to log on all my virtual servers under my private cloud, using ADMINISTRATOR as username and some wrong password, every single minute, 24h/day ? Thank you.

29/06/17 23:01 from OVH:

Hello,

Apologies for the delayed response. I’m not aware of anything off the top of my head that might be doing this. Can you please send me some logs that show
this is happening and I’ll see what I can to do to find out if it is us and ifwe can resolve the resolve the issue.

Let me know and I’ll aim to get this resolved as quickly as possible.

Thank you,

Craig

30/06/17 06:43 from me:

Dear Craig,
I was able to packet capture some requests received from one of my servers, last ones came from IP 37.187.48.9 and IP 51.254.79.9 and they tried several logons on RDP trying to connect to my virtual server with IP XX.XX.XX.XX and using “ADMINISTRATOR” as username with a wrong password.

I don’t know how they are able to reach my servers, considering the OVH firewall rules are set to deny any kind of traffic and allow RDP only from my home IP address.

I like to understand what’s happening.

Best Regards,

Luis.

30/06/17 10:39 from OVH:

Hello,
Thank you for that. I was in contact with our PCC team and they said with confidence that this not related to us in any way.
Please contact abuse@ovh.net and they will assist you in stopping this.
Best Regards,
Craig

 

01/07/2017 18:15 from me:

Dear Craig,
I sent an email to them as per your request, waiting for their answer. In the meanwhile, could you please confirm me that every other customer inside OVH network can reach my virtual servers network “jumping” the external OVH firewall rules and doing network tcp and port scans ? Because I need to take proper actions to improve my security and I really wonder how that IP’s are reaching my servers. This is a very important security detail I would need to know. Thanks.

 

03/07/2017 10:10 from OVH:

Hello,

They should not be able to do that. People on OVH would have to go through the public firewall
if they are trying to access your server. If they can, it would be a bug on our end.

The abuse team will be able to provide you with more information on your problem and be able to resolve it for you.

Thank you for choosing OVH,

Craig

 

14/07/2017 12:25 from me:

Dear support,

Kindly notice that:

– As per your suggestion, I sent an email to abuse@ovh.net regarding the two IP’s I was receiving RDP connection attempts from, but nobody
answered still at today, after 14 days – however, after a few hours I sent the emai, without doing anything, all at a sudden there were no
more RDP connection attempts to all of my servers, at the same time.

– Even if you say that “should not be able to do that” (and I agree with you), I inform you that it is unfortunally currently possible to do
that… and I think this is a huge security issue in OVH because **it is currently possible** to reach every OVH customer server, jumping the OVH
firewall, simply pointing the public IP of the machines, without doing anything special. I just tried again right now with a friend that has a
server in OVH Spain, and he can start an RDP session to all of my servers, even if RDP port is closed on OVH firewall (!!!!). It’s very
easy to test. Just try it… Please tell me if this is considered “normal behaviour” for OVH, so I can take appropriate actions on my side
to better protect my servers.

Awaiting some news,

Luis

Il 03/07/2017 11:10, OVH Customer Support Service ha scritto:
> Hello,
> They should not be able to do that. People on OVH would have to go through the public firewall if they are trying to access your server.
> If they can, it would be a bug on our end.
> The abuse team will be able to provide you with more information on your problem and be able to resolve it for you.
>
> Thank you for choosing OVH,
>
> Craig

14/07/17 15:07 from OVH:

Hello,

Can you please send me the email that you sent to the abuse team and I’ll contact them to get it resolved more quickly.
In regards to the RDP Requests getting through the firewall. That shouldn’t be happening. Can you please send me some proof of this and I’ll do what I can to
get this resolved.

Best Regards,

Craig

 

14/07/17 15:08 from me:

Dear Craig,
Please find below the email I sent on 30 June to abuse.

Regarding the RDP connection:

– Screenshot of current OVH firewall settings for IP of my server XX.XXX.XX.XXX
https://www.dropbox.com/s/XXXXXX

– Screenshot of telnet from a computer OUTSIDE OVH network trying to
connect to TCP RDP port (connection gets stuck, this is OK, OVH firewall
is blocking correctly):
https://www.dropbox.com/s/XXXXXXX

– Screenshot of telnet from a computer INSIDE OVH network that gets a
response to RDP (this is not fine, OVH firewall is being jumped) – this
works also from IP’s of different OVH countries, like when my friend
tried from Spain.
https://www.dropbox.com/s/XXXXXXXX

– Packet capture where you can see RDP connections reaching my server
(search for IP 37.187.48.9 and IP 51.254.79.9 for example)
https://www.dropbox.com/s/XXXXXXXX

Luis.

 

——– Messaggio Inoltrato ——–
Oggetto: security issue trough firewall
Data: Fri, 30 Jun 2017 16:59:10 +0200
Mittente: Luis Dragotto <XXXXX@XXXX.XX>
A: abuse@ovh.net

 

Hello,
As per your collegue request trough the ticket I opened from my panel, I’m forwarding this information to you.

Kindly notice that I’m receiving to almost all my virtual servers hosted in my private cloud on OVH (for example, one of my server’s IP is
XX.XX.XX.XX) several RDP connection attempts with username “ADMINISTRATOR” and a wrong password at a rate of nearly 4 request per
minute from IP 37.187.48.9 and IP 51.254.79.9, which are unknown to me, and both belonging to OVH.

Awaiting your news,

Luis.

 

Luis.Il 14/07/2017 16:07, OVH Customer Support Service ha scritto:
> Hello,
>
> Can you please send me the email that you sent to the abuse team and
> I’ll contact them to get it resolved more quickly.
>
> In regards to the RDP Requests getting through the firewall. That
> shouldn’t be happening. Can you please send me some proof of this and
> I’ll do what I can to get this resolved.
>
>
> Best Regards,
>
> Craig

19/07/17 15:20 from OVH:

Hello,

Apologies for the delayed response. I’m am still waiting for a reply from our abuse team in regards to saying this issue is resolved.

Are you still receiving the connection attempts? If so, can you send me the IP addresses that are making the request and I will investigate it further.

Best Regards,

Craig

 

19/07/17 15:35 from me:

Dear Craig,
Thank you for the update, I hope they’ll answer soon as this seems a very bad security issue for OVH.

I turned on a windows firewall to block the requests so now I’m fine, but the moment I turn it off, in a couple of hours the connection attempts with “administrator” starts again, nearly 4 at a time togheter, every minute.

To be able to catch the IP addresses I should start a packet capture and filter among all the other traffic, because windows doesn’t log the IP automatically (unfortunally) on RDP failures… if you need some more source ip’s I can do some packet capture and analize the results during the weekend.

Let me know,

Luis.

19/07/17 15:54 from OVH:

Hi Luis,

If you have the time to do a packet capture that would be perfect. Once I know where the requests are coming from and if it’s in our network,
I can do something about it. Please send them in whenever you can.
Thank you,

Craig

19/07/17 16:11 from me:

Dear Craig,
I just created a firewall log of discarded packets, it’s updated once every minute more or less, when the firewall buffers out. You can find it here:

http://XX.XX.com/fwdrops.txt

There you can already see some ip’s who are reaching my server, jumping the OVH firewall (you should not consider the UDP traffic attempte to 224.0.0.252 which is “normal” for windows).

For example, just in a few minutes I’m already receiving connection attempts to TCP port 445 from IP:
46.105.42.91
37.187.71.195
158.69.151.73
178.33.108.220
147.135.228.100

Also notice one month ago I had a server hit by a “NSA randsomware”, I never understood from where they came from, now I know. In that case I had a backup and I was able to recover the server and install any updates to prevent future problems.

Best,

Luis.

 

19/07/17 17:10 from OVH:

Hi Luis,

Yeah, it looks like it’s coming in through the network alright. That is a very strange issue.

I contact the abuse team and they told me that you should fill out the following form:
https://www.ovh.ie/abuse/

And paste in the logs of each IP. Once that is done, message me and I’ll get it fixed.

Thanks,

Craig

 

19/07/17 17:32 from me:

Ok, done. Abuse report reference #PVRKZWNFXB

Thanks,

Luis.

 

20/07/17 09:43 from OVH:

Hi Luis,

Thank you for that. I have spoken with the abuse team and it will be investigated and repaired with high priority.

Kind Regards,

Craig

25/07/17 16:08 from me:

Hi Craig,
Have you got some news about this issue ? I’m still receiving more then 650 connection attempts daily from different IP’s all coming from OVH
network…

Kind Regards,

Luis.

25/07/17 16:36 from OVH:

Hi Luis,

I would have assumed it was resolved by now. Let me contact the abuse team and see if I can find out more.

Hopefully I’ll get it sorted quickly. I’ll inform you.

Kind Regards,

Craig

 

25/07/17 16:39 from me:

Thank you, in case you need it I moved the firewall logs here, always updated every minute.

http://www.XXXXX.it/fwdrops.txt

Kid Regards,

Luis.

Il 25/07/2017 17:36, OVH Customer Support Service ha scritto:
> Hi Luis,
>
> I would have assumed it was resolved by now. Let me contact the abuse
> team and see if I can find out more.
>
> Hopefully I’ll get it sorted quickly. I’ll inform you.
>
>
> Kind Regards,
>
> Craig

 

26/07/17 13:51 from OVH

Hi Luis,

Thanks for that. I have been in contact with the abuse team and they said they messaged you.

If you’ve need any help from me, let me know and I’ll do what I can to assist you.

Kind Regards,

Craig

 

26/07/17 14:02 from me:

Dear Craig,
I confirm the abuse team contacted me … asking ME to investigate and fix the intrusion behaviour I’m doing (!!!!) in violation of OVH terms of service, and including as a proof … the same reports I sent to them to proof I’m receiving attacks from a list of IP’s.

Honestly, I don’t know if laugh or cry.

Kind Regards,

Luis.

 

26/07/17 16:34 from OVH:

Hi Luis,

I’m sorry about that. It is indeed crazy. I was talking to one of our network
engineers about the issue and he was saying that the because the IPs are
internal, they’re not blocked by the OVH Firewall as that protects against
outside traffic. The best course of action to protect against these
connections would be to disable RDP on these machines. If RDP is necessary,
only whitelist the necessary IPs. This will resolve the issue.

We’re also going to look into the ips attempting to connect to your server and
get them removed. But I really would consider turning off Remote Desktop if it
is not necessary.

Kind Regards,

Craig

 

26/07/17 16:48

Dear Craig, thanks for the answer. It’s crazy that everybody who has a server inside OVH can do TCP port scan to every other OVH server and try to attack every server, and most of all it’s incredible that the most effort the abuse team can do is … to forward the email to the alleged attacker asking him to kindly stop it 🙂

I tought internal OVH network was much more controlled, secured and isolated… I’ll take note of it and rise up adeguate security rules on my servers.

Thank you and best regards,

Luis.

 

27/07/17 15:40

Hi Luis,

Our internal network blocks out malicious attacks. The reason why this wasn’t
the most threatening email sent from the abuse team that we have to account
for the risk that the customer is legitimate and just typed in an incorrect
IP. If we detect anything that poses a serious threat to our services or
customers, we stop it immediately and close the customers account/services. I
think in this case, because it RDP attempts with no password attempts, I think
they were less malicious.

Anyway, I’m really sorry that it went on for too long. I was told to pass it
to the abuse team and I’ll know better than to do that in the future. If
you’re having any further problems or issues in the future and let me know and
I’ll get it resolved much much quicker than this.

Kind Regards,

Craig