This is a story (or a journey) about CloudFlare which saves my website from crashing. It’s all begin few months back when I decided to build up my online presence again (after the fall of ****.net, I’ll save it for another chapter).
Starting with a domain (this domain ayie.my, purchased and many domains afte that ) and a shared hosting (borrowed from a friend, syednazri.com. I could not afford a hosting that time) I start my build up again from ground up.
No traffic yet coming to the site, so no problem much, but the happy story about to become a nightmare for me as the trouble fire start to spark.
The Crawler Traffic
Be visible and searchable, I need my site crawled by search engine. Submitted my site to Google and few others to be crawled.
Everything was OK until my friend shared hosting was hitting 100% CPU and Memory Usage, which I traced and because of a website I was hosting together.
I decide to chop the website away and move it to another shared webhosting (which I got for free). Everything looks normal until …
An Army of Virtual Private Server
The following day, the second shared webhosting spiking on CPU and Memory usage again (my friends webhosting was ok since the migration). I suspect the website comes with bad PHP coding (It’s a website I got for free when taking over a domain), so I grab a VPS offer and move the site into it.
Why VPS? Because of the CPU and memory resources is higher then any shared hosting I can afford. It was looking good until…
Few days after that my VPS was timed out. Very odd, I had to reboot via console to get it back running and it happen again after that. My thought was a single 2GB 2vCPU VPS unable to handle the traffic and the processing, so i did this [Website Exceeding Web Resources. What Was My Action.].
Find The Cause and Block It
And I am wrong again … the server keeps dying and dying until I decide to take a peek into apache’s log (I should do it long time ago and avoid having sleepless nights). Damn you spiders!!
These spiders were basically crawling and making my resources spiking, requesting bogus URL. Since the website hosts dynamic contents, each page do take small amount of memory to generate. Multiply that by 15 000 request per hour. Man i was pissed, but nothing much I can do except blocking it.
The Firewall Win
Thank god the domain was managed by CloudFlare and CloudFlare do have Firewall features.
The ‘attack’ had decreased now to 9k+ request per 24 hours and it’s done neatly by CloudFlare.
Why not do blocking on server level? I do not want to waste precious VPS resources and reserve them for a legit request. Since it can be done on DNS level, it’s more practical to do so.
Now the website is running fine and 99.9% up time. Turns out only 1 VPS is enough to coup with the load. Now I got underutilized and un-utilized VPS, what should I do?
A new project arise and I’ll put it in another post.