Image not found 404 - hot linking + server load

Kayz

Founder license
License owner
Joined
Nov 2, 2013
Messages
78
Points
58
I have a lot of images which i've had to delete and start afresh unfortunately. The problem i'm facing now is those websites which embedded (hot linked) my hosted images on their websites are no longer able to display them.

However every time the hotlink loads on the 3rd party website it is affecting my servers load. I didn't mind the bandwidth use so much, i also had cloudflare to save me some bandwidth.

I realised if an image is deleted and it is embedded on some popular websites then it will make a lot of call backs to my server.

My host has informed me the site was getting a lot of 404 requests due to images being removed, so they suggested i create a simple ErrorDocument 404 "<H1>Page not found</H1>" in my htaccess alleviate the load.

Yes i can disable hot linking, but this will mean people will not be able to embed onto their websites.

Is there any other way i can alleviate the load my server when the 3rd party website is calling an image which it cannot find on my server?
 

Kayz

Founder license
License owner
Joined
Nov 2, 2013
Messages
78
Points
58
Disable server access (referrals) websites that colapse your website.
It's not collapsing the site, but rather making it sluggish. You're though that is an option....

Do you suppose if i increase the cache life time on cloudflare it will still continue to display the file even if it is not on the origin server?
 

HenrysCat

Core license
License owner
Joined
Apr 18, 2017
Messages
92
Points
58
Location
127.0.0.1
Website
imgrpost.com
I added this to my .htaccess file
Code:
RewriteEngine on
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !-f
RewriteRule .*\.(gif|jpe?g|png|bmp) 404.gif [NC,L]
It displays the 404.gif if a hot linked image is missing, I also made it about 3k in size to alleviate the problem you are experiencing.
 

javajiver

Network license
License owner
Joined
May 20, 2019
Messages
17
Points
53
You probably at least need a VPS or a dedicated solution so you are not abusing resources on shared web hosting.
 

Kayz

Founder license
License owner
Joined
Nov 2, 2013
Messages
78
Points
58
I added this to my .htaccess file
Code:
RewriteEngine on
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !-f
RewriteRule .*\.(gif|jpe?g|png|bmp) 404.gif [NC,L]
It displays the 404.gif if a hot linked image is missing, I also made it about 3k in size to alleviate the problem you are experiencing.
Thank's, i think this is what i needed! Can i cache this file somewhere or better host the 404.gif somewhere else!? :p


You probably at least need a VPS or a dedicated solution so you are not abusing resources on shared web hosting.
Thanks have been on a dedicated server for many years now.
 

Rodolfo

Chevereto Developer
Chevereto Staff
Joined
Oct 7, 2008
Messages
15,526
Points
237
Location
Chevereto HQ
Website
rodolfoberrios.com
A PHP 404 will really slow your website but Chevereto doesn't use any of these for the uploaded content. In fact, this is the expected behavior when you enable not found image replacement: https://demo.chevereto.com/images/2019/05/21/KVSBPjhkjhkjhj0013P-1.md.jpg

By default, Chevereto doesn't include this setting turned on and like I said, it doesn't serve 404 using PHP. The default .htaccess

.htaccess:
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule images/.+\.(gif|jpe?g|png|bmp) - [NC,L,R=404]
^ This tells Apache to simply issue a txt 404 response.

Since Chevereto doesn't use any backend for delivering these 404, I assume that you implemented that on your own, maybe you skipped some server rules (apache or nginx) or the problem is that the traffic is so huge that even returning these 404 is eating your network. That's why I suggested you to block all that traffic right away.

In any case, the problem you are exposing here is entirely server related which is not something that we could help you that much anyway.
 

HenrysCat

Core license
License owner
Joined
Apr 18, 2017
Messages
92
Points
58
Location
127.0.0.1
Website
imgrpost.com
Thank's, i think this is what i needed! Can i cache this file somewhere or better host the 404.gif somewhere else!? :p
How many hits on missing images are you getting? would be interesting to know, I use Awstats and I get around 100-150k per month and my little 404.gif (actually its 2.5k) makes no difference, and I'm on a VPS not dedicated.

 

Kayz

Founder license
License owner
Joined
Nov 2, 2013
Messages
78
Points
58
A PHP 404 will really slow your website but Chevereto doesn't use any of these for the uploaded content. In fact, this is the expected behavior when you enable not found image replacement: https://demo.chevereto.com/images/2019/05/21/KVSBPjhkjhkjhj0013P-1.md.jpg

By default, Chevereto doesn't include this setting turned on and like I said, it doesn't serve 404 using PHP. The default .htaccess

.htaccess:
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule images/.+\.(gif|jpe?g|png|bmp) - [NC,L,R=404]
^ This tells Apache to simply issue a txt 404 response.

Since Chevereto doesn't use any backend for delivering these 404, I assume that you implemented that on your own, maybe you skipped some server rules (apache or nginx) or the problem is that the traffic is so huge that even returning these 404 is eating your network. That's why I suggested you to block all that traffic right away.

In any case, the problem you are exposing here is entirely server related which is not something that we could help you that much anyway.
My host informed me i was getting a lot of 404 requests, they then implemented the following in in my htaccess

Code:
 ErrorDocument 404 "<H1>Page not found</H1>"
This alleviated the load somewhat.

So are you saying if i enabled a not found image replacement it would create server load? Even if it was hosted elsewhere?

Excuse me for my ignorance

Whats the difference between what you have suggested

Code:
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule images/.+\.(gif|jpe?g|png|bmp) - [NC,L,R=404]
and

Code:
ErrorDocument 404 "<H1>Page not found</H1>"
are they both not text 404?

I have not implemented any 404 until my host suggested for me to use: ErrorDocument 404 "<H1>Page not found</H1>"
 

Rodolfo

Chevereto Developer
Chevereto Staff
Joined
Oct 7, 2008
Messages
15,526
Points
237
Location
Chevereto HQ
Website
rodolfoberrios.com
If you see this from load, then both methods perform exactly the same.

But, I must be very clear on this: We already ship the software with a server configuration that delivers the lightest way to deliver a "not found". If your host can't handle the delivery of these 404 then you should point your research to your hosting service provider.
 
  • Like
Reactions: Kayz

Kayz

Founder license
License owner
Joined
Nov 2, 2013
Messages
78
Points
58
Thank you.

If you see this from load, then both methods perform exactly the same.

But, I must be very clear on this: We already ship the software with a server configuration that delivers the lightest way to deliver a "not found". If your host can't handle the delivery of these 404 then you should point your research to your hosting service provider.
The lightest 'not found' message are you referring to this example https://demo.chevereto.com/i/lsdf
 

Rodolfo

Chevereto Developer
Chevereto Staff
Joined
Oct 7, 2008
Messages
15,526
Points
237
Location
Chevereto HQ
Website
rodolfoberrios.com
You are mixing concepts.

1. The embeddable URL is the one that is referenced as "hotlink" and is delivered by your server. Those .htaccess and nginx rules instruct your server to respond a plain 404 when the alleged image doesn't exists.

2. The /i/<id> url is handled by Chevereto because it is a virtual url. Is not a real URL and the system needs PHP to check if the alleged content is there or not. This URL is not embeddable and the traffic drain comes from people clicking these links.

Most likely people gets a broken image and click it anyway. That's why you should use a not found replacement (an image that says "image deleted" or something like that).

Since these urls are fake, you can't skip PHP in order to determine if the content is there or not. If your server gets stressed serving these /i/<id> then options are: a) Use CloudFlare (it caches these 404), b) get a better server. A third option is to manually force 404 for the deleted urls and for that you will need to know all the conflicting urls and add the respective server config. You can detect these in your server logs.

You should start by analyzing if your setup can really handle the load that you are driving there. You didn't mention your server hardware, neither how much traffic we are taking about.

It seems that you assume that the server is fine and that the actual load is irrelevant.
 

Kayz

Founder license
License owner
Joined
Nov 2, 2013
Messages
78
Points
58
You are mixing concepts.

1. The embeddable URL is the one that is referenced as "hotlink" and is delivered by your server. Those .htaccess and nginx rules instruct your server to respond a plain 404 when the alleged image doesn't exists.

2. The /i/<id> url is handled by Chevereto because it is a virtual url. Is not a real URL and the system needs PHP to check if the alleged content is there or not. This URL is not embeddable and the traffic drain comes from people clicking these links.

Most likely people gets a broken image and click it anyway. That's why you should use a not found replacement (an image that says "image deleted" or something like that).
Yes i think i will opt for the latter solution you have mentioned with "image deleted" or similar.

Since these urls are fake, you can't skip PHP in order to determine if the content is there or not. If your server gets stressed serving these /i/<id> then options are: a) Use CloudFlare (it caches these 404), b) get a better server. A third option is to manually force 404 for the deleted urls and for that you will need to know all the conflicting urls and add the respective server config. You can detect these in your server logs.
Already using cloudflare...

Here are my specs running apache

Xeon E3-1270v5 CPU
4x3.6GHz 8MB Cache
RAM - 16GB DDR4
HDD1 - 1TB SATA
HDD2 - 1TB SATA
KernelCare
CloudLinux
Bandwidth - 6TB

You should start by analyzing if your setup can really handle the load that you are driving there. You didn't mention your server hardware, neither how much traffic we are taking about.

It seems that you assume that the server is fine and that the actual load is irrelevant.
Between 2-3k hits per day according to Google Analytics - growing embedded images must be having a greater impact.

I seriously lack server management, i really need to learn to tweak it to manage the site and load despite having CloudFlare supporting me at the back.
 

bilderupload

Network license
Beta tester
Joined
Mar 15, 2019
Messages
17
Points
53
Location
Munich
Website
bilderupload.net
What is your server configuration, what do you use? Nginx, apache, litespeed.. Because, 2-3k hits daily are acutally nothing, even a 500mb vps could handle it without any problems.. Contact me via pm, if i can i will help you with few tweaks and so on.
 

siddharthramakrishnan

Core license
License owner
Joined
May 31, 2015
Messages
9
Points
53
What is your server configuration, what do you use? Nginx, apache, litespeed.. Because, 2-3k hits daily are acutally nothing, even a 500mb vps could handle it without any problems.. Contact me via pm, if i can i will help you with few tweaks and so on.
He is using apache based on the rewrite rule he have posted.
 

bilderupload

Network license
Beta tester
Joined
Mar 15, 2019
Messages
17
Points
53
Location
Munich
Website
bilderupload.net
He can just order an VPS for few dollars in the month on different locations:
amazon aws = 1 year free / i m using it for all my files i link on my websites, files like banners, js scripts, fonts, logos and so on..
contabo.de = cheap vps
netcup.de = cheap vps tho
as example, and use them for load balancing, or just create an subdomain on your domain and put in the http folder of this subdomain your 404 content and host the subdomain on different server as example, before you should put this in your .httacces of your main apache server:
ErrorDocument 503 http://sub.domain.com/404.html
Otherwise this is actually not needed at all if you have setup caching on your server:

And another thing i dont understand, you wrote that u are already on your own dedicated server but at end your provider had to fix this for you, that means for me u dont have the full access to the server, or you are just on managed machine?
Look at hetzner.com (there is no provider who can mess with their hardware and prices in the hosting market...)