Routing Storage Servers for Upload

[siddharth]

Hi Team,

I believe the front web server is used for uploading to all the available web servers. So it is slowing down the uploads when people are uploading 1000 images. Since I own multiple file hosting script, all the script used to make use of the storage servers to upload.

But their storage server setup works a bit different as it requires PHP files, so we will be uploading the same set of files to the storage server as the main website.

So people come to the main website and then they choose the image and image information is passed to storage server via the front end web servers and they take care of the rest, this will result in significant upload speed increase. And this is the optimization being applied to the Chevereto script and it is not a feature.

I owned a file host which had more than 1PB of files and this idea is to make Chevereto better.


Thanks

 

[siddharth]

See, you are understanding in a different context, the example doesn't relative to the one who uses in real life. I just give an example with 100mbits to make an easy understanding. Like people usually mention in the maps (scale 1:10), that doesn't mean it is so small. It is just a reference.

I will give you a real-time example. All my machines at 1gbps and can be upgraded to 10gbps for an additional cost. And these are additional expenses.

Now consider 1 Web Servers is located in the USA, there are 20 Web Servers in Germany. Now consider a client is uploading from ASIA. Now his uploads first go to the USA and then to Germany.

Disadvantage

1) 20 webserver ports are idle until the web server uses it
2) 1 webserver port will get overloaded when tons of people using it and experience slow upload
3) there will be a 1 to 2-second delay in each upload as there are lot of latency involved
4) there is no way to load balance the incoming upload bandwidth, adding LB will make the same concept again as LB have to handle all the upload.

Solution:

Route the uploads directly to storage servers

Advantage

1) huge amount of incoming bandwidth will be available, in our case it will be 20gbits
2) super fast upload speed
3) less waiting time

This is going to be the last message from my end, as I failed to make you understand or failed to convince you. But the above are limitation is true if you consider it or not.

That is why all other popular script works in a different way than yours.

 

[Rodolfo]

Solution:

Route the uploads directly to storage servers

Advantage

Kindly name the disadvantages.

 

[siddharth]

Kindly name the disadvantages.

The storage servers need to have the requirement as web servers to handle the image extensions, not sure if this correct as you know the coding part.

And as an owner, you know the disadvantages more than me. As a consumer, we always think about more features and benefits for our clients.

 

[Rodolfo]

The storage servers need to have the requirement as web servers to handle the image extensions, not sure if this correct as you know the coding part.

Exactly, these simple storage servers now become applications. That means to carry all what that means, including synchronization, maintenance, fallback, etc. It also adds the cost for us to provide ongoing support for that. Without knowing those costs is very easy to simply put benefits and omit the problems of this implementation. You are only assuming benefits, that's why your RFC is so wrong.

The cost of building this own storage system doesn't worth the project limited resources. The benefit, if any, is too small for our user base. At least to my current understanding.

As any RFC, users can vote to show their interest in this and I hope that others can also join with their opinions.