What you think about a distributed self-hosted grow strategy?

[Rodolfo]

Hey there,

You all here already know that once the system gets heavily used it starts to show some pitfalls regarding servicing. Common issues are on storage, traffic and specially the database performance. The more users you got also carries the cost of moderating user-content driven actions, to provide support, contact, etc. 👏🏾 You have managed to keep it reasonable cheap, to earn some profit or at least break even.

I have seen your strategies and is very interesting how most of you tried pretty much all available storage providers, VPS services, configurations, etc. Many of you are really taking advantage of the software, giving it the real purpose I intended when I created it and I'm proud to see what you manage to build with my stuff.

However, I do feel that this precious knowledge and efforts are too isolated as we aren't acting as a distributed group, which make us very weak. It is too selfish, imagine that each one of you is the main person in charge of running a massive service, sounds awesome (we now is cool) but it is also a tall responsibility. That's why I'm opening this topic, because I want to propose a distributed growth strategy and I will really need your feedback on this.

Why is this a problem?​

We know that users will always trend to saturate the system, is just matter of time. We can safely assume that any giving installation won't stop growing, regardless of the rate. Installations are "doomed" to be always growing. Perhaps it is not a problem now, maybe in 2 years? Problem is that while I keep driving the provisioning without this option there will be always a cap on how much income you can get from using this software, and it will be always limited to my capacity of making the software efficient at higher scales.

If you reach a given installation size that I can't optimize, your service will just be less proficient than others and more expensive to run. I believe that key here is specification, to diversify the offering. To provide more unique experiences rather than just one general system, but the problem is how hard is to provision this.

Distributed strategy​

The distribution strategy I want to purpose is really simple and straightforward: To provide the core feature of "account migration", enabling the user to migrate their account to another provider, including their own self-hosted instance. With this, you could get in touch with power users and offer them migrating to their own managed instance or sell them the migration service. Not only you will leverage load from the main system, this new instance will grow on its own and is very likely that the power user will bring and care for more users and for you won't be a problem, you already have plenty experience with the software and its servicing.

✨ You could spawn another business (yet another income) for managing these instances, specially for those big power users.

Rather than making just one big installation and absorbing all costs, you can use what you already know to keep running your main installation and on top of it to sell more expensive services to the users you already got. Imagine if some of these users goes big as your original website? Not only you will be right there, the user will become another peer for the software.

What I'm purposing here is to design a system that will enable to re-distribute your load servicing strategy based on sprouting isolated managed instances. This takes the best from the software, and provides the user a freedom that doesn't exists anywhere.

Your thoughts​

This is the most relevant part here as you represent the big installations. I need to know your honest opinion about this, what you like/dislike. I know that this requires an extra effort.

 

[ashkir]

I really like the idea of being able to redistribute enough load of a website and the amount of traffic. However, one of the most important things to me is the consistency of the URL, especially the direct image. Changing this around breaks images, which ends up being bad for the long term growth of a website in my opinion.

I would love to be able to have a "premium" hosting server, etc, to upload to the faster server, or use the better CDN service for a paying customer.

 

[Rodolfo]

I never noticed the technical limitations.

One alternative for missing URLs could be to pay to get the original service to redirect to the new URLs? Another alternative could be to enable users, from scratch, to define a fixed subdomain url (easier to redirect).

There will be also the trouble of conflicting IDs, either keep all the records and borrow the original crypt salt or go all over from scratch.

 

[mkerala]

Sorry, I am late to the party.

I have already implemented most of what Rodolfo has covered. Without distributing, I can't run a site with 7M images.

The external storage servers have unique URLs with sub domains s1, s2, s3 etc. One thing I noticed is when nearing 2TB, the server becomes very unstable and long term to iterate file list over SFTP. So have to create a new one. Also, I can keep each in different server for load balancing and make multiple active at the same time.

Still not a perfect solution. High traffic can basically knock out the server and upload fails until I manually switch to a different server. Wish the external storage a had an automatic fall back instead of failing.

I was not offering any premium offerings. But few users contacted me asking for it. So recently created a subdomain and installed a fresh chevereto so I can offer more features than standard installation. I wish in feature if this could be done on the standard install.

 

[lovedigit]

At the end it all boils down to supply and demand, and price that makes it profitable for both parties.
Scaling is a trivial infrastructure decision. One wrong move, and you're stuck with inefficient system/infrastructure. Sometimes, it is hard to even move away to new system as things might break. However, chevereto is simple enough to move around, or try new things. In last 5 years of using Chevereto, I have changed things countless number of times.
None of them has proved to be efficient. Sure, one is better than the other, but there is always that one catch. And to provide that to someone else efficiently is totally different ballpark. Sure, at the right price we can provide managed service, but user has to be profitable in the first place to seek for such offerings.

Currently, I am trying out backblaze b2 as one of the external storage, and it is giving me all sort of trouble. unable to connect to pod, Invalid resource type: resource (closed), cURL error 7: Failed to connect to pod, connection refused, Received error from B2: internal_error. On top of that, uploads are incredibly slow, compared to sftp servers. I am not sure if it is B2's fault, or our external storage code has some issue. B2 suggests in case of 503 error we should retry, maybe chevereto is not doing that? I am not sure.

The point is, it is always complicated as you grow. It all comes down to how much you're willing to spend for stability and scalability. Nothing is impossible for a right price. But, for us free image host providers with no other source of income/profitability, it is something out of reach, unless paid plans are introduced, and there is some financial power to expand and grow.

 

[mkerala]

I am looking for something better than SFTP. OpenStack seems to be too complicated.

 

[Rodolfo]

At the right price we can provide managed service, but user has to be profitable in the first place to seek for such offerings.

The profit for the user is that you provide a seamless path to let the user grow. Why you struggle trying to make one system huge if you can drive users towards distribution and have a massive distributed offering?

In a system like Chevereto is more reasonable to grow spawning managed services, not growing huge websites. Not only it allows you to isolate services, you will make way more money with less effort. This is because user accounts are limited compared with a managed installation where the user can drive a more specialized system, all the resources and system usage is completely decoupled. You can save all the hassle responsibilities (storage, transfer, moderation) of all these accounts that grow too big for your monolith system, the user will care about those at their own scale and it will remove that costs from your operation.

 

[lovedigit]

I am looking for something better than SFTP. OpenStack seems to be too complicated.

I think b2 + cloudflare with their new bandwidth alliance is nice alternative. Because you don't pay for the bandwidth, but you get all the extra features that gives you peace of mind. Your images are replicated on backblaze to avoid data loss. But, their ingest is super funky and unstable. I always get few errors on image uploads.
I think it is time we spend more time making b2 extrnal storage more robust. In the event on failure. Chevereto should retry to upload the image.
Right now, it waits for the response from b2, and then throw error.
Other than that, everything is super stable, and fast. You pay roughly the same price that you'd pay for a kimsufi 2TB storage server. + some extra change. But you get superior egress connection speed, redundancy and cdn in return as long as you serve images through cloudflare workers.

Openstack is complicated, and in my opinion it is only better when you've your own hardware.

 

[lovedigit]

The profit for the user is that you provide a seamless path to let the user grow. Why you struggle trying to make one system huge if you can drive users towards distribution and have a massive distributed offering?

In a system like Chevereto is more reasonable to grow spawning managed services, not growing huge websites. Not only it allows you to isolate services, you will make way more money with less effort. This is because user accounts are limited compared with a managed installation where the user can drive a more specialized system, all the resources and system usage is completely decoupled. You can save all the hassle responsibilities (storage, transfer, moderation) of all these accounts that grow too big for your monolith system, the user will care about those at their own scale and it will remove that costs from your operation.

I see what you mean. It can be profitable if there is a market for it. Basically we'll be re-selling host and proving management. But, it will be a peaceful transaction for users who just want to focus on their work, and would like someone else to handle their chevereto install and growth path.

But, what's stopping them from making their website a monolith image host? We can't restrict how many users they can have on their website. And they will be free to open their image host, only we'll end up managing it for them.
It is good profitable business idea, I am just saying with growth comes the complexity, list of headaches and complaints. Whether or not it is worth it, that can only be determined if you make some really flexible system to deploy distributed websites that can enforce restrictions based on terms and plans that we are willing to offer.

 

[Rodolfo]

what's stopping them from making their website a monolith image host?

Nothing, that's the point. Those who go from account > managed > dedicated follow an organic grow path and you leave them go once their size becomes a problem for you.

We all know that the problem are the users who really take seriously the "upload all you want" and that use too much resources for just being another account.

Whether or not it is worth it, that can only be determined if you make some really flexible system to deploy distributed websites that can enforce restrictions based on terms and plans that we are willing to offer.

It does that already isn't? Or you say something like a cPanel experience?

 

[mkerala]

I think b2 + cloudflare with their new bandwidth alliance is nice alternative. Because you don't pay for the bandwidth, but you get all the extra features that gives you peace of mind. Your images are replicated on backblaze to avoid data loss. But, their ingest is super funky and unstable. I always get few errors on image uploads.
I think it is time we spend more time making b2 extrnal storage more robust. In the event on failure. Chevereto should retry to upload the image.
Right now, it waits for the response from b2, and then throw error.
Other than that, everything is super stable, and fast. You pay roughly the same price that you'd pay for a kimsufi 2TB storage server. + some extra change. But you get superior egress connection speed, redundancy and cdn in return as long as you serve images through cloudflare workers.

Openstack is complicated, and in my opinion it is only better when you've your own hardware.

Both Backblaze and Cloudflare won't work for me. I got 10TB of data and 200TB for bandwidth to serve. Cloudflare already kick me out and Blackblaze is expensive. Even if it was not, it won't work for me as as I get around 100 images uploaded every min. So failure rate is much higher for me.

Only option is to run own hardware. No cloud provider can go cheaper than the hardware and maintenance cost they have. So its impossible to find a Cloud provider that is cheaper than hardware. Even if you does find one, it won't be able to handle the load. My SFTP storage servers use the maximum resources whether it is RAM, CPU, bandwidth or storage.

Btw, Hetzner has the cheapest hardware with lots of storage options. But bandwidth is a problem. Even 1Gbps x 2 is not enough at times. Check out their SX series. 64TB for €64 is one hell of a deal.

Dedicated Root Server Hosting - Hetzner Online GmbH

Find your reliable dream server, optimized for high capacity use, ever-growing storage needs or rapid and parallel processing.

www.hetzner.com