The speed is always tied to network/server performance. I'm from Chile so a website in Amsterdam will be always slower than a website in California. It is a physical boundary, you can't fix that with software.
Having said that, even increasing the memory limit to 512MB doesn't make your image upload process any faster. You should ask your server administrator / hosting company about this, ask them to profile your website as it is unusual this slow speed.
My guess is that maybe all your bandwidth is being eaten by serving static content (images). Think about your website as a store, where some people come by to actually purchase stuff and some people just want to lookaround. Problem is that at the door everybody waits in the same line.
You should enable CloudFlare to see if that improve the situation, CF will leverage all the load from the static content (images). Maybe the server just need to be rebooted, it happens all the time. Don't mind asking your hosting company, stuff like this is somehow common due to the large number of "things" that need to work together to put a website online.
Keep in mind that this is software, and I provide software support. I can't help you that much with server issues not related to the software itself.
