A Practical Guide to Buying Compute
Table of Contents
Comparison
People often ask how to enter the AI space if you need to spend thousands of dollars on AWS for every small project. A common solution is to “run it locally”, which can be the best option in many cases. Before getting into the build guide, let's quickly see if that's the right option for you.
Server
- Servers are more extensible and can be changed on the fly. If you need faster disks, just add a raid tier! If you need a better internet connection, get on a better plan. Everything is in your hands, and changes are isolated and small.
- Storage is free compared to cloud storage. One month of 1TB high-performance AWS-EBS costs 20% more than a faster, local 1TB SSD. Even S3 can't compare to HDD prices.
- Traffic is fast and free, depending on your local plan. Instead of paying $90/TB egress on AWS, you pay nothing. You may have a faster connection, as Swiss cities deliver 25gbit to every home, and most servers are on 100 - 1000Mbit.
- It's yours, and nobody can take it away. Not only will it have a resale value higher than 0, reducing TCO, but a local server won't share your data with untrusted third parties. There's always a risk of stolen API keys on vast.ai.
Cloud
- There's no need to manage infrastructure, allowing you to scale more easily to multi-node deployments and never worry about downtime or node accessibility issues. Especially with TPUs, things “just work”, which is exceptionally challenging if you want to scale.
- There are no free trials, and you'll have to pay for the whole system at the start, hoping the prize will be amortized. Small-scale experimentation, such as renting a GPU for a weekend, is not possible, and if the cost is not amortized through use, it will be more expensive.
- Downtime will drive you crazy, and you will stay awake for another five hours to fix that last bug and not waste the pre-paid compute.
- Real servers are loud. Gaming machines might not be, but servers tend to be loud. DIY solutions exist, but the out-of-the-box experience tends to be deafening.
Guide
The most common setup is that you're going to buy a new machine. You've checked eBay for cheap GPUs, found a nice mining rig, or want to buy a gaming PC. But before you do that, consider your budget and goals.