Buffy, Let's Start an AI Company!
We all know how OpenAI's GPT systems have taken the AI and automated sex worker worlds by storm. Maybe we can start our own company and make some cashish!
Let's do some little estimates based on the numbers in my prior article on replicating the human brain's computational performance with GPT-type AI. We saw that the memory requirements for the GPT3 are around 10^11 bytes (B) which requires 5 A100 GPUs. This will fit into a nice 8 A100 system that you can pick up for around $150k (including the CPU cores needed to take care of the GPUs). The throughput of that system is around 10^16 FLOPs.
Now to generate a token probably takes about 100 visits to the network nodes (optimizing the state of the network, more than 10, less than 1000) which requires about 100 times 10^11 = 10^13 FLOPs. So you can generate around 1000 tokens of text output per second with our fancy GPU box. OpenAI is currently charging around 0.2 cents per 1000 tokens (gpt-3.5-turbo), so our box generate revenue at a rate of $0.002 per second, or about $175 per day.
Now you need to pay the electric bill for the 3 kW power requirement for the box, which is going to add up to 60 kWh or about $15 a day. So we net $160 a day. To pay off the $150k box will therefore take $150k / $160 = 1000 days, or 2.6 years.
Well, there are a few other things to take into account. GPT3 only take 5 A100s and we have 8 so there is a factor of 8/5 there. And maybe the optimization per token is faster by a factor of say 3 or so than I have estimated. This cuts time-to-profit down by a factor 5/24 = 0.2, which is now less than a year - so we have a viable business on our hands! Of course you have to build and train the software which OpenAI has been doing for years...
Another way to think about this is that the box is like a worker for the company, and is generating revenue of around $160/day = $58k a year. The cost of the box is like the worker's salary, and if we imagine amortizing the box cost over a 5-year lifetime, it's like we are paying the worker $30k a year (no overhead or benefits!) to net the company $28k per year. This is likely the scariest thing about AI.
GPT4 sounds like it has a factor of 10 larger network which will require, say, ten boxes. Assuming simple proportional scaling of the optimization with boxes you can see you need to charge at least ten times more to make things pay off similarly, which is about the access cost for GPT4 (about $0.04 per 1 kB token input/output transaction). Just based on charges, we can guess that the GPT4 network must be around $0.04 / $0.002 = 20 times larger than that used for GPT3.
I was asked to comment on this page:
ReplyDeleteI have been using GPT-3.5 Turbo for more than 6 months in SL without hosting any GPU hardware. I made the API calls as recommended and the costs per 1K tokens is 0.0035 USD. I don't pay for electricity bills from hosting power hungry GPU as the post suggested.
Medium has a post to speculate the type of hardware for running a ChatGPT session, however, I have no knowledge that openai has released its model to public therefore I do not understand the point of business cost calculations, as you do not (and cannot) have to host these hardware in order to use it.
However, the author use case is sexbot as suggested in the post, since it is in direct conflict with OpenAI ethical guidelines, he'd need to run his own large language model. This LLM is likely to be open sourced instead of ChatGPT. There are many LLMs with the same transformer architecture like ChatGPT that can be run easily on a single RTX4060. The key in high performance during inference is to trade-off floating point precision. This approach is proven to be practical in both inference and training. For example, I am very satisfied with vicuna 13B model quantize to 4bit. I do not notice any difference in the quality of generated tokens, but much easier to run with good response-time.
I have no experience and interests in the use case of sexbot, nor I am recommending to use LLM as such. My interest is practical hardware requirements for convo capable bots in SL. I hope this helps.
Thanks for the comment! I think you misunderstood the idea - which was to estimate electricity use by OpenAI for it's A100-based 400 GB network-size LLMs, not the electricity use of an external API user (which is of course miniscule). We're talking here about setting up a business like OpenAI and we have a pretty good idea of what the size of the 3.5 LLM is and how many customers can be served by an 8-A100 server blade. I agree using a smaller model can work for limited applications, but the fun of OpenAI's system is how big and knowledgeable it is! I have yet to get one of the local open-source less-LLM AI systems set up - mostly due to lack of time - but vicuna sounds like a reasonable option. I have also had the pygmalion runpod implementation https://docs.runpod.io/reference/pygmalion-6b suggested to me as an alternative but access to that endpoint is limited. Guess I will have to buy a new server with 128 GB of memory!
Delete