ekZepp@lemmy.world to Technology@lemmy.worldEnglish · 1 year agoDuckDuckGo is down. Is there any info about it??lemmy.worldimagemessage-square276fedilinkarrow-up10arrow-down10
arrow-up10arrow-down1imageDuckDuckGo is down. Is there any info about it??lemmy.worldekZepp@lemmy.world to Technology@lemmy.worldEnglish · 1 year agomessage-square276fedilink
minus-squarejoneskind@lemmy.worldlinkfedilinkEnglisharrow-up0·edit-21 year agoMost of 7b-8b models run just fine in 4bits quant and won’t use more than 4 or 5 GB of VRAM. The only important metric is the amount of VRAM as the model must be loaded in VRAM for fast inference. You could use CPU and RAM but it is really painfully slow. If you got an Apple Silicon Mac it could be even simpler.
minus-squareVenia Silente@lemm.eelinkfedilinkEnglisharrow-up0·1 year agoI have an Intel Celeron Mobile laptop with iGPU and, I think, 256MB VRAM. How many bs does that get me for the LLM? Only half-joking. That’s my still functional old daily driver now serving as homelab
minus-squarejoneskind@lemmy.worldlinkfedilinkEnglisharrow-up0·1 year agoWell, I got a good news and a bad news. The bad news is you won’t do shit with that my dear friend. The good news is that you won’t need it because the duck is back.
Most of 7b-8b models run just fine in 4bits quant and won’t use more than 4 or 5 GB of VRAM.
The only important metric is the amount of VRAM as the model must be loaded in VRAM for fast inference.
You could use CPU and RAM but it is really painfully slow.
If you got an Apple Silicon Mac it could be even simpler.
I have an Intel Celeron Mobile laptop with iGPU and, I think, 256MB VRAM. How many bs does that get me for the LLM?
Only half-joking. That’s my still functional old daily driver now serving as homelabWell, I got a good news and a bad news.
The bad news is you won’t do shit with that my dear friend.
The good news is that you won’t need it because the duck is back.