"Tinyboxes finally have a buy it now button"(twitter.com) |
"Tinyboxes finally have a buy it now button"(twitter.com) |
For those wondering, 3200W power, in residential / low-end commercial in the US, they say you'll need two separate circuits, they have a built-in power limiting utility in the OS which will let you safely run on one circuit at reduced speed.
The only part of this that gives me pause is interconnect -- over PCIe, 64GB/s stated. This is much, much lower than infiniband -- can any ML engineers comment on using this box for a full finetune of, say, LLama 3.1 / 70b?
Say you are training a 3B parameter model in BF16. That's 6GB of weights, as long as your step time is >=500ms you won't see a slowdown.
This server has two IEC C20 connectors, rated for ~16 amps, each feeding a PSU rated for 1600W (i.e. 16A @ 100v)
If you're plugging in to 110v you shouldn't plug them both into the same outlet, as a 20A circuit can't supply 32A.
As each PSU is rated for 1600W you'll have to plug both in to get 3200W even if you're running on 220v - although they'd only draw ~7.2A each in that case.
US Residential 220v dryer outlets are usually wired one-circuit-to-one-outlet, and multi-way adaptors are discouraged. So although plugging two 7.2A loads into a single 20A feed would work from a current perspective (and indeed it's common in Europe), I don't know how easy it is to do legally.
If you're in a data centre with a 3-phase 220v power you probably know what you're doing. Your UPS guy will probably thank you if you split your load over two phases instead of putting the whole load onto one phase.
Most folks aren’t going to unplug their water heater to turn on their AI.
But yes, a power supply can draw around 240V times 20A = 4800VA, which is nearly 4800W if the power factor is close to 1. An office in an office building is more likely to have 208V.
My experience trying to run machines this powerful in residential settings has been extremely poor.
All of the Seasonic power supplies that go beyond 1kW or so will trip my shitty (i.e. probably defective) Siemens AFCI breakers. Not even the same circuit all the time.
Even after violating local electrical code, I have found that living with a 1500w+ monster inside my house during the summer at 100% utilization is a complete joke. Unless you live in the perfect datacenter climate (i.e. the people who designed the tiny box), this thing needs to be inside. All of that wattage is pure heat being dumped into your home. The HVAC solutions in most residences were not designed for this kind of heat load. It would be like running your oven with the door hanging open all day. For those of us in places like Texas, this machine simply would not be feasible to run for half the year.
Looks like good value, but I wonder if it would get CPU/RAM bottlenecked, especially if you want to train something with a lot of preprocessing in the pipeline. Something comparable I've found with 7x4090 which comes to about $50k, but with much better CPU/RAM (3x CPU, 4x RAM, 5x SSD):
https://www.overclockers.co.uk/8pack-supernova-mk3-amd-ryzen...
So the red is ~$5k in gpu's - where is the other $10k going?
You're free to try building one yourself for cheaper. If you consider your time for researching/assembling/testing it to be worthless, and are happy with a contraption in a miner frame, then you can probably do it.
- Air cooling 6x4090 and a 32 core CPU for sustained peak workloads.
- 3200W total power when a single 4090 can draw close to 600W.
Maybe they are targeting startups who aren't interested in overclocking.
It also seems just weird from a business point of view. He's not going to sell many, he's not going to offer support, he's not at a scale where vendors are going to offer much particular support, and despite being absolutely tiny in scale he's still offering two totally different SKUs.
Shame it wasn't designed for EU sockets. 230V*16A = 3700W, or double that on separate breakers!
And with a distributed training you can end-up with "synchronized" transients over all cards :(.
No it isn't. Capex is only part of the equation. Opex (power and cooling amongst other things) is important. And networking at scale isn't cheap either.
ie: things like this https://gptshop.ai/config/indexus.html
I don’t know much about US electrical standards but aren’t your residential circuits rated for 1800w or 2400w? Here in New Zealand they are 2400w and people regularly plug in 2400w fan heaters without issue.
> The HVAC solutions in most residences were not designed for this kind of heat load. It would be like running your oven with the door hanging open all day. For those of us in places like Texas, this machine simply would not be feasible to run for half the year.
Yes it wouldn’t be pleasant running this 24/7 in summer in any living space. But you could install a heatpump with 7kw of cooling capacity which should handle it (adding to the electricity bill).
The residential AFCI issue I describe isn't about the wattage directly. It's about transient currents generated by large switch-mode power supplies being detected as arc faults. Similar concern as with induction motors.
In my experience, the Siemens AFCI just do that. I recommend switching them out for Eaton AFCI. That fixed all my nuisance tripping, especially from induction in other lines
At that point it isn't super price-efficient, it's an absolute space hog, and you need to maintain a whole bunch of infra. Still might work for you, but it's losing a lot of general appeal
Wikipedia [0] states that PCIe 4.0 x16 has a throughput of ~32GB/s, what does the (64 GB/s) indicate on the website, is this just a typo and you have 6x ~32GB/s or does it mean in total you can "only" expect a throughput of 64GB/s all lanes slots combined?
If so, wouldn't you also be bottlenecked by the PCIe bandwidth (when moving data between CPU and GPU)?
[0] https://en.wikipedia.org/wiki/PCI_Express#Comparison_table
Pedantically, the combined bidirectional bandwidth of PCIe x16 is ~64 GB/s, as it's a full-duplex ~32 GB/s link, but that's an awfully misleading spec if this is the intent (akin to claiming Gigabit Ethernet is 2 Gb/sec).
That's tiny. Can it train/fine-tune 70B models?
If I was serious about this I'd have an electrician and HVAC installer on the way first. A mini split in the computer room with a dedicated 50A/220v circuit.
Part of the fun is planning, researching, putting together the pieces and power it on.
A quick point: transient surges are usually fine. Both cables and circuit breakers are designed to fail (trip or burn out) under sustained overloads. For example, a 16A Class C circuit breaker might take around an hour to trip with a constant 17A load, but a ~80A load would trip it ~instantly.
PS: Of course, everything is a matter of integration over time (heat dissipation in cables mostly).
240V: Split-phase, this gives you 120V between each leg and neutral, and 240V across the two legs.
208V: The interphase in a 3-phase system.
HTH, ducking out :)
You should max. pull 2.7kW.
For everything else you need a blue eu socket or camper socket.
I learned this due to my EV which is able to be charged through a normal socket but it regulates it down due to this on purpose and has a temperature sensor build in as well.
The problem of the form factor will remain. The tinybox is 15U big for compute that you'd normally expect to find in a 4U form factor.
I just don't think it makes sense to connect multiple of them into a "cluster" to work with bigger models, as the networking bandwidth isn't good enough and you'd have to fit multiple of these big boxes into your local space. Then I might as well put up a rack in a separate room.
The main issue I run into mainly is flops vs ram in any given card/model.
Water heater, heat pumps, stove, dryer, hot tub, etc are all 220.
So if you want to plug in a device like this "tinybox" at home, it's going to be a lot easier to find two separate 110v outlets on different circuits than to have a new 220v circuit added, or to unplug your stove every time you want to use it.
It's common for EVs, clothes dryers, ovens, and hot water heaters to use 240V while most other appliances are 120V.
There is nothing in US power system that is 220V.
As for the datacenter (I’ve racked many things with A/B power) the entire point is redundancy which this defeats the purpose of since each PSU is not properly rated. Seems incredibly bizarre to me in so many ways.
Yes - often for the data centre you'd end up with something like [1] with 4x 2700W power supplies, providing redundancy and ample power at the same time. It does mean you need four 220v power feeds though.
[1] https://www.supermicro.com/en/products/system/gpu/4u/sys-421...
It used to be done in kitchens in the US, back when appliances were power hungry. I have done so in my workshop for the same reason.
Houses are wired in split phase 240V, with the neutral in the middle. That is, you have two opposite 120V phases, around the same neutral.
This is a clever way to double the power, while adding a single wire.
In the US the standard outlet receptacle has two outlets. Bring the same neutral to the two outlets, and assign one phase per outlet (outlets have metals tabs you can break off, you don't need any extra wiring).
At the panel, you have a dual breaker. One breaker per phase, with a physical linkage forcing them to trip and arm together at once.
As a benefit; but very unsafe; you can make up a Y that plugs into the two 120V outlets, and gives you a single 240V receptacle. This is unsafe because if you plug only one of the 120V plug, the other one has now 120V on its exposed phase prong! On the other hand, I have both 240V@20A and 2×120V@20A anywhere in the shop ;)
https://store.leviton.com/products/duplex-receptacle-outlet-...
With my setup I have 2×120@20 always available, and 240@20 for the occasional welding.
I could assign a different 120 phase to every other outlet but then I would need some clear identification.
The two phases are assigned to the top and bottom outlets the same way all around the shop. If I need to run two high amperage machines, I only have to remember to use one bottom and one top outlet.
I've been looking for a proper answer to this for a while, because I want to build a similar machine with 8 GPUs (~4500W max load) which would need to be split between two 16A 230V circuits.
Are you planning to build such a machine for your personal home use? If so you should be aware that (a) you might find server hardware hasn't thoroughly tested compatibility with things like suspend; (b) you might find games haven't thoroughly tested compatibility with multi-GPU setups; and (c) you might find the idle power consumption is 200W or more, even while doing nothing.
That is good to know multiple phases can work. Perhaps there would still be a fire risk in case of a short? Like somehow bridging the circuits > breakers don't trip?
Something to keep in mind though is that (at least with consumer-grade PSUs) it is not safe to simply tie the outputs together, even if both PSUs produce 5V, 12V, 3.3V, etc. The voltages will be slightly different and connecting them together will cause current to flow back into one of the PSUs.
You can still use this setup though, the key is that the GPUs do not (or should not) connect the motherboard voltage provided via the card slot to the voltage provided via the power connector. This detail allows you to safely power the motherboard from one PSU and power the GPU from another one, you just have to be careful not to mix connectors on the same card between different PSUs (if it has multiple). Additionally the motherboard should be entirely powered from a single PSU.
- $20
- GFCI, $115
- GFCI + AFCI, $115
A 1-pole GFCI breaker, a 2-pole GFCI breaker, and an ordinary GFCI outlet all have the same clever pair of coils, the same IC, and rather similar trip mechanisms. Yet the costs are quite different, and the costs get _really_ absurd if you want a breaker that trips at a level other than ~5mA.
And yes, they’re all very much worth using. I do wish that electrical codes would at least start encouraging the use of GFCIs for 240V outlets, which might encourage manufacturers to start making them, and those would actually be able to compete with each other.