Introducing “6-pack”: the first open hardware modular switch(code.facebook.com) |
Introducing “6-pack”: the first open hardware modular switch(code.facebook.com) |
The QSFP's are "spaced for optimal airflow." However this spacing seems to neglect cooling the QSFPs themselves. Belly-to-belly mounting of modules is usually the most thermally challenging way to arrange them. The heat dissipated by the QSFP's is generally directed towards the top of the module. By placing open air channels between modules, they have effectively ensured that little to no air flows over the QSFP heat sinks (which is not shown). So there is probably a limitation on which reach codes are supported. My guess is that because of the thermal limitations of this design, it's not truly non-blocking in all reach configurations.
No, they really don't. I've never heard of 48V outside of telco.
However, at no single site (especially my home) do I need a 6U chassis full of switch ports.
Is there a 1U version of this on the horizon ?
Some easy ways to buy are: https://bm-switch.com/ http://whiteboxswitch.com/ http://ipfabrics.net/ http://www.penguincomputing.com/products/network-switches/
- nolan co-founder/CTO Cumulus
I thought about adopting Cumulus for a large telco project, but for the moment it doesn't seem a good fit as we do lots of openflow, L3 and custom application development.
Actually, have a look at it. It's also blue (maybe we should sue facebook :D): http://ripe69.ripe.net/archives/video/177/
Similar to how you can buy a Backblaze Pod ( https://www.backblaze.com/blog/backblaze-storage-pod-4/ ) from http://www.45drives.com/
Looks like a 40 GbE NIC is just $500: http://www.amazon.com/MELLANOX-CONNECTX-3-56GBE-1PORT-MCX313...
Just need an SSD RAID to feed it...
From the responses, it looks like "open hardware" means something like "can install any OS on it" in the network switch world.
That said, I do hope other developments in network gear that will be useful in other markets emerge from this effort.
From my perspective, there's a gap in network gear between the unmanaged, low port-count switches in plastic enclosures, targeted to home and small office consumers, and the lower tiers of Cisco's catalog, targeted toward top-of-rack or wiring-closet-of-a-larger-building type uses. I would love to see a managed switch with say, 8-24 ports, supporting features such 802.11Q VLANs. I would love to be able segment my network at home so different devices with different performance and security needs aren't all stepping on each other's toes. And I'd like the firmware and hardware designs to be open source, so it can readily patched when bugs are found, and easily adapted to new use cases.
I realize that I'm an outlier and that my needs are not common, or there'd probably be equipment on the market that met them. But it is my hope that as a result of Facebook's work here, and similar efforts, that building such a device will become feasible.
/* it's easy to forget just how many iterations of Moore's Law have happened since the mid-90s */
It is still fun watch this switch go around the proverbial track, but I'm happy knowing that I'll never have to configure, build and test a switch of this complexity unless I really absolutely have to, with my largest caveat being AWS disappearing from the face of the earth.
I currently work in the network hardware industry (think Cisco, Juniper, etc). Our boxes sell in the 6-figure price range, each [1]. We sell to your ISP, wireless carrier, datacenter constructor. We're the competition this kind of box is aiming at.
[1] it's highly specific hardware and software for a low-volume market. Individual chips used in the hardware can cost multiple thousands of dollars each.
and 10 years ago, the compute power of a QC i7 would have tens of thousands of dollars.
I've been a "cisco guy" since 2001 or so, but I am so tired of them. Licensing fees kill budgets that could be used on other things, and you end up surrounded by consultants that only ever touch Lozedoze systems insisting that "nobody ever got fired for buying Cisco". Smartnet is a must for some equipment, yes, but I'm so ready for a paradigm shift in networking.
I've really been watching ubiquiti and their switching/routing products, they seem very promising but not quite prod ready. I am impressed with Dells open switches too.
Edit: The Microtik stuff being linked elsewhere here is looking pretty awesome too. Not quite FOSS but still.
As someone who runs a 6509 (almost fully populated with line cards and specialized controllers) as my HOME core switch (with a second one on the way for full redundancy), an open modular switch is very cool.
(I'm currently hacking on OpenFlow -> NETCONF bridging, to bring typical SDN capabilities to legacy Cisco environments). Hence my rather..... extensive home network.
This is very similar to what Cisco is doing with Nexus and the "fabric extender" TOR replacement kit. I'll have to see if Facebook has any of this stuff in GIT and stand it up in a VM environment and play with it (I already do a bunch of OpenFlow stuff on OpenWRT and am looking at implementing an open southbound API on FPGA on the parallela board).
http://opencores.org/project,mac_layer_switch
and some links to digilent dev boards for hardware, however the linked project is a bit more ambitious, all full of custom ASICs and such.
An open source switch made out of COTS FPGA dev boards would be interesting. So you'd use something like
http://www.digilentinc.com/Products/Detail.cfm?NavPath=2,400...
But what to use as a COTS "just unpack it all, plug it all together, upload the firmware and go" backplane?
Almost all "open" projects seem to hit a point where the line between open and closed is quite arbitrary, even if they choose not to see it that way.
If Facebook wants to give me that value for free I won't complain.
Pretty much the entirety of the Arista userspace is python with all of the magic really in their ASICS. Truly, I've not seen many better engineered pieces of equipment. At the time (I did this > 4 years ago), the Arista switches had dual core AMD Athlons with a few G of ram. They were willing to put more RAM in one, or let us upgrade it. For a so called "vendor product" Arista clearly gets it. They have some really solid Linux guys working for them. When you get the pleasure of using their python apis or pull one apart, it shows that they know what they're doing. It isn't completely insane to think of running apps directly on the switches when they are simply x86_64 Fedora boxes with some fancy asics and a lot of interfaces.
1) low individual component cost (which is great when you need 100's of them for a datacenter build out) 2) no frills, high throughput non-blocking backplane which are normally only available from the enterprise grade network vendors at top dollar per individual component, and come with a mountain of features and bugs you DONT NEED 3) software defined networking stack: if you can imagine a scenario/feature that would improve your life, nothing but development costs will get in your way. contrast with enterprise vendors, which requires explaining the problem to begin with, and dangling a wad of money explaining how it will be worth their while to develop it
The talk was on HN a little while ago, worth watching: https://www.youtube.com/watch?v=JIQETrFC_SQ
The scale at FB means they're probably saving hundreds of millions doing this.
I use it for the home lab, where I have a one-armed router serving multiple VLANs, and I have 3x MacMinis running Linux as a "server farm" (The latter I use because they are quite a good gear power-management wise, scaling from ~18W at idle up to ~250W when all cores are busy, and because they are very very quiet, which is handy when the "lab" is next to the bedroom).
The biggest complain I have about this box is that the only way to manage is the Web UI, and especially the 802.1q configuration is a bit unintuitive (though I just learned the firmware is actually upgradable to something with decent IOS CLI, so I will try it out and update here the impressions).
Otherwise, needing just a very simple L2 switching and 802.1q trunking at gigabit speeds, and fanless operation - I am pretty happy with it.
8 ports works well in my setup (the main segmentation/trunking is really in the lab, the rest is either wireless, or directly connected to the "border router").
Where I needed to add more access ports, I used http://www.conrad.com/ce/en/product/976050/CE-Port-Network-S... to help. Also based on my experience seems to be quite a solid building block for a small home/lab network.
full disclosure: I do work for cisco.
EDIT: the upgrade to the latest firmware indeed unearthed the checkboxes to enable telnet/ssh, as well as quite a few new features, comparable if not more than the "bigger brothers". What's pleasant is a quite comprehensive IPv6 support.
My main hesitation is the proprietary firmware. Now I'm not going to disagree with anyone arguing that Cisco knows what they're doing and is competent at putting together firmware for the hardware they sell. Nor will I disagree with anyone arguing that open source is not a magical talisman ensuring quality. No, my main concerns are a) timeliness of critical updates, and useful lifetime of the hardware vs support lifetime.
Being open source cannot prevent bugs, but once found, fixes tend to become available quickly. Also, I have found that the useful lifetime of computing and networking hardware tends to exceed the period of time the vendor will offer support for it. I have gigabit ethernet switches I bought years ago that still work just fine, even though they're no longer sold. That's what I love about OpenWRT. The hardware my home router uses is discontinued, but still does the job just fine, and I can still get updates when I need to.
That said, I think I will be checking the SG300 out. Thanks for the recommendation.
http://www.mikrotik.com/downloadterms.html
I mean, sure, asking for 45 USD for a CD with the source is technically complying with the GPL -- but it does seem a bit strange in this day and age. Not to mention that for the source code to be useful, one would hope one could build a working routerOS image from it -- and it doesn't appear that the CD will enable a user to build a running image -- and therefore not facilitate changing the product.
But apart from that, my first thought was -- can this thing run BSD -- because pf is quite a bit friendlier than iptables (even if the latter has gotten a lot better lately).
Does anyone know if there are any recommended alternatives to soekris for running a bsd switch/router (preferably running at ~gigabit speeds) ?
I'm also scared that I'm going to open up my home intranet to the world every time I tweak some of the advanced settings.
Not trying to dissuade anyone from using them, but it's light years away from a DD-WRT-based router.
OTOH, the ability to run VMs on your router is magical. I have yet to do that, but I'm itching to do something like putting the unifi management tools on it.
the 15W consumption seems to hint it could be, but the datasheet does not mention it.
I have 1 2port VLAN that connects my cable modem to my router on the TP-Link. From that switch I have 1 port cables to a machine with wireshark, and I can configure that port to monitor various VLANs for whatever reason.
I run MRTG for several things, including basic traffic graphing. I display the graphs from the router uplink port and a couple of other key ports in a window on the VMS that also has security cameras on it. From that monitor I can keep on eye on key things (cameras, Internet I/O, some home automation stuff).
Anyway, I haven't found an affordable "perfect" switch for home stuff, but there are a lot of cheap, decent managed switches that give you a lot more flexibility beyond "everything on 1 network".
My home net is essentially segmented into Primary LAN, Security Devices, Guest LAN (mostly just a wifi bridge) and LAB LAN.
I think what would be ideal is something in the vein of the Linksys SFE2000, with an open-source firmware akin to OpenWRT. AFAIK, that, or something approaching that isn't available. However, I would love to be wrong in that regard.
See the bottom of this page for models the 24 and 8 switches are what you are looking for. http://www.cisco.com/c/en/us/products/switches/catalyst-2960...
Also, if you want a cheaper options the Dell Basic switches are fully managed and have just about any feature you could want.
http://www.dell.com/us/business/p/powerconnect-2800/fs
Unless I'm missing something that you are looking for.
http://www.ubnt.com/accessories/toughswitch/
and
http://www.ubnt.com/unifi-switching-routing/unifi-switch/
That's exactly what you are looking for. SOHO friendly (fanless, cheap, supports vlans).
And N years after that you get that functionality for free along with your new TV.
Though, 30 years ago, in Neuromancer, William Gibson wrote about the city of Los Angeles transferring megabytes (yes, whole megabytes) of data each second, so he probably couldn't have imagined what someone would do with the gigabit switch on my desk.
[1] - http://en.wikipedia.org/wiki/High-bandwidth_Digital_Content_...
We don't do OpenFlow, but L3 and custom apps are our main focus.
That said, often times the subset of things that can be done w/ OpenFlow in actual existing hardware are things we can do natively. Feel free to email me if you want to discuss!
nolan@cumulusnetworks.com
+1 on the OpenWRT. Building a custom package that allows you to get a $20 specialised networked appliance is a breeze. That platform absolutely rocks.
But further tinkering revealed it seems to be a quite-close approximation of IOS, but not the same IOS you'd get on the "older brethen" boxes.
It's about 95% the same, with differences in small details - the format of the output, the behavior on "more" prompt, the look of that prompt, the way the certs are stored, etc.
Nonetheless should be close enough to be usable in a geek home environment.
I got the software here, looks like it allows to download w/o logging in: https://software.cisco.com/download/release.html?mdfid=28301...
Since the box I had ran a rather old version, I had to do the intermediate steps as described in http://serverfault.com/questions/622724/error-illegal-softwa...
One caveat I noticed is the ssh seems to not work when connecting from Ubuntu 14.04, works fine from OS X. When I have time, I'll debug it further. (I very rarely do any changes on it, the CLI was more an unexpected bonus I wanted to check out rather than a real need).
It's only a 3 port system though.
I'm with you about the GPL though. It's my biggest complaint about it. Apart from the proprietary hardware (which OK, I guess I can forgive it), they definitely seem to be playing it pretty loose with the terms of the GPL.
Pf is indeed nice and it'd be cool to run OpenBSD on it. That said, as I mentioned I really like their configuration tools and they make iptables actually quite easy to configure.
Your best bet for a BSD router is probably to pick up a cheapish computer and put in one of the Intel or HP quad-port ethernet cards and then plug into a dumb switch. They can be had for surprisingly reasonable prices, I saw some on Amazon for ~$80 (the quad-port cards, that is).
[0]: http://www.maxxwave.com/solutions/security/routermaxx-6-port...
It's kind of stupid to say that today's general-purpose hardware can do what yesterday's special hardware could do at a faction of the cost, because the specialist market has also moved on and wants the performance of today's special hardware.
It is substantially better pricing than list for comparable Juniper / Cisco / etc. equipment. However, Cumulus has no truly low end 1 gig switch (single power supply, limited L3 capability), and you can absolutely negotiate Juniper / Cisco / etc. down to be close to or even below the pricing of the Cumulus solution. That is tougher for 10 gig or 40 gig equipment, which is where the value proposition kicks in for Cumulus.
Maybe Cumulus's approach is enough of a value add for it to make sense to pay a premium, but everyone I talk to is interested in cost savings first and better manageability a distant second. You'll still have to have Juniper / Cisco / etc. in your life to an extent, Cumulus doesn't do routers and they don't have a full range of switch models.
- nolan
That said, RouterOS pretty well documented and isn't hard as long as you know the basics. Mikrotik actually has a pretty vibrant little community around their wiki.
I've been looking at trying to control some of my devices a bit more (eg, SmartTVs).
PFSense looks good, but the dedicated hardware boxes are kinda pricy[1], and I don't really want to build something myself.
A switch with a built in firewall at that price is an interesting option.
I've heard people say it's not really powerful enough CPU wise to cope with a ton of rules (>hundreds), but I have a fairly involved firewall config on mine and it's no trouble.
Funny story, we use a firewall rule to punish roommates for not doing their share of the chores. If they get too far overdue on chores, we have an iptables rule to randomly drop a certain percentage of packets to their machine.
[1]: http://forums.juniper.net/t5/Data-Center-Technologists/Junip...
http://whiteboxswitch.com/products/edge-core-as4600-54t
http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=S...
http://www.amazon.co.uk/Mellanox-3-3ft-QSFP-QSFP-Passive-Cop...
Actually seems the price is finally coming down a bit (compared to what I remember these used to cost, years ago -- but maybe I've just upped my budget ;-).
[1] http://en.wikipedia.org/wiki/QSFP
"The Quad Small Form-factor Pluggable (QSFP) is a compact, hot-pluggable transceiver used for data communications applications. It interfaces networking hardware to a fiber optic cable."
But it also interfaces hardware to copper cables, as I gather.
It is a SFP+ plug on each end (but without all the optical magic), connected with twin-ax cable, which is like coax but with 2 signal paths, one for each direction.
Without this part, everything works the same, but is of course not hardware accelerated. So the 100% open source parts of Cumulus Linux would still make a great Network OS for a router/switch VM.
We don't yet have an official VM version, but that is something we will have in the future.
- nolan co-founder/CTO Cumulus Networks
Or can you actually get fairly low level, like implementing your own algorithms for channel bonding? A while back I wanted to do some L7 inspection, but could only get like 10G per server, and we had 40G coming in. EtherChannel didn't acceptably balance out the traffic. Doing so would have required dealing with one of the network processor vendors and all that mess. Would an open switch platform make this a straightforward exercise?
The big advantages are reusing config management tools like puppet/chef/ansible/etc, and monitoring tools like collectd/graphite/nagios/etc.
Also, it is super easy to run services on the switches. For example, you can easily run isc-dhcpd on each ToR, instead of DHCP relaying back to one mega DHCP server. Distributing services like this scales better, and reduces the blast radius of service failure.
I've been experimenting with the idea of a transparent caching TFTP proxy server running on the top of rack switch, to make PXE scale better to large clusters.
The important thing is that anyone who has the know-how to write a transparent caching TFTP proxy server for Linux can just go ahead and do that on a Cumulus Linux switch! You don't need to come to us and convince us that it is a good idea and then wait for us to actually implement it. Compare that to asking for features from a traditional switch vendor...
- nolan co-founder/CTO Cumulus