OK, so for those of you who are out there looking for an excellent, inexpensive alternative (or supplement in many cases - as in outs) to expensive Cisco routers, there is an excellent alternative that’s mature and stable called MikroTik. It’s no replacement for Cisco routers, but we’ve found it to be ample for setting up complex configurations like VPN between multiple sites, and it runs on inexpensive hardware, is easy to install, can run a massive wireless network, runs radius authentication, and includes the kitchen sink. And it’s US$50 - about the cost of a really nice night out, and possible more interesting. Open Source Alternatives There are some Open Source alternatives, most notably XORP and Vyatta (coverage on launch here). However, I don’t think they’re worth mentioning. What’s amusing is that Vyatta (how do you say that?) is actually supposed to be competition for the Cisco 3825 and Cisco 2821 Integrated Services routers. Sidenote: We use the 3825 as core routing for a portion of our XO facility, due to it’s ability to prioritize different kinds of traffic. Your supposed to be able to support 240 ip-based phones on it, but of course being the Open Source addicts we are, we ended up running on TrixBox because it’s plain cooler and way better integrated into CRM and whatnot.

Anyways, I’m not sure it’s really ready to compete, and because of it’s location in my network, I’m going to let someone else beta test the brand spakin’ new core-router Open Source company.
So, we’re using MikroTik’s awesome little software to do some of the bandwidth shaping for cost-conscious customers, for our resellers (because they don’t need a 4k router to run 3 servers), and for some of our colo customers. MikroTik makes a number of pre-configured boards called RouterBOARDs, which are little MIPs 4Kc core CPU that run RouterOS (MikroTik’s product name) or Linux. For resellers, the RB500 is absolutely perfect, and if you combine it with the 564, you can have 9 (read ‘em: NINE) routable ports, serial console and management ports, 32/64MB RAM, two MiniPCI slots, 128MB NAND Storage (think local disk), all for US$265-280 (depends on RAM) each. And yes, they can be EASILY configured to do VRRP for hot failover.

Not bad. But, you are going to have to shell out another US$20 per unit for a case, and if you use the daughtercard, US$25 more that that case extension. So, real costs for loaded boards with chassis is more along the lines of US$650 bucks. Still, for redundant routers, that’s pretty damn cheap. Oh, and did I mention you can plug in a second hard drive and use this thing as a SQUID proxy server? Works great for caching images, and it puts the content right on the edge (if you put a bunch of memory in these boxes, the caching server flies!). See: kitchen sink, above. The only problem we’ve seen is that, even on the P8SCT SuperMicro board with 2GB of 533DDR2 ram and a p4-3.0 single core (an excessive system for a mere ip router, in almost every case), the max throughput we could get with the PQI 40x 512MB CF card (I reccomend a SANDisk device for production systems - this is my home computer’s card I’m talking about) and it’s IDE to CF adapter card was about 4MB/sec. That’s NOT impressive speed, but good enough for a router (most of which lives in memory most of the time anyways). But, we’re not interested in little embedded systems. At FireBright, we’re interested in insanely high performance systems. That setup is fine if you’re traffic is going to stay before 40Mbit, but if you want to support larger traffic numbers (think: burstable), you’re going to need more horsepower. And if you’re going to use one of these puppies with more horse power at the core of your network, you’re going to want to avoid failure prone configuration. So, what’s a system admin to do?

Go solid state. Right now, all of our MikroTik routers are running on CF (Compact Flash) adapters plugged into the IDE ports of the motherboard. The only moving parts in the system is the power supply. The RouterOS is loaded onto the system with a hard drive, which is then unplugged. If you’re curious as to why this is a good idea, check out the recap from Damn Small Linux’s site. This has been working great. In fact, I now have VPN set up from my house to my servers at the XO facility, so I can administer my boxes in relative safety (SSH is good ’nuff, but MikroTik makes VPN so easy it was like, “Hey, why not?”). A box in my garage provides core routing (replacing a Cisco Linksys Business Router). So, that makes it nice and reliable. But, we’ve found the performance to be a little disappointing. Now, let’s talk about how to take it up a notch. This has been working so well, I’ve been seriously thinking about expanding the product. It is already worth providing it on top of physical systems for colo and consulting customers. But, I would like to be able to use the same technology for deploying customers who may have more demanding requirements, but don’t need the expensive relatively unused features of Cisco equipment.

So, I’ve been looking for SATA to CF adapters - and there weren’t any. Until recently. I’ve finally ordered Addonics’ new SATA to CF adapter through Amazon (better overall price than direct - free shipping!). The adapter:

Addonics is shipping a CompactFlash-to-SATA adapter that can be used to boot a computer from a Linux operating system embedded in a CompactFlash card. The prosaically named “SATA to CF Adapter” is claimed to be among the fastest CF readers/writers available. According to Addonics, the Adapter complies with the Serial ATA 1.0a specification, and can transfer data at speeds up to 1.5Gbps, depending on the speed of the card used. It can be used to boot from Type I CF devices, or from Type II CF devices such as IBM Microdrives.

Now that I have the SATA to CF card adapters, I’ve eliminated the primary performance bottleneck. I want to be able to see 15+ mb/s on these cards, and I want to see the lowest latency routing conceivable. So, to do so, I’ve found a database of CF/SD performance metrics. Unfortunately it’s for photography, which is a completely different task than pushing packets, but it does give us a good idea of the performance of particular cards in a variety of cameras and card readers (computer, this is the interesting bit), a data point which, in the complete absence of routing performance information, is as good a starting point as I can fathom without my own exhaustive testing. Anyways, the results of the testing are here. As you can see, there is a significant variation in the performance of individual cards, magnified by their use in particular card readers (which is why I mentioned the fact that there is no router specific testing). But, if you look for the highest consistent numbers for the CF cards on there, and then look further at the computer card reader (SanDisk Extreme CF FireWire 800 to Mac G5 - big shock, a mac) that turned in the best performance, a clear winner emerges: The SanDisk Extreme IV 2GB and 4GB (there is an 8GB model as well) look like the fastest cards consistently on there. In fact, it looks like SanDisk products are basically sweeping the performance awards - not that surprising considering they are the market leader. Combine that card with a SATA interface, and you’ve got one hell of a fast disk. Rob continues in this article:

Did we mention how fast the Extreme IV cards are at card-to-computer transfers? Here’s a taste: the best CompactFlash cards on the market now are capable of real-world throughput between about 15MB-17MB/second in the best shipping readers we’ve tried. By comparison, the SanDisk Extreme IV 2GB, when inserted in an Extreme FireWire Reader, tops out at a whopping 38.6MB/second, with the Extreme IV 4GB weighing in at 38.4MB/second.

These aren’t synthetic benchmarks, but the actual speediness of Extreme IV when moving JPEG and RAW picture files to a Power Mac G5 here. As you’ll read later in this article, the right combination of hardware is needed to achieve transfer rates like this. Armed with the appropriate gear, however, it’s possible to nip at the heels of the 39.1MB/sec read speed specified by SanDisk for this product. This may mark the first time in technology history that real world performance, at 38.6MB/second, nearly matches manufacturer-specified performance. Note: SanDisk’s official Extreme IV CompactFlash read/write specification is a minimum sustained write speed of 40MB/second (266X) on a device from Testmetrix. The 40MB/second figure assumes 1,000,000 bytes in a megabyte, which is the conversion all storage product manufacturers use in rating the capacity and speed of what they make. We’ve standardized on 1,048,576 bytes in a megabyte for all calculations, however, because this is both the true number of bytes and the number used by devices such as cameras and computer operating systems in doing their calculations. SanDisk’s 40MB/second using the storage industry’s conversion rate, then, is actually closer to 39.1MB/second using everybody else’s, which is why we refer to 39.1MB/second as being the performance level specified by the company for Extreme IV. Oh, and did we mention how fast the Extreme IV cards are at card-to-computer transfers? In a bid to saturate the striped RAID 0 array in our Power Mac G5/Dual 2.0GHz, as well as give the computer’s FireWire infrastructure a workout, we tested the throughput from two Extreme IV cards - a 2GB and 4GB - each in an Extreme FireWire Reader, copying pictures from both cards to the Mac at the same time. The transfer rate for each dropped slightly, to 37.3MB/second and 37.1MB/second, respectively, for a combined simultaneous throughput of 74.4MB/second. Crunching this further reveals it would take 1:20 to transfer the entire contents of these cards using this twin-reader setup. That’s 1 minute and 20 seconds to offload about 5.7GB worth of freshly-shot picture files. Wow.

A cheaper alternative may be the 150x OEM model Amazon sells, but we’re not interested in cheap - we’re interested in insanely high throughput and lack of spindles (hence no micro-drives btw). What I would love to put in place for a high performance customer is a 2GB primary router (essentially 2GB CF card for root linux partition). Then, on the second SATA II controller, an 8GB card and configuring it as a SQUID cache for image, javascript, and other relatively unchanging aspects of the system. Combined with 2GB of onboard ram, and you have one astoundingly simple load handling machine. Blazing fast 30-35 mb/s routing and squid caching from disk, and that’s ONLY if the available memory doesn’t already contain the object. And you’re doing it on a P8SCT with a SuperMicro 14″ chassis and single power for about US$750-900 for the system and memory (depends on where you pick it all up, I’m estimating retail - we buy wholesale), US $35.00 for the adapter, and estimated US$160 for the 2GB card and US$639 for the 8GB card (bear in mind Microdrive performance sucks - solid CF media is the way to go). Grand total: US$1584.00-1734. Kind of getting up there in price, but remove the 8GB cache card and you’re talking US$945 per unit. Why would you need this kind of performance? Well, you probably don’t unless you have ridiculous traffic or are working on a very tight budget but need some higher performance numbers. But it’s only a few dollars more, and the applications are limitless. And, the model is absolutely proven to be the fastest way to build systems (see company TMS - Texas Memory Systems’s Tera-RamSan for a compelling example:

• Up to 1 Terabyte of non-volatile DDRRAM in 24U.
• Unlimited overall capacity.
• Over 3.2 million random I/O requests per second.
• Over 24 GB/second of random sustainable data bandwidth.
• Up to 512 physical LUNs.
• Requires 2,500 watts of power.
• Up to 8 independent non-volatile solid state disks (SSD) modules. Each SSD module is a RamSan-400, including 128 GB of DDRRAM and up to eight 4-Gbit Fibre Channel connections.

For those of you who know how insanely fast that is, and how little power consumption that is for the size of the array, you’re impressed. So, this is what’s been rattling around in my head all day today. Load Balancers The other thing that’s got me interested is load balancing. This card configuration would make an ideal platform for load balancer deployment. We use Coyote Point load balancers right now (with hardware decrypted SSL, a separate add-on, they are reasonable if not outstanding performers, and they “just work”), as well as Squid reverse proxy and POUND for customers. The MikroTik can also do “load balancing”, but it’s not really appropriate for load balancing ip services - it’s more designed for load balancing routes (so called “link load balancing” - see more info here)). Update: You can do load balancing - I just haven’t, and have no idea how well it would work, but it is possible. There’s a lot of confusion over that distinction - you’ll see a lot of posts on lists that reply with something like, “Are you talking about load balancing two (or more) servers? Or link (ISP) load balancing?”. Well, you really don’t need a super high-performance machine to do load balancing. I’ve seen very low-end celeron p3 systems doing a fine job of ip routing for a 10Mbit connection with 3 frontend servers. However, I would LOVE to put this set up together with Zeus’s ZXTM 3.0. I have been very impressed by the Zeus ZXTM & Appliances (and all of Zeus’s products for that matter). It has some really excellent SSL performance to boot. Their documentation says:

How does ZXTM’s cost per SSL transaction compare with other solutions?  ZXTM’s native SSL implementation has a better price / performance ratio than any other load balancer or traffic management solution available. We have achieved this though close collaboration with companies such as hp, Intel and AMD. Product List Price (incl. hardware) SSL tps Cost per SSL tps Saving with ZXTM Cisco CSS 11506 $36,990 1000 $37 89% F5 BIG-IP 6400 $44,990 5000 $9 56% Nortel Alteon 2424-SSL $22,200 310 $72 94% Radware WSD AS III $45,000 700 $64 94% ZXTM 3.0 $18,8501 4500 $4 - Source: respective vendors’ sites Notes: 1 Includes cost of dual AMD Opteron server hardware

Including the cost of a system? Not a bad deal actually, especially given those performance numbers. And with Coyote’s fully loaded Extreme II coming in at ~US$26,000 (US$10,000 base), it’s starting to look like a deal. A big problem for many companies is that, due to the fact that it’s not a hardware device, you cannot get a used “zxtm” box. You’re going to pay retail. However, these prices are well out of the range of your average small web master. And the SSL numbers are not that important to the majority of content sites out there - it’s really only an issue for high security sites, like ecom and such. And while you can always use ours, building a load balancer for under 1000$ with solid state performance may well be a good use of your time. If not, you could always use loadbalancer.org’s (they’re good people, and you can try it out with VMWare here). I still think their stuff is a little pricey for what it is, but it’s not expensive - it’s just not cheap.

Well, anyways, that’s what I’m working on for our routers, and we will probably be packaging up solid state load balancers for a number of customers. And with the XEN and VMWare-based images we already have in production, and the grid computing load balancing that we’ll also soon have on offer, there should be options to fit most any budget. That’s it for the do it yourself-er. Have fun h47k1n6.