Linux for Suits

As I write this, I'm embedded in a hotel in Copenhagen. Besides being jet-lagged (it's past 3am and I'm still wide awake), I'm also cursing the hotel's firewall, which blocks Web sites it thinks are violent (Howard Stern), sexual (one in five results for “brassiere” on Google) or worth forbidding on other grounds (most display ads are blocked). Bandwidth (banned width?) is slow as mud. And both SMTP and SSH are blocked, so even my normal route-arounds of blocked outbound e-mail are thwarted. Naturally, I posted a gripe on my blog under the irresistible headline “Something's censored in Denmark”, and instantly got back an e-mail from my Danish friend Thomas Madsen-Mygdal. He wrote, “Blame the Americans” and pointed to the Web site of WatchGuard (www.watchguard.com), a US-based firewall hardware company. WatchGuard runs on Linux (www.watchguard.com/help/lss/50/handbook/need_f27.htm).

So, I'm faced with the ironic task of writing about embedded Linux while my access to the whole Internet is throttled by an embedded Linux product. Fortunately, I spent much of yesterday morning watching embedded (and other forms of) Linux being put to good use in the grass roots of applied tech in Copenhagen's urban habitat.

The object of my interest was Indienet.dk, a local pure-Internet infrastructure provider. Although they support VoIP phones, they don't sell VoIP as a service. Nor do they sell TV. (Those last two combine with Net service to form the familiar “triple play” of offerings that telcos, cablecos and many of their new muni competitors are pushing in the US.) “We're bottom level”, says Jakob Frederiksen, our host and the company's sales and service chief. “We just provide the base-level connectivity.”

Figure 1. Indienet headquarters—the art on the walls is made from Ethernet cabling.

Figure 2. Jakob Frederiksen of Indienet.dk, outside its offices in Copenhagen

Indienet is better known for its vans than its offices. The vans are bright orange and carry the gear required to install fiber optic and Ethernet cabling in old urban neighborhoods, like Vesterbro, where the company is based. Headquarters is a practical and nondescript upstairs space. Ethernet cabling forming artful shapes is stapled to the walls. This is where Jakob handles business and monitors performance, waiting for trouble that rarely happens. When asked to describe the company's tech, he replies, “It's all Linux.” On the same screen, he shows us a video of the company trenching a street on a Sunday (when vehicle traffic is minimal) to lay conduit for fiber optic cabling. Indienet does most of the grunt work themselves, including re-assembly of the old cobblestone street surface when the trenches are patched back up.

Figure 3. Indienet Van

Down at street level, Jakob shows us Indienet's garage workshop, where wooden spools of green and orange fiber optic cabling sit on floors and shelves. Piled near the door are many smaller spools of CAT6 wiring. Wire channels, conduit, connecting hardware, drills, saws and other tools are neatly arranged in the space behind a collection of foosball tables in shipping boxes. Nearest the door are two pallets of Applied Telecom switches. “Those are the best”, Jakob says—though he saves his highest admiration for Soekris, the Santa Cruz-based maker of small workhorse embedded Linux boxes. “They're simple and extremely reliable.” (I just noticed, while surfing around the Soekris site, that the company's founder and CEO is Soren Kristensen. I'm guessing he's Danish. Coincidence?)

Figure 4. Spools of CAT6 Cabling

We walk down the street to a curbside where one of the Indienet vans is parked. Next to it Jakob's partner Preben Conrad is rolling a smoke before heading down into the basement of an apartment building, where he and another guy are wiring up the place. Preben is the lead technical guy for the company and its chief installer. In the basement, the guys show me the empty rack where the routers, switches and other gear will go, and the metal wiring channels they've been assembling and are now attaching to the ceiling. Next to the channels, several generations of telephone wiring and cable TV co-ax are tacked to floor joists and disappear through holes drilled in the floors above. The difference between installations is notable. Indienet's is built to service and to improve when the need comes. The building, typical for the neighborhood, is seven floors high and about 100 years old. The basements make it easy to run wiring to appropriate points below the stacks of flats above. And, the co-op nature of most apartments makes it easy to deal with one entity.

In most cases, the co-ops themselves are the customers. And, once the installation is complete, the relationship with Indienet is service-based. The company just makes sure the Internet is up and running. In some cases, that involves monitoring usage and doing traffic shaping. This is more likely to be necessary where the shared connection is a single ADSL line. ADSL is the familiar standard Internet offering in Denmark, though Indienet and similar grass-roots outfits would rather change that, especially as more fiber-based connections are deployed. (Fiber deployment is inherently wide open and symmetrical.)

Thomas, who set up the visit and is tagging along, explains how, as Indienet continues wiring and daisy-chaining apartment buildings in districts like Vesterbro, the company and its customers increase their independent buying power, which they bring to the country's backbone transit providers and ISPs.

In the midst of all this is a convergence of different approaches, deals, partnerships and deployment approaches. Thomas gives Bryggenet, Tirkontnet and Parknet as three examples of ideal grass-roots efforts. These typically include 20 to 30 buildings, or a total of 3-4,000 households. Parknet's deployment, he says, is currently a 200-megabit one. By contrast, Indienet, which owns the infrastructure it deploys, is a customer service provider to groups of people who would rather not have to deploy and keep up the infrastructure themselves. Both types of entities typically hook into GlobalConnect, which is a country-wide fiber interconnect provider. GlobalConnect in turn hooks into a variety of wholesale transit providers. At this “back end”, competition also increases, and costs gradually come down across a very complicated board.

In Thomas' own case, his co-op apartment did its own Ethernet deployment, hooked up with one of the transit companies, and distributes an ADSL service to residents. The cost is so low (around $3/month) that it falls under the standard co-op fees to households. Although there is no Indienet to call if something goes wrong, usually nothing does. “The Net goes down maybe once a year”, Thomas says. And, there are plenty of geeks among the residents anyway. Theirs is a totally DIY-IT approach.

Among wholesale transit providers are TDC, which for many decades was Denmark's national PTT (which stood for Postal Telegraph and Telephone, and generically still stands for the old state-owned communications companies). Later, Thomas and I spoke at length with TDC's Per Rasmussen, who made it clear just how complex and competitive the market for Internet deployment and service has become.

For example, in suburban and rural areas, the old cable television head ends are being put to new use by the communities themselves. It's easy to forget today that what we now just call “cable” began as CATV, or Community Antenna TV. In Denmark as well as in the US and elsewhere, cable got its start when communities—on their own or with the help of specialized contractors—put up towers and antennas at some central point (usually where the antennas could see the originating transmitters of TV signals), and amplified those signals down coaxial cabling to the houses of customers. Today, Per explains, these old cables (and newer ones too) are being repurposed for Internet service, with telephony and television riding on top of the Net as services. Meanwhile, a large number of private electric power companies, leveraging war chests of cash gained from selling their power plants, are getting into the phone/Internet/TV triple-play business, driving fiber down many last miles to many homes. With their smaller cash reserves, companies like TDC have to be more careful about how and what they deploy, along with whom and what they connect to.

What gets Thomas most excited is users owning their own infrastructure, and the opportunities for small grass-roots companies like Indienet to grow the Net “from the outside in”—from customers at the edges toward the backbones. All the market fragmentation and competition, he says, serves to force vertical integrators to unbundle their offerings. Hearing this made me envious. Back in the US, most residents have a choice between two completely integrated vertical silos—one from the phone company and one from the cable company. Neither of which seems terribly interested in deploying fiber, much less raw Internet infrastructure. Although TDC and Denmark's power companies might envy that kind of exclusivity and success, Thomas says those are bad models. “The cost of the Net itself is headed toward zero”, he says. “Along the way, it's just a matter of taking on a small infrastructure cost of $300-2,000 dollars for once, instead of paying off a service provider's investment for 50 years with a high premium.”

As I wrap this up, the Net came back up in the hotel, without all the blocked sites and ads (though SMTP and SSH are still blocked). The front desk tells me the problem was a “software mistake”. Seems their service provider installed the wrong firewall. What we had was one intended for a school or a business, rather than for a hotel. “I wish it was simpler”, the guy says.

If the hotel listened to some of the enterprising geeks in the neighborhood, it would be.