Akamai or Backdoor?

Recently my brother contacted me via IM to ask about some strange network behavior on his machine. He was using sysinternals tcpview, and noticed that svchost.exe was opening connections to two IP addresses; one on 80.66.x.x subnet, and another somewhere beneath a different 80.x.x.x subnet. He was concerned because the IP addresses in question showed up as “unassigned EU block” in the RIPE database. The closest assigned block to one of the addresses showed up as being assigned to a company in the Netherlands, and the other to a company in Germany (and GeoIP returned the same information using the original IP addresses).

More interesting was the traceroute. The address that GeoIP reported being in Germany routed to Hurricane Electric in Fremont, California; with the last hop before 80.x.x.x being a 64.x.x.x router in Fremont. Could someone in Germany actually be within one hop of a router in Fremont?

After more investigation, we found a google news posting pointing the finger at Windows Update; and particularly to Akamai servers in the 80.x.x.x range. With a bit more coaxing, we were able to get the RIPE database to reveal that some small subnets within the unassigned blocks were actually assigned to Akamai. I knew that Windows Update and many other MSFT sites contract to Akamai for edge-caching services, so this was a very plausible resolution. However, I am left with a few nagging questions:

  • Are there any better tools or techniques to find out exactly what chunk of code is accessing the network? Knowing that svchost.exe is initiating the connection is not very useful. More useful would be the exact DLL.
  • Akamai works by configuring DNS to resolve differently depending on geographic location (ping download.windowsupdate.com to see this in action). This is a common architecture for our large globally distributed customers’ sites who use routing products like Cisco Global Director and F5 3DNS to accomplish this. However, it leads to a problem — using reverse DNS from an IP addressis rather unlikely to return the same FQDN that was used to resolve the address in the first place. So starting with an IP address like, you have no way of finding out if that was initiated by a call to download122.windowsupdate.com or spywareupload22.gator.com. And considering the way that Akamai provides services to spyware vendors as well as to MSFT, you can’t necessarily trust a network connection just because it is connecting to a block owned by Akamai. It would be ideal if Akamai offered an IP address lookup service that could be used to verify which of Akamai customers was being serviced by a particular IP.

Without at least one of the two above requests, the only way to verify that the connections were indeed made on behalf of Windows Update was to bounce the service and watch the connections die (and assume Windows Update DLL hadn’t been hacked of course).


When I first heard that McDonald’s was planning to launch a new ad campaign themed “Lovin’ It”, I immediately got visions of the horribly tacky “Mentos, the Freshmaker!” commercials. I envisioned some German ad agency telling hapless McDonald’s executives, “We know how to make more teens go to McDonald’s; we’ll use some real groovy stuff and say the words Lovin’ It because then kids will think you are cool!” So today I saw one of the new ads for the first time, and it wasn’t all that bad. Actually it was kind of nice. It’s kind of a feel-good, “happy memories of carefree times” theme, kind of like the Pepsi spots a few years back.

Binary XML

Miguel comments on the “Binary XML” postings from Omri and Dare, pointing out that only two standards would probably be needed (one for size, one for speed) to cover the majority of scenarios. I think this is correct, but in my opinion it’s not the number of encodings that is a problem, but simply the existence of any “standard” encoding beyond XML 1.0.

If you can remember just five short years ago, itwas oncea major decision for IT developers to choose what encoding to persist and send their data:

  • Should it be fixed-width or delimited?
  • Should it be delimited with tabs or commas? What about quotes?
  • Should it be binary or text? ASN.1? DXF? IGES?

Every system used a different encoding technique, and every time you wanted to interop you had to write a parser. Most of ushave written at least a fewparsers for formats like IGES, W3C Log File, and so on. How much money was wasted by people writing parsers?

Now fast-forward to 2003. When a system developer thinks about persisting and sharing data, she automatically thinks “XML”. In 90% of cases, XML is the obvious choice and no debate occurs. Do you think that this happens because XML is a superior format based on size, speed, or any othe technical criteria compared to the options available in 1998? Of course not! XML is the obvious choice because programmers are lazy, many parsers are freely available, and it’s “good enough” for most uses. The fact that XML is ubiquitous leads to plenty of parsing options being available, and more parsing options and tools leads to greater ubiquity. Developers can use XML in most cases and be confident that everyone else in the world will be able to parse out their data with trivial effort. Developers can argue about data schemas now instead of wasting time bickering about parser code and syntaxes. This is a huge contribution!

The thing that many people fail to understand, though, is that none of this virtuous cycle could exist if XML parsers were not trustworthy. XML depends on the fact that well-formed XML can be processed by any parser, and non-wellformed XML can be processed by none. People deploy XML because they know it will “just work” no matter which parser is being used. People deploy XML because they know it will work no matter whether it is IBM or Microsoft in favor that week. Nothing about XML matters more than this promise matters.

So, consider what happens when we introduce some new encodings which are not wellformed XML 1.0, but we call them “XML” anyway. When Jane in the IT department configures her EDI software to send an “XML” file to a partner, and the partner’s machine rejects it, who is to blame? Jane will claim that “my vendor says that XML 1.0bin is a W3C spec, so your vendor is non-standard”, while the partner will claim “my vendor accepts XML 1.0 so your vendor is non-standard”. In fact, it is quite likely that vendors with multiple XML-enabled products would end up in situations where their own products failed to communicate with one another. Note that this danger exists with any variations from XML 1.0, and not just “binary XML”.

Reasonable people might argue that this is OK, and that IT pros will simply have to learn to distinguish between the four different incompatible types of XML (XML 1.0, XML 1.1, XMLfast, XMLsmall) and will have to manage the compatibility mismatches between all of their systems. But that starts to look a lot like 1998 to me. Developers will bicker about which XML to use, and will have to switch parsers based on the choice of data format. Systems will have to offer and consume multiple formats and negotiate formats between one another. I have a good memory, and I remember how badly things used to suck. Having a solid, reliable “obvious choice” like XML 1.0 means freedom from pain for millions of developers. Let’s please don’t mess with that too hastily.

Fair and Balanced?

Bill Gates recently got together with Steve Mills from IBM and demonstrated some web services interoperability between our two companies’ products. It has taken awhile to get to this point, from the initial hype to the point where some of these key scenarios work without smoke and mirrors; so it is nice to see a “status report” like this.

The first response I saw came in the form of this shrill attack piece run on CNET. The author seems stuck in the last century, when people still bought the big lie about “write once run anywhere”. He fails to explain how “runs only on Java” is significantly different from “runs only on Windows”, and completely misses the point that most enterprises have to support both types of systems (and many more) and therefore place a high priority on interop.

The attack piece brought back fond memories of the days when Bob Metcalfe and Jai Singh (now managing editor at CNET) were together at the helm of Infoworld. Then I saw another analysis in CNET, covering the same interop event, but surprisingly balanced, at least in comparison to the first piece. Finally, I found yet another analysis on CNET, again covering the same event; and this one is positively glowingly accurate!

What to make of it? A single presentation by Bill Gates inspires three different pieces in CNET which cover the whole spectrum of opinion. Can’t complain about that.


Tragically, politics shuts down John Poindexter’s data mining program. It seems that only Safeway is allowed to collect that sort of information about U.S. citizens.

Update on List of MSFT Bloggers

Stylesheet has been updated, and the OPML now imports into NewsGator, RSS Bandit, Radio Userland, andprobablyany other aggregator without trouble. I have also added a link to the list in FOAF (RDF) format on the left. Thanks to Leigh Dodds and Dan Brickleyfor hooking me up with the appropriate XSLT. Now that I have a simple XML format that can be edited with Infopath, and XSLT to render to the formats I want, maintaining the listis vastly easier. I’ll refrain from spamming this blog with more update announcements until perhaps we pass the 200-person mark.

America Pop

One of my friends in college was a Japanese guy, here in the U.S. getting his master’s degree in “American Pop Culture”. He had a degree in marketing, andworked for a Japanese marketing agency, and he was taking a break from his work to develop some expertise that would help him more effectively target the American market. Although we haven’t kept in touch, the idea of someone pursuing a graduate degree in pop culture always fascinated me, andI think itmade me pay a bit more attention to pop cultureever since.

Last weekend, I was able to play amateur cultural anthropologist and observe one particular class of pop-culture meme in action. I was having lunch at the airport, and overheard a rather boisterous self-identified geek relating a story to his friends, all of whom were definitely not geeks. The story line was a familiar one; “Once upon a time in the land of Unix, there was a pure-hearted young man named Richard Stallman. The people of the land toiled beneath the crushing weight of the Empire’s brutal reign. The community cried out for a savior to deliver them from oppression andrestore to themcontrol of their own destiny…” The person telling the story missed a few points, exaggerated some others, and fumbled a bit by trying way too hard to get his audience to laugh about the recursive nature of “GNU” (they never got it); but at the end it was satisfying enough. We all know the story.

In some ways, the basic plot is universal, and certainly you can find the same storyas far back asthe Scottish legends of the warriors who fought the Roman occupiers, or even the story of Moses in the Bible. But I think the story plays better in America than most other places. The names vary, but the story is remarkably consistent in its ability to capture our hearts. The key ingredients of the story are this:

  1. The empire is too big and powerful; you can never hope to control it or reason with it head-on.
  2. The empire mindlessly serves its own desires, and you and yourcommunityare collateral damage. You will go extinct and the empire will not cry.
  3. Only through action can you hope to earn survival for yourself and your people. You may never eliminate the empire entirely, but you can heave off the yoke of opression from the neck of your own people.

A very current example of this meme can be found in the story of Sherman Austin, who began serving a one-year term in federal prison last week. At the face, he’s not much different than any other college kid who styles himself as a revolutionary/hacker and gets into trouble with the system. But when you follow the story he spins, he could easily have some serious marketing appeal.He’s sort ofa crossover between Mitnick and Mumia, and although I highly doubt that he has the marketing clout of either the “Free Mumia” or now-defunct”Free Mitnick” campaigns, you have to consider that he is just starting out. One hopes, to be sure, that he reflects while in prison and decides to become a quietly productive and settled member of society. But I am very skeptical of this outcome.

The main proof comes from the comments he makes in every interview, and his speech to the crowd at the Che Cafe benefit last month (in audio). He sounds the way that Bobby Seale must have sounded, and he seems to be drawn to the mic. All bets are that he is back at the mic the minute he gets out of prison, and with his talent for attracting controversy he is sure to continue his rise in notoriety.The anarchistcommunity are undoubtedly salivating over this guy the same way that the democrats are currently salivating over Wesley Clark.

At the Che Cafe benefit, Zack De La Rocha recited From a German War Primer, by Bertolt Brecht. It’sa very well-crafted poem. If you’re familiar with the Japanese Anime Cyborg 009, read the tail part of the poem and tell me if you think there is a resemblance:

It smashes down forests and crushes a hundred men.
But it has one defect:
It needs a driver.

General, your bomber is powerful.
It flies faster than a storm and carries more than an elephant.
But it has one defect:
It needs a mechanic.

General, man is very useful.
He can fly and he can kill.
But he has one defect:
He can think.

Finally, since the topic is pop culture, I should point at this story of a Bosnian town who are erecting a statue of Bruce Lee. The story is funny, profound, and even a bit sad. “To be honest, I get sick every time I tell someone I am from Mostar and they ask me whether I am from the east or west side of the city (the city is divided into the Bosniak east side and the Croat west side),” said Nino Raspudic. “That is one of the reasons for building a statue of Bruce Lee. We are hoping that someone in the future will say: “I knew Mostar. That is the city with the Bruce Lee statue. If we succeed in that, then I can retire.”

OPML, Book Authors

Just got back from a meeting with a bunch of book authors, here for the publishers’ summit. They’re all under NDA, of course, but sometimes I am still surprised at how much some of these people know. Robert Scoble is organizing a bloggers/authors dinner at Crossroads tonight, which I’ll also be attending.


In other news, I finally admitted defeat in my effort to manually track all of the MSFT bloggers. You will notice that my link on the left now points to an OPML file instead. There are well over a hundred, and it was getting too difficult to keep up with it. Now I can just maintain the OPML file, visitors to the site can import it to their news aggregators automatically, and it has an attached XSLT so that it renders nicely in the browser if you view it from my site. I started with an OPML file that Benjamin Voigt kindly constructed for me, then merged with the OPML file from blogs.gotdotnet.com. Strangely enough, I was still missing quite a few names after doing this (for example, Mark Fussell and Arpan Desai did not show up on the gotdotnet OPML, even though their blogs are hosted by gotdotnet; and Dare Obasanjowas not on Benjamin’s list). I suspect that there are at least 20 more MSFT blogs that I am missing, but this should be the most complete list available.

Note that the OPML file is rendered using the stylesheet at /blog/blogOpml.xslt, which is a ridiculously simple XSLT you can modify any way you like. I would like to modify it to sort the blogs by most recent update, but really would rather not do that using XSLT.

Computing Supply and Demand

I was impressed with this interview with Greg Papadopoulos, Sun Microsystem’s new CTO. First comment that caught my attention was when he said, “I think the biggest piece of crap going around right now is that this is a mature industry and innovation doesn’t matter.” He goes on at length about how improvements in computing efficiency impact the cost equation: “You have this exponential improvement in computing, but ERP is not getting exponentially complex. So you would certainly expect it to get cheaper. If you went to a CFO and said, “I can give you a million times more computing power,” they would say, “OK, I’m not going to close my books every millisecond, so you are going to give it to me at a millionth of the cost.” But if you go to an R&D department or to Wall Street, they will say, “Please, bring it on.” I think we can actually free-up dollars from other parts of the business. They would definitely see a competitive return doing better simulation of product development, for instance.”

That’s exactly howI see it, too. People are crazy if they think software is played out. He also has some interesting things to say about “utility computing”, web services, and outsourcing.

Test Dynamics

Recently I was discussing with one of our developers the impact that managed code is having on the product groups. For a group like mine, which develops APIs, the ratio of dev to test is usually 1:1; in other words, one tester for every dev. However, with the advent of managed code, our devs are somewhere close to twice as productive as before — this means that one dev can overwhelm a tester pretty quickly. And although managed code tends to be significantly less buggy, the testers still need to do just as much testing — testing isn’t the sort of thing you can stop just because you aren’t finding many bugs. In fact, the ever-increasing emphasis on quality for enterprise customers and security testing means that testers will be expected to do more testing than before. All of this adds up to a trend where the average product group is going to be spending at least 150% more on test resources than dev resources in the near future.

At least, that’s my prediction. I findthe whole ideahas many interesting implications. For example, we will be spending a larger portion of the budget on test, and this will probably coincide with a reduction in support costs — but much of the quality improvements will be due to managed code rather than increased test expenditures.

On the other hand, there is always the possibility that the ever-increasing test expenditures will not coincide with a reduction in the number or severity of high-profile security and quality incidents. In fact, if you were to graph the number of security incidents over time on one line of a graph and the average test expenditures over time on another line, I suspect that the graph of the most recent years would look rather depressing. Optimistic thinking would suggest that the increase in security incidents is due to heightened attention to security and will level off as investments due to training and testing eventually bear fruit. But it’s going to be interesting to observe this.

Another interesting graph would show marketing expenditures from 1995 to present on one line, and corresponding customer perception/satisfaction on another.


Dare, Andy, and Omri are enjoying their new iPods. I’ve been enjoying mine as well. Iwas previously able tolisten to music on my Siemens XDA using a 256MB SD card, but the 30GB drive of the iPod is nice. I’ve already used about 10.5GB, and now am using the Belkin Car Stereo hookup to listen to music in my car. Since others have covered the high points of an iPod, I’ll share a few of the things that bugged me:

  • When the device is cradled, it enters this “do not remove from cradle” mode. You have to click some menu option on the PC before it is “safe” to remove the device. Of course, I have removed the device “unsafely” a few times and it seems to work fine, but it’s annoying to see the big alarmist flashing graphics.
  • About 900 of my songs were in Windows Media format. It took roughly 36 hours of continuous processing for dbPowerAmp to convert them all to MP3 so that iPod could play them.
  • No facility to save the “on the fly” playlist from the iPod UI. Playlists have to be created on the PC.
  • Power On/Off was unintuitive. Every complicated function I was able to figure out without reading the manual; it’s one of the most usable and intuitive pieces of technology I have ever used. But I used the device for nearly a week before figuring out how to turn it off (hold the play button for five seconds).

These are all minor nits; the thing is well worth the money. There is something magical about having every single piece of music in your collection available with the touch of a finger. I expect the iPod to last me at least two years, or until the inevitable convergence occurs when I can get a single pocket-sized device which combines PPC, Phone, WiFi, 2Mpx Camera, and 30GB drive.


[Updated – It’s the 17th; NOT the 10th!] Rumors are that Scoble is organizing another dinner at Crossroads for a buch of bloggers and book authors who will be in townWednesday the 17th at 6:30.