Clemens Vasters on Longhorn

Exploring Next Generation Web Services with Indigo

  • The Capital of Egypt

    Ok, ok. I've said this over and over to Microsoft people over the past year and I can finally say it out loud. Ah, no, I won't. I'll just link.

    [Update: As it turns out, this post made it fairly high up on Google for "The capital of egypt" and I have tons of high school kids coming by getting a bit frustrated about the fact that I don't tell them what the answer is. So, kids, the answer is .... "Cairo", or, in Egyptian, "Al-Qahira"]

  • Chris Anderson's Challenge

    At the SDC conference in Arnhem (NL), Chris Anderson entered the following task into his Pocket Outlook: "Turn Clemens from a server developer into a smart client developer within the next 15 months." We'll see how that goes. Chris: For that to happen, you'll have to give me something that I can seriously fall in love with.

  • Longhorn In Budapest: The relevance of Indigo today

    Filed under:

    I am in Budapest today and I am just done with my Indigo talk (you can find the slides at under “Talks”), having done it for the 6th time on this tour throughout Europe. After the events Den Haag, Oslo, Copenhagen, Helsinki and Geneva, I still find Indigo a very difficult topic to talk about on this tour. It’s not about technology or because my talk doesn’t work: It’s about whether people think it’s relevant to their work.

    The true challenge is to explain to the developers we meet that Indigo is going to be very important for them down the road. I find that when I talk to developers on this tour or look at their evaluation forms that very many of them apparently still write fairly compact (to avoid the word monolithic) ASP.NET applications or Windows Forms applications that use a conservative client/server approach. All presentation and logic resides in one tier and the only remote component worth mentioning is the database. That means that the majority of the folks sitting in my talks hasn’t even touched one of the existing distributed technology stacks that Indigo is set to replace.

    The difficulty presenting Indigo on this tour – alongside sexy stuff like declarative UI programming with spinning Windows and Videos with alpha-blending in Avalon and googlefast cross-media searches across all of your local storage media as in WinFS – is that Indigo is about things that are hidden inside applications and do not surface to the user. Stuff that drives server-applications is sometimes hard to understand without knowing the architectural background and the motivations. (Sidenote: A while ago I heard a rumor from a usually trustworthy source that the spinning balls in the COM+ Explorer exist because COM+ was horribly hard to demo as well and the spinning balls provided a good way of visualizing that stuff was happening.)

    The ideal talk for an unsuspecting audience with little knowledge in distributed systems would have to sell the whole idea of distributed systems to boot, the experiences and errors made, the reasons for why Web services are a good thing, the problems creating the motivation for and the principles of service oriented architectures, a set of some tangible application examples and use cases along with the solutions that Indigo provides; all of that in the same talk and within 75 minutes. And that in a way that developers get to see code and demos, too. That sort of talk would span about 20 years of distributed computing history. I am not sure this fits in 75 minutes. Therefore I think I will have to be happy with only a fraction of the audience being interested and/or willing to appreciate the things that I am talking about here. 

    Very many folks think that the topics I am talking about are only relevant to “big apps” and have a hard time seeing the benefits of something like Indigo – much in the same way as it is with Enterprise Services or Web Services.

    If you believe Don Box, who said at PDC that Indigo will ship at some point between Whidbey and Longhorn, and think about the implications of that, Indigo is in fact relevant to everyone writing applications that expose functionality to other applications in some way – now or at least quite soon. The first ship vehicle for Indigo will be, if Don’s statement holds water in its consequences, some service pack or upgrade pack for Windows Server 2003 and Windows XP. That means nothing less than the entire application infrastructure of Windows Server 2003 is getting a major upgrade probably in a year or so from now.

    If you are writing applications using ASMX, Remoting or Enterprise Services today, the impact of Indigo’s arrival can be immediate if you want to make it so. If you code your applications cleverly today (following guidelines explained by Joe Long here or in my talk) and don’t play too many tricks on the infrastructure – for instance by using the Remoting extensibility points – you should have a fairly smooth upgrade path to Indigo. The goal is that upgrading code will be simple and mechanical in most cases.

  • Indigo: The evolution of the in-memory message

    Filed under:

    The evolution of in-memory concept of messages in the managed Microsoft Web Services stack(s) is quite interesting to look at. When you compare the concepts of System.Web.Services (ASMX), Microsoft.Web.Services (WSE) and System.MessageBus (Indigo M4), you'll find that this most fundamental element has undergone some interesting changes and that the Indigo M4 incarnation of "Message" is actually a bit surprising in its design.


    In the core ASP.NET Web Services model (nicknamed ASMX), the concept of an in-memory message doesn't really surface anywhere in the programming model unless you use the ASMX extensibility mechanism. The abstract SoapMessage class, which comes in concrete SoapClientMessage and SoapServerMessage flavors has two fundamental states that depend on the message stage that the message is inspected in: The message is either unparsed or parsed (some say "cracked").

    If it's parsed you can get at the parameters that are being passed to the server or are about to be returned to the client, but the original XML data stream of the message is no longer available and all headers have likewise either been mapped onto objects or lumped into a "unknown headers" array. if the message is unparsed, all you get is an text stream that you'll have to parse yourself. If you want to add, remove or modify headers while processing a message in an extension, you will have to read and parse your copy of the input stream (the message text) and write the resulting mesage to an output stream that's handed onwards to the next extension or to the infrastructure. In essence that means that if you had two or three ASMX-style SOAP extensions that implement security, addressing and routing functionality, you'd be parsing the message three times and serializing it three times just so that the infrastructure would parse it yet again. Not so good.


    The Web Services Enhancements (WSE) have a simple, but very effective fix for that problem. The WSE team needed to use the ASMX extensibility point but found that if they'd build all their required extensions using the ASMX model, they'd run into that obvious performance problem. Therefore, WSE has its own pipeline and its own extensibility mechanism that plugs as one big extension into ASMX and when you write extensions (handlers) for WSE, you don't get a stream but an in-memory info-set in form of a SoapEnvelope (that is derived from System.Xml.XmlDocument and therefore a DOM). Parsing the XML text just once and have all processing steps work on a shared in-memory object-model seems optimal. Can it really get any better than "parse once" as WSE does it?


    When you look at the Indigo concept of Message (the Message class in the next milestone will be the same in spirit, similar in concept and different in detail and simpler as a result), you'll find that it doesn't contain a reference to an XmlDocument or some other DOM-like structure. The Indigo message contains a collection of headers (which in the M4 milestone also come in an "in-memory only" flavor) and a content object, which has, as its most important member, an XmlReader-typed Reader property.

    When I learned about this design decision a while ago, I was a bit puzzled why that's so. It appeared clear to me that if you kept the message parsed in a DOM, you'd have a good solution if you want to hand the message down a chain of extensibility points, because you don't need to reparse. The magic sentence that woke me up was "We need to support streaming". And then it clicked.

    Assume you want to receive a 1GB video stream over an Indigo TCP multicast or UDP connection (even if you think that's a silly idea - work with me here). Because Indigo will represent the message containing that video as an XML Infoset (mind that this doesn't imply that we're talking about base64-encoded content in an UTF-8 angle bracket document and therefore 2GB on the wire), we've got some problems if there was a DOM based solution. A DOM like XmlDocument is only ready for business when it has seen the end tag of its source stream. This is not so good for streams of that size, because you surely would want to see the video stream as it downloads and, if the video stream is a live broadcast, there may simply be no defined end: The message may have a virtually infinite size with the "end-tag" being expected just shortly before judgment day.

    There's something philosophically interesting about a message relaying a 24*7*365 video stream where the binary content inside the message body starts with the current video broadcast bits as of the time the message is generated and then never ends. The message can indeed be treated as being well-formed XML because there is always a theoretical end to it. The end-tag just happens to be a couple of "bit-years" away.

    Back to the message design: When Indigo gets its hands on a transport stream it layers a Message object over the raw bits available on the message using an XmlReader. Then it peeks into the message and parses soap:Envelope and everything inside soap:Header. The headers it finds go into the in-memory header collection. Once it sees soap:Body, Indigo stops and backs off. The result of this is a partially parsed in-memory message for which all headers are available in memory and the body of the message is left sitting in an XmlReader. When the XmlReader sits on top of a NetworkStream, we now have a construct where Indigo can already work on the message and its control information (headers) while the network socket is still open and the rest of the message is still arriving (or portions haven't even been sent by the other party).

    Unless an infrastructure extension must touch the body (in-message body encryption or signature do indeed spoil the party here), Indigo can process the message, just ignore the body portion and hand it to the application endpoint for processing as-is. When the application endpoint reads the message through the XmlReader it therefore pulls the bits directly off the wire. Another variant of this, and the case where it really gets interesting, is that using this technique, arbitrary large data streams can be routed over multiple Indigo hops using virtualized WS-Addressing addressing where every intermediary server just forwards the bits to the next hop as they arrive. Combine this with publish and subscribe services and Indigo's broadcasting abilities and this is getting really sexy for all sorts of applications that need to traverse transport-level obstacles such as firewalls or where you simply can't use IP.     

    For business applications, this support for very large messages is not only very interesting but actually vital for a lot of applications. In our BizTalk workshops we've had quite a few customers who exchange catalogs for engineering parts with other parties. These catalogs easily exceed 1GB in size on the wire. If you want to expand those messages up into a DOM you've got a problem. Consequently, neither WSE nor ASMX nor BizTalk Server nor any other DOM based solution that isn't running on a well equipped 64-bit box can successfully handle such real-customer-scenario messages. Once messages support streaming, you have that sort of flexibility.

    The problem that remains with XmlReader is that once you touch the body, things get a bit more complex than with a DOM representation. The XmlReader is a "read once" construct that usually can't be reset to its initial state. That is specifically true if the reader sits on top of a network stream and returns the translated bits as they arrive. Once you touch the message content is the infrastructure, the message is therefore "consumed" and can't be used for further processing. The good news is, though, that if you buffer the message content into a DOM, you can layer an XmlNodeReader over the DOM's document element and forward the message with that reader. If you only need to read parts of the message or if you don't want to use the DOM, you can layer a custom XML reader over a combination of your buffer data and the original XmlReader.

  • On the road on The Road to Longhorn

    Filed under:

    I leaving shortly for Den Haag for the first installment of the Longhorn Developer Preview Tour throughout Europe as part of the Dutch Developer Days 2004. We start tomorrow and I am quite excited since this is the first time I will speak about Indigo in any detail to a larger audience. I've witnessed Indigo "forming" from a distance when the team was still in "stealth mode" and it's great to see how it comes along.

    But be forewarned: In my talk there will be no live demos. I have 75 minutes for the talk and I had to decide whether I concentrate on explaining the "M5" milestone that is currently in development in Redmond and which implements the (likely) final programming model or whether I allocate more time to the M4 model found in the PDC build. The decision that I made was that M4 is so different from M5 that unless you want to get a major degree in Longhorn development history or have way too much time on your hands, learning and therefore showing M4 code is almost pointless. I will show code, but it won't run.

    If you want to check out how this first run of my talk goes (as usual, I don't really rehearse talks so this is as spontaneous, "fresh" and probably embarrassing as it gets on this tour), Microsoft Netherlands will have a live webcast tomorrow that you can log into at

  • Listen to Joe Long!

    Filed under:

    Joe Long, the Product Unit Manager for XML Enterprise Services at Microsoft, talks about the Indigo migration story in this recorded presentation on MSDN. If you weren't at Joe's PDC talk and think you don't have 37 minutes time for this, you can still not afford to miss listening to the prescriptive guidance section starting at slide 60, if you ever have or will cross an application domain boundary with a Remoting, Enterprise Services or Web service call on the current stacks. And now leave here and go there.

  • Reporting Longhorn / Whidbey bugs useful?

    Filed under:

    Because most teams at Microsoft seem already a milestone or two ahead of what the Longhorn and Whidbey PDC builds reflect, how much is it worth to report bugs?

    Hello? Redmond? Comments?

  • Lightweight Transactions - A puzzle (I mean, really..)

    Filed under:

    Between PDC and now, I was in Redmond on Monday and Tuesday at a meeting with the Indigo team. One of the topics discussed were the new transaction management capabilities that are part of Indigo (which, for Longhorn, includes a lightweight transaction manager).

    Ingo was there, too, and we had a little argument about how hard it is to write transaction resource managers. Ingo thought that it would be awfully hard to write them and that average programmers would never do so and wouldn’t see the need for them. I said “hey, it’s really trivial”, explained that I consider transactions a very general programming paradigm for much more than just databases and told him that I would write a little demo to prove it. I wrote the demo on the plane going home in about 3 hours. Here it is.

    The “2 Phase Commit Puzzle” application is a little Windows Forms puzzle that doesn’t use Indigo or the Longhorn bits, but rather employs a little lightweight 2PC transaction manager that Steve Swartz and myself hacked up when we were on our Scalable Applications tour this spring.

    The puzzle uses four resource managers (transaction participants). The TileWorker keeps track of the tiles as they are moved around, always votes “yes” on Prepare, does nothing on Commit and rolls all tiles back into their original (shuffled) state on Abort. The TimeoutWorker votes “yes” if the puzzle is completed (pressing the “Done” button) within the preset time-span and “no” otherwise. It does nothing on either Commit or Abort otherwise. The GridWorker votes “yes” on Prepare if the puzzle is completed (order is correct) and otherwise “no”. It also does nothing on Commit or Abort. The OutcomeContingentMessage is a participant that will always vote “yes” on Prepare and shows a “Congratulations” message on Commit and a “You failed!” message on Abort.

    The great thing about this little puzzle is that I could add arbitrary other success/failure conditions for the outcome of the puzzle (e.g. number of moves) without having to rewrite or even touch the code determining the other conditions or the code emitting the result message. I would just have to hook in the new resource and feed it with information from the grid.

    Transactions aren’t just for databases. The discussion of the theory behind this is of course in our already well-known transaction deck.

    Over on my blog are the download links to the game executable (in IE, you need to right-click and save it to the local disk; it will not work if started directly from IE) and the source code archive, including the simple WorkSet transaction manager. Check it out.


  • Petzoldian-style Development on the road to Avalon Nirvana. Sigh!

    Filed under:

    Avalon is very promising. No doubt about that. I can very well imagine how Visual Studio "Orcas" (the one to follow Whidbey) will add fantastic designers for creating stunning Avalon UIs. However, today, my first steps with XAML remind me very much of my first little baby steps exploring the Win16 API back in 1990 using Charles Petzold's Windows Programming book. Of course, it's all on a vastly elevated level and some of the controls are stunningly powerful, but putting together an Avalon app that actually looks good is pretty difficult right now. I am not complaining -- it's very early and I am happy that Microsoft lets me play with the stuff. I am just so horribly spoiled by visual design tools.

    I figure that Avalon will create a lot of new jobs for designers. In fact, if you don't have a designer, your app will look really, really old, no matter how well you master the Avalon technology. 

    And I figure that I'll have to freshen up my math skills on trigonometry and linear algebra in a big way. Vector graphics is a very different ballgame.

    Oh... and MSBuild does really rock the house.

  • Installing the PDC build

    Filed under:

    I’ve got the PDC build running on my box. Jim Allchin was right; it isn’t exactly screamingly fast – at least in Virtual PC. Here are a few notes:

    ·         Needless to say, but: Have a dedicated box for Longhorn or use Virtual PC. I’ll likely get a new box when I am back at home. Sounds like a machine brutally optimized for 3D gaming is a good bet – along with 2GB of memory.

    ·         If you can’t at least assign 256MB of memory to the VPC machine, forget it. If you can allocate more, go for it. Shut down all apps and services on the host you can and give Longhorn room to breathe.

    ·         The VPC2004 from the PDC disks will remove any previous Virtual PC builds from Connectix. The version from the PDC disks will expire February 28th, 2004. Which means that I just went from a licensed copy to a demo copy. I don’t like that, at all.

    ·         You will need to log on to the server to acquire your product key. The userid and password is in the disk booklet.

    ·         Take your time. Installing into Virtual PC takes a very long time. Expect that your box will take about 2-3 hours and I wouldn’t do too much on it during that time. Expect the box to lock up, requiring a hard reset. It did that several times for me.

    ·         I’ve mounted one of the ISO images from Disk 2 into VPC, which seemed to be the most convenient option.

    ·         On my Dell Inspiron 8100, Longhorn comes up in 4-bit color mode and that’s the only mode available. You will have to install the VPC additions into Longhorn to get a graphics driver that works, reboot and then switch to that one.

    ·         Just right after install, with nothing done, the VPC disk size stands at 3GB. I think you should have some 6-10GB available if you want to do anything with it but looking.

  • PDC: Don's solo session on Monday

    Filed under:

    The typical PDC attendee is very special. PDC is not like TechEd where you get very practical information on today’s shipping products. PDC is about futures and it requires a lot of imagination of how applications could look and could work on the new platform. It’s about building excitement for the things to come. PDC attendees are the folks who will make the first wave of applications happen. They are excited about technology and they love to code.

    Don Box’ talk yesterday afternoon (WSV201) was very much about now. I heard a few people complain that he didn’t show enough new code. I don’t think he should have. I found his talk very important and Don delivered his message very well. Don’s talk was very much about architecture. No matter how much you want to see code, it’s not the 1990’s anymore. Simply hacking up an app won’t let you play in a connected application ecosystem that’s powered by Web services. WinFX will enable better applications by simplifying coding complex applications in a big way and making developers more productive. You’ll code less. Code isn’t all that matters. Architecture matters. Negotiation and contracts matter. Design matters.

    There were four key takeaways from his talk: Boundaries between applications are explicit. Indigo’s programming model is different from previous distributed programming models such as COM and Remoting, because it doesn’t make objects implicitly remote. You need to declare things as being remote. The fact that you’re theoretically able to write a local application and can then write a configuration script that distributes this application across multiple machines using Remoting was a naïve approach. Likewise, writing a COM application that’s built as a local application and reconfiguring it to run as a distributed application using a different registry setup is a naïve approach. With Indigo, you will need to start writing applications explicitly as being remote. If you love objects, you will find a few things very restricting in this world, and at first sight. There are no automatically marshaled callbacks, interfaces and objects. There are messages, not object references going across the wire. The endpoints of communication, called services, aren’t fragments of the same application based on the same types and classes. Services are autonomous units which adhere to compatible data contracts and policy, not dependent units that use identical implementations. We share schema, not type.

    Don recommended, as I’ve done earlier here on the blog, one of the most important Indigo talks for anyone who’s building software on today’s platform (that means: everyone at PDC): WSV203, “Indigo”: Connected Application Technology Roadmap; Wednesday, 11:30am, 409AB.  Go.

  • PDC: Rebooting the blog after the keynote: Indigo in a nutshell

    Filed under:

    Here’s my quick, two sentence definition of Indigo in order to give you an idea about the scope of this thing:

    Indigo is the successor technology and the consolidation of DCOM, COM+, Enterprise Services, Remoting, ASP.NET Web Services (ASMX), WSE, and the Microsoft Message Queue. It provides services for building distributed systems all the way from simplistic cross-appdomain message passing and ORPC to cross-platform, cross-organization, vastly distributed, service-oriented architectures providing reliable, secure, transactional, scalable and fast, online or offline, synchronous and asynchronous XML messaging.

  • PDC: Best keynote ... ever.

    Filed under:

    The PDC keynote, featuring Bill Gates, Jim Allchin, Don Box, Chris Anderson and, on video, John Scully, Marc Andreesen, Bill Clinton, Warren Buffet, Sean Combs (P. Diddy) and lot of other folks ... was easiest the best, most substantial, longest and fun keynote I've ever seen. And I've seen very many.

    Longhorn, the Aero shell, the Avalon programming model, WinFS and Indigo rock already and they are going to get better and better as time progresses.

    Hey, Linux Penguins, here's the new thing to clone. Good luck.

  • Steve Swartz and Doug Purdy set to blog about work.

    My good friend Steve Swartz is giving blogging a second try and this time for real. Given that the stuff he's been working on is/was in the stealthier areas of the Indigo effort (not the public WS-* specs), it was pretty difficult for him to blog about work, but now with PDC things are changing.

    In an effort to get newtelligence's PDC T-Shirt, Doug Purdy switched from Radio to dasBlog as well.

    These two blogs will be very interesting places to watch if you are interested in the Indigo programming model.

    Doug Purdy is the Program Manager for the new serialization framework (which consolidates XmlSerializer, BinaryFormatter and SoapFormatter), Steve Swartz drives the Indigo programming model that all of us will use.

  • PDC Countdown: Use ASMX and Enterprise Services Now For Tomorrow

    Filed under:

    Brad More is asking whether and why he should use Enterprise Services.

    Brad, if you go to the PDC, you can get the definitive, strategic answer on that question in this talk:

    “Indigo”: Connected Application Technology Roadmap
    Track: Web/Services   Code: WSV203
    Room: Room 409AB   Time Slot: Wed, October 29 11:30 AM-12:45 PM
    Speakers: Angela Mills, Joe Long

    Joe Long is Product Unit Manager for Enterprise Services at Microsoft, a product unit that is part of the larger Indigo group. The Indigo team owns Remoting, ASP.NET Web Services, Enterprise Services, all of COM/COM+ and everything that has to do with Serialization.

    And if you want to hear the same song sung by the technologyspeakmaster, go and hear Don:

    “Indigo": Services and the Future of Distributed Applications
    Track: Web/Services   Code: WSV201
    Room: Room 150/151/152/153   Time Slot: Mon, October 27 4:45 PM-6:00 PM
    Speaker: Don Box

    If you want to read the core message right now, just scroll down here. I've been working directly with the Indigo folks on the messaging for my talks at TechEd in Dallas earlier this year as part of the effort of setting the stage for Indigo's debut at the PDC.

    I'd also suggest that you don't implement your own ES clone using custom channel sinks, context sinks, or formatters and ignore the entire context model of .NET Remoting if you want to play in Indigo-Land without having to rewrite a large deal of your apps. The lack of security support of Remoting is not a missing feature; Enterprise Services is layered on top of Remoting and provides security. The very limited scalability of Remoting on any transport but cross-appdomain is not a real limitation; if you want to scale use Enterprise Services. Check out this page from my old blog for a few intimate details on transport in Enterprise Services.

    ASMX is the default, ES ist the fall-back strategy if you need the features or the performance and Remoting the the cheap, local ORPC model. 

    If you rely on ASMX and ES today, you'll have a pretty smooth upgrade path. Take that expectation with you and go to Joe's session.

    [PS: What I am saying there about ES marshaling not using COM/Interop is true except for two cases that I found later: Queued Components and calls with isomorphic call signatures where the binary representation of COM and the CLR is identical - like with a function that passes and returns only ints. The reason why COM/Interop is used in those cases is very simple: it's a lot faster.] 

1 2 Next >