Monday, January 30, 2012

Managed IStreams

Most of the time, when we want to use some unmanaged Windows feature in C#, we have a lot of work to do first. But there's a small set of types that come pre-translated, in the System.Runtime.InteropServices.ComTypes namespace. These types provide basic COM features, like activation, naming, storage, etc.

One of those types is the IStream interface, used by COM to represent a data stream. As interfaces go, IStream is a pretty good choice to be one of the few types Microsoft translates for us. In COM, an IStream is the de facto standard for sending data between COM components. And C# developers love the Stream object, for much the same reason. This sounds like a match made in heaven!

Unfortunately, there are some problems with the Framework's version of IStream that makes it harder to use that it really needs to be. Beyond that, the way COM handles streams is a bit different from the way the Framework does, making the two not-entirely-compatible. These differences make the IStream interface, as designed, less useful that it ought to be. So today, we're going to fix it.

Your Stream Is Not IStream

Oddly enough, despite IStream being readily available in the BCL, none of the managed stream classes actually implement it. That means you cannot pass, say, a FileStream into a COM method that takes an IStream. Part of this is probably just a desire to keep COM interop out of the BCL as much as possible, but part of it stems from differences in how managed streams behave. Even writing a simple "managed IStream" wrapper around the Stream object turns out to be rather complex, unless we're willing to severely limit what kind of Stream we're taking about.

Later on, we'll see a more complete implementation of just such a wrapper, that tries to compensate for some of these issues. But first, lets look at the interface itself, and some issues I have with the way it's been defined.

Return Type Mangling

The translated IStream from Microsoft does, of course, work properly with COM, but they made some annoying choices that I think reduce the usability of the interface. The first of these is their selection of return values. In particular, every method in this interface has undergone an almost mechanical replacement of return values, per the standard signature rewriting rules: every method returns a void instead of an HRESULT.

These rules work in the typical case because COM return values follow a very consistent pattern. Most of the time, a COM method will return S_OK on success, or an error code on failure. Any error code becomes an exception, so if the method returns at all, it must have returned S_OK, and we can just ignore it.

But some methods are more complex than that, among them being IStream::Read(). The documentation for this method (which is inherited from ISequentialStream, but rarely implemented that way) indicates that it can return two different success codes: S_OK if a full read was performed, and S_FALSE if a good but partial read was done. This means the return value has meaning, which we are losing out by having .NET throw it away.

Now, in this specific case, there's another way we can tell what the return value "should" have been, but that's besides the point. We are using COM here specifically to interact with code we didn't write, so we no control over how the other component was written. We need to stick to the documented behavior, or risk introducing bugs in our code. According to the documented behavior of IStream::Read(), a consumer of an IStream we provide, for example, would be perfectly within its rights to assume that an S_OK result code meant a full read, and that there was a full cb number of bytes written into the output buffer.

But the situation is even worse than that. Microsoft took the step of discarding all the return values (including potentially useful information), but did not take the next logical step of moving output parameters into the return value! For example, ISequentialStream::Read() is declared (in objidl.idl) like this:
HRESULT Read(
    [annotation("__out_bcount_part(cb, *pcbRead)")]
    void *pv,
    [in] ULONG cb,
    [annotation("__out_opt")] ULONG *pcbRead);
This could be translated into C# in at least three different ways:
uint Read(
    [Out, MarshalAs(UnmanagedType.LPArray SizeParamIndex = 1)] byte[] buffer,
    uint size,
    out uint read);
 
uint Read(
    [Out, MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 1)] byte[] buffer,
    uint size);

void Read(
    [Out, MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 1)] byte[] buffer,
    uint size,
    out uint read);
The first preserves the signature intact, and returns the HRESULT. The second discards the HRESULT, and moves the output paramter into the return value (making it behave much like Stream.Read()). The third, the least useful of the three, discards the return value for no apparent reason. Of course, it's this third option that the BCL IStream takes, them manages to make it even worse (as we'll see in a bit):
void Read(byte[] pv, int cb, IntPtr pcbRead);
Similarly, the Clone method has a single output parameter that is just begging to be the return value, but its not. Our version will definitely be more judicious in our choice of return values, but before we get there, we have another issue to deal with: that IntPtr parameter in the Read method.

Too Many IntPtrs!

According to the MSDN documentation for IStream, the output paramter in Read is not optional: the caller must supply an unsigned integer reference when calling Read, so we can write the byte count to it. But, as you can see from the previous method signature, Microsoft has opted to use an IntPtr for this parameter (and the corresponding one for Write).

This choice presents two different opportunities for us to accidentally break the constract. First, it makes it possible for us pass in IntPtr.Zero. This would crash any COM code that failed to check for NULL before writing to them. Second, it means in an implementation we write, we could forget to fill in those values, which the caller may rely on. A simple change to the interface would make the compiler remind us to provide the expected unsigned integer value on both callee and caller side.

In general, IntPtr is a failsafe, last-resort kind of thing. We fall back on IntPtr when there's absolutely no other way to get C# to do the correct thing, like when we need bizarre memory allocations or unsupported C language features (which we have coming up). Using it when there's a perfectly good out keyword is just not a good idea.

Doing It Better

Of all the methods in IStream, Read and Write are the two we're most likely to call a lot, so it's silly for them to be as difficult to use as the BCL versions are. Lets look at how we could go about implementing them better. Before we start, we have a bit of a quirky situation here that warrants some explanation. The IStream interface that ships with the BCL includes a Read and Write method, and doesn't inherit from anything (except, implicitly, IUnknown). But both the IDL for the interface, and the IStream documentation, clearly indicate that Read and Write are inherited from a base interface, ISequentialStream. What gives? This is explained in the ISequentialStream documentation:
Note Most applications do not implement ISequentialStream as a separate interface, and you are not required to provide it separately even if you provide an IStream implementation. For example, the compound file implementation of structured storage does not succeed on a QueryInterface method for ISequentialStream but it includes the Read and Write methods through the IStream interface pointer.
ISequentialStream is just two methods, Read and Write, and we'll probably never use it by itself. So we will follow the crowd here and just include our "inherited" members at the top of our child interface. Because of the way a COM interface is structured in memory, this is guaranteed to work, so we have nothing to worry about. Note that we could choose to implement ISequentialStream first, then inherit IStream from it; interface inheritance for ComImport interfaces works exactly as COM expects it to. It's really just a question of usefulness: are we ever going to use the base interface by itself? If not, why bother translating it?

We've already seen what Read looks like, we just need to decide what to return. I'd really like to use option #2 from above, which makes this work just like Stream.Read, but we've already seen the potential problems with discarding the return value. Since we're planning to be a provider of IStream here, not just a consumer, we need to be extra careful to follow all the rules, so we'll just leave the Read signature intact. Write, on the other hand, only has a single successful return code, so we could move it's final parameter into the return value. Here we have to decide between making Write more convenient and keeping the two method signatures the same. I may change this later if it gets to be a pain, but for now preserve them both:
uint Read(
    [Out, MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 1)] byte[] pv,
    uint cb,
    out uint pcbRead);
 
uint Write(
    [MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 1)] byte[] pv,
    uint cb,
    out uint pcbWritten);
Next up is the Seek method. There are several interesting things happening here:
HRESULT Seek(
    [in] LARGE_INTEGER dlibMove,
    [in] DWORD dwOrigin,
    [annotation("__out_opt")] ULARGE_INTEGER *plibNewPosition);
That annotation on the third parameter jumps out first: "__out_opt". The presence of an annotation tips us off that this parameter is special, and the documentation tells us this value can be set to NULL if we don't care about the new position. This is different from Read and Write, where the parameter is mandatory. We now have a situation where a parameter can either be a long, or null, which is illegal in C#. We've seen this before, and we know how to handle it. In this case, Microsoft's choice to go with IntPtr was the right way to go.

Slightly more interesting is that middle parameter. We could mechanically translate that as a uint, or we could dig deeper and get a little smarter. The documented values for that parameter come from the STREAM_SEEK enumeration, defined a bit earlier in the IDL as this:
typedef enum tagSTREAM_SEEK
{
    STREAM_SEEK_SET = 0,
    STREAM_SEEK_CUR = 1,
    STREAM_SEEK_END = 2
} STREAM_SEEK;
Again, we could mechanically translate that, but hopefully those values look vaguely familiar to you. The type, names and values for that enumeration exactly match System.IO.SeekOrigin! And since we know that enumerated values are marshalled as whatever integral type they derive from, the enumeration works perfectly here as the paramter's type:
void Seek(
    long dlibMove, 
    SeekOrigin dwOrigin, 
    IntPtr plibNewPosition);
The SetSize and CopyTo methods are more of the same. For the Commit method, MSDN tells us that it's parameter is a set of STGC flags. Those are defined elsewhere, in wtypes.h, and translate pretty straightfowardly:
public enum StorageCommit
{
    Default = 0,
    Overwrite = 1,
    OnlyIfCurrent = 2,
    DangerouslyCommitMerelyToDiskCache = 4,
    Consolidate = 8
}
We do this same exercise when we reach LockRegion and UnlockRegion, with the LOCKTYPE enumeration defined earlier in the IDL.

The Stat method takes a parameter of type STATSTG, and at a quick glance, defining a StorageStatistics structure doesn't look too daunting. Before we go down that road, though, remember that Microsoft already translated this interface for us. Just because we aren't going to use their interface definition doesn't mean we can't use the other types they've translated. In fact, it's a good idea to browse the namespace documentation for System.RunTime.Interop.ComTypes when doing particularly complex translations, just to make sure you aren't reinventing the wheel. And we find that yes, in fact, there is already a translation of STATSTG there. (I hate the name, but gift horse, mouth, etc.)

A word of caution: there are two STATSTG types defined in the BCL. For the early versions of the Framework, all of this COM interop stuff was in the System.Runtime.InteropServices namespace. In 2.0, some of these were moved into the ComTypes sub-namespace, and the originals are now tagged obsolete. This is one of the rare cases that I will use namespace aliases to avoid the conflicting types.

With that type out of the way, and the StatFlag enumeration also translated, our final IStream interface looks like this:
[ComImport]
[Guid("0000000C-0000-0000-C000-000000000046")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IStream
{
    uint Read(
        [Out, MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 1)] byte[] pv,
        uint cb,
        out uint pcbRead);
 
    void Write(
        [MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 1)] byte[] pv,
        uint cb,
        out uint pcbWritten);
 
    void Seek(
        long dlibMove, 
        SeekOrigin dwOrigin, 
        IntPtr plibNewPosition);
 
    void SetSize(
        long libNewSize);
 
    void CopyTo(
        IStream pstm, 
        long cb, 
        IntPtr pcbRead, 
        IntPtr pcbWrittem);
 
    void Commit(
        StorageCommit grfCommitFlags);
 
    void Revert();
 
    void LockRegion(
        long libOffset, 
        long cb, 
        LockType lockType);
 
    void UnlockRegion(
        long libOffset, 
        long cb, 
        LockType lockType);
 
    void Stat(
        out iopt.STATSTG pstatstg, 
        StatFlag grfStatFlag);
 
    IStream Clone();
Next time, we'll see how to implement this interface in a real class, and start to work through those behavioral differences we talked about earlier

0 comments: