Friday, February 03, 2012

Seeking The Unseekable: Managed IStream Wrapper

Previously, we took a look at one of the most useful COM interfaces around, IStream. We first briefly looked at the built-in BCL translation for this interface, before building up a new and improved version. But there were a couple of other salient points in that post, unrelated to the actual translation of IStream:
  1. Despite being part of the BCL, none of the BCL Stream types implement IStream, and
  2. Implementing IStream is more complex than simply deriving an IStream wrapper from the Stream base class.
The problem is, there is not a one-to-one mapping between the things an IStream can do, and the things a Stream can do. In particular, all IStream implementations are seekable: IStream makes no provision for a non-seekable stream. This means you can't wrap a NetworkStream, for example from WCF, in an IStream wrapper and hand it off to COM.

Today, we're going to see how to solve that problem, and in the process, look at a real implementation of a COM interface in managed code.

Comparing Streams

The following table gives a brief overview of the capabilities of IStream vs a managed Stream class:

Operation IStream System.IO.Stream
Read DataRead()Read(), ReadByte()
Write DataWrite()Write(), WriteByte()
Change PositionSeek()Seek(), Position
Make a CopyCopyTo(), Clone()CopyTo()
Flush ChangesCommit()*Flush()
Trasactional ProcessingCommit(), Revert()-
Restrict access to parts of a streamLockRegion(), UnlockRegion()-
Change the size of the streamSetSize()SetLength()
Retrieve information about the streamStat()-
Determine a stream's capabilities-CanRead, CanWrite, CanSeek
* - See below for when Commit() behaves like Flush()

As you can see, these operations mostly line up evenly. There are a few gaps, mostly on the COM side, so lets take a look at those. You'll start to notice a few common theme here, especially if you read some of the notes on the MSDN pages for these functions. In the COM world:
  1. IStream means "file stream", 
  2. COM file streams can't actually do half of the things that IStream supports
COM IStream Assumptions

The core COM libraries provide a few standard IStream implementations: a global memory stream and two types of file stream. Not surprusingly, IStream's behavior is targetted at compound storage documents, which form the basis of OLE. The assumption is that most IStreams will be either a file or memory block can be seen in the types of operations you can perform on an IStream.

The first missing piece on the managed side are the transactional commands. The COM methods to open a stream on a new or existing file allow you to specify a storage mode of Direct or Transacted, the latter of which buffers any changes you make until they are explicitly committed. Managed streams don't provide this capability by default, so it initially seems like we will need to deal with this ourselves. Fortunately, we don't need to bother, as we find this very common note buried in the documentation for these methods:
The compound file implementation does not support the opening of streams in transacted mode
It turns out that many of IStream's more advanced operations are entirely optional, and in fact, not available for even the most common type of stream: a compound storage file. If we follow COM's lead and implement a direct-mode-only stream wrapper, we can ignore this whole transaction mess completely. In direct mode, a Commit operation is essentially the same as a Flush operation, and a Revert operation does nothing.

The next thing we see on the COM side is the exclusive region locking for parts of a stream. This feature is present to support the concept of complex, compound streams, where locking a single region is equivalent to locking an entire file within a parent directory. (It can also be used to reserve "scratch" space at the end of a memory stream for cross-client communication.) Again, the default compound document implementation doesn't support locking, so our implementation for these two methods just needs to return STG_E_INVALIDFUNCTION and be done with it.

The last missing method really underscores the IStream's assumption that it's talking to a file. The Stat method is the universally accepted name for the operation to retrieve file system statistics about a given file, including such information as the file name, size, create/change/access times, and in our case, some additional mode or state information. Most of this information will only exist if your stream is a FileStream, so our implementation will need to account for that.

Otherwise, we can see how most of the IStream methods can be implemented as pass-through calls to the managed Stream methods. The real problem comes from what's missing on the other side: three seemingly harmless properties CanRead, CanWrite, and CanSeek.

Not All Streams Are Created Equal

In COM, streams mostly exist as wrappers around random-access forms of storage, like a file or memory buffer. In .NET, however, we have a ton of other stream types. The wrapper class I'm about to show you, for example, came about as a way to wrap a WCF MessageBodyStream and send it into a COM shell handler.

The reason this gets complex is because not all managed streams can do all of the things a normal Stream can do; in particular, network-based streams are non-seekable. But COM clients are expecting a file or memory stream, and have no way to determine otherwise. As such, the first thing a typical COM client does when handed a stream is attempt to Seek(0, SeekOrigin.Begin). In my case, the Adobe preview handler was seeking all over my stream trying to pull out the appropriate pieces of a PDF file for previewing.

To get around this problem, we need to find a way to fake a seekable stream out of a non-seekable one. One way to do this is to implement our own internal stream buffer, and simulate seeks via a combination of forward-only reads and buffered memory operations.

Before we get into the buffering parts, though, lets look at implementing the bits we know we can do easily.

A Basic Managed Stream Wrapper

The first thing we want to do is to knock out the operations that we have no intention of implementing. For our purposes, this is LockRegion, UnlockRegion, Commit, Revert, and Clone. For the first two, COM tells us to return a well-defined error HRESULT. As a matter of course, I've already defined an enumeration that lists the HRESULTS, so now I just need a way to return it. Since I'm using the rewritten method signatures for most of these methods, I can't return the value directly. Instead, we'll use a handy feature on the Marshal class to do it for us:
public void LockRegion(long libOffset, long cb, LockType dwLockType)
Brief aside: when translating the HRESULT codes out of the header files, you may notice that C# doesn't like it when you define a negative signed integer constant in hexadecimal format; the way the compiler performs its type conversions, a constant like 0x80030001 (STG_E_INVALIDFUNCTION) is treated as 2147680257, not -2147287039. In C, the "bit pattern" represented by the hex digit sequence is stored into the integer, and because it is the twos-complement of 2147287039, you get the correct value. In C#, the "integral value" represented by the digit sequence is calculated, and is too big for a signed integer. (In C#, if you want to express a negative value in hex, you express it as -0x7FFCFFFF). Fortuantely, we can ask C# to ignore type safety and act like C via an explicit unchecked typecast:
public const int STG_E_INVALIDFUNCTION = unchecked((int)0x80030001);
Now, back to our stream. The next two operations are almost as simple. For a Revert, we just do nothing, and for a Commit, we call a Flush on the underlying stream.
public void Commit(StorageCommit commitFlags)

public void Revert()
Clone is an interesting case. The idea is to create a new stream pointing to the same underlying data, with it's own seek pointer. Normally, this wouldn't be possible for managed streams because they may not be seekable. However, with our buffering enhancement, we could arguably make this work. It would incur a significant amount of overhead, however, and require synchronization among all cloned streams, to ensure they return the exact same bytes. For now, we're going to leave this one as not implemented, by throwing the exception for STG_E_UNIMPLEMENTEDFUNCTION.

Streamed Data Caching

Which brings us to the meat of this class: the internal data cache. The idea here is to make a non-seekable stream behave as if it were seekable, as far as COM is concerned. To do so, we are going to add an internal buffer to our wrapper that will cache all data read from the non-seekable stream, and use that to fulfill any requests to seek backwards. To do this, we'll need to enhance some of the normal stream behaviors to accommodate our wrapper:
  • When a read happens, return as much as we can from the cache before reading more data from the stream.
  • When a read happens that reads from the stream, cache whatever we read.
  • When a seek happens, set a pointer to the position in the cache that matches the new seek position.
  • If a seek happens to a point past out current position, perform a Read and discard the data.
(This "seek forward by reading data" operation is seen a lot in streaming media contexts, and is called "scrubbing".)

For starters, we only want to do this if we have to: if our underlying stream can seek on it's own, we definitely want to let it. So our constructor will check for that case, and set up a cache only if needed:
public CachingStreamWrapper(Stream stream)
    if (stream == null)
        throw new ArgumentNullException("stream", "Cannot wrap a null stream for COM Interop.");

    this.source = stream;

    if (!stream.CanSeek)
        this.cache = new MemoryStream();
The first place we have to do any real work is in the Read method. Our data cache here is a memory stream, so it has its own internal position and length. This makes it easy for us to tell if a read can be made entirely from the cache:
if (this.cache.Position < this.cache.Length - 1)
    var read = this.cache.Read(buffer, 0, remainder);
    remainder -= read;
    totalRead = (uint)read;
At this point, our remainder will contain the number of bytes that we have left to satisfy the read request; if that's zero, we're done. (Remember, too, that totalRead is an output parameter that holds the final read byte count.) If there's more left, we read from the underlying stream and cache that data as well:
var read = this.source.Read(buffer, (int)totalRead, remainder);
if (this.cache != null)
    this.cache.Write(buffer, (int)totalRead, read);
totalRead += (uint)read;
The other part of this solution is in the Seek method. Here, we need to decide for any given seek request how far from the start of the stream our new position is. If this new position is within the data we've already cached, we just Seek our memory stream to that new position and we're done. Subsequent reads will start from there and return cached data. If not, then we move our cache to the end, then start copying data from our underlying stream into the cache until both are at the desired position. This is a key aspect of our solution: our cache's length must stay the same as our underlying stream position, or we will lose data!
this.cache.Seek(0, SeekOrigin.End);

var missing = (int)(offset  - this.cache.Position);
byte[] buffer = new byte[CHUNK_SIZE];
while (missing > 0)
    var size = Math.Min(missing, CHUNK_SIZE);
    var read = this.source.Read(buffer, 0, size);
    this.cache.Write(buffer, 0, read);
    missing -= read;
The remainder of the operations are pretty straightforward. If COM requests something from us that's simply not possible (SetLength on a read-only stream, for example), we return an appropriate error HRESULT; otherwise, we just pass through.

The full source code for this class can be found here; it depends on some of the translations we've done in previous posts, but nothing you shouldn't be able to handle.