Introduction
This article explains the cause of the ambiguous OutOfMemoryException that is common when using MemoryStream with large datasets, and introduces a class, MemoryTributary, which is intended as an alternative to .NET's MemoryStream that is capable of handling large amounts of data.
Background
When attempting to use MemoryStream with relatively large datasets (in the order of tens of MB), it is common to encounter the OutOfMemoryException. This is not due to, as the name would imply, having reached limitations of the system memory, but in fact those of the process' virtual address space.
When a process requests memory from Windows, the memory manager is not allocating address space from RAM, but 'pages' - chunks (typically 4KB) of storage - which can exist in the RAM, or on disk, or anywhere the memory manager decides to store them. The pages are mapped into the address space of the process, so, for example, when the process attempts to access the memory starting at [0xAF758000], it is in reality accessing the byte at the beginning of [Page 496], wherever Page 496 happens to be. The process therefore can allocate as much memory as it likes so long as the disk space holds out, and can map as much of it as will fit into its virtual address space at any one time - provided, that is, these allocations are made in a large number of small chunks.
This is because the process address space is fragmented: large sections are taken up by the Operating System, others for the executable image, libraries, and all the other previous allocations. Once the memory manager has allocated a set of pages equivalent to the requested size, the process must map them into its address space - but if the address space does not contain a contiguous section of the requested size, the pages cannot be mapped, and the allocation fails with the OutOfMemoryException.
The process is not running out of space, or even addresses: it is running out of sequential addresses. To see this (if you are on 64 bit), target the following program at x86 and run it, then target it at x64 and see how much farther it gets.
Read more: Codeproject
QR: 