2. Oktober 2009

Copying Large Files on Windows

Copying large (>50 GB) files seems to be a problem on Windows (XP). Windows Explorer fails after some time with "Insufficient system resources exist to complete the requested service. (error 1450)". I use GoodSync as a folder syncing program. It has the same problem. I even have a commercial license, because I use it for software deployment on WebDAV and other remote file systems. Suprisingly this commercial file copy program also fails after about 50 GB.

GoodSync support told me "i am not sure why you expect that windows can handle 85 gb file". Good joke. This is the 21st century, guys. Why should it not? Apparently, GoodSync uses the Windows function CopyFileEx. Probably the same as Windows Explorer. I don't know what's wrong with CopyFileEx. There is an old MSDN knowledge base article (kb259837), which admits a problem and continues, that it has been fixed in Windows 2000 SP2. Since Win XP is just an updated Win 2k (XP major version 5.1 compared to 5.0, Vista is 6.0), I suppose the problem is also fixed in XP.

I wrote a simple copy program using the POSIX API (fread/fwrite) which uses Win32 ReadFile under the hood instead of CopyFileEx. Just works. It does as many MB/sec as physically possible, and uses only 1.6 MB memory. So, what's the problem?

Considered my half page of code, CopyFileEx really sucks. The funny part is, that while copying large files in Windows Explorer makes the PC unresponsive and is no fun, my small program copies peacefully in the background. I am so good. Or is CopyFileEx just so bad? Probably the latter. There is nothing special with my code. No fancy tricks, just the basic 1st year CS bachelor stuff.

Lets call it "scopy" for simple copy because all things I do are not stupid, but simple:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char* argv[])
{
if (argc != 3) { printf("Usage: %s src dst\n", argv[0]); return 1; }

char const *szSrc = argv[1];
char const *szDst = argv[2];

FILE* fSrc = fopen(szSrc, "rb");
if (fSrc == 0) { printf("Error opening %s\n", szSrc); return 1; }

FILE* fDst = fopen(szDst, "wb");
if (fDst == 0) { printf("Error opening %s\n", szDst); return 1; }

size_t nSize = 1024*1024;
unsigned char* pBuf = new unsigned char[nSize];
if (pBuf == 0) { printf("Error allocating %d bytes\n", nSize); return 1; }

int nCnt = 0;
__int64 nTotal = 0;

bool bDone = false;
while (!bDone) {
size_t nRead = fread(pBuf, 1, nSize, fSrc);
if (ferror(fSrc)) {
perror("Read error"); return 1;
} else {
size_t nWritten = fwrite(pBuf, 1, nRead, fDst);
if (ferror(fDst)) {
perror("Write error"); return 1;
}

nCnt++;
nTotal += nWritten;

if (feof(fSrc)) {
bDone = true;
printf("\n%I64d\n", nTotal);
} else {
printf("%d ", nCnt);
}
}
}

delete[] pBuf;
pBuf = 0;

fclose(fSrc);
fclose(fDst);

//char c = getchar();

return 0;
}

_happy_copying()

Kommentar veröffentlichen