Welcome Guest! You need to login or register to make posts.

Notification

Icon
Error

#large-images #obsolete

Options
Go to last post Go to first unread
Fedor  
#1 Posted : Sunday, February 11, 2007 7:46:00 PM(UTC)
Fedor

Rank: Advanced Member

Groups: Member, Administration, Moderator
Joined: 7/28/2003(UTC)
Posts: 1,558

Thanks: 5 times
Was thanked: 62 time(s) in 61 post(s)
This post is outdated. Graphics Mill 6 and later supports processing of huge images.

When you try to process a huge bitmap with Graphics Mill for .NET, you may encounter the
"Unable to commit memory" error. This articles describes how to work around this problem.


When you try to load a huge bitmap in Graphics Mill for .NET and try to apply some effect to it, you may encounter the following error:

Code:
Unable to commit memory.


Reason

Let's imagine you are trying to load 8000x8000 JPEG image. Each pixel in JPEG takes 24 bits (it is stored in 24-bit RGB pixel format), or, in other words, 3 bytes. The memory buffer size for such bitmap should be 8000x8000x3=192000000, or about 183MB. This is quite block of memory and it is not always possible to allocate it.

Even if the system shows you much more free memory available, it is still may be lacking of memory. The problem is that this memory block should be linear (i.e. continuous). But memory is always fragmented (i.e. all the free memory comprises of a big number of memory portions of different sizes). It is not guarantied that the system will find a memory block of enough size when you try to allocate 183MB.

Moreover, when you apply some transforms to a bitmap, it is often required to create a copy of this bitmap in memory. So even if you were able to load this bitmap, it may fail to allocate memory later, when you try to process it.

General Solution

The only way to avoid this problem is to process it part by part instead of loading entire bitmap at once. Currently Graphics Mill is not able to load a bitmap partially (the only exception is JPEG file format, it is described below more detailed). However you can minimize the memory usage in the following way:
  1. Load an original file.
  2. Split it to several smaller bitmaps using the Crop transform and save them to the temporary files.
  3. Unload the original image.
  4. Apply the necessary effect to each part of the original bitmap.
  5. Create an empty bitmap of the original size.
  6. Load the bitmap parts and draw them onto the empty bitmap.
This way you do not have to allocate huge memory portions at one time when transforming a bitmap (but you have to do it when load and save an image).

This method works fine for certain effects, where each pixel is processed separately. However some transforms (e.g. Unsharp Mask) may have so-called edge effects: some pixels at the edge of the bitmap can be processed differently than in the center. The problem is that such algorithms use neighboring pixels in their calculations, and there are not enough neighbors at the edges. When processing the whole image, these effects appear only at the edges. However, if you apply the approach used above, the effects will also appear at the boundaries of the bitmap chunks. To avoid this, you may “overlap” the parts of the bitmap. In other words, instead of splitting the original bitmap into stripes of the Xwidth, you will have to crop stripes of the X + n width, that is the stripes should be larger on each side. When gluing the parts together, crop the margins out.

Here is an example how to apply Unsharp Mask to an image using the method described above.

Code:
//Load the bitmap.
Bitmap bitmap = new Bitmap(@"D:\large_image.jpg");
            
int width = bitmap.Width;
int height = bitmap.Height;

//Thw width of a stripe
int stripeSide = bitmap.Width / 20;

Crop crop = new Crop();

//As this is just an example, stripes are stored in memory.
Bitmap[] parts = new Bitmap[20];
for (int i = 0; i < 20; i++)
{
	//Each part will have margins on the left and on the right equal to the
	//doubled blur radius
	parts[i] = new Bitmap(stripeSide + 8, height, bitmap.PixelFormat);
	
	//Cut out the part
	crop.Rectangle = new System.Drawing.RectangleF(i * stripeSide - 4, 0,
		stripeSide + 8, height);
	crop.ApplyTransform(bitmap, parts[i]);
	Console.WriteLine("Cut part:" + i.ToString());
}

//Unload the bitmap.
bitmap.Dispose();

//Apply Unsharp Mask
for (int i = 0; i < 20; i++)
{
	parts[i].Transforms.UnsharpMask(1.5f, 2.0f, 0.04f);
	Console.WriteLine("Unsharped part:" + i.ToString());
}

//Create an target empty bitmap.
bitmap = new Bitmap(width, height, parts[0].PixelFormat);

//Paste parts to the target bitmap.
for (int i = 0; i < 20; i++)
{
	//Cut off the margins.
	crop.Rectangle = new System.Drawing.RectangleF(4, 0, stripeSide, height);
	crop.ApplyTransform(parts[i]);
	
	//Paste the part.
	parts[i].Draw(bitmap, i * stripeSide, 0, stripeSide, height,
		CombineMode.Add, 1.0f, InterpolationMode.MediumQuality);
	parts[i].Dispose();
	Console.WriteLine("Pasted part:" + i.ToString());
}

//Save the bitmap.
bitmap.Save(@"D:\large_image_unsharp_mask.jpg");
bitmap.Dispose();


Special Case: JPEG Files

If the images you process are stored in JPEG format, you can improve the algorithm of huge bitmap processing. The key difference of JPEG format is that Graphics Mill is able to crop JPEG image without decompressing it and write a portion of image to a compressed JPEG. To do it, Graphics Mill for .NET provides the LosslessJpegTransform class.

Here is a modified variant of the above algorithm which uses LosslessJpegTransform to process huge JPEG file.

Code:
//Read the width and the height of the image.
//The following will not actually load the bitmap, so the operation will be fast.
JpegReader reader = new JpegReader(@"D:\large_image.jpg");
Frame frame = reader.LoadFrame(0);
int width = frame.Width;
int height = frame.Height;
frame.Dispose();
reader.Dispose();

int stripeSide = width / 20;

//Create a transform for the bitmap.
LosslessJpegTransform transform = new LosslessJpegTransform(@"D:\large_image.jpg");

//Parts if the bitmap will be stored in memory.
System.IO.MemoryStream[] parts = new System.IO.MemoryStream[20];

for (int i = 0; i < 20; i++)
{
	parts[i] = new System.IO.MemoryStream();
	
	//As LosslessJpegTransform cannot crop the bitmaps outside their boundaries,
	//we need to handle special cases.
	System.Drawing.Rectangle rect;
	if (i == 0)
		rect = new System.Drawing.Rectangle(i * stripeSide, 0, stripeSide + 4,
			height);
	else if (i == 19)
		rect = new System.Drawing.Rectangle(i * stripeSide - 4 , 0, stripeSide +
			4, height);
	else
		rect = new System.Drawing.Rectangle(i * stripeSide - 4, 0, stripeSide +
			8, height);
			
	//The crop rectangle should be aligned to the JPEG sample size.
	rect = transform.AlignToSampleSize(rect, JpegAlignToSampleSizeMode.Crop);
	
	//Cut out the part
	transform.WriteCropped(parts[i], rect);
}

for (int i = 0; i < 20; i++)
{
	//Create a temporary bitmap to apply Unsharp Mask.
	Bitmap bitmap = new Bitmap(parts[i]);
	bitmap.Transforms.UnsharpMask(1.5f, 2.1f, 0.04f);
	
	//Cut off the margins.
	if (i == 0)
		bitmap.Transforms.Crop(i * stripeSide, 0, stripeSide, height);
	else
		bitmap.Transforms.Crop(i * stripeSide + 4, 0, stripeSide, height);
	
	//Write tha part to the target bitmap.
	System.Drawing.Point offset = new System.Drawing.Point(i * stripeSide, 0);
	transform.WritePatched(@"D:\large_image_unsharp_mask.jpg", offset, bitmap);

	bitmap.Dispose();
	parts[i].Dispose();
}
transform.Dispose();


Note also that this approach works faster that the general solution described earlier. So if you are sure that you are processing only JPEG files, it is recommended to use it instead of the general one.

Edited by user Wednesday, May 21, 2014 8:15:37 PM(UTC)  | Reason: Not specified

Best regards,
Fedor Skvortsov
Mattias Malm  
#2 Posted : Thursday, April 16, 2009 5:51:36 PM(UTC)
Mattias Malm

Rank: Member

Groups: Member
Joined: 5/4/2006(UTC)
Posts: 17

This post was written in february 2007. Is this problem solved in the current release of Graphics Mill (v. 5.1)?

We frequently need to open large files in different file formats and save them in another format, size or color mode, so the suggestion to split the image in smaller parts doesn't work for us. If this isn't solved yet, can we expect a solution some time soon?

Best Regards
Mattias Malm
Dmitry  
#3 Posted : Sunday, April 19, 2009 3:17:54 PM(UTC)
Dmitry

Rank: Advanced Member

Groups: Member, Administration, Moderator
Joined: 8/3/2003(UTC)
Posts: 1,070

Thanks: 1 times
Was thanked: 12 time(s) in 12 post(s)
This post is outdated. Graphics Mill 6 and later supports processing of huge images.

Hello Mattias,

Unfortunately, 32-bit edition still suffers from this problem, but there is a 64-bit edition of Graphics Mill that can handle huge images.

Edited by moderator Wednesday, May 21, 2014 8:15:50 PM(UTC)  | Reason: Not specified

Sincerely yours,
Dmitry Sevostyanov

UserPostedImage Follow Aurigma on Twitter!
Mattias Malm  
#4 Posted : Sunday, June 7, 2009 5:07:54 PM(UTC)
Mattias Malm

Rank: Member

Groups: Member
Joined: 5/4/2006(UTC)
Posts: 17

Jumping to the 64-bit version would probobly be too big a step for us at this time (with regards to other software needing to be updated). When will you fix the 32-bit version?

Best Regards
Mattias Malm
Dmitry  
#5 Posted : Sunday, June 7, 2009 7:29:27 PM(UTC)
Dmitry

Rank: Advanced Member

Groups: Member, Administration, Moderator
Joined: 8/3/2003(UTC)
Posts: 1,070

Thanks: 1 times
Was thanked: 12 time(s) in 12 post(s)
This post is outdated. Graphics Mill 6 and later supports processing of huge images.

Hello Mattias,

Unfortunately current implementation does not allow support of handling of large images and will require deep refactoring. At this moment we are not ready to step into it.

Edited by moderator Wednesday, May 21, 2014 8:16:03 PM(UTC)  | Reason: Not specified

Sincerely yours,
Dmitry Sevostyanov

UserPostedImage Follow Aurigma on Twitter!
Mattias Malm  
#6 Posted : Monday, September 28, 2009 2:16:10 AM(UTC)
Mattias Malm

Rank: Member

Groups: Member
Joined: 5/4/2006(UTC)
Posts: 17

I know you have written that you aren't ready to step into the modifications required to fix the memory allocation bug. I still would like to raise the subject again though.

This is growing to be a big problem for us and we need to find a solution some time soon. Could you reconsider fixing this problem is the 32-bit version aswell? If not, do you know of any external system tool that could help keep the RAM defragmented on a Windows Server 2003 machine?

/ Mattias Malm
Alex Kon  
#7 Posted : Tuesday, September 29, 2009 9:23:47 PM(UTC)
Alex Kon

Rank: Advanced Member

Groups: Administration
Joined: 1/31/2005(UTC)
Posts: 458

Was thanked: 5 time(s) in 5 post(s)
This post is outdated. Graphics Mill 6 and later supports processing of huge images.

Hello Mattias,

I don't know any tool which is able to defragment virtual address space of the process. And I'm not sure that it even can exist - mapping between virtual address space & physical memory is a really low-level OS stuff.

Unfortunately, the only way I can suggest is to migrate to x64 platform.

Best regards,
Alex Kon

Edited by moderator Wednesday, May 21, 2014 8:16:21 PM(UTC)  | Reason: Not specified

Loose Cannon  
#8 Posted : Wednesday, June 2, 2010 5:21:23 PM(UTC)
Loose Cannon

Rank: Member

Groups: Member
Joined: 7/24/2007(UTC)
Posts: 2

Alex Kon wrote:

I don't know any tool which is able to defragment virtual address space of the process. And I'm not sure that it even can exist - mapping between virtual address space & physical memory is a really low-level OS stuff.


Haha...I transparently defragmented virtual memory as necessary for the games Ultima 8 (1994) and Crusader: No Remorse/No Regret (1995-1996) by Origin Systems/Electronic Arts. If only Windows were smart enough to do today what I was doing for DOS games 15+ years ago....

- TZ
honeylet  
#9 Posted : Wednesday, April 27, 2011 10:20:15 AM(UTC)
honeylet

Rank: Newbie

Groups: Member
Joined: 4/25/2011(UTC)
Posts: 3


Hello,

I am getting this error in this line of code

Aurigma.GraphicsMill.Bitmap bp = _canvasViewer.Canvas.RenderWorkspace(dpi, Aurigma.GraphicsMill.ColorSpace.Rgb);

I am loading into the bitmap, the design region. The sample you gave was loading into the bitmap from a file. The dpi that I set was 600 and the workspace space and height is 1500


Thanks,
Honeylet
Alex Kon  
#10 Posted : Friday, April 29, 2011 12:08:20 AM(UTC)
Alex Kon

Rank: Advanced Member

Groups: Administration
Joined: 1/31/2005(UTC)
Posts: 458

Was thanked: 5 time(s) in 5 post(s)
This post is outdated. Graphics Mill 6 and later supports processing of huge images.

Hello,

1500 pt is approximately 21 inch. With 600 DPI image will occupy about 625 Mb - rather big chunk of memory. Unfortunately current implementation of VObjects doesn't best suit for such dimensions.

Edited by moderator Wednesday, May 21, 2014 8:16:39 PM(UTC)  | Reason: Not specified

Users browsing this topic
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.