Will Insufficient RAM Make Rendering Crash or Just Go Slower?  

In the panorama field, hardware is also part of the success. You can discuss here about it: camera, computer, pano head, anything
no avatar
Track
Member
 
Topic author
Posts: 100
Likes: 0 post
Liked in: 0 post
Joined: Thu Sep 16, 2010 2:04 pm
Info

Will Insufficient RAM Make Rendering Crash or Just Go Slower?

by Track » Sat Sep 25, 2010 8:42 pm

Hypothetically, if I'm rendering a 500,000-image panorama on 12GB on RAM.. will it crash and die because I require 192GB.. or will it just take somewhat longer?

Just wondering if I should get 12GB of fast RAM or 24GB of slower RAM.
Last edited by Track on Sat Sep 25, 2010 8:43 pm, edited 1 time in total.

no avatar
foundation
Member
 
Posts: 276
Likes: 3 posts
Liked in: 2 posts
Joined: Mon Jan 15, 2007 4:43 am
Info

by foundation » Sun Sep 26, 2010 10:40 pm

if you are using more than 12gb of RAM, the 1-3% difference in speed between RAM modules in benchmarks, would quickly be made pointless with any disk activity IMO

no avatar
Track
Member
 
Topic author
Posts: 100
Likes: 0 post
Liked in: 0 post
Joined: Thu Sep 16, 2010 2:04 pm
Info

by Track » Mon Sep 27, 2010 9:45 pm

Right, but what if I'm rendering a huge panorama and it tells me "Error: Insufficient RAM".

Because that happened to me already on a 2GB PC and I was only trying to render 500 images.

User avatar
[bo]
Moderator
 
Posts: 1226
Likes: 0 post
Liked in: 0 post
Joined: Fri May 05, 2006 8:16 am
Location: Bulgaria
Info

by [bo] » Tue Sep 28, 2010 9:18 am

The 26 Gigapixel Paris pano was rendered with only 12GB of RAM, so I'd argue that 3x4 (for triple-channel memory) or 2x8 (for dual-channel) should be enough for most projects out there. APP does not try to fit the whole thing into the RAM - one of the great things about cell-based rendering. You can actually render gigapixel panos with 1GB of RAM, this was tested by Kolor team!
Some of my panoramas, posted in the Autopano Pro flickr group.

no avatar
Track
Member
 
Topic author
Posts: 100
Likes: 0 post
Liked in: 0 post
Joined: Thu Sep 16, 2010 2:04 pm
Info

by Track » Wed Sep 29, 2010 9:17 pm

gerardm wrote:Get 8gb sticks or what ever the largest your mother board supports and buy more if you need them. If you are going to aggressively edit the image you will need much more RAM for photoshop. I have 32gb and fast raids. Big pictures take a couple of weeks or longer to complete. There is also the quick and very dirty( ooops: I meant popular), render to gigapan uploader approach. Regardless of how much ram you are planning to purchase, you will require a large scratch disk for your smartblend/photoshop temp disk. A fast raid will help your performance and will cost the same as maxing out your ram. All that RAM requires something as fast to write to and to read from. Look for further scratch disk solutions in the hardware section.

Like the guy above me said, 12GB is enough for 26GP. So why do I need 96GB, which is what I'd get on my board if I got 8GB sticks.

I'm looking to shoot tens of thousands of photos, so I'm going to get 24GB. I could get 48GB, but they would be a lot slower, where as I'm going to overclock the 24GB to 2250Mhz.

Also, I'm building a RAID 0 from two OCZ Vertex 2 SSD's.. which are currently the fastest there is.
But they're only going to be 120GB put together. I guess I could also get like 8 320GB drives in RAID 0.. but is there really a point?
I mean, how is it going to need 500MB/s of bandwidth? That's 100 photos per second. It can't possibly go that fast, even on my 24 cores.

My netbook can easily put together a 500-image 10GP panorama in several hours on 2GB of RAM.
I think you're just being too careful.

no avatar
mediavets
Moderator
 
Posts: 16414
Likes: 2 posts
Liked in: 130 posts
Joined: Wed Nov 14, 2007 2:12 pm
Location: Isleham, Cambridgeshire, UK.
Info

by mediavets » Wed Sep 29, 2010 9:32 pm

Track wrote:My netbook can easily put together a 500-image 10GP panorama in several hours on 2GB of RAM.

Can we see this pano?
Andrew Stephens
Many different Nodal Ninja and Agnos pano heads. Merlin/Panogear mount with Papywizard on Nokia Internet tablets.
Nikon D5100 and D40, Sigma 8mm f3.5 FE, Nikon 10.5mm FE, 35mm, 50mm, 18-55mm, 70-210mm. Promote control.

no avatar
gerardm
Member
 
Posts: 95
Likes: 0 post
Liked in: 0 post
Joined: Tue Aug 29, 2006 8:24 pm
Location: nyc
Info

by gerardm » Thu Sep 30, 2010 12:41 am

Track wrote:
gerardm wrote:Get 8gb sticks or what ever the largest your mother board supports and buy more if you need them. If you are going to aggressively edit the image you will need much more RAM for photoshop. I have 32gb and fast raids. Big pictures take a couple of weeks or longer to complete. There is also the quick and very dirty( ooops: I meant popular), render to gigapan uploader approach. Regardless of how much ram you are planning to purchase, you will require a large scratch disk for your smartblend/photoshop temp disk. A fast raid will help your performance and will cost the same as maxing out your ram. All that RAM requires something as fast to write to and to read from. Look for further scratch disk solutions in the hardware section.

Like the guy above me said, 12GB is enough for 26GP. So why do I need 96GB, which is what I'd get on my board if I got 8GB sticks.

I'm looking to shoot tens of thousands of photos, so I'm going to get 24GB. I could get 48GB, but they would be a lot slower, where as I'm going to overclock the 24GB to 2250Mhz.

Also, I'm building a RAID 0 from two OCZ Vertex 2 SSD's.. which are currently the fastest there is.
But they're only going to be 120GB put together. I guess I could also get like 8 320GB drives in RAID 0.. but is there really a point?
I mean, how is it going to need 500MB/s of bandwidth? That's 100 photos per second. It can't possibly go that fast, even on my 24 cores.

My netbook can easily put together a 500-image 10GP panorama in several hours on 2GB of RAM.
I think you're just being too careful.

Please, ignore my post. I only speak from experience, D3X tiffs at 16 bit are about 150MB each. I have nothing else to offer.
Last edited by gerardm on Thu Sep 30, 2010 9:22 am, edited 1 time in total.

User avatar
AlexandreJ
Kolor Team
 
Posts: 5988
Likes: 7 posts
Liked in: 10 posts
Joined: Mon Nov 14, 2005 4:56 pm
Location: Francin, France
Info

by AlexandreJ » Thu Sep 30, 2010 3:56 pm

For stitching, the speed will depend on the amount of RAM and this RAM is needed. With 8 Giga, you can stitch 2000 images without any issue.
If you need to stitch more, you'll need more RAM ( a lot of image means a lot of CP created and we need to store in memory all of them to be able to match them together and find relation between them ). But if you use gigapan or papywizard plugin, you won't need as much memory. With them, as we know the relations which are possible, we don't solve for the general case but for a smaller version. It means less memory used.

For rendering, it's another story and far more simple to understand : a lot of memory, everything is done in memory and the speed should be at the max. If the panorama cannot be done in memory, we'll use disk and it will just slow down the process. That's engine v2.5 behavior.

User avatar
gkaefer
Member
 
Posts: 3550
Likes: 0 post
Liked in: 15 posts
Joined: Tue Jun 09, 2009 1:01 pm
Location: Salzburg
Info

by gkaefer » Thu Sep 30, 2010 4:02 pm

your initial question does not cover the complete problem.

I for my self did a >800 images pano - 15 images each stacked (so finally only 2.8gigapixel finally) with win7 64bit and 8gig Ram. no deal pano is possible.

finally 8gb in filesize.

initially loading even only 800 images into APG is only "speedy" (=90minutes to get the group of 800 images), if temp directory was placed on SSD. I tried it on Raid1 C:\ standard OS directory - no chance... to slow.

and same during rendering phase. after all available ram was used swapping started... so here again only with SSDs was final a way to finish the pano.

and the final 8gig file.... try this file to open on 8gig ram PC.... working on the file is not possible because to slow.
(a zoom into the pano in APG editor e.g. takes from 0.8%->100% about 2 minutes....)

so here my preference are def. using as many RAM and as many SSDs as (financially) possible.

Georg
Last edited by gkaefer on Thu Sep 30, 2010 4:05 pm, edited 1 time in total.

User avatar
[bo]
Moderator
 
Posts: 1226
Likes: 0 post
Liked in: 0 post
Joined: Fri May 05, 2006 8:16 am
Location: Bulgaria
Info

by [bo] » Fri Oct 01, 2010 12:06 pm

For rendering ... a lot of memory, everything is done in memory ... If the panorama cannot be done in memory, we'll use disk ... That's engine v2.5 behavior.

Wait a second, I think we need a clarification. Say you have 8GB of RAM - how big of a panorama can "be done in memory"? How much of this 8GB will APP use - all available (say, 7GB)? If we have 6GB of input TIFF files, will the panorama be done in memory?

Also, if we have 12GB of input images, in what chunks APP splits the job, how much of the memory it uses? Does it fill 7GB with a piece of the panorama, compute, then store temp file, then get another chunk, etc?
Some of my panoramas, posted in the Autopano Pro flickr group.

User avatar
AlexandreJ
Kolor Team
 
Posts: 5988
Likes: 7 posts
Liked in: 10 posts
Joined: Mon Nov 14, 2005 4:56 pm
Location: Francin, France
Info

by AlexandreJ » Fri Oct 01, 2010 4:19 pm

For rendering, we are still currently improving that part, so I cannot answer yet to the question : with 8 GB ram, which size of panorama can I do fully in memory. In fact, the answer will be really complicated depending on the overlap percentage, bracketing or not, etc.

In any case, during rendering, if you don't have enough memory, we'll use drive and that's it.
How does it works you may ask ?
- we designed a new image type which is reentrant and which manages its contains in chunk ( 8 MB in size on 64 bits and 4 MB in size on 32 bits ).
- if the image to be stored is over the left free memory, a generic memory manager will make some chunk free.
- every filter / processing algorithms have been rewritten over that system to use this new method of managing pixel in memory chunk.
- processing algorithm can use multithreading and still doing an operation on a big image.

About input images : if you have 12GB input images and every image are just projected on a small 4K x 4K zone, the memory needed will depends on that zone ( panorama size ) and not on the 12GB input images. That has changed too between old and new engine.

User avatar
[bo]
Moderator
 
Posts: 1226
Likes: 0 post
Liked in: 0 post
Joined: Fri May 05, 2006 8:16 am
Location: Bulgaria
Info

by [bo] » Tue Oct 05, 2010 4:16 pm

Thanks a lot for the inside info, delivers some food for thought.
Some of my panoramas, posted in the Autopano Pro flickr group.

User avatar
klausesser
Member
 
Posts: 8828
Likes: 5 posts
Liked in: 63 posts
Joined: Mon May 22, 2006 12:18 am
Location: Duesseldorf, Germany
Info

by klausesser » Tue Oct 05, 2010 8:50 pm

I`m just thinking about putting 32GB RAM into my MacPro. But it needs to be fully-buffered RAM with my model - that´s more expensive than usual RAM
To calculate how much indeed is useful and how much would be just nice-to-have but not essential would be of interest for me - i´m not interested in the gigapixel race but stitching/rendering just big images for very big printings. As i experienced that means usually sizes of 1-2 gigapixels printable on gigaposters for the use of hanging on buildings - which means 20 -30m width and about 5-10m height @50 or 80dpi.

Given those premises of working ergonomically with those images - retouching, composing for example - and the way Photoshop deals with big sized images i guess 32GB should work . . . . would it? What do you think?
Or would i be wiser to use a fast RAID instead of the amount of RAM because 32GB RAM alone wouldn´t do it anyway?

best, Klaus
Last edited by klausesser on Tue Oct 05, 2010 8:53 pm, edited 1 time in total.
Simplicity is the keynote of all true elegance. Coco Chanel

no avatar
tived
Member
 
Posts: 796
Likes: 0 post
Liked in: 0 post
Joined: Fri Jul 11, 2008 3:49 pm
Location: Dane in Western Australia
Info

by tived » Wed Oct 06, 2010 12:55 am

klaus,

how much ram do you have now?

Henrik

User avatar
klausesser
Member
 
Posts: 8828
Likes: 5 posts
Liked in: 63 posts
Joined: Mon May 22, 2006 12:18 am
Location: Duesseldorf, Germany
Info

by klausesser » Wed Oct 06, 2010 1:36 am

tived wrote:klaus,

how much ram do you have now?

Henrik

8GB.

best, Klaus
Simplicity is the keynote of all true elegance. Coco Chanel

User avatar
AlexandreJ
Kolor Team
 
Posts: 5988
Likes: 7 posts
Liked in: 10 posts
Joined: Mon Nov 14, 2005 4:56 pm
Location: Francin, France
Info

by AlexandreJ » Wed Oct 06, 2010 7:39 am

To give some value, our biggest computer desktop computer here is an intel i970 with 8 GB ram. It renders a 4 Gigapixels panorama with antighost in one hour ( with that disk usage : 70 GB I/O ). I'm not sure this kind of panorama would fit in memory even with 32GB ( I cannot test that myself as I don't have such amount of RAM on a single desktop ). Nevertheless, given the timing, 1 hour for 4 Gpixels, I found that really acceptable.

User avatar
AlexandreJ
Kolor Team
 
Posts: 5988
Likes: 7 posts
Liked in: 10 posts
Joined: Mon Nov 14, 2005 4:56 pm
Location: Francin, France
Info

by AlexandreJ » Wed Oct 06, 2010 8:09 am

This night rendering benchmark for a 25 gigapixels on the same desktop computer ( 8 Giga RAM, Intel i970 ) :

Code: Select all
1455 images
Taille: 413187 x 61519
FOV: 329.56 x 43.59
RMS: 6.01
Lentille: Standard
Projection: Cylindrique
Couleur: Aucun

Detection benchmark ( 1455 images ): 7 min:19 s
 Sift extraction: 3 min:15 s
 Kdtree matching: 3 min:20 s
 Validation: 35.317 s
Optimization benchmark: 14.0556 us
 Local approach:    0 ns
 First optimization:    0 ns
 Final optimization:    0 ns
 LDC/MV:    0 ns
Previewing benchmark: 5.80235 s
 ComputePosition: 1.59867 s
 RenderPieces:    0 ns
 ColorEq:    0 ns
 Blending:    0 ns
Rendering benchmark: 6 h:18 min:54 s
 Initialization: 424.862 ms
 Warping: 1 h:5 min:44 s
 Cutting: 38 min:52 s
 Blending: 3 h:45 min:44 s
 Saving: 31 min:20 s
 Temporary data read/write: 571 GB/407 GB
Total benchmark time: 6 h:26 min:19 s

no avatar
Apapane
Member
 
Posts: 140
Likes: 0 post
Liked in: 0 post
Joined: Mon Sep 01, 2008 2:04 am
Location: Pittsburgh, Pennsylvania, USA
Info

by Apapane » Wed Oct 06, 2010 8:26 am

I run WinVista Ultimate 64
AMD Phenom II X4 940 3.01 Ghz
8 Gb RAM

APG2.5B1 rendered a 2.24 Gpixel image in about 2.5 hours.
Details with the pano:

http://gigapan.org/gigapans/60217/

I need a second job to upgrade to Win7, faster processor, and more RAM! - also to get a new camera. :^\

regards,
Richard

User avatar
[bo]
Moderator
 
Posts: 1226
Likes: 0 post
Liked in: 0 post
Joined: Fri May 05, 2006 8:16 am
Location: Bulgaria
Info

by [bo] » Wed Oct 06, 2010 9:36 am

That's great time for the 25GP Paris! Of course the most interesting parts are:

Warping: 1 h... - is that CPU dependent only or amount of RAM influences the speed? Does HDD speed comes in play here?
Cutting: 38 min... - same question as above?
Blending: 3 h... - same question as above?
Saving: 31 min... - I guess that's the time it takes to save the whole thing to the HDD, so probably a fast SSD/RAID will speed it up?
Some of my panoramas, posted in the Autopano Pro flickr group.

no avatar
tived
Member
 
Posts: 796
Likes: 0 post
Liked in: 0 post
Joined: Fri Jul 11, 2008 3:49 pm
Location: Dane in Western Australia
Info

by tived » Wed Oct 06, 2010 10:42 am

Alexandre,

could you give up the spec's on your test desktop machine, I still think a fixed image/pano would be great to have to compare the various configurations.
Also how do you extract the data you have above, with the timing. Sorry if this is obvious

Henrik

User avatar
klausesser
Member
 
Posts: 8828
Likes: 5 posts
Liked in: 63 posts
Joined: Mon May 22, 2006 12:18 am
Location: Duesseldorf, Germany
Info

by klausesser » Wed Oct 06, 2010 11:40 am

The most relevant part for me is detecting and correcting/editing - the final renderings can be done over night or while having some sushi ;)

best, Klaus
Simplicity is the keynote of all true elegance. Coco Chanel

User avatar
klausesser
Member
 
Posts: 8828
Likes: 5 posts
Liked in: 63 posts
Joined: Mon May 22, 2006 12:18 am
Location: Duesseldorf, Germany
Info

by klausesser » Wed Oct 06, 2010 11:42 am

AlexandreJ wrote:Nevertheless, given the timing, 1 hour for 4 Gpixels, I found that really acceptable.

Yes - definitely.

best, Klaus
Simplicity is the keynote of all true elegance. Coco Chanel

User avatar
[bo]
Moderator
 
Posts: 1226
Likes: 0 post
Liked in: 0 post
Joined: Fri May 05, 2006 8:16 am
Location: Bulgaria
Info

by [bo] » Wed Oct 06, 2010 12:36 pm

Henrik, the test machine is listed in the post by Alexandre - i970 with 8 GB ram. The timings are from the rendering report in APP 2.5.
Some of my panoramas, posted in the Autopano Pro flickr group.

User avatar
AlexandreJ
Kolor Team
 
Posts: 5988
Likes: 7 posts
Liked in: 10 posts
Joined: Mon Nov 14, 2005 4:56 pm
Location: Francin, France
Info

by AlexandreJ » Wed Oct 06, 2010 2:04 pm

'[bo wrote:']That's great time for the 25GP Paris! Of course the most interesting parts are:

This was not Paris image, but a gigapan panorama. So the detection time is given also in the benchmark : 7 mins 19s to get the preview with everything quite right.

'[bo wrote:']Warping: 1 h... - is that CPU dependent only or amount of RAM influences the speed? Does HDD speed comes in play here?

Totally CPU depends. RAM, no. HDD a little. If it doesn't fit into memory, HDD will be used for temporary files.
I would say 75% depending on the CPU, 25 % on HDD ( really rough number, okay ! Nothing was measured here )

'[bo wrote:']Cutting: 38 min... - same question as above?

75% CPU, 25% HDD, we need some RAM here to accelerate.

'[bo wrote:']Blending: 3 h... - same question as above?

I would say 100% CPU if only in memory, to 30% CPU / 70% HDD depends if it's really big.

'[bo wrote:']Saving: 31 min... - I guess that's the time it takes to save the whole thing to the HDD, so probably a fast SSD/RAID will speed it up?

Sure. This desktop is a standard computer without any fine tunning in the hardware. Just putting a good RAID 0 on the HDD will boost performance a lot I guess. And with SSD in RAID 5, it could be a beast :)

User avatar
[bo]
Moderator
 
Posts: 1226
Likes: 0 post
Liked in: 0 post
Joined: Fri May 05, 2006 8:16 am
Location: Bulgaria
Info

by [bo] » Wed Oct 06, 2010 2:52 pm

Really big thanks for that clarification! So it seems CPU will be the most important component for 2.5... Even a normal HDD takes only 30 minutes (1/13 part) of the whole rendering, so investing in expensive SSD RAIDS or PCI SSD drives does not seem prudent. I'll have to revise my hardware guide :) something along the lines of:
expensive: dual Xeon + 16GB RAM + system/temp SSD + Raptors in RAID0...
cheaper: i950/970 + 12GB (tripplechan) + whatever HDD you can get, at least WD Black
Some of my panoramas, posted in the Autopano Pro flickr group.

Next

Who is online

Users browsing this forum: No registered users and 1 guest