should a GPU , RAM and solid stateHDto speed up giga panorama process?

[aokaf]

should a GPU , RAM and solid stateHDto speed up giga panorama process?

by aokaf » Wed May 11, 2011 1:00 pm

I have a mac pro 8 cores desktop, 6 GB ram and no GPU
dealing with a small g-pano takes about 72 hours

my GM has money and he is a good guy, he wants to improve my work
if I buy more ram (like 32 GB) and a GPU (like the one with apple.com 1600$) and change to 1 TB solid state HD
will it ease my pain ?

I don't want my GM to know one day that I took it with no profesionality.

Ahmad Alkaff
5D MKII, Sigma 8mm, NN R1 + monopod manfrotto , manfrotto 303 + carbon tripod 045cxpro4, epic gigapan pro, mac pro 8 cores, apple cenima display 30", macbook pro 17" quad , CS5 standard,, ptgui, easypano, kolor, gardengnome

[klausesser]

by klausesser » Wed May 11, 2011 1:07 pm

I have a mac pro 8 cores desktop, 6 GB ram and no GPU
dealing with a small g-pano takes about 72 hours

my GM has money and he is a good guy, he wants to improve my work
if I buy more ram (like 32 GB) and a GPU (like the one with apple.com 1600$) and change to 1 TB solid state HD
will it ease my pain ?

I don't want my GM to know one day that I took it with no profesionality.

Ahmad Alkaff
5D MKII, Sigma 8mm, NN R1 + monopod manfrotto , manfrotto 303 + carbon tripod 045cxpro4, epic gigapan pro, mac pro 8 cores, apple cenima display 30", macbook pro 17" quad , CS5 standard,, ptgui, easypano, kolor, gardengnome

I also use a MacPro 8 core - they usually have a GPU. 6GB RAM is quite low for such a machine - but it works fine nevertheless.

What is a "small g-pano"? What is a "GM"?

best, Klaus

Last edited by klausesser on Wed May 11, 2011 1:09 pm, edited 1 time in total.

[tived]

by tived » Wed May 11, 2011 1:52 pm

I think 6 Gb is standard on a Macpro :-) but as we know not limited to.
Now I am surprised to hear that there is no graphics card (GPU - Graphics Processing Unit aka Graphics/video card) on the Macpro, anyway, choose a graphics card with 1GB or more it does not have to be a nVidia Quadro CX, but can be either AMD or nVidia's desktop graphics card suitable for a Mac.

For pano work, we love everything fast, and we love lots of it, so the more RAM you can add the better. Now, I am not sure getting a 1TB SSD disk will give you the best performance/$ I would rather buy several smaller SSD's and spread the load between them. When you get a bit closer to decision time we can talk about configuration

Henrik

[aokaf]

by aokaf » Wed May 11, 2011 4:42 pm

thanks klausesser
GM is my general manager
g-pano is a small giga panorama that is 0.38 GB size
like this one in my town jeddah, saudi arabia
http://gigapan.org/gigapans/286c4d034af00c11d722e960b25898cd/

yes tived you are right
ssd comes only in 256 GB
so I'm thinking about one 256 SSD to put the OS X system and adobe application and make the rest on a 2TB BARACODA 7200 rpm HD

my graphic card is
ATI Radeon HD 5770 1GB (the default that comes when mac pro with no customizing)

and i'm thinking about this
http://store.apple.com/us/product/TW386LL/A?fnode=MTY1NDA5OQ&mco=MTA4MzU2Nzg

so, should I go for the ram, GPU, SSD ?
please advice me

[tived]

by tived » Wed May 11, 2011 11:31 pm

Aokaf,

what is your budget?

If money is no issue, then get 32gb of ram, get two SSD'sm one for the OS/APPs and one for scratch/temp disk get 2x 2TB disk WD RE4 / for your data storage in a soft RAID-1

The Quadro card is really only shinning if you are doing CAD or similar things, anyway to me, having had two of them, they are just not worth the price.

all the best

Henrik

[aokaf]

by aokaf » Thu May 12, 2011 12:10 am

If money is no issue, then get 32gb of ram, get two SSD'sm one for the OS/APPs and one for scratch/temp disk get 2x 2TB disk WD RE4 / for your data storage in a soft RAID-1

thanks man!
this is what i'm looking for
32 GB ram 1000 $
2 SSD 1000 $

one virtual tour should cover the price

and the GPU is like you say (not useful) ! waw! thanks man, I would have fallen for that!

[tived]

by tived » Thu May 12, 2011 10:25 am

GPU is useful, but there are other faster for less $$$

Look for Crucial's new SSD's, Intel is good, OCZ Vertex 3 will be handy too. get the latest generation drives as they are twice as fast as the previous one. You can always use them in your next Macpro

Henrik

[aokaf]

by aokaf » Thu May 12, 2011 11:11 am

waw! can you suggest a better GPU with less price ?

[foundation]

by foundation » Sat May 14, 2011 8:24 pm

the GPU will only speed up the previews of the panos, not the rendering. Having the temp drive on a SSD is great if you are shooting raw images and for blending images larger than your RAM. The RAM upgrade will help the most.

[tived]

by tived » Sun May 15, 2011 4:36 am

Hi,

Not to be really picky, but doing raw files and having SSD, have no direct relation on performance, apart from having an extra step in the process, however file size do have an impact.

System bottlenecks are currently,
storage speed and transfer
not enough ram
cpu speed
Most people today have a processor (CPU) which is relative fast, and probably atleast 4GB of ram, but they most often just look at harddisk/data storage as a size thing over how fast it is, therefore it also becomes the biggest bottleneck.

If I were to upgrade my mac I would just get the faster AMD card, its a bit of a shame that there are no 6xxx models available for the Mac's yet, as far as I know. The nVidia Quadro are nice cards, don't get me wrong, but they are heavily overpriced for the performance they give the average user or non-CAD user. Foundation is right in that the GPU will currently only affect the speed of you previewing your pano's and also help you with the same in Photoshop if you use that program.

I am finding it a bit hard to find good fast compatible GPU's for mac, so if someone else can chip in here that would be great

Henrik

[foundation]

by foundation » Wed May 18, 2011 5:12 am

The RAW files are converted to TIFFs and written as TIFFs to the temp folder. This step is a bottleneck in my workflow. Having an SSD greatly speeds up this specific step. With enough RAM most of the other steps can avoid writing to disk, and thus with enough RAM the SSD doesn't help as much unless you are using RAW files which require writing to disk no matter how much memory you have.

[KreAture]

by KreAture » Sat Jul 30, 2011 3:07 am

I think the ammount of file data for input has a large impact on the memory requirements too.
Using bracketing of 3 layers for instance tripples ammount of input data. With 32 GB memory and only about 30 of that available to APx you'd reach the limit at around 7 Gpix input data making final pano around 3-4 Gpix assuming no bracketing, or 1-1.2 Gpix for 3x bracketed. Anything larger will start using the scratch. This is ofcource not counting storing the controlpoints etc too.

A good ammount of RAM is a must, but if the projects are really really big, I think very fast scratch is more important.
Suggest the new 500MB/s+ speed gen 3 SSD's as I've tested one myself and they really help. (You do need a proper SATA III chipset for this to be worth it though, Marvell based SATA III does not cut it, and will be a source of pain, bluescreens and bottlenecking. If you only have SATA II, go for cheaper gen 2 disks and raid em instead.)

My 6.1 Gpix output, 3x bracketed pano with 2214 input images at 16.2 Mpix each, required 143 GB scratch while APP had access to 12 GB memory btw. (To give a comparison.)

[HansKeesom]

by HansKeesom » Mon Aug 01, 2011 11:51 am

Wondering whether the following rule of thumb will apply. If so we can use it for people that are asking about things here.

I would like everyone here to read it with the focal length they normally use in the back of their mind. I am interested to hear whether you think this will work for the focal lenght you use.

Start with the focal length of the lens you use (Full-frame equivalent, f.e. 12 mm).

Important is how the focal length compares to the nr of GB's of RAM you have. F.e. I have 6 GB and use a 8 mm lens on a 1.6 camera which translates into 13 mm. So I have a memory:focal lenght situation that is close to 1:2

-1:1 :If you have as many GB's RAM as the mm's of the focal length ( f.e. 12 GB for 12 mm) you have more then enough RAM, no need to add anymore. If you want to speed up the processing you might consider adding a SSD big enough to hold all the files of one of your project. Copy all photos to this SSD before you start working on them. Depending on the speed of the HDD you were using this will speed up things more or less.

-1:2 : if you have only half the GB's RAM's as the mm of the focal length (f.e. 6 GB for 12 mm) you will be able to detect, edit and render your panorama's in decent time.
You might hear your harddisk a lot. This can be a workable situation but if you have the money and your computer allows it, you will enjoy buying enough RAM to go to the 1:1-situation above.
If your computer's RAM can not (easily) be expanded consider buying a SSD to be used a first temp-directory. Any size SSD will do as you can still use your normal harddisk as second and third temp directory. But as a rule of thumb take the number of panoramas you normally have in one project (f.e. 10) and multipy it by the focal lenght (f.e.12 ). The result is the size of the SSD you want to buy (f.e. 10 * 12 = 120 GB). Less GB's will work, Autopano might overflow into using the second and third temp-directory though which will cause it to slow down.

-1:n with n = 3 or 4 (f.e. you use 12 mm lens and have 4 of even 3 GB RAM), Autopano will get the job done but you will experience it takes much longer to get things done. If you are on a budget and/or not making money with your panoramas, there is no real hurry to upgrade. Otherwise think about upgrading your RAM to get a better 1:n

-1:n with n >4 (F.e. you have 2 GB RAM and use a 12 mm equivalent lens) Autpano will have a hard time getting your job done and is even likely not able to finish the job.

Warning : check to see whether you motherboard can take the memory and whether the operating system you are using can address it. If not, go for an SSD

I did not mention CPU and GPU as I think they are not that important. Also CPU is hard to upgrade and GPU is not much of influence.

[UK Pano]

by UK Pano » Mon Aug 01, 2011 3:06 pm

If you want to speed up the processing you might consider adding a SSD big enough to hold all the files of one of your project. Copy all photos to this SSD before you start working on them. Depending on the speed of the HDD you were using this will speed up things more or less.

I had read something similiar in the past about have the APG application, source images and designated temp folder all upon the same physical drive. My plan has to invest in a 240Gb vertex 3 SSD which will be the main C:\ and also be used for the above as stated.

Does this sound correct or is the performance that much greater if I were to separate the source images and designated temp folder to an equivalent second dedicated SSD purely for APG processing?

[HansKeesom]

by HansKeesom » Mon Aug 01, 2011 4:24 pm

hi UK Pano,

I did not suggest putting everything on one drive but it is an option, feel free to use it.

However I never understood the idea of having a SSD as your C:\-drive. You computer will problably start up very quickly but after that it seems like an expensive way to store files you not often use.

Having seperate SSD-s for source, temp and even page-files is likely a faster option but we are problably talking about the difference between very fast and very fast.

Adding a SSD to a current system is likely to be an easy and not too expensive way to increase the throughput. I did so with and i7 6 GB system, added a 60 GB SSD and enjoyed it ever since.

[KreAture]

by KreAture » Mon Aug 01, 2011 10:35 pm

I think your suggestion lacks a very important factor Hans, which is the fov.
It really only applies to 360 tours and you do not account for your ammount of overlap.

I've tested a bit with different input ammounts and so far they seem to dictate the scratch/memory needs more precisely than the actual focal lengths.
I do small sections at 900mm and they render in just 26 mins delivering 1.5 Gpix. The actual ammount of input data seems to be transformed to 32bit form first, which has to be held in ram+scratch. Then as the rendeirng progresses data is clipped and later stacked/blended and fed back to the scratch. It appears the clipping actually reduces memory usage again. The blending needs to read data from a LOT of files (or memory blocks if held in mem) pr thread in order to blend, especially when stacking.

I wonder if Kolor can chime in here... It appears that the layers are made before stacking, or that stacking happens together with blending. Would it not be faster to do the stacking individually before blending? Maby not due to the anti-ghost processing?

Anyways, if people can test my "<input image rez> * 4 * <input image count> = scratch" idea by noting how large their scratch gets as warping finishes it would be helpfull.

[HansKeesom]

by HansKeesom » Mon Aug 01, 2011 10:47 pm

I think your suggestion lacks a very important factor Hans, which is the fov.
It really only applies to 360 tours and you do not account for your ammount of overlap.

I've tested a bit with different input ammounts and so far they seem to dictate the scratch/memory needs more precisely than the actual focal lengths.
I do small sections at 900mm and they render in just 26 mins delivering 1.5 Gpix. The actual ammount of input data seems to be transformed to 32bit form first, which has to be held in ram+scratch. Then as the rendeirng progresses data is clipped and later stacked/blended and fed back to the scratch. It appears the clipping actually reduces memory usage again. The blending needs to read data from a LOT of files (or memory blocks if held in mem) pr thread in order to blend, especially when stacking.

I wonder if Kolor can chime in here... It appears that the layers are made before stacking, or that stacking happens together with blending. Would it not be faster to do the stacking individually before blending? Maby not due to the anti-ghost processing?

Anyways, if people can test my "<input image rez> * 4 * <input image count> = scratch" idea by noting how large their scratch gets as warping finishes it would be helpfull.

Yes I am talking 360*180 here.
fov is directly connected to focal length. Overlap is generally something like 25%, but I did not want to make things too complex.

When you talk of scratch are you talking about the temp-directory?
Scratch will be bigger then the RAM you have indeed.
Yes when stacking multiply everything with nr of stacks, again, I was keeping things simple and i fuse them before loading them into autopano.

Would be nice to see if your formula is more precise and can be integrated in my descriptions of situations and advices. So your formula giving us a number of GB's which we can then compare to the RAM someones has.

Last edited by HansKeesom on Mon Aug 01, 2011 10:58 pm, edited 1 time in total.

[KreAture]

by KreAture » Tue Aug 02, 2011 12:02 am

Yes. My formula gives the number of GB's directly.
I am unsure if one can subtract most of the "use memory" ammount of ram from the result and the rest will be scratch of if it will always create all images on scratch if it needs to create any (as it will need to be swapping em out and in anyways...)
However, scratch should be 0 if you have more "use ram" than my formula calculates from input images. (If the formula is correct hehe.)

[KreAture]

by KreAture » Tue Aug 02, 2011 1:07 am

I just ran a pano with 1098 (3x stacks) 16.2 Mpix images through a 100% scale render and it used 67 GB scratch.
According to my estimate it should have used 16.2 * 1098 * 4 = 71150.4 MB or 69.48 GB. I missed my mark by 2.4 GB and APP was using 12.8 GB memory so it probably kept a few sets there.
As soon as it started cutting, the number went down again.

[HansKeesom]

by HansKeesom » Tue Aug 02, 2011 9:04 am

sounds like your formula is working.
Now what happens when you want to process multiple panorama's? Will the memory-use stay around 12.8 GB and scratch be relative to the number of panorama's?

[KreAture]

by KreAture » Tue Aug 02, 2011 1:41 pm

It only renders one pano at the time. They are placed in the batch-queue.
I see it deletes everything after render is done, so I assume this would be no issue.

[UK Pano]

by UK Pano » Tue Aug 02, 2011 3:31 pm

hi UK Pano,

I did not suggest putting everything on one drive but it is an option, feel free to use it.

However I never understood the idea of having a SSD as your C:\-drive. You computer will problably start up very quickly but after that it seems like an expensive way to store files you not often use.

Having seperate SSD-s for source, temp and even page-files is likely a faster option but we are problably talking about the difference between very fast and very fast.

Adding a SSD to a current system is likely to be an easy and not too expensive way to increase the throughput. I did so with and i7 6 GB system, added a 60 GB SSD and enjoyed it ever since.

Hi Hans,

Thanks for replying. Lol! No I wasn't suggesting that it has been yourself on the C: but it was something that I had heard in another post and wanted to ask given we were talking hardware as I've been planning a new PC build for a while (nearly started to build the machine a month or so ago but funds now needed elsewhere for a while) and like to keep on top of the latest recommendations.

Does makes sense to have a HDD rather than SSD for the C:\ rather than a shiny new SSD that isn't going to be used as much. So for source, temp and (possibly) page file, look at a couple of separate SSD's as an option, one large covering source and temp and a smaller for page-file / overflow?

Look forward to hearing your (and anyone else's) thoughts

Regards,
Paul

[HansKeesom]

by HansKeesom » Tue Aug 02, 2011 3:45 pm

Will autopano always use the temp-directory of will it not do so when enough ram is there?

The idea we have now is to first calculate the scratch space using the formula

"<input image rez> * 4 * <input image count> = scratch"
for example, shooting at 16,2 megapixel, making 1098 photos we calculate 16.2 * 1098 * 4 = 71150.4 MB or 69.48 GB

Then we compare the the scratch needed to the memory we have, so "memory you have : scratch as calculated"


-1:1 :If you have as many GB's RAM as scratch calculated, you have more then enough RAM, no need to add anymore. Autopano problably will not even use the temp-directory. This is really a dreamsystem.

-1:2 : if you have only half the GB's RAM of what was calculated you will be able to detect, edit and render your panorama's in a decent time. This is a normal system and good workable.
You might hear your harddisk a lot, If you have the money and your computer allows it, you will enjoy buying enough RAM to go to the 1:1-situation above.
Alternatively you might consider buying a SSD to be used a first temp-directory in the settings. Any size SSD will help as you can still use your normal harddisk as second and third temp directory. To keep things speedy however you want a ssd at least the size of the calculated scratch size (minus the RAM you have).

-1:n with n = 3 or 4 Autopano will get the job done but you will experience it takes much longer to get things done. If you are on a budget and/or not making money with your panoramas, there is likely no real hurry to upgrade. Otherwise think about upgrading your RAM to get a better 1:n or adding a SSD as temp.

-1:n with n >4 (F.e. you have 2 GB RAM and use a 12 mm equivalent lens) Autpano will have a hard time getting your job done and is even likely not able to finish the job.

Warning : check to see whether you motherboard can take the memory and whether the operating system you are using can address it. If not, go for an SSD

I did not mention CPU and GPU as I think they are not that important. Also CPU is hard to upgrade and GPU is not much of influence.

[HansKeesom]

by HansKeesom » Tue Aug 02, 2011 3:59 pm

........
Does makes sense to have a HDD rather than SSD for the C:\ rather than a shiny new SSD that isn't going to be used as much. So for source, temp and (possibly) page file, look at a couple of separate SSD's as an option, one large covering source and temp and a smaller for page-file / overflow?

Look forward to hearing your (and anyone else's) thoughts

Regards,
Paul

if you have to choose between HDD and SSD I would go for a HDD. You problably need a big HDD to begin with. SSD is too expensive to hold all your programms and data.
I would later add a SSD just for storing the pagefile and as temp-dir of autopano.

[KreAture]

by KreAture » Tue Aug 02, 2011 8:41 pm

I agree with Hans on this really, unless you have plenty of extra SSD space relating to the scratch needed, you should not waste the ssd with the OS.
Once the APx is running the OS should not be involved much, and since APx doesn't use that much virtual memory (except due to bugs?) it won't be that swap dependant either.
It does however rely on heavy use of the scratch directory as soon as the pano data can't fit in memory.

My tests suggest it writes everything out, even if somne would fit in mem and only a tad overflows to disk. I guess this is due to the need to go over everything multiple times in blending etc.