You are not logged in.



#1 2010-05-26 17:05:06

Aeriscera
Member
From: Bath, UK
Registered: 2008-03-11
Posts: 557
Website

Memory vs. Disks

Hello Everyone,

2.0.8 seems to have a more robust implementation of Smartblend - great work guys!

At one time it was the case iirc that for Multiband you wanted big fast disks whereas for Smartblend you needed lots of RAM. Or have I got that the wrong around?

Anyway, I haven't given up my gigapanography habit and am still working with multigigapix images and what I'd like to know is what is the situation with respect to (a) 2.0.8 and (b) subsequent versions? I.e. if have money to spend on upgrading my machine should I go for RAM or disks and how does the answer depend on the blending algorithm?

Cheers,

Aeris

Last edited by Aeriscera (2010-05-26 17:05:30)

Offline

 

#2 2010-06-01 13:01:40

AlexandreJ
Absolute beginner
From: Challes les eaux, France
Registered: 2005-11-14
Posts: 7362
Website

Re: Memory vs. Disks

Aeriscera wrote:

2.0.8 seems to have a more robust implementation of Smartblend - great work guys!

Thanks smile

Aeriscera wrote:

At one time it was the case iirc that for Multiband you wanted big fast disks whereas for Smartblend you needed lots of RAM. Or have I got that the wrong around?
Anyway, I haven't given up my gigapanography habit and am still working with multigigapix images and what I'd like to know is what is the situation with respect to (a) 2.0.8 and (b) subsequent versions? I.e. if have money to spend on upgrading my machine should I go for RAM or disks and how does the answer depend on the blending algorithm?

This was more the answer to the version 2.0.* :
- multiband : you need RAM so that everything can be done on the fly without any temporary file.
- smartblend : you need fast disk for temporary files caching but also RAM memory.

For v2.5 series, the answer will change :
- The old multiband is not available anymore. This was a fast but limited multiband algorithm.
You now have access to a full multiband algorithm which have the same demand as smartblend.
- Smartblend engine has been renamed : it's now only a part of the new global rendering engine.
Yes, it will use disk if needed. But if everything fit in memory, you don't need fast drive. It will depends on
the targeted size. My guess is that for gigapixels images, you will first need fast drive because it won't fit in memory.

Offline

 

#3 2010-06-01 16:35:06

Aeriscera
Member
From: Bath, UK
Registered: 2008-03-11
Posts: 557
Website

Re: Memory vs. Disks

AlexandreJ wrote:

My guess is that for gigapixels images, you will first need fast drive because it won't fit in memory.

Thanks Alexandre. Is it possible for you to say whether the 2.5+ versions will allow you designate a particular disk to use when there is not enough RAM? I am thinking of getting some striped raptors for 2.0.* APP scratch space.

Aeris

Offline

 

#4 2010-07-23 17:34:03

HansKeesom
Member
Registered: 2010-07-19
Posts: 706
Website

Re: Memory vs. Disks

AlexandreJ wrote:

Aeriscera wrote:

2.0.8 seems to have a more robust implementation of Smartblend - great work guys!

Thanks smile

Aeriscera wrote:

At one time it was the case iirc that for Multiband you wanted big fast disks whereas for Smartblend you needed lots of RAM. Or have I got that the wrong around?
Anyway, I haven't given up my gigapanography habit and am still working with multigigapix images and what I'd like to know is what is the situation with respect to (a) 2.0.8 and (b) subsequent versions? I.e. if have money to spend on upgrading my machine should I go for RAM or disks and how does the answer depend on the blending algorithm?

This was more the answer to the version 2.0.* :
- multiband : you need RAM so that everything can be done on the fly without any temporary file.
- smartblend : you need fast disk for temporary files caching but also RAM memory.

For v2.5 series, the answer will change :
- The old multiband is not available anymore. This was a fast but limited multiband algorithm.
You now have access to a full multiband algorithm which have the same demand as smartblend.
- Smartblend engine has been renamed : it's now only a part of the new global rendering engine.
Yes, it will use disk if needed. But if everything fit in memory, you don't need fast drive. It will depends on
the targeted size. My guess is that for gigapixels images, you will first need fast drive because it won't fit in memory.

Hi Alexandre,

When you say "Yes, it will use disk if needed" the following practical question comes to mind.

How would you upgrade a typical standard system with f.e. 1 TB harddisk-space and 6 GB of ram. CPU multicore.

Say you have 500 euro to spend to upgrade the machine to work with autopano.
Would autopano profit more from adding 16 GB of RAM (assuming the OS can handle it) or from adding a 128 GB SSD disk (which might also be used to store originals and resulting pano)?

I know it might be hard to tell and you might want both options ( :-) ) but we would appreciate you letting us know if you have a clear preference in such situations. Problably a lot depends on the total filesize of your originals, not?

Can we say something like :
- if you have more free memory then the total size of your original files, you will have no tempfiles and autopano will run at maximum speed.
-If you have less free memory then the total size of your original files, you will have tempfiles and having a fast disk is important?

Also, can we say the following?
-If autopano eats up more then 80% of your cpu cycles most of the time, then you have enough memory.
-If autopano eats up less then 20 % of your cpu cycles most of the time, you problably have too little memory

Last edited by HansKeesom (2010-07-23 18:27:59)


Regards, Hans Keesom
You can ftp your pictures to me on hanskeesom.x4all.nl user dump password dump I will try to stitch them for free and you can download the results for free (although my paypal is : hans@hanskeesom.com  ;-)  )

Offline

 

#5 2010-07-24 13:10:33

[bo]
community overseer
From: Bulgaria
Registered: 2006-05-05
Posts: 1815

Re: Memory vs. Disks

HansKeesom, it's too early to tell in regard of 2.5. I really think you will benefit from reading this thread:
http://www.autopano.net/forum/t9421-so, … dual-cpu-s


Some of my panoramas, posted in the Autopano Pro flickr group.

Offline

 

#6 2010-07-24 20:10:48

HansKeesom
Member
Registered: 2010-07-19
Posts: 706
Website

Re: Memory vs. Disks

[bo] wrote:

HansKeesom, it's too early to tell in regard of 2.5. I really think you will benefit from reading this thread:
http://www.autopano.net/forum/t9421-so, … dual-cpu-s

Hi Bo,

I have just finished reading the whole thread and did not find much to make a decision which direction to invest. The thread is about computers and their speed in general, not so much about APP and how to finetune one's computer for it.

Post #17 made a good point, splitting the whole workflow into phases.

For me the 3rd phase is the one that I am doing work behind the computer. During the other phases I will do other stuff, off-computer, maybe asleep and do not care to much about speed. But part 3 has to be fast. See post #15 "Speedwise, I was sick of waiting for the CP editor to optimize, colors to equalize, etc". I fully agree here.
Or as post #8  says "Now in my reality, people are the really expensive commdity".

So what I would like is a method on how to calculate what is needed to make phase 3 fast, given a certain size of the set of orginals.

Meanwhile I am leaning towards a workflow where I first have all pano's rendered in first version. This can run while I do other work or sleep. Next day I go through all generated pano's to see which need editing. The edited ones I put again in the renderbatch which can run in background/overnight again. Works well for me. Hope one day I can make pano's in daytime, work on computer in evening and sleep during night while computer computes and continues next day while I make new pano's :-)

Last edited by HansKeesom (2010-07-25 00:32:42)


Regards, Hans Keesom
You can ftp your pictures to me on hanskeesom.x4all.nl user dump password dump I will try to stitch them for free and you can download the results for free (although my paypal is : hans@hanskeesom.com  ;-)  )

Offline

 

#7 2010-07-25 18:13:18

tived
Member
From: Dane in Western Australia
Registered: 2008-07-11
Posts: 684

Re: Memory vs. Disks

Hi Hans,

More RAM and bigger and faster hard drives :-) and I think that for Phase 3 as I described in the other thread mentioned above, a good GPU won't go astray. If 500 euro can do it?? Dunno!
However, I think it is wisely to follow the sound advise from Bo, which is to wait and see what really happens when the next big release is out (ver 2.5).

Mvh

Henrik

Offline

 

#8 2010-07-26 02:32:19

HansKeesom
Member
Registered: 2010-07-19
Posts: 706
Website

Re: Memory vs. Disks

Hi Tived,

If a GPU is that important for phase 3? If so, that is rather important information. I am running a rather cheap ATI RAdeon HD 4650 as my machine was more meant for batchprocessing.

If replacing the 50 euro card brings me a lot while editing, I will be very happy.

Hope you can shine your light on this and yes I will wait for 2.5. No need to rush and spend 500 euro on a good running machine.


Regards, Hans Keesom
You can ftp your pictures to me on hanskeesom.x4all.nl user dump password dump I will try to stitch them for free and you can download the results for free (although my paypal is : hans@hanskeesom.com  ;-)  )

Offline

 

#9 2010-07-27 17:17:12

[bo]
community overseer
From: Bulgaria
Registered: 2006-05-05
Posts: 1815

Re: Memory vs. Disks

That's the main "problem" - CURRENTLY we don't have a reliable set of results and we don't even have a reliable software version to test on big_smile

If a GPU is that important for phase 3?

We don't know. I do hope than rendering will utilize GPU fully, but we still have no definite information or tests to show how much faster (if any) will the rendering be with or without the GPU.


Some of my panoramas, posted in the Autopano Pro flickr group.

Offline

 

#10 2010-08-03 14:45:33

AlexandreJ
Absolute beginner
From: Challes les eaux, France
Registered: 2005-11-14
Posts: 7362
Website

Re: Memory vs. Disks

1. If I had to spend 500$ on something, it would be memory and not the SSD drive.
2. GPU for cortex. For the moment, we don't use it. It will stay this way for some moment. Nevertheless, we already have converted some part of the new rendering engine to OpenCL and it is amazingly fast.

Offline

 

#11 2010-08-03 16:47:13

[bo]
community overseer
From: Bulgaria
Registered: 2006-05-05
Posts: 1815

Re: Memory vs. Disks

Could you please elaborate?

"For the moment we don't use GPU for Cortex (rendering)" - what is "for the moment"? Version 2.5? So the final 2.5 won't benefit from GPU in terms of rendering?

"we already have converted some part of the new rendering engine to OpenCL" - if you have already converted it and it won't be into 2.5, when do you plan to release it? 2.6 or maybe 3.0? What's more, if it really is "amazingly fast", why not release it as a part of 2.5?

Thanks!


Some of my panoramas, posted in the Autopano Pro flickr group.

Offline

 

#12 2010-08-03 17:45:52

AlexandreJ
Absolute beginner
From: Challes les eaux, France
Registered: 2005-11-14
Posts: 7362
Website

Re: Memory vs. Disks

[bo] wrote:

Could you please elaborate?

Sir, yes sir !

[bo] wrote:

"For the moment we don't use GPU for Cortex (rendering)" - what is "for the moment"? Version 2.5? So the final 2.5 won't benefit from GPU in terms of rendering?

2.5 stable won't have any GPU support in the rendering code ( it's ready for it, but is not integrated ).

[bo] wrote:

"we already have converted some part of the new rendering engine to OpenCL" - if you have already converted it and it won't be into 2.5, when do you plan to release it? 2.6 or maybe 3.0? What's more, if it really is "amazingly fast", why not release it as a part of 2.5?

I don't have the answer yet. The why is easy : we need time to integrate it. It cannot make it for 2.5 final.
Then the when. I don't know, we didn't schedule that integration yet.

Offline

 

#13 2010-08-03 18:13:28

tived
Member
From: Dane in Western Australia
Registered: 2008-07-11
Posts: 684

Re: Memory vs. Disks

Bo,

Thanks for asking the hard questions.

Alexandre is there a Autopano roadmap? even a rough one :-)

Henrik

Offline

 

#14 2010-08-03 21:07:26

[bo]
community overseer
From: Bulgaria
Registered: 2006-05-05
Posts: 1815

Re: Memory vs. Disks

Well, the good news is that eventually GPU will greatly speed up the rendering! I'm glad you got it working, even if it's not going into 2.5!

From the other thread (50GP Paris) I gather that CPU and HDD config has much more profound effect on render times and I'm also glad to see similar Xeons giving a nice boost.

Basically that means that there's no point in investing big money in GPU now (SLI is a total waste) in terms of APP speedup. For large project, I'd put my money on SSD RAIDs and/or PCIX SSD boards. No point to talk about RAM if 12-16GB is more than enough for 50 gigapixel panorama and 8GB is now commonplace.

So a commonplace PC for pano rendering should have a quad-core i7, 8 or 16GB RAM and at least 2 SSD drives (OS and Project). While a higher end should rely on dual CPU, quad-core Xeon to your liking, 16 or 24GB RAM and some expensive multi-SSD RAID.


Some of my panoramas, posted in the Autopano Pro flickr group.

Offline

 

#15 2010-08-08 07:53:14

tived
Member
From: Dane in Western Australia
Registered: 2008-07-11
Posts: 684

Re: Memory vs. Disks

[bo] wrote:

....
So a commonplace PC for pano rendering should have a quad-core i7, 8 or 16GB RAM and at least 2 SSD drives (OS and Project). While a higher end should rely on dual CPU, quad-core Xeon to your liking, 16 or 24GB RAM and some expensive multi-SSD RAID.

Bo, I love it when you talk dirty :-)


Henrik

Offline

 

#16 2010-08-08 11:18:06

HansKeesom
Member
Registered: 2010-07-19
Posts: 706
Website

Re: Memory vs. Disks

AlexandreJ wrote:

1. If I had to spend 500$ on something, it would be memory and not the SSD drive.
2. GPU for cortex. For the moment, we don't use it. It will stay this way for some moment. Nevertheless, we already have converted some part of the new rendering engine to OpenCL and it is amazingly fast.

Hi Alexandre,

Thanks for the clear answer. I have meanwhile noticed that when trying to do to many things at my computer while APP has  a lot running in the background, my computer does run out of it's 6 GB and things can became very slow. So memory would indeed be the first thing to look at. Just a pity all three memory slots are already used I have to replace the current 6 GB instead of adding memory. But that the investment I will have to make if I move to bigger panorama's. And actually, it looks like memory is dirt cheap nowadays :-)

Yesterday I did my first commercial project, 14 pano's of 25 photos each, shot in JPEG+RAW. 9 hours after we left the site I had already a virtual tour of good quality, based on the jpegs. Only 3 pano's needed some manual editing and had to be rendered again.
The important thing is that from these 9 hours I was only behind the computer for about 4 hours, the rest I was doing things not related to the project. This really allows me to combinate making pano's with all other things in my life :-)

At the moment 14 groups based on the RAW files are running. I do not have to attend that process, can do other stuff. I am really impressed with APP.

Last edited by HansKeesom (2010-08-08 12:00:07)


Regards, Hans Keesom
You can ftp your pictures to me on hanskeesom.x4all.nl user dump password dump I will try to stitch them for free and you can download the results for free (although my paypal is : hans@hanskeesom.com  ;-)  )

Offline

 

#17 2010-08-08 11:47:58

tived
Member
From: Dane in Western Australia
Registered: 2008-07-11
Posts: 684

Re: Memory vs. Disks

Hi Hans,

that is one of many reasons for going for the dual processor workstation mainboards, in that they can take a lot more memory, with 12-18 slots between two cpus's - and yes, it does come at a high ticket price initially, but it does also open up for more flexibility down the track. at least IMHO :-)

I do think that when it does come down to it, you will have to invest in three of four areas, CPU, RAM and Hard drives, the forth being GPU, which will become important down the track.
Though I don't see $500 going very far!

Henrik

Offline

 

#18 2010-08-08 17:20:21

HansKeesom
Member
Registered: 2010-07-19
Posts: 706
Website

Re: Memory vs. Disks

Hi Hendrik,

When making pano's becames a big business for me, I will certainly and timely invest in a dual (or more) processor machine. Money is not so much an issue, but that is because I never spend it without throrough investigation :-)

At the moment it is more a matter of max one tour a week and I can have the machine run a few days generating all pano's belonging to the tour. AVerage tour being 15 pano's created from 24 jpg/RAW's

My current workstation has a i7 920 processor, which is used for just 30% while rendering. Windows indicates the disk-activity is close to 100% active. Most busy files are the two pagefiles I use and a number of e:\temp\kolorapg22.... files.

This indicates to me lots can be won by scaling up the memory. Moving from 3 bars of 2 GB to 3 bars of 4GB should cost less then 500 euro.
I guess AlexandreJ advices to go for the memory as that will solve things by the rootcause and create less need for swapfiles and tempfiles. An SSD would allow for faster reading from and writing to swap and tempfiles but nothing is faster then having no need to do so.

In real life it will be important to have a speed that can finish rendering a whole project in one night while asleep so the next day I can do some manual work on it.  Now it all depends of course on the size of the average project one is doeing..... :-)

cheers
Hans


Regards, Hans Keesom
You can ftp your pictures to me on hanskeesom.x4all.nl user dump password dump I will try to stitch them for free and you can download the results for free (although my paypal is : hans@hanskeesom.com  ;-)  )

Offline

 

#19 2010-08-13 16:55:10

[bo]
community overseer
From: Bulgaria
Registered: 2006-05-05
Posts: 1815

Re: Memory vs. Disks

Here's a recent exhaustive review on the topic of single GPU performance vs SLI. Not really needed now that it has become evident that APP 2.5 won't utilize the GPU in the rendering process; still a good read if you're looking to get a decent (and recent) GPU: http://techreport.com/articles.x/19404/11 (link leads to the last page of the review, in case you don't have time for the whole material).

In short, you can get an amazing GPU for under 200$ (GTX460) and a SLI of those will beat almost anything within a reasonable range!


Some of my panoramas, posted in the Autopano Pro flickr group.

Offline

 

#20 2010-08-14 17:39:50

HansKeesom
Member
Registered: 2010-07-19
Posts: 706
Website

Re: Memory vs. Disks

Looking very promissing as it will be a great way to upgrade a current system.
So we wait for an APP version that does support GPU-use while rendering and not invest in GPU's for the moment?

Does a GPU work in it's own memory and will it be independent of the general RAM (while being used for rendering in APP 2.x (x >5) )?


Regards, Hans Keesom
You can ftp your pictures to me on hanskeesom.x4all.nl user dump password dump I will try to stitch them for free and you can download the results for free (although my paypal is : hans@hanskeesom.com  ;-)  )

Offline

 

#21 2010-08-15 13:03:30

hankkarl
Member
From: Connecticut, USA
Registered: 2006-02-21
Posts: 1923
Website

Re: Memory vs. Disks

GPU works great for previews.  You can always let the render run overnight, but when you're adding and deleting CPs and playing with color anchors, the GPU speedup is valuable.

Offline

 

#22 2010-08-18 01:35:17

HansKeesom
Member
Registered: 2010-07-19
Posts: 706
Website

Re: Memory vs. Disks

That is a good point as the time of me as a person is much more expensive then the time of the computer. And sometimes I find myself waiting a lot while editing a pano.

Meanwhile I tuned APP a bit to use my computer as it is in a decent way. Leaving half a gig of memory in the settings and rendering in idle priority seems to be the way to go.
I can leave my computer on, even do some work on it, and it will run for days doing work. It might not be the fastest setting perse but it is one that goes on and on without a hitch which I like more then one that crashes from time to time.


Regards, Hans Keesom
You can ftp your pictures to me on hanskeesom.x4all.nl user dump password dump I will try to stitch them for free and you can download the results for free (although my paypal is : hans@hanskeesom.com  ;-)  )

Offline

 

#23 2010-08-21 11:43:19

HansKeesom
Member
Registered: 2010-07-19
Posts: 706
Website

Re: Memory vs. Disks

Hope AlexandreJ is still watching this thread because I would like to ask the "memory versus 530 MBPS SSD" question instead of "memory vs harddisk" question

As others told me earlier 500 euro is not buying me the memory I need. It will allow for buying 3*4 GB dimm3 which will bring me from 6 GB to 12 GB total memory. Not bad but in the long run maybe not enough.

OCZ is selling now PCI-E based RAID O 120GB SSD's for less then 500 euro capable of 530MBps. I wonder whether that be a great temp directory for AGG.

So the question is here not "memory vs disks" but "memory versus PCI-E SSD RAID 0at 530 mbps". Which will bring me more speed now? *

Only problem I see is that meanwhile my temp-directory has grown to 799 GB. Could it be APP left some tempfiles behind which it did not clean up? How do I decide what to delete.


*Long term I can always buy 3*8GB giving me the 24 GB that will really speed up things :-) For the moment my panoramas are not that big and I have not earned enough to invest the 1500 euro that it will cost to buy 24 GB RAM. Problem with memory is that if I now would buy 3*4 GB I cannot use that anymore when buying 3*8 GB once I have the money. A SSD you can leave in the computer even is you buy something faster later.

Last edited by HansKeesom (2010-08-21 13:32:15)


Regards, Hans Keesom
You can ftp your pictures to me on hanskeesom.x4all.nl user dump password dump I will try to stitch them for free and you can download the results for free (although my paypal is : hans@hanskeesom.com  ;-)  )

Offline

 

#24 2010-09-12 12:38:05

HansKeesom
Member
Registered: 2010-07-19
Posts: 706
Website

Re: Memory vs. Disks

Meanwhile I cleaned up my C-drive which is a 15.000 RPM raptor and the fastest disk in my system.
This allowes me to have my pagefile on it and this seems to make autopano fly. Tempory files for autopano are still on my E drive which is a normal 2 TB drive*.

So far I only tested pano's made up of less the 30 photos as these are my general size at the moment.
The CPU % that is now dedicated to autopano giga went up from 15 % to 85 oto 95% while rendering. For me this is enough "maxing out" of my CPU for the moment.***

Likely I will need to do something** once my pano became bigger but so far so good.

Looking at the pricelevel of these small and fast disks running at 15000 rpm, I would advice people to consider these as a good way to speed up their system. They might not be as fast as SSD but are larger and cheaper and likely more durable.


*I like to keep it there because having so much space (the temp directory currently is over 750 GB big) allows me to quicly reload old .pano files which saves me time when I have to repair little error in older projects.
**Upgrade memory to 24 GB.

*** I assume here it it a good thing when autopano is using such a high percentage of the CPU, interpreting it as autopano not having to wait much for the harddisc or other components.

Last edited by HansKeesom (2010-09-12 12:53:14)


Regards, Hans Keesom
You can ftp your pictures to me on hanskeesom.x4all.nl user dump password dump I will try to stitch them for free and you can download the results for free (although my paypal is : hans@hanskeesom.com  ;-)  )

Offline

 

#25 2010-09-12 14:01:17

tived
Member
From: Dane in Western Australia
Registered: 2008-07-11
Posts: 684

Re: Memory vs. Disks

Hans,

Lets make the Raptor a 10.000RPM :-) only SAS and SCSI is 15.000RPM.

Having your page file on the same disk as you OS, isn't the most effective way, Ideally you would need to have a Disk for OS/Apps, one for Page file, one for Autopano/Photoshop temp/scratch disk and then your data disks. Obviously these can all be combination's of various RAID.

Depending on the type of READ/Writes you may find that a RAID-0 of Fast SAS/SATA disks will out perform a SSD disk, in particular if you only partition the outer part of it which is your first partition (less the 30%).

Lots of memory is great, but it needs support to drop off the data quick, and this will only be helped my really fast drives or arrays of drives (be it SAS/SATA or SSD) what you have to work out is how much you can afford, and if you have room in your box for so many disks. Then have large RAID arrays are usually catered for better by dedicated RAID controllers, with battery backup.

Now we are looking in the many thousands of dollars here :-). I priced my dream system to AUD$17.000 which included 20 disks (My data array was a RAID-10 (8R10) with 8 WD RE4 2TB disks, though if I have a back up for this, it may be better to have it as one RAID-0 with 8 disks (8R0) The remaining 12 disks were all SSD's for the tasks described above, OS/APP's, Page-file, TEMP/Scratch, 4 disks for each in (4R0) RAID-0.

Needless to say, I will be dreaming for some time :-)

anyway, I hope this is of some help

Henrik

PS: If you are looking for SSD disks, I would choose between INTEL and disks with SandForce controllers within them, and also Crucial's C300 with Micron controller is very good, all these are now 2nd gen tech (of current SSD's) (as SSD has been around for almost 10-15 years :-) just not affordable till now (if you can call it affordable now :-) )

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2005 Rickard Andersson