Calculating "Disk Space" for a gigapixel panorama

[Zurich.Gigapixel]

Calculating "Disk Space" for a gigapixel panorama

by Zurich.Gigapixel » Sun Jul 29, 2012 9:39 am

Hi

Is there a way to calculate how much hard disk space will be needed by Autopano Giga to render a gigapixel panorama? (100% size)

What I know:
- How many pictures I will take
- The resolution and file size (more or less) of the pictures
- How much overlap I have x-axis & y-axis

What I'd like to know:
- How much "Hard Disk Space" will be needed to render it.


I'll have to buy new hard disks for this project but I'd like to know how big they have to be...

Thanks for any help in advance!

Regards from Switzerland

[klausesser]

by klausesser » Sun Jul 29, 2012 1:04 pm

I'll have to buy new hard disks for this project but I'd like to know how big they have to be...

Depends on what you exactly mean saying "Gigapixel": 3, 6, 12, 18, 20 or more GPx? I use 3x 2 TB disks and a 180GB SSD at the moment (had a RAID 0 before, but 1 SSD broke).
And my average size is 3 - 4 GPx.

best, Klaus

[HansKeesom]

by HansKeesom » Sun Jul 29, 2012 3:49 pm

Hello Zurich,

Yes there is, I wrote a formula at http://www.kolor.com/forum/p87033-2011-08-11-11-53-58#p87033
It is to designed to advice you how to build a well running system but it should also give you a feeling how your computer will run doing a certain size of panorama.

So if you give us the answers to the questions of Klaus, I can do the math for you ;-)

[Zurich.Gigapixel]

by Zurich.Gigapixel » Sun Jul 29, 2012 11:56 pm

Hi Hans

I read your post about "building a well running system", really informative! Thanks for that one.

In my case it looks like this:

- nearly 10'000 Pictures
- each 36.2 MP (7.360 à— 4.912)

So "the formula" would be: 36.2 * 10'000 * 4 = 1'448'000 MB or 1414 GB or 1.4 TB

In this case I would need to have at least about 1.4 TB of free space, right?
If 1.4 TB is enough: Is there any advantage I would get by buying a 3 TB Harddisk or would 2 TB do the same?

May I ask you: Doesn't the overlap % make a difference on calculating the temp space?

By the way: The system looks like this:

Mac Pro ( http://support.apple.com/kb/SP589 )
- Two 2.4GHz Quad-Core Inter Xeon E5620 "Westmere"
- Hyper-Threading for up to 16 virtual cores
- Two ATI Radeon HD 5770 with 1GB of GDDR5
- 16 GB Ram

Regards from Switzerland!

[gkaefer]

by gkaefer » Mon Jul 30, 2012 12:40 am

Hi Hans

I read your post about "building a well running system", really informative! Thanks for that one.

In my case it looks like this:

- nearly 10'000 Pictures
- each 36.2 MP (7.360 à— 4.912)

So "the formula" would be: 36.2 * 10'000 * 4 = 1'448'000 MB or 1414 GB or 1.4 TB

In this case I would need to have at least about 1.4 TB of free space, right?
If 1.4 TB is enough: Is there any advantage I would get by buying a 3 TB Harddisk or would 2 TB do the same?

May I ask you: Doesn't the overlap % make a difference on calculating the temp space?

By the way: The system looks like this:

Mac Pro ( http://support.apple.com/kb/SP589 )
- Two 2.4GHz Quad-Core Inter Xeon E5620 "Westmere"
- Hyper-Threading for up to 16 virtual cores
- Two ATI Radeon HD 5770 with 1GB of GDDR5
- 16 GB Ram

Regards from Switzerland!

so you dont think Hans part of the formula describing the relationship between above calculated GB (1414) and the existing Memory (16GB) is no problem for you?
I mean Hans describes a ration of 1:4 as more or less anoing endless waiting...

and I wonder if your "36,4" Megapixel per source image is correct. (So this could be a Phase One P40 you use?)
according to Hans' formula having less than 364GB RAM Memory your project will take a really long time...

Georg

Last edited by gkaefer on Mon Jul 30, 2012 12:41 am, edited 1 time in total.

[Briger]

by Briger » Mon Jul 30, 2012 4:38 am

@Georg Nikon D800 i think he use.

[tived]

by tived » Mon Jul 30, 2012 5:18 am

Switzerland,

if you do go ahead and shoot this pano, could you do us all a favor, and record the process. How long its taking on your particular machine, if you encountered any problems, and what they were.
As well as what sort of workflow you used.

And congrats on your D800, saw the D800E in action the other day, very impressive files.

all the best

Henrik

PS: I foresee that it will take quite some time given the computer configuration, but don't let that stop you. :-) add more RAM if you can/want to. Look at using a couple of SSD's and make a RAID-0 of two 2 or 3TB Disks to improve your read/write speed.

The thing is that your 10.000 image files will result in as many 100megabyte 16bit TIFF files, I belive when one of your fellow country men, did something similar, he broke his pano up into 60 degree sections and processed 6 of these, which eventually got merged together into one (72Gigabytes from memory)

All the best

[gkaefer]

by gkaefer » Mon Jul 30, 2012 12:14 pm

@Georg Nikon D800 i think he use.

ah ok. much cheaper than the phase one ;-))
but with 10.000 images a 36MP with a Computer with 16GIG RAM... I would be really surprised (in positive) when this will work.
I know it with a 8gigRAM PC and using 1200 Images... if its a hobby than you can spend the time... if you do it for money than you should calculate twice ;-))
Georg

[klausesser]

by klausesser » Mon Jul 30, 2012 1:17 pm

May I ask you: Doesn't the overlap % make a difference on calculating the temp space?

Hey Switzerland!

Again: which amount of Gigapixels do you have in mind? What lens are you going to use? What head?
What´s the goal to shoot a pano of 10000 images, each 36MPx? Did you ever do Gigapixels before?
How do you think to edit an resulting equirectangular image in Photoshop? What is your projected time frame?

best, Klaus

P.S.: which kind of "pano" are we talking about anyway? Spherical or rectangular/flat?

Last edited by klausesser on Mon Jul 30, 2012 1:19 pm, edited 1 time in total.

[Zurich.Gigapixel]

by Zurich.Gigapixel » Mon Jul 30, 2012 1:37 pm

@ Briger:
Yes this is correct, we're working with the Nikon D800 wich has 36.2 MP

@ gkaefer:
I didn't miss the part about the ratio between 'temp space' and 'RAM'.
In this case the ratio is (well lets say...) pretty bad.
But this is an unique project. I'm not planning doing this even twice

I'm more concerned to know, if it will work, not if it will be fast, beacause it won't be fast anyway... I can't install 364GB Ram

@ tived:
The D800 is just unbelievable ;D And we will record the process, no worries
I heard about breaking the panorama in to individual parts and then merging them...
...but to be honest I would'nt know how to do that.
Do you have a link or something about the guy who did that? Or do you know it's done?


I still have following questions:
- Doesn't the overlap % make a difference in calculating the 'temp space'?
- Will our system (mentioned above) + a 3TB Harddisk manage to render it? (Even if it takes some days)

Kind Regards from Switzerland
Miguel

[HansKeesom]

by HansKeesom » Mon Jul 30, 2012 1:43 pm

Hi Hans

I read your post about "building a well running system", really informative! Thanks for that one.

In my case it looks like this:

- nearly 10'000 Pictures
- each 36.2 MP (7.360 à— 4.912)

So "the formula" would be: 36.2 * 10'000 * 4 = 1'448'000 MB or 1414 GB or 1.4 TB

In this case I would need to have at least about 1.4 TB of free space, right?
If 1.4 TB is enough: Is there any advantage I would get by buying a 3 TB Harddisk or would 2 TB do the same?

Ideal would be to have 1.4 TB of memory......... as you have 16 GB RAM you have a 1: 100 ratio. As I decribed in my post : "-1:n with n >4 Autopano will have a hard time getting your job done and is even likely not able to finish the job.
"
Actually, this sounds like something impossible to finish unless the people of Intell are willing to work with you....... you need to consider shooting or converting at/to a lower resolution and fewer pictures.
Are these 10.000 pictures in brackets? If so, how many photos in one bracket. 1.4 TB divided by that number might look much better.

May I ask you: Doesn't the overlap % make a difference on calculating the temp space?

Makes no difference a.f.a.i.k.

By the way: The system looks like this:

Mac Pro ( http://support.apple.com/kb/SP589 )
- Two 2.4GHz Quad-Core Inter Xeon E5620 "Westmere"
- Hyper-Threading for up to 16 virtual cores
- Two ATI Radeon HD 5770 with 1GB of GDDR5
- 16 GB Ram

Great system but seems to me that for Autopano these XEON's could use some more GB's of RAM and some SSD's

Last edited by HansKeesom on Mon Jul 30, 2012 1:45 pm, edited 1 time in total.

[Zurich.Gigapixel]

by Zurich.Gigapixel » Mon Jul 30, 2012 2:14 pm

Again: which amount of Gigapixels do you have in mind? What lens are you going to use? What head?
What´s the goal to shoot a pano of 10000 images, each 36MPx? Did you ever do Gigapixels before?
How do you think to edit an resulting equirectangular image in Photoshop? What is your projected time frame?

best, Klaus

P.S.: which kind of "pano" are we talking about anyway? Spherical or rectangular/flat?

Amount of Gigapixel: Not quiete sure about that one...
(Because of the fact that there is more than one way to count pixels when it comes to this kind of size)
Lens: 600mm + 2xTC
Head: Seitz Roundshot VR Drive
Goal: World Record attempt
Gigapixels before: I'm sure this one isn't the first
Editing in Photoshop: with KROutils
Time frame: Something between 5 to 8 hours
Projection: cylinder

... you need to consider shooting or converting at/to a lower resolution and fewer pictures.
Are these 10.000 pictures in brackets?

Lower resoltion or fewer pictures is not an option for this one.
No brackets, just nearly 10'000 normal pictures.

...Actually, this sounds like something impossible to finish unless the people of Intell are willing to work with you...

So in your opinion this system won't do the job? Even if it runs some days?


Thanks for all answers so far
Kind regards from Switzerland
Miguel

[gkaefer]

by gkaefer » Mon Jul 30, 2012 3:16 pm

I think this try has to fail on many frontiers:

Lens: 600mm + 2xTC

so - you can do this,

if you neglect quality factor like f8 & resulting exposure times (to get it done in 5 hours)
if you rent a 1200 lens instead (with better values f2.x so you can use f8 at usable exposure times)
if you plan >10 shooting days with similar weather conditions
I would also shoot every image multiple times (with identical f-stop and exposure times) so you can stack them to reduce noise

Head: Seitz Roundshot VR Drive

good one - also good enough for all world records...

Goal: World Record attempt

well so 112GP or more (if you do not wanna go above Microsofts Terapixel project ;-) ...)

Editing in Photoshop: with KROutils

well - a 16GB PC will never give KROutils the power to split a 112GP pano. (a 8GB PC & KROutils fails on a 10GP pano and even smaller ones too)

Time frame: Something between 5 to 8 hours

lol - yah you can use the speed mode of the VR drive (using your 600mm x2TC @f2 and exposure time of 1/100s)

but also to be honest - I cant say I did try myself - so try it - you will at last win a lot of experience

Liebe Gruesse,
Georg

Last edited by gkaefer on Mon Jul 30, 2012 3:18 pm, edited 1 time in total.

[klausesser]

by klausesser » Mon Jul 30, 2012 4:06 pm

. . . - you will at last win a lot of experience

YES, definitely! :cool:

best, Klaus

[HansKeesom]

by HansKeesom » Mon Jul 30, 2012 4:19 pm

"Goal: World Record attempt"

some investigation into what computers have been used for previous record attempts...

http://news.softpedia.com/news/Explore-London-in-a-80-Gigapixel-Panorama-the-World-s-Largest-167328.shtml
speaks of 192 GB ......

They did 7,886 images, not sure at which resolution the images where made so no telling which 1:n ratio they went with.


If you build a system according to my advice and give it 1:4 ratio memory (384 GB), a RAID SSD of at least 1.4 TB I do think you can do the panorama you are looking for.....apart from the problem of doing the shoot itself.

However, this is just theory, no telling what kind of special problems might surface......

Last edited by HansKeesom on Mon Jul 30, 2012 4:30 pm, edited 1 time in total.

[klausesser]

by klausesser » Mon Jul 30, 2012 4:30 pm

"Goal: World Record attempt"

Please do some investigation into what computers have been used for previous record attempts

http://news.softpedia.com/news/Explore-London-in-a-80-Gigapixel-Panorama-the-World-s-Largest-167328.shtml
speaks of 192 GB ......

This would be wise to watch also:

http://blog.paris-26-gigapixels.com/en/?p=110
http://blog.paris-26-gigapixels.com/en/?p=114
http://blog.paris-26-gigapixels.com/en/?p=115

best, Klaus

[Zurich.Gigapixel]

by Zurich.Gigapixel » Mon Jul 30, 2012 7:31 pm

if you plan >10 shooting days with similar weather conditions

The color shift would be too strong, I think...
And the weather in Zurich changes quite rapidly, so this is probably no option

I would also shoot every image multiple times (with identical f-stop and exposure times) so you can stack them to reduce noise

Literally impossible to do brackeing in our case and would be completely of no use, the D800 makes good enough pictures. Noise is no problem.

well so 112GP or more (if you do not wanna go above Microsofts Terapixel project ;-) ...)

Correct, above 112GP (if you count it with a simple picture overlap model)
But at this point I can't say how much above...
Because counting pixels with a simple overlap model has its flaws.

well - a 16GB PC will never give KROutils the power to split a 112GP pano. (a 8GB PC & KROutils fails on a 10GP pano and even smaller ones too)

Thanks for pointing this out. (That's why it didn't work some weeks ago....) Didn't know that.
Is there a formula to know how big a .kro file can be so KROutils still can split it?


@ HansKeesom & klausesser



Math for the Shanghai 112 GP version:
12'000 Pictures * 18 MP * 4 = 843.75 GB Temp Space
843.75 GB Temp Space / 32 GB RAM = ratio 1:26

Math for the Sevilla 100 GP version:
9750 Pictures * 21.1 MP * 4 = 803.61 GB Temp Space
803.61 GB Temp Space / 40 GB RAM = ratio 1:20

Math for the the Sugar Loaf 92 GP version:
12238 Pictures * 17.9 MP * 4 = 855.7 GB Temp Space
855.7 GB Temp Space / 24 GB RAM = ratio 1:36

[I hope there's no math error, if so please tell me]

As you can see all the last world records had ratios way over 1:4

So if I do the math for our project, it looks like this:
10'000 Pictures * 36.2 MP * 4 = 1'414 GB Temp Space
1'414 GB Temp Space / 16 GB RAM = ratio 1:88

So that looks pretty bad, to say the least...
I still don't know if we can upgrade the RAM.
If we will be able to do it, it looks like this:

24 GB RAM = ratio 1:59
36 GB RAM = ratio 1:39
64 GB RAM = ratio 1:22

What do you guys think?

Last edited by Zurich.Gigapixel on Mon Jul 30, 2012 7:38 pm, edited 1 time in total.

[gkaefer]

by gkaefer » Mon Jul 30, 2012 8:22 pm

Literally impossible to do brackeing in our case and would be completely of no use, the D800 makes good enough pictures. Noise is no problem.

I dont write about bracketing. I wirte about what oloneo names "HDR denoise". shooting one and the same picture with totally equal settings several times. Than take these "identical" images and zoom to 150% or so. compare one and the same 10x10 pixel area. 100% different results. So now take your "identical" images and do that "HDR denoise" and you will get a reall better image.
http://www.oloneo.com/en/page/products/photoengine/hdr-denoise.html

and did you ever try at a average 100% couldfree day to climb up a small hill to shoot with a 1200mm an object 2km away?
so above method will give you here same better results simulating a shooting at a very rare 100% cloudfree day some hours after a big raining.

Georg

Last edited by gkaefer on Mon Jul 30, 2012 8:23 pm, edited 1 time in total.

[Zurich.Gigapixel]

by Zurich.Gigapixel » Mon Jul 30, 2012 8:37 pm

shooting one and the same picture with totally equal settings several times. Than take these "identical" images and zoom to 150% or so. compare one and the same 10x10 pixel area. 100% different results. So now take your "identical" images and do that "HDR denoise" and you will get a reall better image.

Ok I see what you mean, I used this method once, but not for panoramas.
But we won't make multiple shots of the same area anyway, it's simply too time consuming.
Even the amount of pictures we're taking now takes too much time.

But thanks for the input anyway!

Last edited by Zurich.Gigapixel on Mon Jul 30, 2012 8:49 pm, edited 1 time in total.

[klausesser]

by klausesser » Mon Jul 30, 2012 11:30 pm

What do you guys think?

What i think? I think hauntig for records is boring Really.

That´s a kind of challenge which can be handled by money aka equipment. Show me fine photography - that could impress me . . . doesn´t matter which size it is.

best, Klaus

Last edited by klausesser on Mon Jul 30, 2012 11:33 pm, edited 1 time in total.

[lumelix]

by lumelix » Tue Jul 31, 2012 9:27 am

The whole world is running for new records.
And for that they dopes and cheats. Quality is no longer an issue most of time.
Many of the "record" panoramas out there have a record-lousy quality, even when reduced to 50% (four times fewer pixels) it is still unusable.
I think many of them are artificially inflated or created with technical equipment that is completely inappropriate.

Currently I reduce most of my panoramic images to 50%, even when taken with the D800E and a prime lens, to get really sharp results.
The advantages are that noise and moiré is no longer an issue and the process continues at least four times faster.
To do so loses little to none of the image information, but new pixel records can't be set up this way.

For new Gigapan-records we should strictly applied new methods for measure the effective resolution and quality of the panorama.
It's like in the olympics games with the doping control

Last edited by lumelix on Tue Jul 31, 2012 9:28 am, edited 1 time in total.

[Zurich.Gigapixel]

by Zurich.Gigapixel » Tue Jul 31, 2012 10:47 am

That´s a kind of challenge which can be handled by money aka equipment.

Like every other technical record. But money alone won't do the job.

I think hauntig for records is boring Really.
...
Show me fine photography - that could impress me

It's like the olympics... nothing really beautiful to look at...
but they're trying to reach the highest perfomance in one specific area.


But please, this discussion is not the topic of this thread.

For new Gigapan-records we should strictly applied new methods for measure the effective resolution and quality of the panorama.

I would love it if there was some strict rules for counting effective pixels!
Because the way we're comparing the Gigapixels Records has many flaws!
For instance:
If you use an simple picture overlap model to count the "optical pixels" of a panorama and pay no attention to the quality of the pictures or take the vertical FOV into consideration, then it's extremly easy to cheat...

Back to the topic:

Do you guys think our system might be able to render the picture if we upgrade the RAMs?
Taking in consideration that all other World Records had an ratio of 1:20 or worse.

Last edited by Zurich.Gigapixel on Tue Jul 31, 2012 10:48 am, edited 1 time in total.

[klausesser]

by klausesser » Tue Jul 31, 2012 12:24 pm

I would love it if there was some strict rules for counting effective pixels!

very easy - look at Paris 26 Gigapixel: zooming in to 100% doesn´t show any artefacts. That must be the goal: absolutely "clean" @"true"100%. Not more, not less.

Many giga-pixels exceed 100% - especially on Gigapan.org. And that is what make them unreliable - and in my eyes unacceptable in terms of the numbers they
name.

best and good luck for your project,
Klaus
(i´m curious how it will work out )

[Zurich.Gigapixel]

by Zurich.Gigapixel » Tue Jul 31, 2012 1:06 pm

very easy - look at Paris 26 Gigapixel: zooming in to 100% doesn´t show any artefacts. That must be the goal: absolutely "clean" @"true"100%. Not more, not less.

Well that's very subjective, to say the least...
There should be a way of mesuring it by calculation...
Otherwise there would be an endless discusion about when that "true" "clean" "100%" is achieved.

Many giga-pixels exceed 100% - especially on Gigapan.org.

Well that's a (problem a.k.a.) feature of gigapan.org.
Even if you upload a super high quality image, the viewer lets you zoom in more than 100%, that can be pretty annoying.

good luck for your project

thanks!


Question of this thread:
Do you guys think our system (mentioned in post #4 & #17) might be able to render the picture if we upgrade the RAMs?
Taking in consideration that all other World Records had an ratio of 1:20 or worse.

[HansKeesom]

by HansKeesom » Tue Jul 31, 2012 2:38 pm

if you plan >10 shooting days with similar weather conditions

The color shift would be too strong, I think...
And the weather in Zurich changes quite rapidly, so this is probably no option

I would also shoot every image multiple times (with identical f-stop and exposure times) so you can stack them to reduce noise

Literally impossible to do brackeing in our case and would be completely of no use, the D800 makes good enough pictures. Noise is no problem.

well so 112GP or more (if you do not wanna go above Microsofts Terapixel project ;-) ...)

Correct, above 112GP (if you count it with a simple picture overlap model)
But at this point I can't say how much above...
Because counting pixels with a simple overlap model has its flaws.

well - a 16GB PC will never give KROutils the power to split a 112GP pano. (a 8GB PC & KROutils fails on a 10GP pano and even smaller ones too)

Thanks for pointing this out. (That's why it didn't work some weeks ago....) Didn't know that.
Is there a formula to know how big a .kro file can be so KROutils still can split it?


@ HansKeesom & klausesser

http://img6.imagebanana.com/img/zh0shjrs/5284344733_51a7a5430d_b.jpg

Math for the Shanghai 112 GP version:
12'000 Pictures * 18 MP * 4 = 843.75 GB Temp Space
843.75 GB Temp Space / 32 GB RAM = ratio 1:26

Math for the Sevilla 100 GP version:
9750 Pictures * 21.1 MP * 4 = 803.61 GB Temp Space
803.61 GB Temp Space / 40 GB RAM = ratio 1:20

Math for the the Sugar Loaf 92 GP version:
12238 Pictures * 17.9 MP * 4 = 855.7 GB Temp Space
855.7 GB Temp Space / 24 GB RAM = ratio 1:36

[I hope there's no math error, if so please tell me]

As you can see all the last world records had ratios way over 1:4

So if I do the math for our project, it looks like this:
10'000 Pictures * 36.2 MP * 4 = 1'414 GB Temp Space
1'414 GB Temp Space / 16 GB RAM = ratio 1:88

So that looks pretty bad, to say the least...
I still don't know if we can upgrade the RAM.
If we will be able to do it, it looks like this:

24 GB RAM = ratio 1:59
36 GB RAM = ratio 1:39
64 GB RAM = ratio 1:22

What do you guys think?

1:22 is getting closer. We don't say it will work or not work. I just say that anything beyond 1:4 is unlikely to work.

But feel free to try first and upgrade later. You might find a ratio higher then 1:4 that makes it to the end.

You might want to ask people with big machines ( present on this forum) if they want to give it a try, that might be cheaper then buying everything yourself