 |
|
|
|
|
|
 |
| ||||||||
 |
 
|
|
|
|
|
|
|
|
|
| ||||||||
|
|
| 
| User list | You are not logged in.
Hi Everybody,
There were a lot of such report recently about inconsistent crash during smartblend. First, let me say that we are aware of that issue ( it's a really old issue since v1 for sure ). It is really complicate to fix and here's the situation about that crash.
The crash : smartblend algorithm
The smartblend issue is a tricky one. We know why but it's really hard to fix. Let's explain it in detail.
How smartblend works :
* take image 1 and 2, calculate the transition seams, do the blending. You'll get new blended image 1_2
* then SB blends the 1_2 with image 3 and you'll get image 1_2_3
* etc for all images
To be able to work on big images we use memory mapped files. It's a good technic that allows to work on terabytes with low memory.
The issue we have is that along the process, the memory got fragmented. And even if we use low memory in smartblend, it often happens that we can get any contiguous memory to be able to use the file map. I did an analysis yesterday on a provided crash case, and the crash occurs really at the end of the process :
* total memory used by the application ? 400 MB
* total memory installed : 2 Giga.
* free memory : 1600 MB ( 2 GB - 400 MB ).
* Cause of crash ? I cannot allocate 100 MB. In this big 1600MB, block, there were not any single contiguous 100 MB block. That's why smartblend is crashing. We need to reduce memory fragmentation, but it's not that simple.
When does that happen ?
The crash often happen at the end of the smartblend process when pictures to blend together become really big. A rendering can work directly or fail.
On a 2GB laptop with duo core, we did intensive testing by rendering a 1.6 Gigapixels panorama. 10 renders, 1 crash. Once you got it, further rendering continue to crash ( memory is fragmented and you need to reboot to try a new rendering ).
It can happen on all OS. I've asked for expertise from top expert in memory management and then explained to me that it is a more and more common issue in today applications. We asked this expert to help us on solving this issue in the current smartblend.
Note 1 : One issue we have is reproducibility : several people sent us their files and .pano and most of time, we were able to render them. But that's normal, because the issue is not in smartblend code but in the system that can decide to manage the memory each time differently.
Note 2 : It's totally unrelated to Qt.
* http://www.autopano.net/forum/t7655-aut … lend-stage
* http://www.autopano.net/forum/t7441-run … sion-2.0.4
How to prevent it ?
I would really like to get an answer to that. It depends on the hardware and system.
So first reboot the computer and just run autopano, nothing else ( and not photoshop, this one has a memory technic that really make smartblend fail ).
Be sure to reduce cache to minimum.
Of course, using multiband will prevent that issue, but you won't have any anti-ghost technology.
Why do have this issue now and not before ?
That's a good question and a part of the answer will come because of the size of panorama beeing rendered actually.
2 years ago, we didn't have much such crash, only sometime. But today, with the introduction of automatic panorama head, it so easy to get hundred images for a panorama that the average number of image used in a panorama has raised a lot.
So smartblend is put into a huge pressure now. It's a old algorithm and wasn't really designed for that aim. It needs to be recreated from scratch. That's our top 1 priority for autopano engine 2.1 ( this work has already started ).
Status of solution / Future
We are waiting for the result of the expert about resolution of this issue in current smartblend implementation.
I had a date of solution for mid-January. We then need to integrated the solution, so this should give an Autopano 2.0.7 for end January.
In the meantime, we are working on smartblend 2, the new design that matches better current size of panorama and has a totally different memory model.
For end January, I hope we'll probably have a working version of this new design. Nevertheless, this new algorithm won't be integrated into 2.0.7 serie but only
in 2.1 because, it's part of the big final rendering revamp. Everything here has changed.
Alexandre
Offline
Well that settles is, everybody go buy 16Gigs of RAM 
Has anyone tried RAM Def / Memory Defrag for Mac or similar tools?
Offline
These tools are just useless. Then cannot interfere with the general OS memory manager.
Offline
Why try to eat the whole bowl of Speghetti in one bite? And surely the "servings" aren't going to get any smaller.
If the memory fragmentation is the issue, I would rather reboot my machine a few times during the process and have it pick up each time where it left off, rather than start from the beginning each time with my fingers crossed hoping for it to finish.
Wouldn't that be a simpler (and faster) problem to solve and still achieve the goal of having a finished image? Are you not able to ascertain during the process when the fragmentation becomes an issue and then close everything up and write it all out to disk for continuation later?
Offline
AlexandreJ wrote:
* total memory used by the application ? 400 MB
* total memory installed : 2 Giga.
* free memory : 1600 MB ( 2 GB - 400 MB ).
* Cause of crash ? I cannot allocate 100 MB. In this big 1600MB, block, there were not any single contiguous 100 MB block.
If the above is not an over-simplification of the actual problem then a "safe mode" where a large piece of memory (100 MB in the above example) is "locked" by autopano before Smarblend and used in the final step only could help (this would not be a brilliant solution, but...)
30 or 40 years ago a fellow worker explained to me that the ramasse-miettes (table tidy) problem was one of the hardest and most important in programming: it looks like he was right!
Offline
That sounds nice, but is there a way to accurately predict how much memory Smartblend will need? If yes, then APP could warn the user even before the beginning of the rendering. "Memory too fragmented, please reboot your system and try the rendering again."
Offline
How about just using fixed size blocks of memory? Or do other programs eat into the memory you've de-allocated?
And why are you deallocating and reallocating memory instead of just re-using it?
Offline
You cannot tell to the file mapping mecanism which area of memory to use. Of course, this idea of having a block of 200 MB ready for that is a good idea. The issue is that you cannot say to the system, "he, use this block for the next memory mapped !". One idea I had was to reserve a block, unallocate it just before the file mapping, the file mapping should then not fail because I know I just released a block large enough just before. And you know what ? I doesn't work either. Seems that memory management behind is not handling standard memory and file mapping memory the same way.
Offline
In theory, I don't see how memory fragmentation could be an issue for the entire computer. The virtual memory manager will map blocks of dispersed memory to a contiguous space. Therefore I can't see that rebooting the computer would clean up anything but if someone can explain why it would I'd be interested.
Within a process, for example the execution of an exe file like SmartBlend.exe, memory fragmentation could lead to problems like this:
Let's say that 1 GB of RAM is available.
1) Allocate 10 100 MB blocks. All blocks are referenced by pointers and can not be relocated by the operating system.
2) Release every other 100 MB block. 500 MB are now free.
3) Try to allocate a 110 MB block. This will fail because there is nowhere a free block large enough.
One solution would be to terminate and re-launch the program after processing one or a few partial image(s). For this, a helper program would have to be created that keeps track of what's been processed and and that launches the program.
Summary: I believe memory fragmentation can only occur within the address space of a process.
Last edited by Progi8 (2009-12-04 18:48:38)
Offline
AlexandreJ wrote:
You cannot tell to the file mapping mecanism which area of memory to use. Of course, this idea of having a block of 200 MB ready for that is a good idea. The issue is that you cannot say to the system, "he, use this block for the next memory mapped !". One idea I had was to reserve a block, unallocate it just before the file mapping, the file mapping should then not fail because I know I just released a block large enough just before. And you know what ? I doesn't work either. Seems that memory management behind is not handling standard memory and file mapping memory the same way.
just googled "windows memory fragmentation". APP is not the only program to have this issue.
Just out of curiosity, why would you deallocate then reallocate a block of memory instead of just clearing it and reusing it?
Offline
hankkarl wrote:
Just out of curiosity, why would you deallocate then reallocate a block of memory instead of just clearing it and reusing it?
So true. That would work if you do memory allocation. With file mapping, you cannot say, "he, reused this block". And that resume the whole situation.
For smartblend 2, we changed the design completely and and we have a file mapping system that correct this. In fact, the file mapping has to be done by us instead of the system. The early results are quite good.
Offline
The only crashes i have are with APT - not with APG and never while rendering even big panos.
I strictly don´t have any other program open when i use APG and have the cache set to default.
I always use an external FW800 drive for temps with free 500GB.
best, Klaus
(Mac Pro, Macbook Pro)
Offline
klausesser wrote:
The only crashes i have are with APT - not with APG and never while rendering even big panos.
I strictly don´t have any other program open when i use APG and have the cache set to default.
I always use an external FW800 drive for temps with free 500GB.
best, Klaus
(Mac Pro, Macbook Pro)
Klaus,
I'm experiencing SmartBlend crashes while using a dedicated 4TB (yes, that's 4TB) FW800 scratch drive.
I think there's something more to this phenomenon...
- Mark
Offline
Single-row stitchers are often able to stitch with very little memory because they know as they move along that they have fully rendered everything to the left and can write it to disk, not leaving either the original images or the resulting blend in ram. That usually means the result is rendered by columns and must then be transposed when done, one quick pass over the pixels written to disk.
With multi-row, I think this can still be done, but you need to look at the aspect ratio and decide whether to do chunks horizontally or vertically, whichever is going to perform better. But you will come to a point where you can decide that everything to the left of a certain column or above a certain row is now fully rendered and can be released or un mapped. Then the chunks must be assembled.
Offline
Are there any special problems reported with Smartblend on Windows 7 64 ?
I`ve had a nightmare with Windows 7 / Smartblend rendering, even with the fresh installed Windows 7 not booting after Smartblend crashes.
Offline
I'm on W7 x64 with APP 2.06 and no problems whatsoever... And I'm using W7 since the public RC!
Offline
PanoramaPrinters wrote:
think there's something more to this phenomenon...
you bet there is! 
I just told my own experiences.
Don´t you think 4TB for scratch might be a bit oversized?
best, Klaus
Offline
I wonder if those with much problems have some variable in common.
The only software I have installed is PS CS3/4 and APP Pro/Giga 2.06.
I render to .psb over 4 gig sizes with default smartblend seettings and usually more than one render in que.
May Windows swap/memory/cache settings or any other Windows settings ? ( on Win7 )
Offline
[bo] wrote:
I'm on W7 x64 with APP 2.06 and no problems whatsoever... And I'm using W7 since the public RC!
Do you render 4+ gig files to .psb with default smartblend settings ?
Last edited by JohnM (2009-12-08 19:29:42)
Offline
Alexandre, Thank you for posting this explanation of the Smartblend algorithm and problem. I had been searching for a better explanation of what is going on without reading through code.
Offline
I do not. And I don't think Smartblend was designed to take that kind of stress anyway, so I'd use Multiband if I had to. The same way as I wouldn't print my 4000-copies full-color leaflet on the office inkjet.
Offline
Bo, I understand your argument, but there are cases where we might want to use smartblend on gigapixel shots with filesize over 4 gb.
Maybe its possible to do the rendering in several steps, especially for full spherical images where we might render 6 cubefaces with smartblend to use with krpanotools. The challenge is to make a script that works with the APP to send 6 cropped cubefaces to smartblend.
Maybe APP should have a feature to render as cubefaces ? Is it something that should be requested ?
Offline
What is the actual goal here is replacing Smartblend with something else, something better. Which is exactly what team Kolor are doing right now. So that's that. I'd just wait and see. To dedicate time "fixing" the current Smartblend is a foolish waste of resources, so they're building it anew.
Offline
I do have access to a Mac with 30-40 gb ram. Is it likely to have problems with memory fragmentation too ?
Offline