Renderfarm

[HansKeesom]

Renderfarm

by HansKeesom » Mon Sep 05, 2011 5:23 pm

A number of 3D applications are already connected to a renderfarm. An example can be seen at http://www.rebusfarm.com/

I like that idea, even though I don't have large enough panoramas to make it worthwhile.....

Kolor could pick it up themselves or at least make the software suitable for something like this.

[tived]

by tived » Wed Sep 07, 2011 2:02 am

Hans,

I like your thinking ;-)

Henrik

[HansKeesom]

by HansKeesom » Mon Sep 12, 2011 1:58 pm

And at the same time, would be nice if the renderfarm concept can also be used locally, so over the network, when having your own little renderfarm on which you can have your stuff rendered. I should of course be shareable with other people that are processing panoramas, whether they are in your LAN, WAN or WWW. This way you can do editing on a relatively normal PC and put all power into the renderfarm machine.

[HansKeesom]

by HansKeesom » Mon Sep 12, 2011 1:59 pm

Thank you, Tived

And at the same time, would be nice if the renderfarm concept can also be used locally, so over the network, when having your own little renderfarm on which you can have your stuff rendered. I should of course be shareable with other people that are processing panoramas, whether they are in your LAN, WAN or WWW. This way you can do editing on a relatively normal PC and put all power into the renderfarm machine.

[HansKeesom]

by HansKeesom » Mon Sep 12, 2011 7:51 pm

Interesting remark at http://www.tomshardware.com/reviews/render-farm-node,2340-8.html

"Say your workstation is a Core i7-940-based machine, which turns a CineBench score of 14,544. For the price difference between that processor and the Core i7-965 Extreme, you can build an entire Core 2 Duo E7200 render node, which would give you a combined CineBench score of 19,762, far exceeding the performance of a Core i7-965. If you had a workstation using a Core i7-920 instead, the savings, when put in your render node, would allow you to build the node using a Core 2 Quad Q9550 and give you a combined CineBench score of 24,389, which is very close to doubling performance. Note: the ability to show combined performance by simply adding the CineBench scores is why it was used for this comparison.

The other performance advantage to having render nodes is that they can be rendering while you are working on your workstation, thus maintaining full interactivity and enabling you to use your processor's resources while the nodes are crunching away on a rendering task. For a complex project, this capability can become very important."

Now the nodes they are talking about are not filled with lots of RAM and for rendering you do need more RAM. But the statements hold I think as you would have to put this memory in your workstation otherwise.
For me personal, I would keep my current workstation as workstation and add the renderstation with lots of GB's. Locally in my LAN or at a renderservice. Would need a higher speed upload then off course.

[AlexandreJ]

by AlexandreJ » Tue Sep 13, 2011 7:40 am

Interesting facts

[HansKeesom]

by HansKeesom » Tue Sep 13, 2011 1:11 pm

Hi Alexandre,

Thank you.


As this is the autopano future section I use it as a dump for all the ideas that i get while working on panoramas. It will be up to you to decide what is a good idea and what not, what is easy to build and what not. I guess this is not an easy one.

If only two copies of the batch renderer running on different computers inside a LAN could communicate with each other and do some work for eachother..... one would then just startup a second machine (or even more) startup autopano on these and have a farm up and running.

The user would then rightclick on the projectname of a project that is not running yet in the list of the Batch renderer and choose an option like "run on" which shows a list of machines the project can be send to. The intelligent of where things are best run is then left to the user. This could also be a dropdown on the window Render Panorama.

After finishing all generated data is send back and placed in the local directory choosen for this project

An external online renderfarm will allow for many people to create much larger panoramas in shorter time, although uploading all images might take a while. Would be great to have a setting which allows you to tell autopano at which size of output-file it should call the external render.

This will position autopano as software that automates a lot of the work and allows you to do things that other software just can t.

However, if this is to be a very big thing to build, I can imagine other work goes first.

Last edited by HansKeesom on Tue Sep 13, 2011 3:40 pm, edited 1 time in total.

[pauldu79]

by pauldu79 » Tue Sep 13, 2011 3:37 pm

very interesting.
Thank you for posting, it was very usefull !

[HansKeesom]

by HansKeesom » Tue Sep 13, 2011 3:46 pm

Hi Paul,
Thank you

Usefull yes, for me to get rid of all these ideas :-)

[CheeseAndJamSandwich]

by CheeseAndJamSandwich » Sat Mar 03, 2012 10:09 pm

This could solve the priority problem we have here: http://www.kolor.com/forum/t11676-rendering-priority-in-settings

If the rendering engine is a separate application, which is given a .pano project file to render, then it could be either local or out in the farm (many computers)... The farm could be managed by another little app.

So the render dialog box would have this option within it, local or farm.

The local rendering process would be able to have it's own priority set, fixinging the above thread's topic...

[Phil Howard]

by Phil Howard » Sun Mar 04, 2012 3:56 am

It shouldn't be that hard to add multi-machine network rendering management. But I would suggest that it not depend on keeping the desktop running. So one should be able to shut down the desktop, then next day start it up, connect to the render manager, and see what's been done so far.

[CheeseAndJamSandwich]

by CheeseAndJamSandwich » Sun Mar 04, 2012 5:25 am

We could even think like SETI/Folding/etc
screensavers on the directors/CEO's PCs, doing piece work for the render job, whilst they're out playing golf, taking backhanders

[HansKeesom]

by HansKeesom » Tue Mar 06, 2012 1:39 pm

Seti is a volunteernetwork for good cause. Don't think people will want to do this kind of work with there private machine. Also most pc's will not be strong enough to run the pano's we might want to have them render.

[CheeseAndJamSandwich]

by CheeseAndJamSandwich » Tue Mar 06, 2012 4:38 pm

Seti is a volunteernetwork for good cause. Don't think people will want to do this kind of work with there private machine. Also most pc's will not be strong enough to run the pano's we might want to have them render.

I was joking

And thinking of all the PCs on a private office network, rather than everyone out on the interwebs.

But

If you have a big computing task that can be broken down into small pieces, like SETI, Folding, etc, then having lots of 'normal' PCs available is very, very effective. And realistically, almost all computers these days are capable of helping out... as long as the chunks of data only take, say a couple of hours at most say to process, so they can do a few chunks per night...

When at work, i've wondered why we couldn't use the 100 or so PCs we had sitting idle for 16 hours of every working day, and all weekend... We had to wait for big FEA & CFD jobs to run (Finite Element Analysis, Computational Fluid Dynamics), which were run on an expensive unix workstation... but if we had the 'farm', then we could run jobs much, much quicker, or more complex/accurate jobs overnight that we couldn't on just one workstation. That would be much more cost effective, without the need for over-priced top-end hardware.

This is easy with open-source or free software, as there's no problem installing the farm client on all the office PCs you want, but with commercial products, you often pay per processor, so it very quickly becomes too expensive.

i don't know if processing gigapixel panos can be broken down into discreet chunks suitable for this kind of farming, but it's interesting to dream of

[AlexandreJ]

by AlexandreJ » Tue Mar 13, 2012 9:45 am

i don't know if processing gigapixel panos can be broken down into discreet chunks suitable for this kind of farming, but it's interesting to dream of

That's the real issue : is it or not ?

Some part of the rendering is doable but with some early estimation, sending or getting back huge images will just kill the new CPU you've got from the renderfarm.
The key point to make it scalable is workflow : what should be done where, what does it need and what should be send back.
You have 2 bottlenecks : CPU power and bandwidth. In CG renderfarm, the issue is CPU. For gigapixels rendering, I think the bottleneck is bandwidth ( because of the data size ).

[HansKeesom]

by HansKeesom » Tue Mar 13, 2012 1:04 pm

Good thing is that more and more people are moving to fiber or to 50/100mbit cable here. Wondering if that would change the picture a lot.

I already have one photographer uploading all her panorama photos to my server. Yes it takes long to upload but as it no problem to run through the night there is no problem. The next day I process them, though she can do it herself too, remotely.

[AlexandreJ]

by AlexandreJ » Tue Mar 13, 2012 4:23 pm

Uploading is one part (1000 RAW is around 20GB), but then getting back 200GB because it's a giant KRO file is something else ...

[Hellkeeper]

by Hellkeeper » Tue Mar 13, 2012 5:06 pm

this would only make sense for local networks - connected with 1/10GBit ...
let's say you could break up a large Gigapixel like the one I work on right now (~80 gpix = ~344GB final KRP) on our 10 workstations in our office it would be 34,4GB of data to be transferred per node ... guess that would be about 6-10 Minutes in 1 Gbit ... x2 (up and down) = 10-20 Minutes.

The final render time for this gpix was 12h 51min 1s ... this splittet by 10 nodes would be ~1:30h rendering time per node ... not calculated the overhead in stitching the final parts of the nodes together in the end, and not taken into account that the nodes have different CPUs/RAM ...

Well - guess it would be a very nice technical feature - but as I know how hard load distribution over multiple nodes is - I guess it's not worth implementing it ... ;-)
Especially when a good Workstation is able to render a 80 Gigapixel Image in under 13 Hours ... ;-)

Just my 2 cents! ;-)

Andreas

[klausesser]

by klausesser » Tue Mar 13, 2012 5:10 pm

Uploading is one part (1000 RAW is around 20GB), but then getting back 200GB because it's a giant KRO file is something else ...

Right. In the end the question is: who really needs more than, let´s say, 30GPix? Maybe it´s nice to have some terapix . . . but what for? For the Guiness book?

Everybody who thinks about gigapix must know the demand on hardware rises exponentially. And also rises the time to spend with it.
So it´ll never commercially earn enough money to compensate the effort.

In my mind it would be more important to make interesting panos instead of extra-large panos of always the same and somewhat boring snowy mountains and blue seas with some stones in the foreground or one marketplace and ancient castle following the other . . . :rolleyes::cool:

I mean: the motiv mus justify the size of a pano! There surely are motives which are great to see them extremely detailed! The panos from Julian of the church in vienna, the "Last Supper" and so on. Here very big stitches really do attract me and give me relevant informations.

The 26megapixel from Paris do it also: very interesting scenes. But would it be even more interesting having 100mpx? I doubt.

best, Klaus

Last edited by klausesser on Tue Mar 13, 2012 5:20 pm, edited 1 time in total.

[lkwpeter]

by lkwpeter » Sat Mar 24, 2012 10:14 pm

I also would love to have multi-machine support for Autopano.
We've got a bunch of high-speed machines here, which are connected by 1, 10 GBit/s and Fibre.

A problem would be the licensing: would there be higher costs for a multi-machine license?

[CheeseAndJamSandwich]

by CheeseAndJamSandwich » Sat Mar 24, 2012 11:03 pm

Uploading is one part (1000 RAW is around 20GB), but then getting back 200GB because it's a giant KRO file is something else ...

When a gigapano is rendered, currently, how it it broken up from processing??? It can't do everything at once, so it must surely do discrete chunks of it?
...which could be farmed out and returned when done...
So the 'Farmer', the editing computer which has all the data, would send out 'small' chunks to each member of the 'team', perhaps sequentially to prevent thrashing of it's disk and saturation of the LAN... Perhaps sending out chunks to members for processing after they finish the chunk they're currently doing... i.e. Caching... And the finished chunks would be coming back at various times, not all at the same time.... The farming out/caching could be tweaked and tuned depending on size of job, priority needed, etc... So the load on the network and farm members could be managed... So if running over night, you'd let it rip, saturating the LAN to farm it out and running the rendering at normal priority on the members... OR, farming out 'slowly' so there's little impact on the LAN, and running the rendering at Idle Priority, so you could have it running in the background when someone is using the member PC!

It would be a lot of data, yes... But then it's a big job! But undoubtedly it would be done quicker than a single machine...

And this would then open up the practical possibility of Terapixel image creation by us more mortals! (APT!)

[CheeseAndJamSandwich]

by CheeseAndJamSandwich » Sat Mar 24, 2012 11:15 pm

Just thought...

If chunks are then bought together for processing into bigger chunks... then the farm members could then be instructed to send data peer-to-peer,... taking some load off the 'farmer'... so then one peer gets all the chunks to process the bigger chunk... ?

If you guys coded something like this... then it's going to be almost infinitely scalable

And you could sell multi-machine licences of different sizes for monthly, quarterly, yearly periods... as some people might just have one big project to do... so might spend a month editing, rendering, tweaking it...

[HansKeesom]

by HansKeesom » Wed Jan 30, 2013 4:59 pm

meanwhile I have been playing around with virtual machines in the cloud.
Data can be transfered to it using dropbox or other services. When uploaded to dropbox I start one machine configured 1 processor and 1 GB memory. Once everything is downloaded onto this machine, I reconfigure it to became a 8 core 64 GB RAM machine. Piece of cake.

Having this machine turned off costs me only 1,60 euro a month, having it run costs up to 50 euro a day.

I can start one machine but there is no problem starting 256 machines from a template. Every machine costs 50 euro when configured at 8 core xeon with 64 GB.
They are in one network/ ip range so should not be to difficult to have them communicate with one another.

In theory should be easy to quickly create a huge renderfarm the moment you need it and to remove all machines when done. Would be great if you don't even need to logon to the slave machine, just start them and they automaticly add theirself to a central machine on which one does the main work.

Dreaming? I think the more technocal people can do wonders with these dreams ;-)

[klausesser]

by klausesser » Wed Jan 30, 2013 8:55 pm

Dreaming? I think the more technocal people can do wonders with these dreams ;-)

Maybe - but only when they never learned thinking business . .

256 machines for 50.- € a day = 12800.-€ PER DAY + costs of traffic and a lot of time for transporting lots of gigabytes online . . . or adding the cost for a symmetrical high speed upload . .

For 10 full days of use you can easily build your own renderfarm - a real cool one . . :cool:

I wonder whether you know that there are facilities, where you can hire REAL renderfarms for the render-time you need?
A facility here in Düsseldorf provides up to 1024 processors containing xxx cores. And that´s no usual servers - they´re dedicated 3D/image render-farms.

You give them your data on a HD - and off it goes. Takes a fraction of the time as in the cloud-model, i bet.
Even driving 500km to the next facility saves time and costs compared to the cloud-model.

That´s because of the most expensive item of all items: time. Driving 4 hours to take the HD to the facility and let them run the thing, sending the result to you via their hispeed connection
you´ll have it done faster than only the upload of your data via your ADSL connection.

Cloud-computing is a fascinating method, no question. But for processing very large data/panos/images/movies in the shortest possible time?

I doubt.

But time is what counts most - because time means money in a producer´s world.

best, Klaus

[beetwo77]

by beetwo77 » Thu Jan 31, 2013 1:57 pm

The concept of a local renderfarm is interesting to me but I agree with Klaus, I doubt there is a strong commercial driver for it. I use engineering software for sewer modelling and flood modelling. It now has a simulation server that splits single simulations across multiple PC's on LAN. If graphic cards are powerful enough, the simulation engine can use graphics card for processing. I am not an expert in programming but I imagine conceptually that if a single numerical simultion of a grid type problem can be split between many PC's and use graphics card acceleration, it is hard to see why rendering of a panorama could not be but I would think Kolor would have to charge more for it and I am sure they would.

My commercial focus is limited. I sell landscape photographs in sizes up to 70 inches. I don't need gigapixel images and lately I've been dropping back to only 2 or 3 shots per pano and still getting very good quality prints up to at least 50 inches. I am sure somebody has a commercial need for such high resolution imaging but I think it is limited. I'd need a better image editor for a start!