Quadro K6000 vs. Tesla K40

X-COM writes:
> I'm a small manufacturer of Hi-End custom built computers in the Czech Republic and I want to expand
> my offering with multi GPU workstations so I’m mapping the options. http://www.x-computers.eu/

Best of luck with your venture!! 8)

Until 2010 I mainly dealt with SGI systems (still do), but then I began to branch out into custom PC
builds for people, everything from simple daily desktops to top-end non-XEON pro-type systems,
making the most sensible use of 2nd-hand parts and new-item auctions to minimise costs for solo
professionals, small businesses and hobbyists on a budget, eg. builds such as this. I guess this is not
at the level you're working on, but I've had to wade through many of the same issues.


> Premiere Pro is a program I'm actually using myself so I have the best grasp on its requirements.

Adobe is behaving rather oddly atm with respect to how it's making use of GPU acceleration. It's
making a lot of noise about OpenCL, but at the same time it updated something recently for
Premiere as regards CUDA. I wish they support both equally, give people the real choice, but
right now the situtation is a bit muddled. It's also hard to find direct performance comparisons;
many sites reviewing new GPUs won't include CUDA results for NVIDIA cards, which is crazy.
All they do is affirm what's already obvious, that a typical AMD gamer card is better than NVIDIA
for OpenCL, even though the same NVIDIA card might blast through the same app massively
faster ith the relevant CUDA plugin. Especially irritating when it's something like AE being tested.


> I have already concluded it does not benefit from Quadro or Tesla cards in performance. So
> Titan Blacks are the most powerful while the 780Ti are a more sensible choice because of price.

Indeed! Titan does have a larger VRAM, though I wouldn't be surprised if this doesn't matter much
unless one is at least working with heavy HD material. I guess if one was going to be working with
4K, then Titan's 6GB would make a lot more sense, assuming the app in question doesn't have
some dumb legacy 32bit limit on how much VRAM it can exploit.

Since I've been working on budget-limited builds, I've obtained lots of GTX 580 cards instead. Good
performance and less upfront cost, at the expense of greater heat output and power consumption.
Naturally, as a video artist progresses, s/he can upgrade/replace such cards later to increase system
efficiency, performance, or both.

If the budget permits though, it's definitely best to jump straight to the 700 series, Titan & newer Quadros.


> Quadros in PP are just for 10bit color output, right? ...

Quadros certainly offer deeper colour modes, but I'm not sure to what extent the various Adobe
apps can make best use of these higher modes. Adobe's site should have the details.


> Can I combine a Quadro for 10bit output with a GeForce for additional rendering acceleration? ...

Yes, I've done this many times.

Issues to watch out for though:

- It's wise to ensure the two different driver sets are as closely matched as possible, even if this
means using somewhat older drivers. At the moment I'm using the 310.90 release, because this
version was made available for both Quadro and consumer cards. I think there's a slightly newer
combination one could use aswell, but I forget offhand, though note I've not checked any newer
possibilities since Mar/14. Note I'm not saying that using different driver versions won't work,
merely that it's a safe assumption one is more likely to encounter issues if the versions are
significantly different. In my case, because I'm using 500 series cards (GTX 580), reliability is
maintained by not using anything newer than 314.22. If I was mixing a newer Quadro with a 700
series card though, then it would be more sensible to explore a driver combo which made use of
much newer releases. I don't have any 700 cards yet, so I've not examined newer driver combos
which would be feasible.

- Install the Quadro and relevant drivers first. Once done, unpack the Geforce driver archive, let
it get as far as the License Agreement question, then quit from the archive. Or, as I do, just copy/paste
the two folders previously prepared elsewhere (I use an installation SSD that holds all the various
downloads, utils, etc.)

- If installing from original driver archives, rename the display driver folder numeric names so they're
more meaningful & distinct, eg. after I install the Quadro 310.90 driver, inside the C:\NVIDIA\DisplayDriver
folder there is a folder called 310.90, so I rename this to quadro, or quadro_310.90, something like that.
Likewise, once the desktop driver archive is unpacked, I rename the new 310.90 folder (yeah, NVIDIA
uses the same name) to desktop, or gtx, or gtx_310.90 (whatever your preference). Once done, and
with the 580s fitted, I use Device Manager to access the properties panel for the 580s, select the first
one, choose Update Driver, manually select the relevant folder within the desktop folder mentioned
above, let it install the 580 desktop driver. It will automatically do the same for the other cards.

- In the NVIDIA Control Panel (NCP), there are two important settings: one to specify the OpenGL
rendering GPU, the other to define which GPUs are used for CUDA. If you only want certain cards
to be part of the CUDA pool, remember that changing these options in the wrong order will result
in the CUDA setting being reset or inverted. Thus, always set the OpenGL rendering GPU option
first, then set which GPUs are used for CUDA. Do this after sorting out all other settings such as AA,
texture quality, etc. An example of where one would not want to use all GPUs for CUDA is the combo
I've used several times in the past, namely a Quadro 4000 and multiple GTX 580s; I set the Quadro
to to be the OpenGL GPU, while only the 580s form the CUDA pool. Checking the GPU info panel
in AE is an easy way to ensure the corrects settings are in effect. In other words, I reckon it's best
not to include a Quadro card in the CUDA pool if its CUDA performance is rather mistmatched
vs. the Geforce cards. By contrast, a more recent build I did with two GTX 580s and a K5000 had all
three cards in the CUDA pool since the K5000 is only slightly slower than a 580 for CUDA.

- Remember to install a custom 'ray tracing supported cards' file in the relevant folder if you want
to make use of cards not supported by default. I don't have the one I use to hand atm, but if you PM
me I'll send you a copy, or I can post it here later this week if you like.


> But my costumers use other programs like After Effects, Photoshop, Illustrator, Blender, Catia,
> Maya, AutoCad, 3Ds Max, Rhino, ZBrush, Petrel...

AE certainly benefits to a huge degree for CUDA, that's the main app I've been investigating. Photoshop
and Blender also make good use of CUDA (the latter via the Octane Render plugin if need be). The
others I'm less familiar with, but check the Octane Render plugin info page, it has a list of which
apps can use the plugin for CUDA (I'm sure 3DS Max is one of them).


> Some of those programs benefit from Quadros (Catia, Maya, Petrel). ...

Indeed.


> ... But the question is: would whose programs benefit from more than one card?

I'm not sure... the older versions of these apps (via Viewperf 11) showed no gain from SLI,
but I only tested with gamer cards. So far I've not tried testing with Quadros in SLI, partly
because it's hard to sort out a setup where that will work. I do have a Dell T7500, but
annoyingly, Dell restricted which Quadros will work in SLI on the system, to an extent that
even NVIDIA was not aware of (so I was told by someone at NVIDIA when I asked why a
particular Quadro SLI combo didn't work even though the T7500 is a certified SLI system).

From the tests I've done so far though, I suspect those apps would respond more strongly
to a better CPU setup, with stronger IPC. I can't say for sure though yet as I've not done
enough tests with stock speed CPUs to form conclusions, but here's my data so far:

http://www.sgidepot.co.uk/misc/viewperf.txt

NB: Don't be fooled by the 7970 results, I found out the numbers are high because the AMD
drivers ditch an awful lot of precision to trade off for higher performance, ie. the image quality is
poor compared to pro card results.


> For what task would you recommend a Tesla? ...

Scientific computing, tasks related to fields in physics, chemistry, cosmology, where 64bit fp
and ECC memory matters more. It's not universal though, some codes are hard to port to
GPUs, others only need 32bit fp, some don't need ECC, and so on. Oddly enough, financial
transaction processing is better with Teslas as ECC is absolutely essential (a Titan might be
ok for initial development work, but not for deployed production systems).


> Has it to be a program specifically written for Tesla?

I should imagine custom coding is the way to get the most out the tech for sure, but no doubt
there are available bundled libs to make things easier in various fields.

Ian.