I found an interesting tid-bit regarding an Asus X79, but the claim in the block diagram was from Intel's website regarding their own boards, so it is general.  Notice how one of the PCIe 3.0 lanes is being used for two additional USB 3.0 ports...what's the EVGA layout look like?  We can't assume all the lanes are used just for PCIe anymore.
 

eduncan911
 
Your generalization that only 2% is lost is incorrect with CUDA applications, and you are even quoting older 480 hardware opposed to 580s running at 975 Mhz (which, mine are). 

If I Google "CUDA PCIe scaling analisys" guess what I get?  Nothing relevant, why?  Not enough demand, not enough people using CUDA to their limit question PCIe scaling to have a reviewer do the analysis for all to see.  Most people are concerned with gaming PCIe scaling, hence if I search for that I get many more hits.  My 2% "generalization" was an average and only based on gaming and benchmarks since I don't use CUDA, thus if I talked about it I'd be ignorant, wouldn't I? 
 
Per link above, the loss at 2560x1600 for GTX480 is even lower, at 1% at x8 and 5% at x4, it isn't linear.  So your equivalent loss at x8 isn't much just because your fill rates are 40% higher due to more cores running faster-- of course you're not playing a game with it.
 
That said, how different is the rendering computation of a massive resolution with high-details from the back-and-forth computation of using the graphics processor for parallel applications?   I feel whether the output is with multiple frames per second (like a game), a single-frame render (CAD model) or a massive text file doesn't make a difference with respect to the processing power and communication accross the PCIe slot.  I get the impression from your reactions that your specific CUDA uses need more bandwidth (i.e. more lanes) than a game at some extreme setting.   Are you too impatient to show/inform me so I can better understand the situation as you see it?
 
Maybe you can make more sense of this FAH analysis.  Tried looking into this, but nothing exceptional about PCIe scaling performance.  Found a game-oriented link thinking GPU-intensive has more in common with CUDA uses than CPU-intensive such that 3-way SLI GTX580's at 2560x1600 don't perform much differently X79 or P67 so the lane losses are equally minimal with a GTX480-- but of course it's a game thus you can ignore that too.
 
Here's a thought:  If you change your board configuration from x16 to x8 to x4 to x1 for all four of your HC's, how does your CUDA performance change?  I suspect x4 on down may show a sharper drop, but I doubt the difference between x16 and x8 is 40% to account for the difference between a stock GTX480 and your GTX580 HC's.  But if you are worried about there being any difference at all, then there is nothing I can do to calm you.

One of the distributed computing projects I run is Einstein@Home. I measured a significant loss in performance by moving the card from a 16x to 8x slot which I've tested multiple times. I measured similar loss in both Linux and Windows operating systems. Here are the results I had with this particular application using the Linux GPU application:
 
16x: 2131 seconds
8x: 2984 seconds
 
In the case of the 16x slot, the tasks completed significantly faster. Multiple others users have reported similar performance loss so these results have been confirmed. Not all CUDA apps are affected but I have seen performance loss by switching to an 8x slot with some applications particularly those applications which have a greater dependence on the CPU.
 
If I was only gaming, I would not worry about the PCI-E lane allocation but I use my GPUs for other purposes as well. Both CUDA and OpenCL are becoming increasingly popular as time goes on and as a result users will become more dependant on additional PCI-E lanes that the latest processors can be provide.

What about 2-way SLI plus Physx? I am planning an X79 upgrade and want to use EVGA mainboard. I want 2x580 SLI and a GTX 285 for physx. Can I use the GTX 285 for physx and keep my 2 580's @ x16?
 
Thanks

FCooley


What about 2-way SLI plus Physx? I am planning an X79 upgrade and want to use EVGA mainboard. I want 2x580 SLI and a GTX 285 for physx. Can I use the GTX 285 for physx and keep my 2 580's @ x16?

Thanks

And there, my friend, is exactly the issue.  The answer is no.  If you run just two cards, you 'll get x16.  But if you stick anything into the other two slots, say a card for PhysX, your two cards will run in x16-x8, and the PhysX card will be x8.  Absolutely no different than the X58 platform, and actually goes against the specifications of the X79 platform itself. 

Thanks for the clarification. Even if it did support X16, X16, X8 I would plug in a X1 sound card anyway. I don't think for gaming its gonna matter much. I found an article on Hardocp where they compared SLI 480's with X16 & X8 and only really high res 5760x1200 had an impact in performance.
I'm still going to go EVGA for an x79 build. I'd hate to have to deal with any other Mainboard manufacturer's support, and a possible RMA would be a nighmare.

FCooley


Thanks for the clarification. Even if it did support X16, X16, X8 I would plug in a X1 sound card anyway. I don't think for gaming its gonna matter much. I found an article on Hardocp where they compared SLI 480's with X16 & X8 and only really high res 5760x1200 had an impact in performance.
I'm still going to go EVGA for an x79 build. I'd hate to have to deal with any other Mainboard manufacturer's support, and a possible RMA would be a nighmare.

And I game at 6000x1080, yep it affects me directly.  And to think, they were using stock 480s.  My 580s hydros are overclocked 33%.  Btw there are two non-direct-PCI-E slots you can use for the 1x sound card (on the FTW board, the SLi board 1).  Those do not affect the x16-x16 performance of using only two cards.  E.g. place your two cards separately on the two furthest slots, and put the 1x sound card into a slot between them marked as 1x or 4x.
 
This was not documented and I've now decided to return my EVGA X79 FTW.  Problem is, they are trying to charge me a restocking fee for their mistake (all pieces, even the mobo, is unopened - I only opened the box to verify it had the LGA2011 protector as those without it are not covered under warranty - now, I am being punished for inspecting it to make sure I had warranty on the product). 
 
This is turning into a real nightmare.  $60 restocking fee, for inspecting if the board was covered under their warranty? 
 
 
 

I have a stupid question, I have 2 gpus and the new X79 Classified mobo what PCI SLOTs do I have to put them in for both to remail at 16X

-see post 24 of 37

*ahem*  anyone wanna buy an X58 with 3 16X... with the NF200?

N200 chip rules, think N200 chip should go to x79 platform reason is Intel acting stupid and not providing enough PCIe lines.

lradunovic


N200 chip rules, think N200 chip should go to x79 platform reason is Intel acting stupid and not providing enough PCIe lines.

er they don't do pci-e 3.0

lradunovic


N200 chip rules, think N200 chip should go to x79 platform reason is Intel acting stupid and not providing enough PCIe lines.

Im pretty sure X79 has not only MORE PCIE lanes available, but also double the bandwidth with PCIE 3.0. So how is Intel "acting stupid" exactly?

lehpron


Looking at the X79 FTW visual guide page 2:

Slot 1:   x8/x16 (switch)
Slot 2:   x1 (perm)
Slot 3:   x8 (perm)
Slot 4:   x4 (perm)
Slot 5:   x8/x16 (switch)
Slot 6:   x8 (perm)

I wonder if it is possible that slots 3 & 6 share bandwidth with 1 & 5 such that if they are populated, then they switch, thus 3-way SLI + PhysX as x8+x8+x8+x8.  But JacobF didn't cover 2-way + PhysX, so it should be possible using slot 4 for PhysX and maintain dual x16's.

Looking at the X79 Classified visual guide page 2:

Slot 1: x8/x16 (switch)
Slot 2: x1 (perm), same as FTW
Slot 3: x8 (perm), same as FTW
Slot 4: x4 (perm), same as FTW
Slot 5: x8/x16 (switch)
Slot 6: x8/x16 (switch)

JacobF says it 8-8-16 for 3-way SLi, so similar to FTW, then 1 is affected by 3 and 6 is affected by 5.  Except if nothing in 3 or 6, 1 and 5 remains x16.   If wanting 2-way SLi, must use slot 1 & 5.  Any other add-in PCIe like PhysX or audio must go in 2 or 4, otherwise you loose dual x16.

Looking at the X79 regular SLI visual guide page 2:

Slot 1: x8/x16 (switch)
Slot 2: nothing
Slot 3: x8/x16 (switch)
Slot 4: x1 (perm)
Slot 5: x8 (perm)
Slot 6: x1 (perm)

JacobF says up top that 3-way SLI will enabled 16-8-8, so why is the top slot 1 allowed to switch, shouldn't it be permanent x16?  Clearly a card occupying slot 5 affects slot 3.  So unlike FTW, 2-way + PhysX will loose dual x16's.

daleorama78

You completely missed the point........Every other board manufacturer used 32 also for the X58. Every other board manufacturer is using 40 lanes for the X79. Your point is moot.

I had a lot of points, don't generalize.  Quite frankly, when PCIe 3.0 gets certified, there is reason to worry about that because each x8 3.0 will function like x16 2.0.  Like I said, it will be several years before 3.0 spec graphics cards come to succeed 8 lanes, by then X79 will be dethoned.

 
A correction on your Classified chart. I didn't get it myself before, but I do now, why both slots 4 and 5 are labeled PCIE_X16/X8. The type of switch between these slots works both ways. If you look at the 2-way SLI connector from the pictures, it looks longer than the 3-way SLI connector and as long as the 4-way SLI connector, making me believe that you use your very first and very last slots for 2-way SLI, a rather smart design for OCing. In this case, your lane distribution would be: X16 (GPU), X1, X0, X4, X0, X16 (GPU). For 3-way SLI, you would use your top most PCIe X16 slots leading to a lane distribution of: X8 (GPU), X1, X8 (GPU), X4, X16 (GPU), X0. For 4-way SLI, you would use all your PCIe X16 slots that are evenly spaced apart leading to a lane distribution of: X8 (GPU), X1, X8 (GPU), X4, X8 (GPU), X8 (GPU). Therefore, if you want to load in a PhysX card or any other card that uses more than 1 lane, it is best to use that X4 slot (physically X16), as it is on its own, dedicated lane. Remember, PhysX is not needy when it comes to PCIe lanes, so even a PCIe 2.0 X4 connection is good enough. Of course, this being a PCIe 3.0 X4 slot, newer cards will see a slight benefit for PhysX. For cards that only require one lane, use the X1 slot. Another note, all the red slots are of course PCIe 3.0, while the black slot is PCIe 2.0 (from the chipset).
 
And yes, it looks like eVGA could have used that extra X4 bandwidth to bump up 3-way and 4-way SLI. My guess is, it was supposed to be used for SAS until Intel cancelled that for X79. Perhaps they never got around to rewiring the board, in which case, don't expect a BIOS update to fix that. Not really a big deal for gamers considering how fast PCIe 3.0 is, although this may be quite annoying for other people who need the bandwidth.