Discussion and research below have lead to updates to post #1.

All posts from #2 to #14 are now irrelevant as they related to the older version of this topic, and you don't have to read them as everything that was discussed there is now covered in post #1.

Yes that's right, 24 GB was about twice what I would need. ;-) I work in design and audio production, and both of those uses are happy with around 9GB of RAM without ever filling that up. I just bought that full set of 12 sticks (24 GB) on a whim, thinking "meh, I have money, why not max it out".

Having more RAM eliminates swapping/paging so you should ALWAYS pursue being a bit above your maximum required RAM amount, but going above that is pointless, and as just mentioned, 12 GB will be SAFELY above the needs for my uses. Going above 12 GB will not only not give me any benefit, it will actually downclock the RAM as described in post 1.

For my uses, there's a LOT more focus on the SHUFFLING of data back and forth from RAM, particularly audio production, where you have hundreds of plugins working on the same piece of data simultaneously in REALTIME, causing lots and lots and lots and lots and lots and LOTS of travel back and forth in and out of RAM, as each plugin does its work.

Therefore, it's more important to have the maximum available bandwidth to cut down on latency (allowing lower latencies in the production software, which is helpful for realtime monitoring), since 24 GB was way overkill already.

Hope this thread helps other people decide what they need: Capacity (12 sticks) or Performance (6 sticks).

Yes you are right that if you can maintain a RAM amount over and above the RAM needed it SURELY is better using fewer dimms per channel.

And i can see that for your audio processing needs the ram speed will matter quite alot. And you sure should be making very good use of the sr-2, in that field.

I would still do some overclocking, even for a 24/7 workstation use. At least that is the only way i see any need for the sr-2 above ordinary 1366 boards (except of course the 7 PCIe slots)

IMO one could easily overclock to say a 180 BCLK and still maintain full 24/7 stability (also without extraordinary cooling, 2x Corsair H50 would solve it no problem, or any other good aftermarket cooler)... Meaning your RAM could run at a cool 1440mhz (with all dimms populated, 8x multiplier) and your processor at something like 3,60-3,96ghz.

I mean, come on, if you are worried about the impact on your performance if your memory runs at 1066mhz instead of 1333mhz.. THINK about the positive impact on your performance if you get your ram running at 1400, 1600 or say 1800 mhz? !! :D Also, man, get those cpus overclocked! They will rock your processing speeds. 

xeonite
 I would still do some overclocking, even for a 24/7 workstation use. At least that is the only way i see any need for the sr-2 above ordinary 1366 boards (except of course the 7 PCIe slots) 

 


Bingo! ;-) There are two reasons I am going for the SR-2. The first one is that I prefer overclocking boards because they tend to have the highest quality components, leading to the most stability and versatility, and often much more features than other boards.

The second reason is that I don't live in the 1990s anymore, why does everyone insist on PCI slots?! I havent had ONE SINGLE PCI card since about 2005, if not earlier. Meanwhile, I *really* need the PCI express slots, and that's where the NF200 PCI express lane multipliers come in handy, giving access to 7 PCI express slots, which is enough to run 4x Audio DSP processing cards, 1x Graphics card, and have 2 slots free for something else. :-)

Those are the real reasons I'm choosing the SR-2. Every Intel 5520 board has about 2-4 PCI express slots and a bunch of PCI slots. It's sickening.

xeonite 
IMO one could easily overclock to say a 180 BCLK and still maintain full 24/7 stability (also without extraordinary cooling, 2x Corsair H50 would solve it no problem, or any other good aftermarket cooler)... Meaning your RAM could run at a cool 1440mhz (with all dimms populated, 8x multiplier) and your processor at something like 3,60-3,96ghz.

I mean, come on, if you are worried about the impact on your performance if your memory runs at 1066mhz instead of 1333mhz.. THINK about the positive impact on your performance if you get your ram running at 1400, 1600 or say 1800 mhz? !! :D Also, man, get those cpus overclocked! They will rock your processing speeds. 

I haven't completely ruled out overclocking, since I've done it successfully with a home computer (Q6660) that runs pretty much 24/7, and hasn't crashed in about 2 years (literally no crashes, not a single bluescreen, I only reboot for updates).

Speaking of overclocking, the 2.66 GHz X5650 model with its 20x multiplier is constantly mentioned as horrible for overclocking due to two reasons: One, the multiplier being too low (20x), meaning you have to crank the bus too much to reach high speeds, which stresses the whole system. Two, Intel processors have a history of unstable overclocking if they have an even multiplier. Odd multipliers such as 21 and 23 will ALWAYS overclock better. Therefore, the 2.80 GHz X5660 is a stabler overclocking choice with its 21x multiplier. This is just what I've heard from a lot of overclocking forums discussing the new Westmere Xeons.

This is an interesting topic. I'll probably end up with 2x5650 and ZERO overclocking, just for stability, or 2x5660 and mild overclocking. Stability is the main priority, I can't afford to lose a project due to an overclocking related crash. Although I'm open to mild overclocking, which is usually very stable.

Update: All references to returning 6 of the 12 sticks in this post are now inaccurate, as explained in post #1, since my RAM sticks were single ranked it turns out I actually NEED two per channel AND overclocking in order to reach the full 1333 MHz bandwidth. All explained in post #1.

xeonite
Yeah i agree, the sr-2 certainly a lot more features. And yeah, who cares about other slots than PCIe these days... :)

Exactly. I look forward to Intel realizing this and putting twice the number of PCI express lanes in each CPU, so that they can have fully decked out PCI express slots without needing NF200s. :-) 

xeonite 

You could aim for a very mild overclock, say 166 BCLK. That will give you the same 1333mhz speed on your ram, even if you should decide to use all 12 dimm slots.

Hmm, that's not a bad idea (keeping the RAM and using overclocking to get it back to speed). Although 12 GB is already about 3 GB more than the highest capacity I'll ever need in either design or audio production, and the value of the 6 sticks I'm returning will pay for the difference in going from 2xX5650 to 2xX5660 and leave some over. Quite a good tradeoff, for RAM I wasn't going to need anyway. People are saying that it's REALLY worth it to go for odd-multiplier CPUs, since even-multiplier CPUs consistently seem to require more voltage to overclock (this is just what I've read dozens of times from overclockers discussing the Westmere Xeons, so I'm suspecting there's some truth to it).

I REALLY like the idea of a mild 166 BCLK overclock though, that's excellent, it would take the 21x Multiplier X5660 from 2.80 GHz per core (33.6 GHz for all 12 cores) to 3.486 GHz per core (41.832 GHz for all 12 cores), or a 25% number crunching gain, and should be stable. Now THAT is why the SR-2 is so nice, unlocked bus clock for a dual socket motherboard. :-)

xeonite 

Yes it is a very interesting topic :D Will be VERY interesting to see how well the different westmeres overclock (and on the sr-2). There is thread with westmere overclocking experiences on xtremesystems. Lots of validations from x5650s doing 4-4.5ghz on both good air and water cooled. So im not really disheartened regarding their overclocking potential, at least not yet. But will be interesting to see more experience with them. An trying them out oneself on the sr-2. GODDAMN thats a nice looking motherboard. Couldnt help saying it!

That's better results than I've been reading for the X5650. Most people seemed hung up on the multiplier being 20x, and requiring more voltage to overclock than an X5660. They said this "multiplier bug" (their word) is also on the Core i7 series. I really wish I had full details about this. It may just be overclocker's superstition or hearsay. Not to be rude but a lot of them are stupid little kids that will repeat any rumor they hear as if it was fact. I haven't been able to find anyone explaining exactly why even-multiplier CPUs would be bad, they just keep claiming that they require higher voltages for stability than odd-multiplier CPUs (click for typical Google result with loads of pages talking about how even multipliers don't overclock well, and require higher voltages, but not a single person has explained EXACTLY why that would be case, so it could be a case of overclocker's hearsay as mentioned).

Anyway, nice chat man, thanks for all the suggestions.

I'm returning 6 of the sticks as originally planned, using that money to go from 2xX5650 to 2xX5660 (mainly to be safe if there's truth to that "multiplier bug"*) and 12GB of RAM, and doing that 166 BCLK overclock to gain a little bit more performance than a stock-clocked X5680. Very nice of you to help me with these decisions. :-) Sounds like a plan. I'll probably lower the RAM multiplier to 8x to achieve 8x166 = 1328 MHz RAM, which should be safer since it lacks heatsinks and I think it would be quite warm if I let it run at 10x166 = 1660 MHz when it's made for 1333 MHz.

* There's one more reason to consider X5660 over X5650: The X5650, X5660, X5670 and X5680 (and possibly more) are all the exact same chip design, the only difference is in how stable they were during post-production testing. The most stably overclocking CPUs get binned as X5680, and the worst get binned as X5650. The fact that the X5650 is the "leftovers" of the production isn't exactly inspiring confidence in it. So not all X5650s will overclock equally well, it has nothing to do with model number, it's simply the fact that it's the lowest-performing tier of all the new 6-core Xeons, and all tiers have quality variations, but at least with the X5660 you aren't at the very bottom.

As for cooling, I'm definitely going to air-cool it (water cooling requires yearly changing of fluid, maintenance, and carries the risk of leaks), probably with the "Xigmatek S-1283V Dark Knight" cooler which has received great reviews due to its small design, excellent cooling (matching much larger coolers), and a fan that pushes a lot of air even at low RPMs. Sounds ideal. I build each computer with noise level in mind, since silence is important in a recording studio.


EDIT: Oh and I agree, goddamn this is a nice motherboard. ;-)

nullack
Excellent topic, good one mate :) Hey, lets see some real world scores of memory bandwidth in both cases if someone thats lucky enough to have the sr-2 already can do that please.

Hi. Having a high memory bandwidth isn't something you notice AT ALL in everyday use (browsers, email, games). You could get by fine with 800 MHz RAM and not notice a thing.

It's when you use applications that are EXTREMELY heavy on the RAM read/writes that you notice the benefit of faster RAM. It's hard to test exactly what the benefit is, but it basically works like this:

The faster your RAM operates (i.e. 1333 MHz vs 800 MHz), the faster your CPU can read/write the necessary data to it and carry on with the next step of processing, instead of having to wait for the RAM. In my example with audio production, you have chains of hundreds of plugins that EACH read and write a LOT of data, repeatedly, nonstop, in realtime. It's a very heavy task and benefits greatly from *fast* RAM (and of course fast CPUs since it's all very heavy mathematical work), leading to lower latencies for faster realtime monitoring.

The benefits depend entirely on the application. Basically any application that uses a lot of data over and over, reading and writing nonstop, will benefit from faster RAM speeds.

Also remember that you have TWELVE real, physical cores all reading/writing data at the same time (when using a heavily multithreaded application), and that is yet another reason why you want the fastest RAM you can get, so that the CPUs can all retrieve/store their data in RAM in a timely manner.

raja: I learn something new every day! Thank you so much for bringing this factor to my attention. I have done research and have some great information now and will update the original post.  Populating all DIMM slots per channel will *actually* allow the processor to queue up commands for TWO DIMMs at once instead of one, and this, depending on the RANK of the installed memory, COULD result in better access rates. It's all explained in post #1, in a hopefully clear manner.

I just emailed the vendor AGAIN and said to cancel my return request, I'll be keeping the memory (it was single rank memory). You'll see why in post #1.


Everything below this post (#14) will be relating to the new, updated post #1.

Thank you too, xeonite, for the discussion and help earlier. ;-)

Just some clarification:

I have 6x2gb 1600 DDR3 modules.
I have 2x5650 cpu's
What is the issue?
Putting more than 2x 2gb modules in the 6 slot memory banks is bad?
Can the cpus control 6gbs or all the red slots filled
This will be my first intel board.

Maybe someone could post a legend of the terminology.
channel=memory slot etc.
I'm sure I'm not the only AMD defector buying this board...Help us out.

Excellent thread BigBrother!

with 6 ram sticks you get 10.6 GB/s per stick so 6 X 10.6 =  63.6 GB/s total  

with 12 ram sticks you get 8.5 GB/s per stick so 12 X 8.5 =  102 GB/s total

with 18 ram sticks you get 6.4 GB/s per stick so 18 X 6.5 = 117 GB/s total

of course none of this matters becuse the intel xeon 5600 and 5500 series have a max memory band with of 32 GB/s  so you actully need to populate all the dimm slots to get the 64 GB/s total memory bandwith

The final nail in the coffin that nullack's source is incorrect:

Intel themselves:

http://webcache.googleuse...clnk&client=safari
(must use this Google Cache link since the original page requires an Intel developer login)


"DIMMS Populated Per Channel: 1
Speeds: 800, 1066, 1333

DIMMS Populated Per Channel: 2
Speeds: 800, 1066

DIMMS Populated Per Channel: 3
Speeds: 800"


Everything in the original post is therefore still correct and needs to be taken into account when choosing memory.

nullack
 Thanks for the clarification, but a shame the article I found is incorrect :)

Since I need ECC ram, it seems I need to take pot luck with the DRAM I buy - ECC registered DRAMS arent binned like overclock high performance non ECC ram. I cant buy it in matched pairs/triples either - the multi stick ECC "kits" that are sold are just kits, not actually binned by test speed.

No problem. Actually, I can help you out with finding a good triple-matched Registered ECC kit that's made to run at 1333 MHz. All my examples below relate to a single-ranked stick, which is the most common type. You'll see the differences for a dual-ranked stick in the first post (mainly that a dual-ranked stick can achieve maximum bandwidth at just 1 stick per channel without even needing overclocking).

As has been mentioned in the thread I'll be running this for a design and audio production workstation and need stability, so I too understand the value of Registered ECC for both error correction and stability of RAM communication. In case you don't know what each does, ECC protects against single-bit errors (where bits become flipped due to electrical interference, usually caused by radiation), and Registered means that there's a register (buffer) between the actual RAM modules and whatever is accessing them. This buffer adds a layer of stability to the data access and also lowers stress (for the computer, not you).

I've bought 4 packs of triple-matched memory (so 4x3 = 12 sticks in total). If I was to place 3 per channel (6 sticks), I would get 1333 MHz RAM speed at default bus clock (133 MHz) but since it's single-ranked memory I would only be able to use 90% of that bandwidth. If I put in all 12 sticks (still at default 133 MHz bus clock), the RAM multiplier would be lowered so that I get 1066 MHz memory, but thanks to the simultaneous queuing of commands to two DIMMs at once it will end up giving me 91% of the bandwidth of a 1333 MHz rate, even though each individual stick is now running at 1066 MHz.

So what's the solution to get full bandwidth? Well, it's mentioned in the first post. :-) Raise the base bus clock from 133 MHz to 166 MHz, thereby slightly overclocking the entire system (since everything derives from the bus clock). This is a very mild overclock that's going to work for *everyone*, isn't going to cause instability (unless you've got some seriously bum component in the computer that can't take it, but I doubt that), and will do the following:

Your CPU will gain 25% clock speed since it's based off a multiplier of the system bus. For instance, a 2.8 GHz X5660 uses a 21x multiplier (21x133 = 2793 MHz) and would become ~3.5 GHz (21x166), making it faster than a non-overclocked X5680 (which uses a 25x multiplier)! Now, consider that X5660 and X5680 are the exact same silicon, the only difference being how Intel classified them at the factory (they are sorted and marked based on overclocking stability and get a permanent multiplier etched into them, but are otherwise identical). You're getting X5680 performance, with IDENTICAL chips (remember), for FREE. The SR-2 is the first dual-Xeon server board that has serious overclocking features, you should use it since that 25% faster CPU will get more work done in less time. :-)

Not only that, your 1066 MHz-downclocked RAM (~8x133) suddenly becomes about 1328 (~8x166) MHz. Giving you 99% of the bandwidth of 1333 MHz communication (thanks to having two sticks per channel thereby giving the same effect as dual-ranked chips, and now running at ~1333 MHz again). Your RAM won't really be overclocked either, since it will result in an operating rate of ~1333 MHz, which is what it's built to withstand, so this is an overclock it can handle. ;-)

So to sum it up: You achieve 99% RAM bus bandwidth and gain 25% CPU performance, as well as a very stable system since 133 -> 166 is a very mild overclock (most people go for 180/190, but that's more likely to give instability).

Now, the sticks I'm about to show you may seem expensive if you look at the total cost, but consider that they are triple-packs of sticks, and if you break down cost per gigabyte they are actually very well priced as far as $ per GB of RAM. Reg-ECC 1333 MHz RAM is just expensive, that's the way it is.

The sticks are:

Crucial DDR3 1333MHz 6GB KIT REG/ECC
Kit w/3x DDR3 2GB, 240pin, CL9, 1.5V, 256x72, Single Ranked
http://www.crucial.com/store/partspecs.aspx?imodule=CT3KIT25672BB1339S  

Don't be shocked by the CL9, that's typical for DDR3, they are much slower at access than DDR2 but make up for this by running at higher rates, which gives you more bandwidth in the end.

This stick also has siblings with lower (3x1 GB) and higher (3x4 GB) capacity per stick:

3x1 GB Single Ranked http://www.crucial.com/st...ule=CT3KIT12872BB1339S
3x4 GB DUAL RANKED http://www.crucial.com/st...dule=CT3KIT51272BB1339

Since the last stick is dual ranked, you can get full bandwidth without overclocking with just 6 of that one, since each stick basically contains two RAM modules in one, that's what dual ranked means (it means two separate "banks" of chips on a single DIMM stick, but the CPU can only access one bank at a time; the end result is a lot like having two single-ranked sticks installed which is why it gives higher performance in a single-DIMM-per-channel installation, but single-ranked is better if you have multiple sticks per channel). 

Also, 6x dual-ranked sticks are going to be much worse if overclocking, since you won't get the overclocking benefit of a lowered RAM multiplier (multiple sticks = lower multiplier = you can raise base bus clock higher without making the RAM go out of its proper operating rate). If you plan on doing the 25% overclock, get 12 single-ranked Reg-ECC sticks. If you don't want overclocking (why wouldn't you? it's stable at low overclocks... ;)), get 6 dual-ranked Reg-ECC sticks.

Lastly a note to anyone reading this: Don't even look at Quad-ranked, they run so slowly that they have to be run at 800 MHz at normal bus clock.

I hope this extensive answer helped. :-)

Yeah I know that the triple pairs aren't speed-matched in Corsair's case (which is something you'd want when overclocking to ensure they all overclock equally well), but the entire example still works out because they are single ranked meaning that once you've installed all 12, you get a lower RAM multiplier which in turn lets you overclock the CPU easily, and bring the RAM back up to ~1333 MHz.

It still runs within spec, that's the beauty of it, you don't need speed-matched sticks. ;-) I don't think you should take Registered ECC sticks above their specification, they may become unstable, or at the very least that buffer would run extremely hot.  

By the way, the highest end (1333 MHz-capable) Westmere Xeons are locked to two different RAM multipliers, them being 2:8 and 2:10.


You posted a good question in a PM to me, I'll quote it here:

nullack 
thanks mate awesome 

cant wait for my sr-2 :) 

would be good to know what memory spec the lower end 32nm 4 cores run like the e5620/30/40 - others might find this useful too as the six core 32nm 56XXs cost a fair bit more and with two physical slots its all good anyway 

cheers 

 
The PDF linked in post #23 contains a chart on page 5 detailing the memory bandwidth of each Westmere Xeon, even the quad core models. :-) The E5620, E5630, and E5640 all run RAM at 1066 MHz with a single stick installed, rather than 1333 MHz. They're also crippled in other ways such as inter-CPU communication (5.86 GT/s vs 6.4 GT/s). The upside is that they cost less and cost less to run (lower wattage).


:-)

Brb, I need to get some taco... that's not a euphemism.

Those were some amazing tacos. I feel like there's a party in my mouth and nobody else was invited. I was dancing with myself, playing with myself, good times were had at the party, but I digest...

nullack
Cool.

Haven't seen any reviews comparing 5.86 vs 6.4 gt on 2 cpu systems. Especially establishing if the interlink can be in anyway a bottleneck with only 4 cores per slot as the 6 core 56XXs have the higher 6.4 but they have 6 per slot. Anyone?? So this may not be an issue being only just 5.86 for 4 core cpus.

With the lower mem clock of the E series 56xx over the X series - since the sr-2 allows for overclocking that can be overcome.....what I'm getting at is that for those where 8 physical and 16 logical cores are enough, can the E series be the ideal price/performance point? I can buy two E5630's for the price of only the slowest 6 core.....and it's a safe bet which one would win even without overclocking in multi threaded apps like video encoding.

Very good point; putting it like that, we have 1/3 less cores per processor but retain ~92% of the QPI speed. It should mean that the link speed is enough for four cores. It's worth noting that the processors also use the link when they need to access the RAM that belongs to the other processor, so it's not as simple as that, but it looks like a good QPI speed.

Likewise, as you say, the lower memory clock of the E series can be overcome when you raise the bus speed. At 3 DIMMs per processor @ 1066 MHz, raising the bus from 133 to 166 would give you the same 1333 MHz RAM. You should try to get dual-ranked 1333 MHz RAM if you are going for 3 DIMMs per processor (6 in total), for maximum bandwidth.

I don't know of any benchmarks comparing the low-watt E-series against the W-series, but check around Google and report back if you find anything. However, considering that the link speed looks good for the lower number of cores, the RAM being overclockable, and the superior energy efficiency of the E-series (meaning it'll cost less to use that computer 24/7), it definitely looks like the best price/performance choice.

Another option is to buy 1x X5650 and 6 single-ranked sticks of RAM to fully populate one side of the motherboard; then as soon as you can afford another processor you set up the RAM so that you have 3 sticks per processor, and finally you get another 6 sticks when you can afford that. It's a plan worth thinking about if it turns out that the E-series is much slower.