(Note: I've actually got one of these cards, and as far as I can tell it seems to work pretty well. I just don't like crappy studies and misleading data.)
Eh, correct me if I'm wrong, but I'm fairly certain that isn't a performance improvement. The lag compensation/prediction system used in most games relies on a stable, but not necessarily low ping; a 10ms drop in ping time, even if it's an average 10ms (not like the CS graph up top), is utterly useless if the standard deviation (and I guess by extension, the ping jitter) is significantly greater than other network cards.
I ran some quick numbers for the TF2 and CS:S data in the white paper, which is something that I really shouldn't have had to do; the paper presents itself as a scientific study, and utterly fails in that regard. The methods and test beds sections are alright, but they give no indication of what procedure they used, which should have included some sort of information about the sample rate (What unit is used along the bottom of the graph? Minutes? Seconds? Inches? Obviously some unit of time, but still), and when they started taking said samples. It looks to me like they started taking samples as soon as the game entered the server, which makes sense given the spike at the beginning of each graph (first 2 units, in all cases). This could be construed as either bias, poor testing practices, etc, as in all cases shown, the greatest difference between the control and the Xeno (and from what I can tell by eyeballing, is enough to skew the results as an outlier in the CS:S no load, TF2 no load, and L4D under load tests.)
Some data, just because I ran it through the calculator and didn't feel like throwing it out.
Counter-Strike : Source
CONTROL
mean = 30
std.dev. = 1.69967
XENO
mean = 25.6
std.dev. = 2.87518
Just under a 70% (69.16%) increase in ping range/variability on the Xeno in exchange for a 4.6ms drop in average ping. That's really not that great of a deal.
TF2
CONTROL
mean = 43
std.dev = 2.08
XENO
mean =42.5
std.dev. = 1.71
1.011% ping improvement (interesting that the Peddie study reports 1.2%... either they have more data to play with, or they're doing some creative rounding there), and a 21.6% decrease in ping range/variability for the Xeno. These results are a bit better, but as I mentioned earlier, they don't say when they started sampling, and if my suspicions are correct, the large spike in the first 2 units is what's responsible for the difference.
Conclusions and comments...
I don't like misleading, badly done studies that appear to be biased in favor of the products under review and lack crucial bits of information that make large differences in the experimental outcomes. After reading carefully through the study and thinking a bit, I'm actually considering ditching the Xeno in favor of the onboard nic, as the ping stability on a standard card appears to be better than that of the Xeno Pro, and from what (admittedly somewhat limited) understanding I have of lag compensation and prediction, improved ping stability/consistency (less jitter, lower standard deviation) is far more important than marginally decreased latency (the prediction system relies on a stable ping in order to... predict properly. Wildly fluctuating ping is much, much worse than a stable ping even 20-30ms higher). I could really care less about how it performs if I'm downloading torrents or large files in the background, because to me that's just bizarre in a "why on earth would I want to do that?" sort of way, so I'll leave that alone (on the bright side, except for the horribly skewed L4D, that part of the card appears to work quite well).
Takeaway Points:
- Lack of a detailed testing procedure calls into question the results, especially given the consistent skew caused by the first 2 unknown units on the graph. My guess is they fired up the game and start taking measurements as soon as the level loaded. The Xeno may have an edge for that period of time, but it's misleading, and not a real world situation. (I'm assuming that the Windows network stack delays a bit before giving the game network priority or something to that effect; something the Xeno doesn't have to deal with as it's already giving a higher priority to UDP packets.)
- Consistent pings are more important, especially for hit registration in FPS games, than are low pings. This should be examined a bit more by the guys over at Bigfoot Networks. The variability on a large scale also makes me wonder what it's like on shorter time scales, say millisecond/sub-millisecond range, which is really what all of this network stack offloading and such is supposed to be about. (Aside: Fire up Quake Live and take a look at the netgraph output. It flickers back and forth between a lower and higher number (around 10ms apart) so quickly that it starts to blur @ 120hz. That does not bode well.)
- Skewed studies are good for no one. Even if the big letters on the graphs and simplified bits make the Xeno look better, once somebody takes a closer look and sees that (a) it looks like someone paid for the results (note results, not study. funding a study and paying for results are 2 entirely different things.) and (b) the results actually make the card look bad (is it? I'm not sure anymore, and quite obviously can't rely on feel, as one's perceptions are easily biased after spending 100$ on a network card). Instead fund high-quality research (either in-house or outsourced) that, in the event the card actually makes things worse, will tell you what happened, and give you a starting point for figuring out the why part. (In fact, you're more than welcome to hire me. Seriously.)
- I'd love to hear some feedback from the developers/pr guys/whatever, but if I hear, anywhere in the response that "the numbers don't show it, you have to rely on how it feels when you play," I'm going to have to assume that the only reasonable explanation is that the study was a joke, the card does very little, and the company (Bigfoot most likely, but you never know) is trying to hide it. (Which, by the way, directly contradicts this whole thread and study thing. I'd love to quote the times I've seen it said on the forums, but everything seems to have disappeared...)
-Matt