I am seeing some odd behaviour with linux.
the settings I am looking at are the following sysctl's.
net.ipv4.tcp_rmem (controls RWIN size, 3 values, first is min used when low on memory, 2nd is default used on initial connection and 3rd is what autouning can grow the RWIN size to if required for full throughput)
net.ipv4.tcp_moderate_rcvbuf (enables/disables autotuning, presumably when this is disabled the above settings should only ever use the default value)
net.ipv4.tcp_sack (enables/disables selective acks)
now if I increase the default RWIN to a value above 65536 so window scaling is used and SACK's are disabled then performance plummets, its as if window scaling and SACK is tied together, the same also happens when sending from the server to a server that has a RWIN above 65535. If window scaling is enabled and SACK is enabled there is no problem. Given that SACK is typically most useful when packet loss is occuring does this suggest routers en route for the server are not supporting scaling properly and SACK is simply working round the problem?
the oddity I get with net.ipv4.tcp_moderate_rcvbuf is that (a) it has to be turned off whilst SACK is off to stop performance plumetting, this sort of makes sense as I have left the max RWIN size in the autotuning value as above 65536 so with it off it should be capped at 65536. But what doesnt make sense is if I use the analyzer on this site it shows a very small RWIN value which suggests some sort of autotuning is in affect. Reports a RWIN size of 6144 when 65536 is configured on the server without autotuning.
any comments, thanks.