Validity of performace metrics?

Many security appliance vendors and ADC vendors provide performance metrics with respect to throughput and connection rate. But if you look deeply, none of these metrics are valid for actual deployments. At most, they are useful for developers to identify the bottlenecks in the software.

If you look at closely, throughput numbers are given with UDP traffic directed to some arbitrary port. Connection rate is given with respect to number of TCP connection establishments (3 way handshake) and termination by TCP RST packet. None of these are real deployment scenarios. How much of traffic is really UDP in real world? How many connections are really TCP with just connection establishment? How often the connections are terminated by TCP RST packets. So, the numbers being given by device vendors are useful for themselves to figure out the device or software bottlenecks.  But these are not sufficient to make end users to relate to their networks and traffic.

Many security devices and ADCs have application protocol capabilities.

• Firewall devices have ALGs (Application Level gateways) for each complex protocol to open holes in the firewall for dynamic data connections. Examples: FTP, SIP, MGCP, H.323, RTSP, DNS, L2TP, PPTP etc..
  
• IPS devices have application intelligent protocols. Similar examples as firewall.
• Anti Virus functionality typically implements proxies to collect the data for scanning. Example proxies: HTTP, HTTPS, FTP, SMTP, PoP3, IMAP etc..
• URL content filtering feature not only interprets HTTP protocol data, but also does quite a bit of content matching (regular expression search).
  
• ADC devices also contain application protocols such as HTTP, HTTPS, FTP, SIP, RTSP etc..

It is fair to expect the throughput and connection rate for all application protocols which are supported by the device.

Connection rate should not only involve TCP connection establishment, but also should have the data that is normally interpreted by the device. In addition, the size of data is reasonable. It is true that the size of data can be different in different deployments, but using some reasonable size for data during connection establishment is required.

Throughput numbers are also expected to be given with respect to application protocols. Whatever size of the data is chosen should be clearly documented. My suggestion to device vendors to give throughput numbers for different sizes of protocol data. Another aspect  is number of connections used in throughput measurements. I observed that device vendors use very less number of connections with large amount of data to get the best numbers. They do give good numbers as most of the connection state information resides in the cache. Any medium sized deployment would have more than few thousand of connections going through simultaneously.  So,  it would be good if device vendors give throughput numbers with large number of sessions.

So, don't go by marketing numbers. Ask for more performance data with respect to applications, number of flows etc.. You would be surprised to see the difference in results. Really..