Each message family contains multiple different types of logs. For example, there could be multiple types of logs in intrusion message family. Typically, each signature in IPS is one type of log. In signature based IPS, there are as many logs as number of signatures. Each type of log message is typically represented by message IDs.
UTM devices generate logs during its operation. There could be large number of incidents which generate similar type of log multiple times. Messaging system component of UTM devices provide multiple controls for administrator to control logs. Some of the controls are:
- Message ID level control on enable/disable: If a message ID is disabled, any log incidents of that message ID are not processed, that is, they are not stored or exported to external log collectors.
- Message ID based Log frequency control: It allows logs, but controls the number of logs generated for processing. Typically it takes two parameters - Log threshold count and log threshold time. At most one log is processed for every 'log threshold count' logs or within the 'log threshold time'. That is, if log threshold count is 100 and log threshold time is 5 minutes, then if number of logs generated are more than 200, but less than 300 within 5 minutes, then it emits three logs - 1st log, 101st log and 201st log.
- 5 Tuple based Log frequency control: Message ID based log frequency control is good for non-network based logs such as system errors, warnings etc.. But, for connections, this kind of control is not good enough. If there is same intrusion detected in traffic going to multiple victims, then administrator would like to know at least one instance of intrusion going to each victim. With message ID based log frequency control, all victims will not be reported if the intrusions happen within 'log time threshold'. To avoid this, 5 tuple based log frequency control is required. 'Source IP', 'Destination IP' , 'Protocol', 'Source Port' and 'Destination Port' constitute 5 tuple. Each item in tuple can be enabled/disabled. 'Log threshold count' and 'log threshold time' described above is individually valid for each combination of 5-tuple items enabled. Let us say that, if administrator enabled 'source IP' and 'destination IP' addresses in the 5 tuple for a given message ID, then for logs generated for this combination of network connections are processed as per 'log threshold count' and 'log threshold time'. Using this, log systems don't miss attacks and other events happening on individual victim machines and also don't miss reporting 'attackers'.
Though many aspects of this article is based on Intoto iGateway product family concepts, these are generic concepts and valid for any security products. The typical flow of logs is:
- Logs are generated by applications such as firewall, IPS, AV/AS, WAF etc..
- Log throttling system discards logs and processes only some logs based on controls configured by administrator.
- Logs are then stored, exported.
- Log analyzers (local or external) analyzes logs and even create reports. Also, they provide 'search' functions based on different criteria.
Exported logs are sent in a format for log analyzers to easily extract values for different fields. WELF is one format that became quite popular. Though iGateway product family uses WELF syntax, but it does not use keywords as specified by WELF as they are incomplete. Some of the rules it follows are:
- Each log forms one line with each line containing multiple fields.
- Each field is formed as keyword
=value . Keywords are defined by corresponding message family. Each keyword and value pair is separated by one or more spaces. keyword and values should not have any spaces. They must not have any quotes and '=' sign characters. If the value needs to have spaces, then the value string must be enclosed in double quotes.
- Mandatory keywords across all message families
- time: Date and time in double quotes.
- priority: Priority of the message. One of values from 1 to 7.
- id : Identity of the device sending the logs. It is configured by administrator on each device.
- mtype: Message family.
- mid: Message ID
- Generic keywords: There are some generic keywords which are valid across multiple message families. Though these are not mandatory, these generic keywords must be used wherever they are needed.
- vsg : Virtual instance.
- fromzone: Zone in which the connection is originated.
- tozone: Zone in which the connection is terminated.
- userid : User name, if this information is known.
- usergroup: User group.
- sip : Source IP address in dotted decimal form.
- dip: Destination IP address in dotted decimal form.
- protocol: Protocol of the connection as integer.
- sport: Source port
- dport: Destination port.
- Other keywords are specific to each message family.