Thursday, June 23, 2011

Chechpoint Firewall / Smartcenter server - Error "Out of Memory: Killed process ()"


These messages may appear in one of the following locations:

* Console

* SmartView Tracker

* messages or dmesg file

If and when encountered, please contact Check Point Support.



"Out of Memory: Killed process ()"


This message appears in the dmesg file and in /var/log/messages files on SecurePlatform.

It means that no more memory is available to Linux in the user space. As a result, Linux starts to kill processes.

"FW-1: Capacity problem detected

Memory consumption has exceeded X%" (Console message)

"Capacity notification: Memory consumption has exceeded X%" (SmartView Tracker message)

These messages indicate that the memory consumption has increased beyond what was defined as the Aggressive Aging threshold (for more details about Aggressive Aging, refer to 'NGX R65 What's New' document, under 'Firewall & SmartDefense', page 2).


"FW-1: Capacity problem detected"

"Connections table capacity has exceeded X%" (Console message)

"Connections table capacity has exceeded X%" (SmartView Tracker message)

These messages indicate that the Connections table capacity has increased beyond what was defined as the Aggressive Aging threshold.

For more details about Aggressive Aging, refer to 'NGX R65 What's New' document, under 'Firewall & SmartDefense', page 2.

"Main database file <(database file name)> is missing - cannot start fwm. If you wish to reset the DB please run 'cpdb new'." (Console message)

The appearance of this error message might indicate a corruption in $FWDIR/conf/objects_5_0.C. As a result, fwm will not start.

"State synchronization is in risk. Please examine your synchronization network to avoid further problems" (in /var/log/messages file)

This message may indicate that the sync network is overloaded. Overloading the sync network can cause traffic loss, unsynchronized kernel tables, and connectivity problems. For more information, refer to sk23695.



"fwlddist_adjust_buf: record too big for sync" (in /var/log/messages file and on the Console)

This message may indicate problems with the sync network. It can cause traffic loss, unsynchronized kernel tables, and connectivity problems. For more information, refer to sk35466.

"Cluster_info: (ClusterXL) member is down" (in SmartView Tracker)

This log message may indicate that the ClusterXL failed over. You can check the member's status and the failed device using the commands cphaprob stat, cphaprob list and cphaprob -a if. For more information regarding the usage of these commands, refer to CheckPoint NGX ClusterXL User Guide.pdf.

If this failover was not initiated on purpose, please contact Check Point Technical Support (as described above).

"Dead loop on virtual device sync, fix it urgently" (in dmesg file)

This message is a SecureXL notification on the outbound connection that may cause the gateway to lose sync traffic. For more information, refer to sk32765.



"FW-1: bpush: push block size error..." (Console message)

For more information, refer to sk32753 , sk59124.


"FW-1: fw_runfilter: illegal kfunc" (Console message)

These message may indicate system instability and should not be ignored.



"FW-1: fw_runfilter: stack overflow" (Console message)


This message may indicate that the number of rules in the firewall has exceeded its limit. If more rules are required, please contact Check Point Technical Support (as described above).

"FW-1: fw_runfilter: stack underflow" (Console message)

This message may indicate memory corruption problems detected by the system. System stability may be impacted.

"FW-1: fw_runfilter: wrong number of arguments..." (Console message)

This message may occur after an unsuccessful upgrade and could precede a system panic.

"FWD Error: Log(s) discarded due to unification process failure" (in SmartView Tracker)

A single "unified" log record is produced by the FireWall-1 kernel driver from a number of "basic" log records. If for some reason the building process (unification process) fails, there is a log discard followed by this error message.

This means that logs are discarded from the system and therefore will be lost.



"Database space check failed. There may not be enough disk space or it may have failed to obtain database capacity information" (in Eventia Reporter's $RTDIR/opt/CPrt-R65/log_consolidator_engine/log/lc_rt.log file).

This message means that the process is not communicating with the mysql process or there might be a problem identifying the disk. The logs will be consolidated once the problem is solved. Please check the disk capacity. Check Point also recommends checking disk sanity (for example, by using the check disk utilities).

"FW-1: panic <(x)>: " (Console message)

<(x)> represents the level of panic induced. represents the message that is associated in the code with this panic.

This error message indicates that fw_panic was called.

"FW-1: fw_kfree: memory already freed at 0x. caller is sz=" (Console message)

represents the pointer. represents the name of the function where the error happened.
This error message indicates that released memory is trying to be released again.

This is forbidden and might lead to a potential panic event.

"FW-1: fw_kfree: wrong magic number at tail end of 0x (0x) caller is sz=" (Console message)

"FW-1: fw_kfree: wrong magic number at 0x. caller is sz=" (Console message)

represents the pointer. represents the name of the function where the error happened.
These error messages might indicate an error in the way memory was handled, which might lead to a potential panic event.



"FW-1: hmem_init: unable to allocate the minimum <(x)>" (Console message)

represents the minimum memory size that is needed.

This error message indicates that the firewall's memory management module is not loaded.



"FW-1: b_create: fw_kmalloc(x) failed" (Console message)

represents the pointer.

This message may indicate an error in memory allocation for binary tables. System stability may be impacted.

"ex_init_timer: Failed to initialize timer" (Console message)

This message might indicate an error in the timer mechanism initialization. This may impact system infrastructures and cause inconsistent behavior.

"FW-1: b_create: fw_kmalloc(x) failed" (Console message)

This message might indicate a critical error in allocating a table. System stability may be impacted.

"FW-1: b_create: fw_kmalloc(x) failed" (Console message)

This message may indicate memory allocation problems. System stability may be impacted.

"fwconn_get_bits: invalid bit category: (x)" (Console message)

This error may indicate a critical error in reading the connections table. It may have a serious impact on connectivity.

"fwconn_set_bits: failed to get bit value for bit category (x)" (Console message)

This error may indicate a critical error in writing to the connections table. It may have a serious impact on connectivity.

"fwconn_chain_fill_bits: invalid bit category: (x)" (Console message)

This error might indicate a memory corruption.



"FW-1: : data connection "FWCONN6_FMT" already exists in connections table" (Console message)

This error might indicate synchronization problems between clusters.



"FW-1: : failed to get info from %s table" (Console message)

This error might indicate memory management problems.

"FW-1: : fwconn_chain_lookup failed" (Console message)

This error might indicate an inconsistency within the connection table. This may lead to connectivity problems.

"FW-1: illegal access to connections table" (Console message)

This error might indicate connectivity problems.



"FW-1: : Cannot change aggressive timeout without setting the timeout (timeout=, aggr_timeout=)" (Console message)

This error might indicate a program error that might cause Aggressive Aging not to work properly.

"Failed to build the objects schema while initializing database manager with error 0x% ('')" (Console message)

represents internal value of a pointer.

This message can indicate that there is a problem with the schema file that could prevent the fwm process from starting.

fwm not running on the system may lead to problems with connecting with the GUI client and installing the policy.

"fwhandle_get(vpn_tag.c:1275): Table kbufs - Invalid handle f5139a5c (bad pool)" (Console message)

This error message might indicate an error in the way memory was handled. It may cause RemoteAccess Connectivity issues.

"Policy install commit function was unsuccessful due to timeout" (Console message)

This error message appears during security policy installation. It indicates that policy installation on the gateway takes too much time. There is no way to know whether policy was successfully installed or not.

"Failed to get password for connection RT_Database, reason: Could not find path to database socket" (Console message)

This error means that the location of the mysql.sock file is missing. It will prevent the Eventia Reporter from connecting to the database. This problem can be fixed by doing the following:

1. Open the file $RTDIR/Database/conf/my.cnf and copy the directory that the "datadir" is referring to.

2. Run rmdstop (make sure that the database processes are down).

3. Run cd $RTDIR/Database/ .

4. Run the following commands:

bin/mysqld_safe --basedir="$RTDIR/Database"

--ledir="$RTDIR/Database/bin"

--datadir=""

--socket="$RTDIR/Database/mysql.sock" --user=root --log-error=$RTDIR/Database/err.log &

If mysql is not starting up, check the file $RTDIR/Database/err.log for errors.

Note that mysqld_safe and my_print_defaults are not part of the installation. You can download them (in WinZip format) here:

* mysqld_safe
* my_print_defaults


No comments: