Introduction

 
Every time an issue/bug is reported at a customer's place on a staging or production environment, the numero uno question asked from any developer or supporting guy is the application log. In Access Gateway(AG), managing logs is done using Advanced options. From the AG documentation severity of logs are configured using option:

LogLevel <loglevel>
Possible loglevel values are emerg, alert, crit, error, warn, notice, info or debug

The most favored loglevel for most of the developers is "debug", but the recommended or default loglevel is "warn" for the obvious reason of negative impact on application performance. AG's logging in debug mode not only increases the size of the log files quickly, in a highly loaded system, it can also lead to request or connection timeout and can slow down the response time.

The biggest drawback of setting AG's advanced option is the automatic restart of the AG process/services.

The Solution

 
This cool solution will allow the user to change the severity level of logs on the fly, without restarting AG. This will help developers to debug an intermittent issue by enabling the debug level of logs only when needed without restarting Access Gateway process/services and avoid adverse impact of setting debug log level all the time. It would also help in the situation where the issue disappears as soon as services are restarted.

Steps to follow:

 
On an SLES/RHEL environment, Access Gateway sends logs to syslog. This cool solution takes help of syslogs's filter mechanism. Depending on which syslog (syslog-ng or rsyslog) is being used, the steps will differ. Access Gateway on SLES-11 uses syslog-ng whereas on SLES-12 & RHEL it is rsyslog.

First and foremost, make sure the AG advance option for logging is set to:

LogLevel debug

For syslog-ng

1. 1. Login to AG system

1. 1. Open syslog conf file
      

      vim /etc/syslog-ng/syslog-ng.conf

1. 1. Comment out all occurrences of the following line
      

      #log { source(src); filter(f_user); destination(agsmessages); flags(final); };

      
      
      

1. 1. Go to MAG section update as follows. Newly added lines are between #cool-solution-start & #cool-solution-end. The rest are existing lines.
      

      #MAG
      filter f_user { facility(user); };
      destination agsmessages { file("/var/log/novell-apache2/error_log"); };
      #log { source(src); filter(f_user); destination(agsmessages); flags(final); };
      #cool-solution-start
      filter f_user_defined { level(warn, error, crit, alert, emerg) and facility(user); }; 
      log { source(src); filter(f_user_defined); destination(agsmessages); flags(final); };
      #cool-solution-end
      
      filter f_local6 { facility(local6); };
      destination httpheaders { file("/var/log/novell-apache2/httpheaders"); };
      log { source(src); filter(f_local6); destination(httpheaders); flags(final); };
      
      filter f_local5 { facility(local5); };
      destination soapmessages{ file("/var/log/novell-apache2/soapmessages"); };
      log { source(src); filter(f_local5); destination(soapmessages); flags(final); };
      
      #
      # Enable this and adopt IP to send log messages to a log server.

1. 1.  The above setting is the same as the default setting where severity level is set to warn and below. Anytime user is looking for logs with better verbosity,  replace the corresponding filter line (between #cool-solution-start & #cool-solution-end) with any one of the following:
      
      

      filter f_user_defined { level(notice, warn, error, crit, alert, emerg) and facility(user) };
      filter f_user_defined { level(info, notice, warn, error, crit, alert, emerg) and facility(user) };
      filter f_user_defined { level(debug, info, notice, warn, error, crit, alert, emerg) and facility(user) };

      
      
      

1. 1. Similarly, the opposite is also possible when the user wants to reduce the verbosity of logs. Replace the line with lesser log level. For e.g.
      

      filter f_user_defined { level(crit, alert, emerg) and facility(user) };

1. 1. Restart syslog service
      

       service syslog restart

      
      The above steps allow the user to change the severity of logs on the run without restarting AG processes and services.

Another useful filter that could help to block redundant logs based on string matching is:

filter f_user_defined { level(debug, info, notice, warn, error, crit, alert, emerg) and facility(user) and not match('CFilter'); };

This filter does not allow logs containing 'CFilter' string.

For rsyslog

1. 1. Login to AG box

1. 1. Open rsyslog's nam specific conf file
      

      vim /etc/rsyslog.d/nam.conf

1. 1. Comment out 2 lines as follows:
      

      local0.* @@164.99.162.16:1290;ForwardFormat
      & ~
      #user.* -/var/log/novell-apache2/error_log
      #& ~
      local6.* -/var/log/novell-apache2/httpheaders

1. 1. Open rsyslog conf file
      

      vim /etc/rsyslog.conf

1. 1. Search for following line
      

      if $syslogfacility-text == 'user' then /var/log/novell-apache2/error_log

      
      and replaced with
      

      if (\
       ($syslogfacility-text == 'user') and \
       ($syslogseverity <= 4) \
      ) \
      then /var/log/novell-apache2/error_log

      
      
      Here is the severity code for each log level.
      
      

      emerg	0
      alert	1
      crit	2
      error	3
      warn	4
      notice	5
      info	6
      debug	7

      
      
      So by default $syslogseverity is set to <= 4 (warn). To enable maximum verbosity (debug), set it to $syslogseverity <=7

1. 1. Restart syslog service
      
      

      service syslog restart

      
      
      The above steps allow the user to jump between verbosity of AG logs on the fly without impacting AG service/process.

Similar to syslog-ng, to block logs based on string matching, the following example can be taken for reference. To block logs having string "a.com:8181/IDPUpdate" use:

if (($syslogfacility-text == 'user') and ($syslogseverity <= 4) and not ($msg contains 'a.com:8181/IDPUpdate')) then /var/log/novell-apache2/error_log

Please note that searching logs for string would have some impact on performance. More filter options can be found at the syslog documentation. Please remember to restart syslog service every time its configuration is changed.

The syslog configuration guide describes many more rich sets of filter options. Please refer to the official documentation for more detailed info.

Reference

1. 1. syslog-ng documentation https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide

1. 1. rsyslog documentation https://www.rsyslog.com/doc/v8-stable/configuration/index.html