troubleshooting problems — rsyslog 8.18.0.master documentation (2024)

Malformed Messages and Message Properties¶

A common trouble source are ill-formed syslogmessages, which lead to to all sorts ofinteresting problems, including malformed hostnames and dates. Read thequoted guide to find relief. A common symptom is that the %HOSTNAME%property is used for generating dynafile names, but some glibberishshows up. This is caused by the malformed syslog messages, so be sure toread the guide on syslog parsingif you face that problem. Justlet me add that the common work-around is to use %FROMHOST% or%FROMHOST-IP% instead. These do not take the hostname from the message,but rather use the host that sent the message (taken from the socketlayer). Of course, this does not work over NAT or relay chains, wherethe only cure is to make sure senders emit well-formed messages.

Configuration Problems¶

Rsyslog has support forconfiguration checking. It offers a special command line switch (-N<value>)that puts it into “config verfication mode”. In that mode, it interpretsand checks the configuration file, but does not startup. This mode can beused in parallel to a running instance of rsyslogd.

The value is a set of binary values. Currently, there only is

Where 2 automatically turns on config checking mode, if not given. In thatsense -N2 and -N3 are equivalent.

When set to check include files, some conditions are relaxed. For example,rsyslog usually requires that at least one action is defined somewhere inthe configuration. For obvious reasons, it would not make much sense to runan instance without any action. However, when an include file is checked,it may happen that it contains no actions as all. As such, the requirementto include one action has been lifted in include file checking.

To check a full rsyslog configuration, run rsyslog interactively as follows:

$ /path/to/rsyslogd -f/path/to/config-file -N1

You should also specify other options you usually give.Any problems experienced are reported to stderr [aka“your screen” (if not redirected)].

If you would like to check just an include file, instead use:

$ /path/to/rsyslogd -f/path/to/config-file -N3

Asking for Help¶

If you can’t find the answer yourself, you should look at these placesfor community help.

• The rsyslog mailinglist. This is alow-volume list which occasional gets traffic spikes. The mailinglist is probably a good place for complex questions.This is the preferred method of obtaining support.
• The rsyslog forum.

Debug Log¶

If you ask for help, there are chances that we need to ask for anrsyslog debug log. The debug log is a detailled report of what rsyslogdoes during processing. As such, it may even be useful for your very owntroubleshooting. People have seen things inside their debug log thatenabled them to find problems they did not see before. So having a lookat the debug log, even before asking for help, may be useful.

Note that the debug log contains most of those things we consideruseful. This is a lot of information, but may still be too few. So itsometimes may happen that you will be asked to run a specific versionwhich has additional debug output. Also, we revise from time to timewhat is worth putting into the standard debug log. As such, log contentmay change from version to version. We do not guarantee any specificdebug log contents, so do not rely on that. The amount of debug loggingcan also be controlled via some environment options. Please seedebugging support for further details.

In general, it is advisable to run rsyslogd in the foreground to obtainthe log. To do so, make sure you know which options are usually usedwhen you start rsyslogd as a background daemon. Let’s assume “-c5” isthe only option used. Then, do the following:

• make sure rsyslogd as a daemon is stopped (verify with ps -ef|greprsyslogd)
• make sure you have a console session with root permissions
• run rsyslogd interactively: `/sbin/rsyslogd ..your options.. -dn >logfile`where “your options” is what you usually use. /sbin/rsyslogd is thefull path to the rsyslogd binary (location different depending ondistro). In our case, the command would be`/sbin/rsyslogd -c5 -dn > logfile`
• press ctrl-C when you have sufficient data (e.g. a device logged arecord)NOTE: rsyslogd will NOT stop automatically - you need to ctrl-c outof it!
• Once you have done all that, you can review logfile. It contains thedebug output.
• When you are done, make sure you re-enable (and start) the backgrounddaemon!

If you need to submit the logfile, you may want to check if it containsany passwords or other sensitive data. If it does, you can change it tosome consistent meaningless value. Do not delete the lines, asthis renders the debug log unusable (and makes Rainer quite angry forwasted time, aka significantly reduces the chance he will remainmotivated to look at your problem ;)). For the same reason, make surewhatever you change is change consistently. Really!

Debug log file can get quite large. Before submitting them, it is a goodidea to zip them. Rainer has handled files of around 1 to 2 GB. Ifyour’s is larger ask before submitting. Often, it is sufficient tosubmit the first 2,000 lines of the log file and around another 1,000around the area where you see a problem. Also, ask you can submit a filevia private mail. Private mail is usually a good way to go for largefiles or files with sensitive content. However, do NOT send anythingsensitive that you do not want the outside to be known. While Rainer sofar made effort no to leak any sensitive information, there is noguarantee that doesn’t happen. If you need a guarantee, you are probablya candidate for a commercial supportcontract. Free support comes without anyguarantees, include no guarantee on confidentiality [aka “we don’t wantto be sued for work were are not even paid for ;)]. So if you submitdebug logs, do so at your sole risk. By submitting them, you acceptthis policy.

Segmentation Faults¶

Rsyslog has a very rapid development process, complex capabilities andnow gradually gets more and more exposure. While we are happy aboutthis, it also has some bad effects: some deployment scenarios haveprobably never been tested and it may be impossible to test them for thedevelopment team because of resources needed. So while we try to avoidthis, you may see a serious problem during deployments in demanding,non-standard, environments (hopefully not with a stable version, butchances are good you’ll run into troubles with the developmentversions).

In order to aid the debugging process, it is useful to have debug symbolson the system. If you build rsyslog yourself, make sure that the -goption is included in CFLAGS. If you use packages, the debug symbols comein their own package. It is highly recommended to install that packageas it provides tremendous extra benefit. To do so, do:

yum install rsyslog-debuginfo

Obviously, this is for RPM-based systems, but it’s essentially the samewith other packaging systems, just use the native commands. Note thatthe package may be named slightly different, but it should always befairly easy to locate.

Active support from the user base is very important to help us trackdown those things. Most often, serious problems are the result of somememory misadressing. During development, we routinely use valgrind, avery well and capable memory debugger. This helps us to create prettyclean code. But valgrind can not detect everything, most importantly notcode pathes that are never executed. So of most use for us isinformation about aborts and abort locations.

Unfortunately, faults rooted in adressing errors typically show up onlylater, so the actual abort location is in an unrelated spot. To helptrack down the original spot, libc later than 5.4.23 offerssupportfor finding, and possible temporary relief from it, by means of theMALLOC_CHECK_ environment variable. Setting it to 2 is a usefultroubleshooting aid for us. It will make the program abort as soon asthe check routines detect anything suspicious (unfortunately, this maystill not be the root cause, but hopefully closer to it). Setting it to0 may even make some problems disappear (but it will NOT fix them!).With functionality comes cost, and so exporting MALLOC_CHECK_ withoutneed comes at a performance penalty. However, we strongly recommendadding this instrumentation to your test environment should you see anyserious problems. Chances are good it will help us interpret a dumpbetter, and thus be able to quicker craft a fix.

In order to get useful information, we need some backtrace of the abort.First, you need to make sure that a core file is created. Under Fedora,for example, that means you need to have an “ulimit -c unlimited” inplace.

Now let’s assume you got a core file (e.g. in /core.1234). So what to donext? Sending a core file to us is most often pointless - we need tohave the exact same system configuration in order to interpret itcorrectly. Obviously, chances are extremely slim for this to be. So wewould appreciate if you could extract the most important information.This is done as follows:

$ gdb /path/to/rsyslogd$ core /core.1234$ info thread$ thread apply all bt full$ q # quits gdb

Then please send all information that gdb spit out to the developmentteam. It is best to first ask on the forum or mailing list on how to dothat. The developers will keep in contact with you and, I fear, willprobably ask for other things as well ;)

Note that we strive for highest reliability of the engine even inunusual deployment scenarios. Unfortunately, this is hard to achieve,especially with limited resources. So we are depending on cooperationfrom users. This is your chance to make a big contribution to theproject without the need to program or do anything else except get aproblem solved.