How to Setup Central Log Server

This article is how to setup Central Log Server to collect rsyslog from different Linux machine and analysis them from a Central location..

What is Syslog Server is used for :-
Central Syslog server is the primary requirement of a Linux Administrator, It help to an Administrator analysis rsyslog time to time and validate is everything going in well manner with server.

Requirement:- In my Case I am using Ubuntu 16.04 TLS

1- Central Log Server    :-    192.168.138.129
2- Linux Web01 Server  :-    192.168.138.132
3- Linux Web02 Server  :-    192.168.138.133

Steps :- 

Central Server Side Configuration :-

Check rsyslog package using command dpkg. 

root@LXD:~# dpkg --list rsyslog
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                      Version                   Architecture              Description
+++-=========================================-=========================-=========================-=======================================================================================
ii  rsyslog                                   8.16.0-1ubuntu3           amd64                     reliable system and kernel logging daemon

root@LXD:~#

By-default rsyslog package come pre-installed. If not exist install using below command.

root@LXD:~# apt install rsyslog

Now make required changes in rsyslog.conf file. Open file and add highlighted line below.

root@LXD:~# vi /etc/rsyslog.conf

#  /etc/rsyslog.conf    Configuration file for rsyslog.
#
#                       For more information see
#                       /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
#  Default logging rules can be found in /etc/rsyslog.d/50-default.conf


#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
module(load="imklog")   # provides kernel logging support
#module(load="immark")  # provides --MARK-- message capability

# provides support for local system logging
$ModLoad imuxsock

# provides kernel logging support (previously done by rklogd)
$ModLoad imklog

# provides UDP syslog reception. For TCP, load imtcp.
$ModLoad imudp

# For TCP, InputServerRun 514
$UDPServerRun 514

# This one is the template to generate the log filename dynamically, depending on the client's IP address.
$template FILENAME,"/var/log/centrallog/%HOSTNAME%/syslog.log"

# Log all messages to the dynamically formed file. Now each clients log (192.168.138.132, 192.168.138.133, etc...), will be under a separate directory which is formed by the template FILENAME.
*.* ?FILENAME

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

# Enable non-kernel facility klog messages
$KLogPermitNonKernelFacility on

###########################
#### GLOBAL DIRECTIVES ####
###########################

#

# Use traditional timestamp format.

# To enable high precision timestamps, comment out the following line.

#

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages

$RepeatedMsgReduction on

#

# Set the default permissions for all log files.

#

$FileOwner syslog

$FileGroup adm

$FileCreateMode 0640

$DirCreateMode 0755

$Umask 0022

$PrivDropToUser syslog

$PrivDropToGroup syslog

#

# Where to place spool and state files

#

$WorkDirectory /var/spool/rsyslog

#

# Include all config files in /etc/rsyslog.d/

#

$IncludeConfig /etc/rsyslog.d/*.conf

Save and Exit from File

Restart rsyslog service

root@LXD:~# systemctl restart rsyslog

We have define log path - /var/log/centrallog/ 
You will see Central Log server log file will be there with hostname, Check through below command.

root@LXD:~# ls -l /var/log/centrallog/
total 4
drwx------ 2 syslog syslog 4096 Jul 25 15:23 LXD



Client Server Configuration :-

Client End Configuration need to add following lines, go to web01 server and made below changes.


root@web01:~# vi /etc/rsyslog.conf

ModLoad imuxsock

$ModLoad imklog

# Provides UDP forwarding. The IP is the server's IP address

*.* @192.168.138.129:514 

Save and exit from file.

Restart rsyslog service 

root@web01:~# systemctl restart rsyslog

Now go to Central log server and run below command to see web01 hostname file inside log path folder.

root@LXD:~# ls -l /var/log/centrallog/

total 8

drwx------ 2 syslog syslog 4096 Jul 25 15:23 LXD

drwx------ 2 syslog syslog 4096 Jul 25 15:38 web01

Logon to  web02 server and add below line into rsyslog.conf file.

root@web02:~# vi /etc/rsyslog.conf

ModLoad imuxsock

$ModLoad imklog

# Provides UDP forwarding. The IP is the server's IP address

*.* @192.168.138.129:514 

Save and exit from file.

Restart rsyslog service 

root@web02:~# systemctl restart rsyslog

Both the Client web01 and web02 started forward logs to Central log server. Use command to see file and analysis log file.

root@LXD:~# ls -l /var/log/centrallog/

total 12

drwx------ 2 syslog syslog 4096 Jul 25 15:23 LXD

drwx------ 2 syslog syslog 4096 Jul 25 15:38 web01

drwx------ 2 syslog syslog 4096 Jul 25 15:41 web02

To check log use command Cat or tail.

root@LXD:~# tail -f /var/log/centrallog/web01/syslog.log

2016-07-25T15:38:07+05:30 web01 systemd[1]: Starting System Logging Service...
2016-07-25T15:38:09+05:30 web01 systemd[1]: Started System Logging Service.
2016-07-25T15:38:09+05:30 web01 rsyslogd-2039: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com/e/2039 ]
2016-07-25T15:38:09+05:30 web01 rsyslogd-2007: action 'action 15' suspended, next retry is Mon Jul 25 15:38:39 2016 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
2016-07-25T15:39:01+05:30 web01 CRON[1254]: pam_unix(cron:session): session opened for user root by (uid=0)
2016-07-25T15:39:01+05:30 web01 CRON[1255]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
2016-07-25T15:39:01+05:30 web01 CRON[1254]: pam_unix(cron:session): session closed for user root
2016-07-25T15:39:45+05:30 web01 dhclient[279]: DHCPREQUEST of 192.168.138.132 on eth0 to 192.168.138.254 port 67 (xid=0x5c3e468d)
2016-07-25T15:39:45+05:30 web01 dhclient[279]: DHCPACK of 192.168.138.132 from 192.168.138.254

2016-07-25T15:39:45+05:30 web01 dhclient[279]: bound to 192.168.138.132 -- renewal in 717 seconds

That's all

!!! Cheers !!!