php - rsyslog performance optimization -

what should done setup rsyslog best performance?

• we can allow items lost on server crash or lost.
• we going save logs mysql db.
• we able handle @ least 100 log writes per second latency 0.001 - 0.005 second.
• we writing logs php application.

thank help.

we went through similar exercise using mongodb database, i'll document did , hope helps you.

it our first time using rsyslog, took bit of effort find right documentation , piece together. in end, our test drivers (we're using soapui) able 1000 tps through php web service uses rsyslog write summary record of transaction.

we found following articles got started:

• http://www.rsyslog.com/doc/rsyslog_high_database_rate.html
• http://www.rsyslog.com/tag/mongodb/

the overview you'll enable rsyslog's queue infrastructure writing incoming messages disk when daemon's memory queue full. in our case, enabled $actionqueuesaveonshutdown, sounds don't need. you'll configure rsyslog ruleset parse incoming messages , pass them output handler mysql. finally, php script use openlog() , syslog() write whatever data want log. oh, had compile rsyslog source, in order enable json/mongo plugins, , exercise in itself. we're using rsyslog 7.4.5 on ubuntu 12.04.

i'm not expert on rsyslog, can give our config files , code starting point. again, they're mongodb, gives idea of , change things implementation.

good luck!

/etc/rsyslog.conf:

$modload imuxsock # provides support local system logging $modload imklog   # provides kernel logging support (previously done rklogd)  # load modules mongodb integration: json parser , mongodb output driver module(load="mmjsonparse") module(load="ommongodb")  # use traditional timestamp format. # enable high precision timestamps, comment out following line. $actionfiledefaulttemplate rsyslog_traditionalfileformat  # filter duplicated messages $repeatedmsgreduction on  # set default permissions log files. $fileowner syslog $filegroup adm $filecreatemode 0640 $dircreatemode 0755 $umask 0022 $privdroptouser syslog $privdroptogroup syslog  # place spool files $workdirectory /var/spool/rsyslog  # use queue decouple db writes default message handling # http://www.rsyslog.com/doc/rsyslog_high_database_rate.html $mainmsgqueuefilename mainq     # set file name main queue, enables disk mode $actionqueuetype linkedlist     # use asynchronous processing $actionqueuefilename mongodbq   # set file name mongo db queue, enables disk mode $actionresumeretrycount -1      # infinite retries on insert failure $actionqueuesaveonshutdown on   # write queue data disk when rsyslogd                                 #   terminated (default off)  # include config files in /etc/rsyslog.d/ $includeconfig /etc/rsyslog.d/*.conf 

/etc/rsyslog.d/10-mongo.conf:

input(type="imuxsock" socket="/dev/log")  template(name="mongodblocal" type="subtree" subtree="$!")  # use json parser "local0" facility messages,  # if parsed run template load # message mongodb database. if $syslogfacility-text == 'local0' {         action(type="mmjsonparse")         if $parsesuccess == "ok" {                 # set local vars appended onto                  # document that's written mongodb                 set $!time = $timestamp;                 set $!sys = $hostname;                 set $!procid = $syslogtag;                 set $!syslog_fac = $syslogfacility;                 set $!syslog_sever = $syslogpriority;                 set $!pid = $procid;                 action(type="ommongodb" server="127.0.0.1" db="test" collection="syslog" template="mongodblocal")         } } 

/etc/rsyslog.d/50-default.conf: note: disables "local0" messages default handling.

# first standard log files.  log facility. auth,authpriv.*         /var/log/auth.log  # don't write "local0" messages syslog,  # they're processed using ommongodb (see 10-mongo.conf) *.*;local0,auth,authpriv.none   -/var/log/syslog  kern.*              -/var/log/kern.log mail.*              -/var/log/mail.log  # logging mail system.  split # easy write scripts parse these files. mail.err            /var/log/mail.err  # logging inn news system. news.crit           /var/log/news/news.crit news.err            /var/log/news/news.err news.notice         -/var/log/news/news.notice  # emergencies sent logged in. *.emerg                                :omusrmsg:* 

php web service related calls:

// open syslog, include process id , open connection logger  // immediately, , use user defined logging mechanism local0 openlog($script_name, log_pid | log_ndelay, log_local0); // note: calling closelog() optional, , don't use  ... // construct $doc, logged, change appropriate // implementation; here $ary_headers request's http headers, // , $request/$response posted/returned $doc = array("headers" => $ary_headers             ,"request" => $request             ,"response" => $response             ); ...  // write log entry syslog, queues , writes mongodb // note: need '@cee: ' prefix rsyslog json parser process // see:  http://www.rsyslog.com/doc/rsyslog_conf_modules.html/mmjsonparse.html  // json_bigint_as_string = encodes large integers original string value. // json_numeric_check = encodes numeric strings numbers. // json_unescaped_slashes = don't escape "/".  syslog(log_info, '@cee: ' . json_encode($doc, json_bigint_as_string | json_numeric_check | json_unescaped_slashes));