Using Lua for Log Message Filters

Here is how the public API of fplog looks like:

//One time per application call.
FPLOG_API void initlog(const char* appname, fplog::Transport_Interface* transport = 0);

//Mandatory call from every thread that wants to log some data.
//Done to increase flexibility:
//each thread will have its own filters configuration and
//can decide independently which stuff to log.
FPLOG_API void openlog(const char* facility, Filter_Base* filter = 0);

//Optional but advised to call from each thread that logs data right
//before stopping and exiting the thread.
//Doing this will free some memory (amount is really small)
//that is needed to support logging from a given thread.
//Not doing closelog() does not lead to memory leak
//because everything will be deallocated in any case on app exit,
//fplog frees its resources when implementation class instance is destroyed.
FPLOG_API void closelog();

//Scope of filter-related functions is a calling thread -
//all manipulations will apply to calling thread only.
FPLOG_API void add_filter(Filter_Base* filter);
FPLOG_API void remove_filter(Filter_Base* filter);
FPLOG_API Filter_Base* find_filter(const char* filter_id);

FPLOG_API const char* get_facility();

//Should be used from any thread that opened logger,
//calling from other threads will have no effect.
FPLOG_API void write(Message& msg);

It resembles syslog closely however extended with per-thread message filtering capabilities along with few other things. Today I would like to concentrate on the filters because I was playing around with several ideas in this regard for couple of days now and the idea that has beaten the competition is to use Lua for log message filtering.

The thing is log message could contain many fields in addition to mandatory that only the log message author could know, therefore it would be nice to allow a fellow programmer to create as complicated filter passing condition as he likes. Well, but we do not need Lua for that, right? There is Filter_Base class that basically has 1 method that accepts log message and as a result tells true/false meaning message should pass further or be suppressed.

It works in case the only way we would be making adjustments in filtering conditions is by modifying the source code: creating new subclasses of Filter_Base that implement the filtering conditions we see fit. However as soon as there is an idea to change filter without modifying the source code the need for loading filters from something like config file arises.

I was thinking about some simple filters where you have only a small subset of logical operations and comparisons you could do, about regular expressions or some other pattern matching but everything what initially seemed simple turned into rather inflexible and/or complex stuff. Therefore step by step I approached the idea of using some existing scripting solution for filters and Lua was the obvious choice – lightweight and easily embedded into C++ code.

So, it goes like this:

  • there is an instance of Lua interpreter running all the time inside fplog – it is not created and destroyed for each message;
  • json.lua module in addition to all the standard Lua libs is loaded;
  • fplog_message object is available from Lua script, this object is an actual log message converted from json to native Lua tables structure (nested tables are possible – same as nested json objects);
  • Lua filter script should call filter_result function with true/false parameter in the end to inform the C++ code to either pass the message further or suppress it.

Of course the performance might be an issue, especially taking into account my intent of trying out fplog on embedded devices that could run Linux. Each log message would need to go via one or more Lua-based filters before it is potentially transported to a log collecting daemon via IPC. I will definitely need to run some tests and of course Lua-based filtering would be optional – if you do not need to be able to fine-tune logging “on-the-fly” via config files or some other methods then you will not be forced to use Lua filters.

Actually even without Lua you could implement several filters inside your application the will be pre-made and hardcoded but still it does not prohibit you from using config file to select a combination of pre-made filters to apply.