Seldon Fluentd Configuration
When running multiple Seldon servers at scale we need to stream the activity logs to a central location such as AWS S3, HDFS or a central file store. We have found Fluentd from Treasure Data is idle for this. We will discuss here what you need to do to set this up.
You will need to install td-agent which is a packaged version of Fluentd. See here.
Seldon server has four log files:
- “restapi.log” : all REST API calls
- “ctr.log” : Impressions and Clicks for recommendations
- “ctr-alg.log” : Detailed Impressions and Clicks for recommendations with algorithms, and returned items
- “actions.log” : All action calls
If you have several Seldon servers running you would want to collect these logs and transfer them to a central location. The td-agent.conf configuration file for this can be found in
seldon-server/td-agent/seldon-server-td-agent.conf.base.example and is shown below. You should replace
<TOMCAT_HOME> with the location of your Apache tomcat home folder:
The above config does the following:
- tails the 4 core log files
- forwards the logs to one of two central td-agent servers called “tdagent1” and a reserve server “tdagent2”.
At the central td-agent server you will need to set the config to forward the logs to a central backing store such as AWS S3, or HDFS.
An example central config can be found at
seldon-server/fluentd/seldon-td-agent.conf.central.example and shown below, which forwards to AWS S3 and Kafka the restapi logs:
Store Actions in Redis Store
If you wish to use Redis as an action store and to store the actions using fluentd you can use our fluentd plugin which can be installed with:
The source github repo can be found here. It is a fork of https://github.com/pokehanai/fluent-plugin-redis-store
You can then add to your central td-agent a match condifuration like: