Setting up a multi-tiered log infrastructure Part 1 -- Getting Started

Logging Infrastructure Overview

Setting up a multi-tiered logging infrastructure can be a little more complicated than just spinning up an rsyslog server and shipping logs to it (although that is an option). A few products come up repeatedly when looking for logging stack solutions, those are elasticsearch, logstash, kibana, and graylog, along with rsyslog, nxlog, syslog-ng and OSSEC.

The ELK stack uses elasticsearch, logstash, and kibana. The graylog and elasticsearch stack or GELP (Graylog Enhanced Logging Platform) for lack of an existing acronym utilizes Graylog to perform the same functions as Logstash and Kibana. Graylog and rsyslog can also utilize an elasticsearch backend for storage. Nxlog, rsyslog, syslog-ng, and OSSEC are client-side log shippers, where OSSEC, rsyslog, and syslog-ng are also servers for receiving logs. OSSEC is not exactly a logging application but it can be used to analyze logs using predefined rules and then alert based on those rules.

The list is certainly not exhaustive and does not contain information about commercial logging stacks such as SPLUNK, Arcsight, or Manage Engine’s Eventlog Analyzer among others.

Design Considerations

When I first started digging into this, there were no specific thoughts about an overall design. I was merely interested in getting something to work. Now that I’ve setup a few different stacks, my thoughts turned to how might this be implemented in an enterprise network.

Endpoints might include Windows, Linux, AIX, Cisco UCS, ESXi hosts, netscalers, ILO, DRAC, IPMI etc. Then on top of that, some Apache logs, some IIS logs, email logs, WAN routers, ASA’s, other firewalls, and a bunch of internal networking devices. There may be a need/interest for printers, power infrastructure, building automation systems, and some medical devices too.

So, what is the best way to get logs from an endpoint (EP) to the Central Log Repository (CLR) then to the Elasticsearch Database Server (ESS) so a Frontend UI Server (UIS) can allow viewing them? The overall setup is EP => CLR => ESS <=> UIS. There are many alternative solutions, which all have their merits but this configuration seems easier to implement and maintain.

Just to discuss some alternative configurations, one can use rsyslog with logstash grabbing logs from a file. Other setups can pipe logs directly to logstash prior to dumping logs into elasticsearch. Sending logs directly to Graylog is yet another alternative. And another is shipping logs to rsyslog, which in turn, sends logs to elasticsearch. Add in implementing OSSEC and wanting to use TLS for shipping logs when possible, one can see how rolling out an implementation of this sort can get a little complicated.

Selected Design

The overall design allows shipping logs from endpoints to two aggregation nodes (setup as an HA cluster). Logs pass to a central log repository running rsyslog prior to being shipped to the elasticsearch backend via the log parser and search frontend. One reason for using rsyslog in between the endpoint and log parser is the backend can be changed without affecting the initial configuration, guarding against product lock in.

Logging_Stack_Overview REV2

4 Responses to“Setting up a multi-tiered log infrastructure Part 1 -- Getting Started”

  1. tristanrhodes
    February 4, 2015 at 6:53 pm #

    Is it correct that you have your endpoints send a copy of their logs to both log aggregators? If so, how do you prevent duplicate logs from being saved in Elasticsearch?

    I was thinking about having endpoints send logs to an active-passive load-balancer. This load-balancer would monitor the health of Graylog servers and distributes the logs to them.

    Thanks for doing this write-up!


    • admin
      February 4, 2015 at 7:41 pm #

      The aggregation nodes use a shared IP so when performing maintenance, the IP can be failed over to whichever host will remain active. Only one host is master at any one time the other is there as a backup.

  2. tristanrhodes
    February 4, 2015 at 9:39 pm #

    Ok, that makes sense. It sounds very similar to my active-passive load-balancers that share an IP.

    However, the benefit of using rsyslog is that you can locally cache logs if the downstream log server goes down, right?

    I still don’t understand the function of #3 (Central Log Repository). Is it mainly for long-term compressed storage of logs, which take up much less space the Elasticsearch logs? If you didn’t need this long-term storage, you could simply forward logs from your log aggregators directly to OSSEC and Graylog.

    What is your opinion of OSSEC for log security? Have you also tried Sagan?

    • admin
      February 4, 2015 at 11:08 pm #

      Yes, we discussed using a round robin type scheme as an option too, either will work since the two aggregator nodes just pass log data through to the Central Log Repository. The CLR is actually functioning as a shared disk for the two HA nodes in a sense. Since this entire setup is using VM’s, it was just as easy to spin up a third server as it would have been to setup shared storage on the HA nodes.

      As far as other functions of the CLR, one is like you suggest, long-term storage. What makes this useful is storing RAW log data from each endpoint prior to parseing logs and passing it to the search front-end (#4). The CLR can also be used as a traffic cop, which allows for easier maintenance on other nodes within the cluster. Granted, this can be handled by the HA nodes too. Something that is more preference than a real need is I like knowing I only have to go to one place to find all the RAW log data.

      We used OSSEC as a HIDS on a few different external FTP servers but I honestly don’t know how it will perform in the log analysis role I plan for it (I have been slacking and it isn’t built yet). I have not come across sagan before and will take some time to look it over. Thanks for the link.

You must be logged in to post a comment.

Proudly powered by WordPress   Premium Style Theme by