Setting up a multi-tiered log infrastructure Part 11 -- Cluster Tuning

This is post 11 of 11 in the series “Setting up a multi-tiered log infrastructure” Tuning Graylog, Elasticsearch, and MongoDB for optimized cluster performance This has been an article a long time in the making. One problem with making changes to a complex clustered environment is that you may have to wait long periods of […]

Setting up a multi-tiered log infrastructure Part 10 -- HA Cluster Setup

This is post 10 of 11 in the series “Setting up a multi-tiered log infrastructure” Setup HA Cluster Services on CentOS 7 Install HA Cluster components Install pacemaker and the cluster control software on both nodes that will be part of the cluster(corosync is pulled in as a dependency) yum install pacemaker pcs Enable and […]

Setting up a multi-tiered log infrastructure Part 9 -- Rsyslog HA Setup

This is post 9 of 11 in the series “Setting up a multi-tiered log infrastructure” Setup rsyslog aggregator nodes (Optional) Setup Note:  As part of the overall design, an HA cluster allows aggregating logs to the Central Log Repository with as little loss of logs as possible due to downtime or maintenance. Below are steps […]

Setting up a multi-tiered log infrastructure Part 8 -- Rsyslog Setup

This is post 8 of 11 in the series “Setting up a multi-tiered log infrastructure” Setup rsyslog node Setup Note: there are two possible build options. Option 1 is to build a single server that will handle log reception and storage. This is perfectly acceptable if losing a few logs during maintenance is okay. Option […]

Setting up a multi-tiered log infrastructure Part 7 -- Graylog WebUI Setup

This is post 7 of 11 in the series “Setting up a multi-tiered log infrastructure” Additional Setup for master node Setup Graylog Web UI on master node Setup Note: newer versions of graylog do not require a separate install for the web interface anymore so we can make a few firewall rule changes and be […]

Setting up a multi-tiered log infrastructure Part 6 -- Graylog Setup

This is post 6 of 11 in the series “Setting up a multi-tiered log infrastructure” Additional setup for master node Setup graylog-server on master node Install instructions from http://docs.graylog.org/en/2.2/pages/installation.html Setup Note: This deployment is not using a prebuilt rpm package, many of the next steps will be moving files, creating directories, creating additional files, and […]

Setting up a multi-tiered log infrastructure Part 5 -- MongoDB Setup

This is post 5 of 11 in the series “Setting up a multi-tiered log infrastructure” Additional Setup for master node Install mongodb on master node Install instructions from https://docs.mongodb.com/manual/administration/install-on-linux/ Create repo file for mongodb vi /etc/yum.repos.d/MongoDB-3.4.repo Insert this text [mongodb-org-3.4] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.4/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-3.4.asc Install mongodb yum install mongodb-org Set mongod to start […]

Setting up a multi-tiered log infrastructure Part 4 -- Elasticsearch Setup

This is post 4 of 11 in the series “Setting up a multi-tiered log infrastructure” Setup Elasticsearch cluster nodes Install Elasticsearch In this example we are building out a three node cluster but this can scale up to fit whatever cluster size you choose. View Elasticsearch setup and configuration docs https://www.elastic.co/guide/en/elasticsearch/reference/2.4/index.html Install Java yum install […]

Setting up a multi-tiered log infrastructure Part 3 -- System Build

This is post 3 of 11 in the series “Setting up a multi-tiered log infrastructure” Partitioning layout Setup Note: The partition layout for elasticsearch nodes may vary depending on how much data storage is needed. It makes sense to create mount points with the idea they may grow in the future. Worksheet for estimating proper […]

Setting up a multi-tiered log infrastructure Part 2 -- System Overview

This is post 2 of 11 in the series “Setting up a multi-tiered log infrastructure” System Build Overview The next steps are to build the environment; starting with the elasticsearch (ES) nodes and the log parser/search frontend because they require certain components to be identical. The process assumes the use of a minimal OS install […]

Proudly powered by WordPress   Premium Style Theme by www.gopiplus.com