How Logstash and Kibana Empower Blackboard Learn Logs

Greetings everyone! Sorry that I haven’t been able to post but I have recently been dealing with a Blackboard production environment that has spent much of its time in various rings of hell. This has happened on and off since the start of the semester (about 14 weeks ago as of the writing of this post). This has been a very difficult time, but there has been one ray of sunshine for us, our implementation of Kibana, ElasticSearch, and Logstash proved invaluable during troubleshooting.

In my opinion, these three applications are the holy trinity of logging for any application like Blackboard Learn. They allow an admin to unlock the information kept in logs at a cost that most IT budgets can handle (the three tools are open source). I learned about these tools from our central server group (CSG) who had heard about the tools during PuppetCon. (See the video below)

They had started testing them in their “skunkworks” area. During a meeting, I lamented the difficulty searching log files on six different application servers. These difficulties made me create the log collection script which I blogged about earlier this year. One of the team members suggested Logstash, ElasticSearch, and Kibana.

The next week I was at my server administrator training at Blackboard HQ in Washington, DC. During that week, one of the attendees from SUNY mentioned these tools as well. These two discussions had sold me. Back on campus, I quickly moved to test out how helpful and easy it would be to implement these applications within our Blackboard Learn environments.

The three tools Kibana, ElasticSearch, and Logstash all have different parts that they play in the log puzzle.

Logstash – This application brings the logs and event data from one or many systems to one location. The open source product runs on top of ElasticSearch.

ElasticSearch – This application is the engine of Logstash. It collects, indexes, parses, and searches the log data.

Kibana – The frontend of Logstash and ElasticSearch, this application is the Google search for all your logs. The user can be as basic or as advanced in how they want to search to find log information.  The latest release of Kibana adds more visuals based on collected data.

We were already using shared network storage to hold all the log files from our production, quality assurance, and development instances. This meant that 95 percent of our files wouldn’t be difficult to collect. We could simply mount the shared storage to collect the data. We then had to use a shipper for the other 5 percent that don’t sit with the network storage, such as tomcat logs. The shipper runs on the application server and sends all data written to that file for indexing and storage. We spent a few weeks making sure that the shipper didn’t put any additional load on the application servers. Next we had to learn how to grok.

Grok is a pattern set used by the Logstash application to breakdown logging input and make it searchable. (See Grok Patterns) We created multiple fields for items such as email subject lines, IP addresses, etc. that appeared in the logs. The fields gave us the ability to leverage the data collected and the application’s indexing abilities.

Once all the data was being collected, groked (or parsed), and indexed. The Kibana application stepped into the spotlight. While Kibana has a simplistic interface, it is powerful and sometimes cryptic. I had to create a Kibana Cheat Sheet for myself and my team on how to use the web application to find logging data.

Once implemented, a few team members started to use the information to collect data about our environment. However our fall semester started out with our environment crashing on a regular basis, nearly as well scheduled as a subway train schedule. The only humor was that we would regularly be able to call the time for these failures. “Will the 11:03 Blackboard train wreck be on time today?” but I digress. During the following weeks we worked with Blackboard Support on various fixes. The use of Logstash, ElasticSearch, and Kibana really helped us find information and learn more about what was going on within our environment. We were able to find ActiveMQ and file locking issues quickly and discovered issues with session spray using our searches. (See some screen captures of Kibana below.)

[EasyGallery id=’kibana’]

The only negative thing I can say about Kibana is that it doesn’t have built in user management or the ability to alert users when specific data appears in the logs. GrayLog2 is an open source project that uses ElasticSearch and Logstash. However it does have the components that Kibana is missing. We might look at moving to it in the future. For right now, Kibana meets our needs.

In the end, these three tools have greatly improved how we can use data to address issues and fix changes within our Blackboard environment. I hope that it helps you when trying to decide how to empower your log file data.

Technically yours,

The Blackboard Guru