Logging today is mostly done too unstructured; each application developer has his own syntax for the logs, optimized for his personal requirements and when it is time to deploy, ops consider themselves lucky if there is even some logging in the application, and even luckier if that logging can be used to find problems as they occur by being able to adjust verbosity where needed.
I’ve come to the point where I want a really awesome piece of logging from the get-go – something I can pick up and install in a couple of minutes when I come to a new customer site without proper operations support.
I want to be able to be able to search, drill down into, filter out patterns and have good tooling that allow me to let logging be an obvious support as the application is brought through its life cycle, from development to production. And I don’t want to write my own log parsers, thank you very much!
That’s where semantic logging comes in – my applications should be broadcasting log data in a manner that allow code to route, filter and index it. That’s why I’ve spent a lot of time researching how logging is done in a bloody good manner – this post and upcoming ones will teach you how to make your logs talk!
Here’s the outline of this post.
Vagrant says it’s used to “create and configure lightweight, reproducible, and portable development environments”. Vagrant means hobo, or someone living without a home. In this context, Vagrant is an excellent tool for packaging your development environment in a way, so that you can repeatedly set it up. Imagine how much time you currently spend on making a developer’s computer come up to speed.
Vagrant orchestrates VirtualBox through its executable ‘VBoxManage’. It is also responsible for tunnelling the output from inside the VM into the console.
Further, the tool uses server provisioning tools that can be equally well used in production as in a test environment, meaning that if you invest time in setting up your development box well, you’ll gain some of that time back when deploying with puppet or chef.
In this post I’m going to use puppet, as I prefer the state-based description language that it uses over the imperative ‘recipe’ like way of chef.
VirtualBox is a free virtualizing product, owned by Oracle, with a corresponding command line tool that vagrant uses to create and destroy the virtual machines.
You are now ready to initialize vagrant and have a look at the finished result (before you go through the steps there).
The following bash script will download the git repository corresponding to this blog entry, run vagrant in that directory (while waiting for vagrant to finish) and then start a browser to see the kibana interface. Remember to shut down any RabbitMQ, port 8080, elasticsearch, logstash or kibana that you may have running locally, and answer yes to the admin prompts
You’ll be setting up a RabbitMQ broker (http://127.0.0.1:55672) as messaging fabric. You can log into the admin panel using guest/guest. (On Windows without ‘git bash’? Why?? In that case you’ll have to manually download the files – look in each of the download.sh files in modules ‘elasticsearch’ and ‘logstash’.)
ElasticSearch (http://127.0.0.1:9200) as a search engine:
LogStash (http://127.0.0.1:9292) for routing the logs:
and finally Kibana (http://127.0.0.1:8080) for watching the logs:
Although, the browser is completely devoid of data, since it was just created.
Let’s add some sample data!
to insert some random data. Let the process work for as long as you’re interested in investigating the virtual machine.