In this post we will describe how to use MongoDB as a log store from your .NET application with very little work or code. We are going to be using log4mongo, a log4net appender to MongoDB. The sample application that we create in this post is available on GitHub.
MongoDB is well suited for logging and can be preferred over logging to text files or to a relational database due to some of the following reasons:
- Simple asynchronous inserts a.k.a fire-and-forget.
- Use of MongoDB capped collections means old logs don’t need to be deleted manually. Logs can quickly fill up disk.
- Log schema doesn’t have to be defined, new information can be added adhoc.
- Off-site access easier than files.
- Query logs using MapReduce or Aggregation Framework.
- Logging to your relational database, assuming you are using one, increases load and disk space usage.
- If some logs are written in transactions and these transactions rolled back, you lose the logs.
Let’s get started.
First we need to install and run a MongoDB instance, if you are familiar with how to do this, continue to the next step, Preparing our .NET Application, where we install log4mongo-net with NuGet.
Setting up MongoDB
Download MongoDB for your platform. If you are on windows, the installer will install MongoDB to C:\Program Files\MongoDB 2.6 Standard\bin by default assuming you are using the 64 bit version and have installed version 2.6. If you are installing it on another platform, please consult the MongoDB docs.
Once you are installed, open up a command prompt and cd to your MongoDB install path. The MongoDB server is called mongod. If you run mongod without any arguments you will probably receive a message saying that you need to define the dbpath.
If you receive a Windows Security Alert go ahead and Allow access.
If you see “waiting for connections on port 27017” you are good to go. You have an empty MongoDB server up and running, let’s fire up Visual Studio.
Preparing our .NET application
Create a new Visual Studio application or open an existing solution you wish to add logging to. We need to install log4mongo and its dependencies; log4net and mongocsharpdriver. log4net is a popular logging framework for .NET and mongocsharpdriver is the C# driver we need to talk to MongoDB.
There are a number of ways to use your logger from your application such as injecting a shared instance using your favourite IoC Container, use a Singleton logger, a static logger, a base class logger. My preferred method is to inject a shared instance via constructor injection and an IoC Container. We will not describe how to do this in this post as we are focusing on log4mongo. You can find more information on logging and the various ways to do it in this StackOverflow question.
We are going to setup the log4net configuration file as well as log4mongo’s configuration file. If you are not familiar with log4net, see this CodeProject article. In a nutshell, log4net needs 3 things before it can work; a configuration file, a call to its static factory LogManager.GetLogger() method and a call to XmlConfigurator.Configure().
The log4net configuration is to be included in your app.config or web.config file. We are going to put the log4net appender in its separate configuration file, you don’t have to do this but I think it’s cleaner as app.config and web.config files can grow large and harder to read the more you add.
You need to add two lines to your configuration file:
The first line tells the configuration file that we are creating a log4net section called “log4net” and the second line is the log4net section itself. This line just refers to our other log4mongo appender configuration file (that we will create next). If you don’t wish to create a separate log4mongo configuration file, the contents would go here instead.
Now create a new application configuration file in the root of your Visual Studio project and replace with:
Here we have a log4net appender named “MongoDBAppender”. You can name it whatever you like but remember to reference it in the root appender-ref element. We define a connection string that points to the mongod instance we started. If you started mongod with the default port you can use mongodb://localhost. Next, the name of the collection that is going to hold our logs. The field elements define what properties each log document will contain, here we have the timestamp, the log level, the thread, the logger and the message. You can create your own custom fields, see the log4mongo-net GitHub documentation for more information. Finally, the root element where we define our log level and reference our MongoDBAppender.
If Visual Studio 2013 tells you that “The ‘log4net’ element is not declared.”, just ignore it. It’s defined in our configuration file that references this one!
That’s it for configuration files. Just make sure your application can find them at run time. If you moved them, take the necessary steps to tell log4net where to find them.
We need to create our logger. You can choose to wrap the log4net logger in your own logger interface or not. The log4net ILog interface is a facade to it’s own implementation so it’s a nice interface to use, however you might want to wrap it so you can stub it or mock it in tests or even replace log4net with another logger in the future. I like to be able to able to inject fake implementations of my dependencies so I will wrap it even though we have to write a bit more code and can potentially loose some of log4net’s features if we do not expose them in our wrapper. I chose the following simple wrapper that exposes each log level with an overload that takes in an Exception.
Let’s implement our ILog wrapper:
We have an instance of log4net’s ILog and we are setting it using log4net’s static factory method, LogManager.GetLogger(). We are passing in the type of the class that is calling the logger as a string. This string can be set by your IoC Container.
That’s it for setup. Let’s instantiate our logger and log a message. I am logging in an ASP.NET MVC application so I am injecting an instance of my logger wrapper into my HomeController constructor:
I am calling our Info() log method three times, once in my index action, again in my about action and again in my contact action. If you are logging in a Console application, a WPF or Win Forms app just make sure you get an instance of ILogger and call some log methods on it.
Putting it all together
Let’s see if we are able to log to our mongod instance. Make sure your mongod instance is still running, if it’s not start it up again using the mongod –dbpath C:\data command. Start a mongo shell, this is the binary that’s simply named mongo. By default this will connect to your mongod running on the default port. If you did not start your mongod on the default port you need to run mongo –port 27017 to specify the port.
When our Visual Studio application launches and logs our info message it will try to find a database called log4net as we defined in the MongoDBAppender. If that database does not exist it will create it. We don’t have to worry about creating it manually. Inside the log4net database will be a collection called logs.
Run your application and go to your mongo shell. You can use the command show databases from the mongo shell. With a bit of luck you will see a database listed as log4net. Next, type use log4net. This switches the mongo shell to that database. Type show collections, you should see a collections called logs. Finally type:
The mongo shell will show you the message that your application logged. Notice that the documents are in the same form as we defined in our log4mongo configuration file; timestamp, level, thread, logger and message.
In the mongo shell query, db is the database we have set (in our case the log4net database), logs is the collection in that database that we want to query, find() gets all the documents in the collection and pretty() formats the Json in the shell so it is easier to read.
Finally, we should make our logs collection a MongoDB capped collection. A capped collection is a fixed size collection that when full makes room at the start of the collection for new inserts. This is ideal for logging as we don’t have to manually remember to delete old logs. Given that we already let our application create our non-capped collection, we need to change it to be capped:
To check that the collection is capped, run db.logs.isCapped() from the mongo shell.
If you are not seeing a log4net database go back and double check that your log4net and log4mongo configuration files are correct, that you are calling XmlConfigurator.Configure() in your wrapper and that your MongoDb connection string is correct. The default should be mongodb://localhost.
That’s it! You have a simple and easy way to log from your application. The calls to the log methods are asynchronous on mongo’s end, we don’t need to wait for a response. Once mongod receives our log document our application can continue without blocking. If you are using C# 5 you could make the calls to log4net asynchronous at the application level, I will leave that as an exercise for the reader.
If you would like to learn more about MongoDB I suggest you head on over to MongoDB University. MongoDB University offers free online Developer and DBA classes and certificates. I would also like to recommend Kristina Chodorow’s book MongoDB: The Definitive Guide.
If you liked this article, please follow me on twitter. @sammleach