Elasticsearch/External Logging - DEPRECATED

The information below is either outdated, or no longer considered best practice at Kalamuna.

This has a lot of good information and is a good candidate for being updated, but as it stands a lot of the information is out of date.Different methods for using external services like Elasticsearch to store, sort, and access Drupal logs.

This is important when you need to log lots of data, have it readily accessible for search, and may want to aggregate multiple kinds of log entries in a single index.

WARNING: I've tried the Qbox service and it works flawlessly on my local, but have had some issues getting it to work on a Pantheon multidev environment. Will report further when I see the issue, suspect it may be some sort of cURL difference between the environments.

Setting Up An Elasticsearch Server

Hosted Solutions

These are nice because they come with services like Kibana and Logstash that will allow analysis of Elasticsearch logs out of the box:

• Qbox

• Found

DIY

For security compliance (like HIPAA), you may want to roll your own Elasticsearch server.

Amazon will sign a BAA for HIPAA, however, you must communicate data between your clusters using SSL, which requires a patch to Elasticsearch: https://github.com/elasticsearch/elasticsearch/pull/2105

Connecting to Drupal

These are the steps I took to use the Qbox hosted Elasticsearch service for Drupal logging:

1. Install the elasticsearch_connector module.

2. Enable the elasticsearch_connector_easy_install module; this will include all the required libraries. Alternatively, you can install these libraries in sites/all/libraries as normal.

3. Enable elasticsearch_connector and elasticsearch_watchdog.

4. Configure the connection to your elasticsearch cluster at admin/config/elasticsearch-connector/clusters/add

   1. Remember to have the port on the end of the URL (don't be scared if the "processing" symbol stays up for a while or if you get an AJAX error; that just means something is wrong with your URL and it will need to be re-entered)

   2. Remember to insert HTTP Basic Auth credentials (or other authentication)

5. Configure the watchdog settings: admin/config/elasticsearch-connector/watchdog

   1. Select the cluster you just connected to

   2. Add an index

      1. Number of Shards/Replicas: I set 2 shards and 2 replicas to experiment, see this post for a decent guide on what these terms mean.

      2. If you're having trouble adding through the modal, you can go to the index management screen (which conveniently isn't on the menu structure for the elasticsearch_connector module): admin/config/elasticsearch-connector/clusters/[CLUSTER_NAME]/indices

6. Log into Qbox, go to your cluster, and click on "View data with Kibana" or use the "Monitor with royrusso/elasticsearch-HQ" link to see if entries are being created as you browse pages on your Drupal site that would generate log entries.
   
   
   

Alternatives to Elasticsearch

https://www.drupal.org/project/gelf (uses graylog2, which actually does use Elasticsearch under the hood)

Review History