Elasticsearch/External Logging - DEPRECATED
DEPRECATED
The information below is either outdated, or no longer considered best practice at Kalamuna.
The information below is either outdated, or no longer considered best practice at Kalamuna.
This has a lot of good information and is a good candidate for being updated, but as it stands a lot of the information is out of date.Different methods for using external services like Elasticsearch to store, sort, and access Drupal logs.
This is important when you need to log lots of data, have it readily accessible for search, and may want to aggregate multiple kinds of log entries in a single index.
WARNING: I've tried the Qbox service and it works flawlessly on my local, but have had some issues getting it to work on a Pantheon multidev environment. Will report further when I see the issue, suspect it may be some sort of cURL difference between the environments.
Setting Up An Elasticsearch Server
Hosted Solutions
These are nice because they come with services like Kibana and Logstash that will allow analysis of Elasticsearch logs out of the box:
DIY
For security compliance (like HIPAA), you may want to roll your own Elasticsearch server.
Amazon will sign a BAA for HIPAA, however, you must communicate data between your clusters using SSL, which requires a patch to Elasticsearch:Â https://github.com/elasticsearch/elasticsearch/pull/2105
Connecting to Drupal
These are the steps I took to use the Qbox hosted Elasticsearch service for Drupal logging:
Install the elasticsearch_connector module.
Enable the elasticsearch_connector_easy_install module; this will include all the required libraries. Alternatively, you can install these libraries in sites/all/libraries as normal.
Enable elasticsearch_connector and elasticsearch_watchdog.
Configure the connection to your elasticsearch cluster at admin/config/elasticsearch-connector/clusters/add
Remember to have the port on the end of the URL (don't be scared if the "processing" symbol stays up for a while or if you get an AJAX error; that just means something is wrong with your URL and it will need to be re-entered)
Remember to insert HTTP Basic Auth credentials (or other authentication)
Configure the watchdog settings:Â admin/config/elasticsearch-connector/watchdog
Select the cluster you just connected to
Add an index
Number of Shards/Replicas: I set 2 shards and 2 replicas to experiment, see this post for a decent guide on what these terms mean.
If you're having trouble adding through the modal, you can go to the index management screen (which conveniently isn't on the menu structure for the elasticsearch_connector module):Â admin/config/elasticsearch-connector/clusters/[CLUSTER_NAME]/indices
Log into Qbox, go to your cluster, and click on "View data with Kibana" or use the "Monitor with royrusso/elasticsearch-HQ" link to see if entries are being created as you browse pages on your Drupal site that would generate log entries.
Alternatives to Elasticsearch
https://www.drupal.org/project/gelf (uses graylog2, which actually does use Elasticsearch under the hood)
Review History
Who | When | Status |
---|---|---|
 |  |  |
Bob | 20230530 | Deprecated |
Â