Sensu Event Handlers

Reference documentation

What is a Sensu event handler?

Sensu event handlers are actions executed by the Sensu server on events, such as sending an email alert, creating or resolving an incident (e.g. in PagerDuty, ServiceNow, etc), or storing metrics in a time-series database (e.g. Graphite).

Handler types

There are several types of handlers. The most common handler type is the pipe handler, which works very similarly to how checks work, enabling Sensu to interact with almost any computer program via standard streams.

  • Pipe handlers. Pipe handlers pipe event data into arbitrary commands via STDIN.
  • TCP/UDP handlers. TCP and UDP handlers send event data to a remote socket (e.g. external API).
  • Transport handlers. Transport handlers publish event data to the Sensu transport.
  • Handler sets. Handler sets (also called “set handlers”) are used to group event handlers, making it easy to manage groups of actions that should be executed for certain types of events.

The default handler

Sensu expects all events to have a corresponding handler. Event handler(s) may be configured in check definitions, however if no handler or handlers have been configured, Sensu will attempt to handle the event using a handler named default. The default handler is only a reference (i.e. Sensu does not provide a built-in default handler), so if no handler definition exists for a handler named default, Sensu will log an error indicating that the event was not handled because a default handler definition does not exist. To use one or more existing handlers as the default, you can create a Set handler called default and include the existing handler(s) in the set.

Pipe handlers

Pipe handlers are external commands that can consume event data via STDIN.

Example pipe handler definition

{
  "handlers": {
    "example_pipe_handler": {
      "type": "pipe",
      "command": "do_something_awesome.rb -o options"
    }
  }
}

Pipe handler commands

What is a pipe handler command?

Pipe handler definitions include a command attribute which are literally executable commands which will be executed on a Sensu server as the sensu user.

Pipe handler command arguments

Pipe handler command attributes may include command line arguments for controlling the behavior of the command executable. Most Sensu handler plugins provide support for command line arguments for reusability.

How and where are pipe handler commands executed?

As mentioned above, all pipe handlers are executed by a Sensu server as the sensu user. Commands must be executable files that are discoverable on the Sensu server system (i.e. installed in a system $PATH directory).

NOTE: By default, the Sensu installer packages will modify the system $PATH for the Sensu processes to include /etc/sensu/plugins. As a result, executable scripts (e.g. plugins) located in /etc/sensu/plugins will be valid commands. This allows command attributes to use “relative paths” for Sensu plugin commands;

e.g.: "command": "handler-irc.rb"

TCP/UDP handlers

TCP and UDP handlers enable Sensu to forward event data to arbitrary [TCP or UDP sockets][t] for external services to consume (e.g. third-party APIs).

Example TCP handler definition

The following example TCP handler definition will forward event data to a TCP socket (i.e. 10.0.1.99:4444) and will timeout if an acknowledgement (ACK) is not received within 30 seconds.

{
  "handlers": {
    "example_tcp_handler": {
      "type": "tcp",
      "timeout": 30,
      "socket": {
        "host": "10.0.1.99",
        "port": 4444
      }
    }
  }
}

The following example UDP handler definition will forward event data to a UDP socket (i.e. 10.0.1.99:444).

{
  "handlers": {
    "example_udp_handler": {
      "type": "udp",
      "socket": {
        "host": "10.0.1.99",
        "port": 4444
      }
    }
  }
}

Transport handlers

Transport handlers enable Sensu to publish event data to named queues on the Sensu transport for external services to consume.

Example transport handler definition

The following example transport handler definition will publish event data to the Sensu transport on a pipe (e.g. a “queue” or “channel”, etc) named example_handler_queue. One or more instances of an external process or third-party application would need to subscribe to the named pipe to process the events.

{
  "handlers": {
    "example_transport_handler": {
      "type": "transport",
      "pipe": {
        "type": "direct",
        "name": "example_handler_queue"
      }
    }
  }
}

Handler sets

Handler set definitions allow groups of handlers (i.e. individual collections of actions to take on event data) to be referenced via a single named handler set.

NOTE: Attributes defined on handler sets do not apply to the handlers they include. For example, filter, filters, and mutator attributes defined in a handler set will have no effect.

Example handler set definition

The following example handler set definition will execute three handlers (i.e. email, slack, and pagerduty) for every event.

{
  "handlers": {
    "notify_all_the_things": {
      "type": "set",
      "handlers": [
        "email",
        "slack",
        "pagerduty"
      ]
    }
  }
}

Handler configuration

Example handler definition

The following is an example Sensu handler definition, a JSON configuration file located at /etc/sensu/conf.d/mail_handler.json. This handler definition uses the mailx unix command, to email the event data to example@address.com, with the email subject sensu event. The handler is named mail.

{
  "handlers": {
    "mail": {
      "type": "pipe",
      "command": "mailx -s 'sensu event' example@address.com"
    }
  }
}

Handler definition specification

Handler name(s)

Each handler definition has a unique handler name, used for the definition key. Every handler definition is within the "handlers": {} configuration scope.

  • A unique string used to name/identify the check
  • Cannot contain special characters or spaces
  • Validated with Ruby regex /^[\w\.-]+$/.match("handler-name")

HANDLER attributes

The following attributes are configured within the {"handlers": { "HANDLER": {} } } configuration scope (where HANDLER is a valid handler name).

type
description
The handler type.
required
true
type
String
allowed values
pipe, tcp, udp, transport, set
example
"type": "pipe"
filter
description
The Sensu event filter (name) to use when filtering events for the handler.
required
false
type
String
example
"filter": "recurrence"
filters
description
An array of Sensu event filters (names) to use when filtering events for the handler. Each array item must be a string.
required
false
type
Array
example
"filters": ["recurrence", "production"]
severities
description
An array of check result severities the handler will handle. NOTE: event resolution bypasses this filtering. NOTE: specifying “ok” in handler severities will not cause the handler to recieve “ok” results from standard checks.
required
false
type
Array
allowed values
ok, warning, critical, unknown
example
"severities": ["critical", "unknown"]
mutator
description
The Sensu event mutator (name) to use to mutate event data for the handler.
required
false
type
String
example
"mutator": "only_check_output"
timeout
description
The handler execution duration timeout in seconds (hard stop). Only used by pipe and tcp handler types.
required
false
type
Integer
default
10
example
"timeout": 30
handle_silenced
description
If events matching one or more silence entries should be handled.
required
false
type
Boolean
default
false
example
"handle_silenced": true
handle_flapping
description
If events in the flapping state should be handled.
required
false
type
Boolean
default
false
example
"handle_flapping": true
command
description
The handler command to be executed. The event data is passed to the process via STDIN. NOTE: the command attribute is only supported for Pipe handlers (i.e. handlers configured with "type": "pipe").
required
true (if type == pipe)
type
String
example
"command": "/etc/sensu/plugins/pagerduty.rb"
socket
description
The socket definition scope, used to configure the TCP/UDP handler socket. NOTE: the socket attribute is only supported for TCP/UDP handlers (i.e. handlers configured with "type": "tcp" or "type": "udp").
required
true (if type == tcp or udp)
type
Hash
example
"socket": {}
pipe
description
The pipe definition scope, used to configure the Sensu transport pipe. NOTE: the pipe attribute is only supported for Transport handlers (i.e. handlers configured with "type": "transport").
required
true (if type == transport)
type
Hash
example
"pipe": {}
handlers
description
An array of Sensu event handlers (names) to use for events using the handler set. Each array item must be a string. NOTE: the handlers attribute is only supported for handler sets (i.e. handlers configured with "type": "set").
required
true (if type == set)
type
Array
example
"handlers": ["pagerduty", "email", "ec2"]

socket attributes

The following attributes are configured within the {"handlers": { "HANDLER": { "socket": {} } } } configuration scope (where HANDLER is a valid handler name).

NOTE: socket attributes are only supported for TCP/UDP handlers (i.e. handlers configured with "type": "tcp" or "type": "udp").

EXAMPLE
{
  "handlers": {
    "example_handler": {
      "type": "tcp",
      "socket": {
        "host": "10.0.5.100",
        "port": 8000
      }
    }
  }
}
ATTRIBUTES
host
description
The socket host address (IP or hostname) to connect to.
required
true
type
String
example
"host": "8.8.8.8"
port
description
The socket port to connect to.
required
true
type
Integer
example
"port": 4242

pipe attributes

The following attributes are configured within the {"handlers": { "HANDLER": { "pipe": {} } } } configuration scope (where HANDLER is a valid handler name).

NOTE: pipe attributes are only supported for Transport handlers (i.e. handlers configured with "type": "transport").

EXAMPLE
{
  "handlers": {
    "example_handler": {
      "type": "transport",
      "pipe": {
        "type": "topic",
        "name": "example_transport_handler"
      }
    }
  }
}
ATTRIBUTES
type
description
The Sensu transport pipe type.
required
true
type
String
allowed values
direct, fanout, topic
example
"type": "direct"

NOTE: types direct, fanout and topic are supported by the default RabbitMQ transport. Redis and other transports may only implement a subset of these.

name
description
The Sensu transport pipe name.
required
true
type
String
example
"name": "graphite_plaintext"
options
description
The Sensu transport pipe options. These options may be specific to the Sensu transport in use.
required
false
type
Hash
default
{}
example
"options": {"durable": true}

Start your FREE trial today.

Already using Sensu Core? Upgrade to Sensu Enterprise today to take advantage of its enterprise console, added-value features, built-in integrations, FREE annual training, and enterprise-class support.