cmd/bosun: major notification overhaul

[captncraig]

Makes notifications more individually configurable.

BREAKING CHANGES:

Notifications no longer support body inline template, or useBody flag. Instead, you can supply bodyTemplate to select any pre-rendered template key on the alert's template:

Old:

notification n {
  post = https://whatever.com/alert
  body = {"text": "{{.}}"}
}
template t {
 subject = something broke
 body = real bad
}
alert a {
 template = t
 critNotification = n
 ...
}

New:

notification n {
  post = https://whatever.com/alert
  bodyTemplate = postData
}
template t {
 subject = something broke
 body = real bad
 postData = {"name": "{{.Subject}}"}
}
alert a {
 template = t
 critNotification = n
 ...
}

Templates can now include arbitrarily named keys (not just subject and body anymore), and they will all be pre-rendered when the alert fires, with the full context of the alert.

Features

1. Can add arbitrary keys to templates, as explained above.

2. Templates can "inherit" another template, essentially copying all of their keys/values that are not already defined:

    template postToChat {
        postData = {"text": "{{.Subject}}", "context": "{{.Alert.Name}}"}
    }
    template t {
      inherit = postToChat
      subject = Something bad happened
    }
   

3. Notifications now support getTemplate and postTemplate to select a template to use for building urls. They have bodyTemplate to select a template to use for post or email body (default is body) and emailSubjectTemplate to select a template to use for the email subject.

[kylebrandt]

Notifications preview needs some formatting love:

Table line up, space between status code and result number.

Edit: Sorry, the stupid just happened to the name of my template since the alert was "stupid", so don't read into that :-P

[kylebrandt]

Since this is meant to build JSON, should be a map[string]interface or it won't marshal into JSON

[kylebrandt]

Need to think, but avoid return errors here because we don't wan't to completely break the template. See the docs at http://bosun.org/definitions#template-error-handling

Somewhat related, we might need an "error" template type for fallback with templates error... same case with JSON.

[kylebrandt]

When I set the content type with the following:

$fakeServiceKey = 1234

alert test {
    template = test
    warn = avg(series("host=server01", epoch(), 1))
    warnNotification = fakePDuty
}

template baseFakePDuty {
    pduty = `{{ makeMap "service_key" (V "$fakeServiceKey") "incident_key" .Id  "event_type" "trigger" "description" .Subject "details" (.Get "details")| json }}`
}

template test {
    inherit = baseFakePDuty
    pduty = `{{ .Set "details" (makeMap "host" .Group.host) }}{{ template "baseFakePDuty" . }}`
    subject = `test`
    body = `test`
    
}

notification fakePDuty {
    post = http://localhost:9998/
    bodyTemplate = pduty
    contentType = application/json
}

I see the content type displayed in the the UI, but when I test the notification I don't see the Content-Type header set with using httputil.DumpRequest(r, true)

[captncraig]

Oops, wasn't setting headers. fixed.