# System Monitor

Starting in Mitto 2.9, users can add a `system` job that monitors
system resources including memory, disk space and services.

To create this job, click on the `Add Job` button on the bottom left
of the Zuar Runner UI, and select "Generic Job".

Enter a title like: `[system] monitor memory and disk`, under `type`
select "System".


Use the following config:

```json
{
    // job type should be `system`
    // writes to a file named `system_status.jsonl`
    "json_file": "system_status",
    "max_disk": 90,
    "max_memory": 90,
    "write_json": "true"
}
```

The system job will fail if the resources on the machine are above the
set thresholds, or if any of the system's services are down.

If `write_json` is set to true, every time the system job runs it will
write JSON data to a JSON lines file defined by `json_file`. This file
will include the amount of disk and memory used, as well as any jobs
that were running at the time.

`max_memory` and `max_disk` set the thresholds at which the system job
will fail for memory and disk usage. The default setting for each is
90 %.

A common practice for using system jobs is to set up a webhook on job
failure to receive notifications when the resources on the box are
failing.

To create a webhook, click the little orange edit pencil next to
`Webhooks`. Under `Event` select "JOB_FAILURE". Enter the URL for the
webhook and the JSON to send to it.

To test the webhook, set the "max_disk" to a number below the current disk
usage found in the UI on the Settings Tab. Ensure that the Event selected
is "JOB_FAILURE" and click the Test link in the Edit Webhook modal. Remember
to reset the "max_disk" to 90 after the test.

Below is an example config that works for webhooks sent to a slack
channel, including the current memory and disk usage and the URL of
Zuar Runner:

```json
{
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": ":rotating_light: Mitto Job Failed!"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "Mitto URL: https://${system['fqdn']}\nJob Title: ${job['title']}\nJob URL: https://${system['fqdn']}/#!/job/${job['id']}"
      }
    }
  ]
}
```

The json above will send a notification similar to this to the specified Slack channel upon job failure:

.. image:: assets/mitto_systems_monitor_slack.png

For more example job configs including configs for pulling the data out of
the JSON lines file, see the [Zuar Runner job templates
repo](https://github.com/zuarbase/mitto-job-templates/tree/master/system).