{"id":1245,"date":"2019-07-16T10:16:06","date_gmt":"2019-07-16T08:16:06","guid":{"rendered":"https:\/\/bluemind.purpl-web.com\/?p=1245"},"modified":"2019-10-14T12:04:07","modified_gmt":"2019-10-14T10:04:07","slug":"monitoring-a-bluemind-install","status":"publish","type":"post","link":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/","title":{"rendered":"Monitoring a BlueMind install"},"content":{"rendered":"\n

When a system breaks downs or malfunctions, finding the cause can often be tricky. This is why BlueMind 3.5.9 integrates a full (open-source) platform metrics, alerting and viewing stack <\/p>\n\n\n\n

A BlueMind\nsystem — or any email server in general — is extremely complex. It is made up\nof many components with very specific yet sometimes obscure roles. When the\nsystem breaks down or malfunctions, it is often very difficult to find the\ncause. <\/p>\n\n\n\n

Email\nsystem breakdowns typically triggers a panic because it is a critical service\nfor most users. Emergency measures are taken, often involving system reboots which\n(sometimes) solves the problem but will make the post-mortem analysis\nparticularly difficult. <\/p>\n\n\n\n

What happens now<\/h1>\n\n\n\n

A BlueMind\nsystem, made up of many services and Java virtual machines, usually tries not to\ndie quietly.<\/p>\n\n\n\n

You will find:
\n\u2022 \/var\/log\/java_pid1234.hprof<\/em> files; memory dumps automatically\ngenerated when the system faces a memory shortage.
\n\u2022 Log files — many of them \u2013 where you will find very ugly things such as elastiscsearch\nindexes saying \u201cdisk high watermark reached, disabling shards\u201d i.e. the hard disk\nis full. <\/p>\n\n\n\n

In any\ncase, the issue isn\u2019t addressed immediately and a reboot or a disk clean-up are\nmade before any anomaly is reported for analysis. <\/p>\n\n\n\n

Letting a\nhard disk fill up when you\u2019re managing a system isn\u2019t a good idea. You\u2019ll end\nup with the following situation: the system administrator will clean up the\nhard disk and then report an anomaly indicating that the search doesn\u2019t work anymore.\n<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

This\nwhite lie may be an issue, but what we\u2019re really interested in is preventing\nthis from happening in the first place. <\/p>\n\n\n\n

Monitoring<\/h1>\n\n\n\n

Monitoring\nsystems do exist, but until now, they had to be deployed by the people monitoring\nthe BlueMind install. Monitoring Java systems is complicated and general\nmetrics are neither relevant nor revealing. <\/p>\n\n\n\n

With\nBlueMind 3.5.9, we\u2019ve integrated a full stack of metrics, alerting and platform\nstatus visualisation. <\/p>\n\n\n\n

Open\nsource being our raison d\u2019\u00eatre, we decided to turn to an open-source solution\nfor this: TICK stack (Telegraf, Influxdb, Chronograf, Kapacitor) http:\/\/influxdata.com<\/a>. A Telegraf agent is installed on each\nnode of your installation. Telegraf collects and sends metrics to Influxdb,\nKapacitor crunches them into actionable alerts and Chronograf displays the\ninformation to help anticipate problems.<\/p>\n\n\n\n

Our integration<\/h2>\n\n\n\n

With BlueMind 3.5.9, the whole stack is available but it isn\u2019t deployed by default. You have to install the bm-tick-full package on your server \u2013 your main server with bm-core or a dedicated monitoring server.<\/p>\n

The bm-tick-full package deploys the whole TICK stack \u2013 which is built into our repository to monitor the versions used. You must install bm-tick-node on every server of your BlueMind install. If you do this before updating to v.3.5.9, the update wizard will take care of everything. Otherwise, it is best to restart and force-reconfigure the entire stack from your bm-core server:<\/p>\n

curl -H “X-Bm-ApiKey: $(cat \/etc\/bm\/bm-core.tok)” -XPOST http:\/\/localhost:8090\/internal-api\/tick\/mgmt\/_reconfigure<\/p>\n

Java Agent<\/h1>\n

Some administrators might have noticed that BlueMind\u2019s Java processes run with an additional option:<\/p>\n

\/usr\/lib\/jvm\/bm-jdk\/bin\/java -Dlogback.configurationFile=\/etc\/bm\/local\/bm-core.log.xml -Dio.netty.native.workdir=\/var\/lib\/bm-core\/work -Xloggc:\/var\/log\/garbage-collector\/bm-core\/gc.pause.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=4 -XX:GCLogFileSize=4M -XX:+PrintGCApplicationStoppedTime -server -Xms812m -Xmx812m -Xss256k -XX:+UseCompressedOops -XX:MaxDirectMemorySize=812m -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=\/var\/log -Djava.net.preferIPv4Stack=true -Dnet.bluemind.property.product=bm-core -javaagent:\/var\/lib\/bm-metrics-agent\/bm-metrics-agent.jar -Djava.awt.headless=true -Dosgi.noShutdown=true -Duser.timezone=GMT -cp \/usr\/share\/bm-core\/plugins\/org.eclipse.equinox.launcher_1.3.201.v20161025-1711.jar org.eclipse.equinox.launcher.Main -registryMultiLanguage -debug -application net.bluemind.application.launcher.coreLauncher<\/p>\n

This agent, which is attached to your virtual machines, allows you to expose application metrics to Telegraf. What does it do?<\/p>\n

It opens one unix socket for every JVM in \/var\/run\/bm-metrics:<\/p>\n

\"\"<\/p>\n

Telegraf queries these unix sockets to get metrics for each component:<\/p>\n

curl –unix-socket \/var\/run\/bm-metrics\/metrics-bm-lmtpd.sock http:\/\/bm\/metrics<\/p>\n

Which returns — as a simplified version:<\/p>\n

\"\"<\/p>\n

This data is consumed by Telegraf every 10 seconds and\u2026<\/p>\n

Dashboards!<\/h1>\n

\"\"<\/p>\n

Here is an example of an email flow monitoring dashboard as it is pre-set in BlueMind 3.5.9.
But it can show many other metrics: system (disks, workload, network traffic), postgresql (QPS, shared buffers usage), nginx.<\/p>\n

Data history is collected over 7 days, so no more mysteries on the hard disk that was still full 2 hours before an incident report.<\/p>\n

Alerting<\/h1>\n

Knowing the number of active IMAP processes is good. Comparing it to the maximum set value is better.<\/p>\n

\"\"<\/p>\n

This metric is being monitored constantly by Kapacitor (the imap-connections script is configured by BlueMind) to compare the number of processes running at a given moment with the number set in BlueMind.<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

You can see these alerts on the monitoring\/tick\/ homepage of your BlueMind console using the installation wizard password.<\/p>\n

\"\"<\/p>\n

Annotations<\/h1>\n

With alerts come annotations which signal points in time (or timeframes) on your graphs.<\/p>\n

If an administrator restarts the Core, the monitoring system archives it and it is shown on all the graphs:

<\/p>\n

\"\"<\/p>\n

Programmed tasks running on your system are also shown on the graphs when they take longer than 30 seconds. That way it\u2019s easy to diagnose when system slow-downs reported by users are caused by a backup or a directory import. <\/p>\n

What\u2019s next<\/h1>\n

BlueMind records a large number of metrics but not all are used in graphs or alerting scripts yet. Dashboards and alerting scripts can be added to BlueMind via plugins and future versions should bring new ones and offer \u201cself-healing\u201d features (when a given problem arises, the solution is known and therefore applied automatically).<\/p>\n

BlueMind\u2019s full documentation is here<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

When a system breaks downs or malfunctions, finding the cause can often be tricky. This is why BlueMind 3.5.9 integrates a full (open-source) platform metrics, alerting and viewing stack A BlueMind system — or any email server in general — is extremely complex. It is made up of many components with very specific yet sometimes […]<\/p>\n","protected":false},"author":8,"featured_media":740,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[179,11,183],"tags":[36],"class_list":["post-1245","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-solution-en","category-usages-en","category-technical","tag-installation"],"acf":[],"yoast_head":"\nMonitoring a BlueMind install - Bluemind<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Monitoring a BlueMind install - Bluemind\" \/>\n<meta property=\"og:description\" content=\"When a system breaks downs or malfunctions, finding the cause can often be tricky. This is why BlueMind 3.5.9 integrates a full (open-source) platform metrics, alerting and viewing stack A BlueMind system — or any email server in general — is extremely complex. It is made up of many components with very specific yet sometimes […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/\" \/>\n<meta property=\"og:site_name\" content=\"Bluemind\" \/>\n<meta property=\"article:published_time\" content=\"2019-07-16T08:16:06+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-10-14T10:04:07+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2018\/07\/image-8.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1314\" \/>\n\t<meta property=\"og:image:height\" content=\"798\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/\"},\"author\":{\"name\":\"\",\"@id\":\"\"},\"headline\":\"Monitoring a BlueMind install\",\"datePublished\":\"2019-07-16T08:16:06+00:00\",\"dateModified\":\"2019-10-14T10:04:07+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/\"},\"wordCount\":964,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/#organization\"},\"image\":{\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2018\/07\/image-8.png\",\"keywords\":[\"installation\"],\"articleSection\":[\"BlueMind's Solution\",\"Practices\",\"Technical\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/\",\"url\":\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/\",\"name\":\"Monitoring a BlueMind install - Bluemind\",\"isPartOf\":{\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2018\/07\/image-8.png\",\"datePublished\":\"2019-07-16T08:16:06+00:00\",\"dateModified\":\"2019-10-14T10:04:07+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#primaryimage\",\"url\":\"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2018\/07\/image-8.png\",\"contentUrl\":\"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2018\/07\/image-8.png\",\"width\":\"1314\",\"height\":\"798\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/bluemind.purpl-web.com\/en\/homepage\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Monitoring a BlueMind install\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/#website\",\"url\":\"https:\/\/bluemind.purpl-web.com\/en\/\",\"name\":\"Bluemind\",\"description\":\"Collaborative messaging\",\"publisher\":{\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/bluemind.purpl-web.com\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/#organization\",\"name\":\"BlueMind\",\"url\":\"https:\/\/bluemind.purpl-web.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2017\/06\/logobm_300.png\",\"contentUrl\":\"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2017\/06\/logobm_300.png\",\"width\":300,\"height\":100,\"caption\":\"BlueMind\"},\"image\":{\"@id\":\"https:\/\/bluemind.purpl-web.com\/en\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Monitoring a BlueMind install - Bluemind","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/","og_locale":"en_US","og_type":"article","og_title":"Monitoring a BlueMind install - Bluemind","og_description":"When a system breaks downs or malfunctions, finding the cause can often be tricky. This is why BlueMind 3.5.9 integrates a full (open-source) platform metrics, alerting and viewing stack A BlueMind system — or any email server in general — is extremely complex. It is made up of many components with very specific yet sometimes […]","og_url":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/","og_site_name":"Bluemind","article_published_time":"2019-07-16T08:16:06+00:00","article_modified_time":"2019-10-14T10:04:07+00:00","og_image":[{"width":1314,"height":798,"url":"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2018\/07\/image-8.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Written by":"","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#article","isPartOf":{"@id":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/"},"author":{"name":"","@id":""},"headline":"Monitoring a BlueMind install","datePublished":"2019-07-16T08:16:06+00:00","dateModified":"2019-10-14T10:04:07+00:00","mainEntityOfPage":{"@id":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/"},"wordCount":964,"commentCount":0,"publisher":{"@id":"https:\/\/bluemind.purpl-web.com\/en\/#organization"},"image":{"@id":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#primaryimage"},"thumbnailUrl":"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2018\/07\/image-8.png","keywords":["installation"],"articleSection":["BlueMind's Solution","Practices","Technical"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/","url":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/","name":"Monitoring a BlueMind install - Bluemind","isPartOf":{"@id":"https:\/\/bluemind.purpl-web.com\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#primaryimage"},"image":{"@id":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#primaryimage"},"thumbnailUrl":"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2018\/07\/image-8.png","datePublished":"2019-07-16T08:16:06+00:00","dateModified":"2019-10-14T10:04:07+00:00","breadcrumb":{"@id":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#primaryimage","url":"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2018\/07\/image-8.png","contentUrl":"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2018\/07\/image-8.png","width":"1314","height":"798"},{"@type":"BreadcrumbList","@id":"https:\/\/bluemind.purpl-web.com\/en\/monitoring-a-bluemind-install\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/bluemind.purpl-web.com\/en\/homepage\/"},{"@type":"ListItem","position":2,"name":"Monitoring a BlueMind install"}]},{"@type":"WebSite","@id":"https:\/\/bluemind.purpl-web.com\/en\/#website","url":"https:\/\/bluemind.purpl-web.com\/en\/","name":"Bluemind","description":"Collaborative messaging","publisher":{"@id":"https:\/\/bluemind.purpl-web.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/bluemind.purpl-web.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/bluemind.purpl-web.com\/en\/#organization","name":"BlueMind","url":"https:\/\/bluemind.purpl-web.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/bluemind.purpl-web.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2017\/06\/logobm_300.png","contentUrl":"https:\/\/bluemind.purpl-web.com\/wp-content\/uploads\/2017\/06\/logobm_300.png","width":300,"height":100,"caption":"BlueMind"},"image":{"@id":"https:\/\/bluemind.purpl-web.com\/en\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":""}]}},"_links":{"self":[{"href":"https:\/\/bluemind.purpl-web.com\/en\/wp-json\/wp\/v2\/posts\/1245","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bluemind.purpl-web.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bluemind.purpl-web.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bluemind.purpl-web.com\/en\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/bluemind.purpl-web.com\/en\/wp-json\/wp\/v2\/comments?post=1245"}],"version-history":[{"count":1,"href":"https:\/\/bluemind.purpl-web.com\/en\/wp-json\/wp\/v2\/posts\/1245\/revisions"}],"predecessor-version":[{"id":1860,"href":"https:\/\/bluemind.purpl-web.com\/en\/wp-json\/wp\/v2\/posts\/1245\/revisions\/1860"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/bluemind.purpl-web.com\/en\/wp-json\/wp\/v2\/media\/740"}],"wp:attachment":[{"href":"https:\/\/bluemind.purpl-web.com\/en\/wp-json\/wp\/v2\/media?parent=1245"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bluemind.purpl-web.com\/en\/wp-json\/wp\/v2\/categories?post=1245"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bluemind.purpl-web.com\/en\/wp-json\/wp\/v2\/tags?post=1245"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}