Integrate Telegraf with Tiger Cloud

Telegraf is a server-based agent that collects and sends metrics and events from databases, systems, and IoT sensors. Telegraf is an open-source, plugin-driven tool for the collection and output of data.

To view metrics gathered by Telegraf and stored in a hypertable in a Tiger Cloud service:

• Link Telegraf to your service: create a Telegraf configuration
• View the metrics collected by Telegraf: connect to your service and query the metrics table

Prerequisites

Best practice is to use an Ubuntu EC2 instance hosted in the same region as your Tiger Cloud service as a migration machine. That is, the machine you run the commands on to move your data from your source database to your target Tiger Cloud service.

Before you migrate your data:

• Create a target Tiger Cloud service.

  Each Tiger Cloud service has a single database that supports the most popular extensions. Tiger Cloud services do not support tablespaces, and there is no superuser associated with a service. Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window.

• To ensure that maintenance does not run during the process, adjust the maintenance window.

• Install Telegraf

Link Telegraf to your service

To create a Telegraf configuration that exports data to a hypertable in your service:

1. Set up your service connection string

   This variable holds the connection information for the target Tiger Cloud service.

   In the terminal on the source machine, set the following:

   export TARGET=postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require

   See where to find your connection details.

2. Generate a Telegraf configuration file

   In Terminal, run the following:

   telegraf --input-filter=cpu --output-filter=postgresql config > telegraf.conf

   telegraf.conf configures a CPU input plugin that samples various metrics about CPU usage, and the PostgreSQL output plugin. telegraf.conf also includes all available input, output, processor, and aggregator plugins. These are commented out by default.

3. Test the configuration

   telegraf --config telegraf.conf --test

   You see an output similar to the following:

   2022-11-28T12:53:44Z I! Starting Telegraf 1.24.3
   2022-11-28T12:53:44Z I! Available plugins: 208 inputs, 9 aggregators, 26 processors, 20 parsers, 57 outputs
   2022-11-28T12:53:44Z I! Loaded inputs: cpu
   2022-11-28T12:53:44Z I! Loaded aggregators:
   2022-11-28T12:53:44Z I! Loaded processors:
   2022-11-28T12:53:44Z W! Outputs are not used in testing mode!
   2022-11-28T12:53:44Z I! Tags enabled: host=localhost
   > cpu,cpu=cpu0,host=localhost usage_guest=0,usage_guest_nice=0,usage_idle=90.00000000087311,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=6.000000000040018,usage_user=3.999999999996362 1669640025000000000
   > cpu,cpu=cpu1,host=localhost usage_guest=0,usage_guest_nice=0,usage_idle=92.15686274495818,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=5.882352941192206,usage_user=1.9607843136712912 1669640025000000000
   > cpu,cpu=cpu2,host=localhost usage_guest=0,usage_guest_nice=0,usage_idle=91.99999999982538,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=3.999999999996362,usage_user=3.999999999996362 1669640025000000000

4. Configure the PostgreSQL output plugin

   1. In telegraf.conf, in the [[outputs.postgresql]] section, set connection to the value of $TARGET.

      connection = "<VALUE OF $TARGET>"

   2. Use hypertables when Telegraf creates a new table:

      In the section that begins with the comment ## Templated statements to execute when creating a new table, add the following template:

      ## Templated statements to execute when creating a new table.
      create_templates = [
        '''CREATE TABLE IF NOT EXISTS {{ .table }} ({{ .columns }}) WITH (tsdb.hypertable, tsdb.chunk_interval='1 week')''',
      ]

View the metrics collected by Telegraf

This section shows you how to generate system metrics using Telegraf, then connect to your service and query the metrics hypertable.

1. Collect system metrics using Telegraf

   Run the following command for 30 seconds:

   telegraf --config telegraf.conf

   Telegraf uses loaded inputs cpu and outputs postgresql along with global tags, the intervals when the agent collects data from the inputs, and flushes to the outputs.

2. View the metrics

   1. Connect to your Tiger Cloud service:

       psql $TARGET

   2. View the metrics collected in the cpu table in tsdb:

      SELECT*FROM cpu;

      You see something like:

      time         |    cpu    |               host               | usage_guest | usage_guest_nice |    usage_idle     | usage_iowait | usage_irq | usage_nice | usage_softirq | usage_steal |    usage_system     |     usage_user
      ---------------------+-----------+----------------------------------+-------------+------------------+-------------------+--------------+-----------+------------+---------------+-------------+---------------------+---------------------
      2022-12-05 12:25:20 | cpu0      | hostname |           0 |                0 | 83.08605341237833 |            0 |         0 |          0 |             0 |           0 |   6.824925815961274 |  10.089020771444481
      2022-12-05 12:25:20 | cpu1      | hostname |           0 |                0 | 84.27299703278959 |            0 |         0 |          0 |             0 |           0 |   5.934718100814769 |   9.792284866395647
      2022-12-05 12:25:20 | cpu2      | hostname |           0 |                0 | 87.53709198848934 |            0 |         0 |          0 |             0 |           0 |   4.747774480755411 |   7.715133531241037
      2022-12-05 12:25:20 | cpu3      | hostname|           0 |                0 | 86.68639053296472 |            0 |         0 |          0 |             0 |           0 |    4.43786982253345 |   8.875739645039992
      2022-12-05 12:25:20 | cpu4      | hostname |           0 |                0 | 96.15384615371369 |            0 |         0 |          0 |             0 |           0 |  1.1834319526667423 |  2.6627218934917614

      To view the average usage per CPU core, use SELECT cpu, avg(usage_user) FROM cpu GROUP BY cpu;.

For more information about the options that you can configure in Telegraf, see the PostgreSQL output plugin.