Implementing Virtual Tables

Introduction

There are two types of table defined within Zeek Agent:

1. Built-in, implemented in the source code
2. Automatically imported on startup
   • from osquery core
   • from osquery extensions

This small tutorial will focus on implementing a new built-in table. For the other method, please refer to the official osquery documentation and examples folder in the source repository.

The Zeek Agent Database SDK

The SDK can be found in the components/zeekdatabase folder of the Zeek Agent source code. This library exports a new include folder when linked against another target, making the ivirtualdatabase.h and ivirtualtable.h headers available.

The Zeek Agent executable already links against this library, so the required includes are already reachable by the main project. If we were to implement the table inside a new component, we would have add target_link_libraries(our_target PUBLIC zeek_database) to our CMakeLists.txt file.

At the time of writing, the IVirtualTable interface declaration reads as follows:

class IVirtualTable {
public:
  using Ref = std::shared_ptr<IVirtualTable>;

  using Variant = std::variant<std::int64_t, std::string, double>;
  using OptionalVariant = std::optional<Variant>;

  using Row = std::map<std::string, OptionalVariant>;
  using RowList = std::vector<Row>;

  enum class ColumnType { Integer, String, Double };
  using Schema = std::map<std::string, ColumnType>;

  virtual ~IVirtualTable() = default;
  IVirtualTable() = default;

  virtual const std::string &name() const = 0;
  virtual const Schema &schema() const = 0;
  virtual Status generateRowList(RowList &row_list) = 0;

  IVirtualTable(const IVirtualTable &other) = delete;
  IVirtualTable &operator=(const IVirtualTable &other) = delete;
};

Method	Description
const std::string &name()	This method must return a unique table name. Standard SQL rules apply; it should not contain any space and start with a letter.
const Schema &schema()	The schema is used to create the actual table. Each item in this map defines a single column of either one of the following types: ColumnType::Integer, ColumnType::String, ColumnType::Double
Status generateRowList(RowList &row_list)	This method is invoked every time the table is hit with a query by the Zeek instance. It contains a list of rows which in turn are a list of column values. Each value is declared using an optional variant value, in order to support NULL fields (note that this however is not yet supported by Zeek).

Declaring a new table

In order to declare a new table, we need to declare a new class that inherits from IVirtualTable:

#include <atomic>
#include <zeek/ivirtualtable.h>

namespace zeek {
class ExampleTable final : public IVirtualTable {
public:
  ExampleTable() = default;
  virtual ~ExampleTable() = default;

  virtual const std::string &name() const override {
    static const std::string kTableName{"example"};
    return kTableName;
  }

  virtual const Schema &schema() const override {
    static const Schema kTableSchema = {
      { "string_column", IVirtualTable::ColumnType::String },
      { "integer_column", IVirtualTable::ColumnType::Integer },
      { "double_column", IVirtualTable::ColumnType::Double }
    };

    return kTableSchema;
  }

  virtual Status generateRowList(RowList &row_list) override;
};
}

It is important to always populate all columns; values that are supposed to be NULL can be left empty by assigning {}.

Status ExampleTable::generateRowList(RowList &row_list) {
  row_list = {};

  static std::atomic_int64_t counter{0};

  for (auto i = 0U; i < 5U; ++i) {
    auto counter_value = ++counter;

    Row row = {};
    row["string_column"] = std::to_string(counter_value);
    row["integer_column"] = counter_value;

    if (i == 3U) {
      row["double_column"] = 1.0;
    } else {
      row["double_column"] = {};
    }

    row_list.push_back(std::move(row));
  }

  return Status::success();
}

It is considered bad practice to block inside the row generation callback; in case the process of generating new items is time consuming, it is best to do the work inside another thread. A mutex can then be used to empty the row list at query time when the callback is invoked.

Registering the new table

Table plugins can be activated after the IVirtualDatabase object has been initialized. In the Zeek Agent, this is performed inside the ZeekAgent class constructor.

Registering the table is easy:

IVirtualTable::Ref example_table;

example_table.reset(new ExampleTable());
virtual_database->registerTable(example_table);

Tables (especially if they rely on external services) should always be unregistered before the database object is released:

virtual_database->unregisterTable(example_table->name());