diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/index.md b/docs/integrations/data-ingestion/clickpipes/mysql/index.md index 02ba06cea64..d0966fc46e7 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/index.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/index.md @@ -36,7 +36,11 @@ To get started, you first need to make sure that your MySQL database is set up c 3. [Cloud SQL for MySQL](./mysql/source/gcp) -4. [Amazon RDS MariaDB](./mysql/source/rds_maria) +4. [Generic MySQL](./mysql/source/generic) + +5. [Amazon RDS MariaDB](./mysql/source/rds_maria) + +6. [Generic MariaDB](./mysql/source/generic_maria) Once your source MySQL database is set up, you can continue creating your ClickPipe. diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/source/generic.md b/docs/integrations/data-ingestion/clickpipes/mysql/source/generic.md new file mode 100644 index 00000000000..438cda38455 --- /dev/null +++ b/docs/integrations/data-ingestion/clickpipes/mysql/source/generic.md @@ -0,0 +1,139 @@ +--- +sidebar_label: 'Generic MySQL' +description: 'Set up any MySQL instance as a source for ClickPipes' +slug: /integrations/clickpipes/mysql/source/generic +title: 'Generic MySQL source setup guide' +--- + +# Generic MySQL source setup guide + +:::info + +If you use one of the supported providers (in the sidebar), please refer to the specific guide for that provider. + +::: + +## Enable binary log retention {#enable-binlog-retention} + +Binary logs contain information about data modifications made to a MySQL server instance and are required for replication. + +### MySQL 8.x and newer {#binlog-v8-x} + +To enable binary logging on your MySQL instance, ensure that the following settings are configured: + +```sql +log_bin = ON -- default value +binlog_format = ROW -- default value +binlog_row_image = FULL -- default value +binlog_row_metadata = FULL +binlog_expire_logs_seconds = 86400 -- 1 day or higher; default is 30 days +``` + +To check these settings, run the following SQL commands: +```sql +SHOW VARIABLES LIKE 'log_bin'; +SHOW VARIABLES LIKE 'binlog_format'; +SHOW VARIABLES LIKE 'binlog_row_image'; +SHOW VARIABLES LIKE 'binlog_row_metadata'; +SHOW VARIABLES LIKE 'binlog_expire_logs_seconds'; +``` + +If the values don't match, you can run the following SQL commands to set them: +```sql +SET PERSIST log_bin = ON; +SET PERSIST binlog_format = ROW; +SET PERSIST binlog_row_image = FULL; +SET PERSIST binlog_row_metadata = FULL; +SET PERSIST binlog_expire_logs_seconds = 86400; +``` + +If you have changed the `log_bin` setting, you NEED to RESTART the MySQL instance for the changes to take effect. + +After changing the settings, continue on with [configuring a database user](#configure-database-user). + +### MySQL 5.7 {#binlog-v5-x} + +To enable binary logging on your MySQL 5.7 instance, ensure that the following settings are configured: + +```sql +server_id = 1 -- or greater; anything but 0 +log_bin = ON +binlog_format = ROW -- default value +binlog_row_image = FULL -- default value +expire_logs_days = 1 -- or higher; 0 would mean logs are preserved forever +``` + +To check these settings, run the following SQL commands: +```sql +SHOW VARIABLES LIKE 'server_id'; +SHOW VARIABLES LIKE 'log_bin'; +SHOW VARIABLES LIKE 'binlog_format'; +SHOW VARIABLES LIKE 'binlog_row_image'; +SHOW VARIABLES LIKE 'expire_logs_days'; +``` + +If the values don't match, you can set them in the config file (typically at `/etc/my.cnf` or `/etc/mysql/my.cnf`): +```ini +[mysqld] +server_id = 1 +log_bin = ON +binlog_format = ROW +binlog_row_image = FULL +expire_logs_days = 1 +``` + +You NEED to RESTART the MySQL instance for the changes to take effect. + +:::note + +Column exclusion is not supported for MySQL 5.7 because the `binlog_row_metadata` setting wasn't yet introduced. + +::: + +## Configure a database user {#configure-database-user} + +Connect to your MySQL instance as the root user and execute the following commands: + +1. Create a dedicated user for ClickPipes: + + ```sql + CREATE USER 'clickpipes_user'@'%' IDENTIFIED BY 'some_secure_password'; + ``` + +2. Grant schema permissions. The following example shows permissions for the `clickpipes` database. Repeat these commands for each database and host you want to replicate: + + ```sql + GRANT SELECT ON `clickpipes`.* TO 'clickpipes_user'@'%'; + ``` + +3. Grant replication permissions to the user: + + ```sql + GRANT REPLICATION CLIENT ON *.* TO 'clickpipes_user'@'%'; + GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%'; + ``` + +:::note + +Make sure to replace `clickpipes_user` and `some_secure_password` with your desired username and password. + +::: + +## SSL/TLS configuration (recommended) {#ssl-tls-configuration} + +SSL certificates ensure secure connections to your MySQL database. Configuration depends on your certificate type: + +**Trusted Certificate Authority (DigiCert, Let's Encrypt, etc.)** - no additional configuration needed. + +**Internal Certificate Authority** - Obtain the root CA certificate file from your IT team. In the ClickPipes UI, upload it when creating a new MySQL ClickPipe. + +**Self-hosted MySQL** - Copy the CA certificate from your MySQL server (typically at `/var/lib/mysql/ca.pem`) and upload it in the UI when creating a new MySQL ClickPipe. Use the IP address of the server as the host. + +**Self-hosted MySQL without server access** - Contact your IT team for the certificate. As a last resort, use the "Skip Certificate Verification" toggle in ClickPipes UI (not recommended for security reasons). + +For more information on SSL/TLS options, check out our [FAQ](https://clickhouse.com/docs/integrations/clickpipes/mysql/faq#tls-certificate-validation-error). + +## What's next? {#whats-next} + +You can now [create your ClickPipe](../index.md) and start ingesting data from your MySQL instance into ClickHouse Cloud. +Make sure to note down the connection details you used while setting up your MySQL instance as you will need them during the ClickPipe creation process. \ No newline at end of file diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/source/generic_maria.md b/docs/integrations/data-ingestion/clickpipes/mysql/source/generic_maria.md new file mode 100644 index 00000000000..3cbdf150ca6 --- /dev/null +++ b/docs/integrations/data-ingestion/clickpipes/mysql/source/generic_maria.md @@ -0,0 +1,106 @@ +--- +sidebar_label: 'Generic MariaDB' +description: 'Set up any MariaDB instance as a source for ClickPipes' +slug: /integrations/clickpipes/mysql/source/generic_maria +title: 'Generic MariaDB source setup guide' +--- + +# Generic MariaDB source setup guide + +:::info + +If you use one of the supported providers (in the sidebar), please refer to the specific guide for that provider. + +::: + +## Enable binary log retention {#enable-binlog-retention} + +Binary logs contain information about data modifications made to a MariaDB server instance and are required for replication. + +To enable binary logging on your MariaDB instance, ensure that the following settings are configured: + +```sql +server_id = 1 -- or greater; anything but 0 +log_bin = ON +binlog_format = ROW +binlog_row_image = FULL +binlog_row_metadata = FULL -- introduced in 10.5.0 +expire_logs_days = 1 -- or higher; 0 would mean logs are preserved forever +``` + +To check these settings, run the following SQL commands: +```sql +SHOW VARIABLES LIKE 'server_id'; +SHOW VARIABLES LIKE 'log_bin'; +SHOW VARIABLES LIKE 'binlog_format'; +SHOW VARIABLES LIKE 'binlog_row_image'; +SHOW VARIABLES LIKE 'binlog_row_metadata'; +SHOW VARIABLES LIKE 'expire_logs_days'; +``` + +If the values don't match, you can set them in the config file (typically at `/etc/my.cnf` or `/etc/my.cnf.d/mariadb-server.cnf`): +```ini +[mysqld] +server_id = 1 +log_bin = ON +binlog_format = ROW +binlog_row_image = FULL +binlog_row_metadata = FULL ; only in 10.5.0 and newer +expire_logs_days = 1 +``` + +You NEED to RESTART the MariaDB instance for the changes to take effect. + +:::note + +Column exclusion is not supported for MariaDB \<= 10.4 because the `binlog_row_metadata` setting wasn't yet introduced. + +::: + +## Configure a database user {#configure-database-user} + +Connect to your MariaDB instance as the root user and execute the following commands: + +1. Create a dedicated user for ClickPipes: + + ```sql + CREATE USER 'clickpipes_user'@'%' IDENTIFIED BY 'some_secure_password'; + ``` + +2. Grant schema permissions. The following example shows permissions for the `clickpipes` database. Repeat these commands for each database and host you want to replicate: + + ```sql + GRANT SELECT ON `clickpipes`.* TO 'clickpipes_user'@'%'; + ``` + +3. Grant replication permissions to the user: + + ```sql + GRANT REPLICATION CLIENT ON *.* TO 'clickpipes_user'@'%'; + GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%'; + ``` + +:::note + +Make sure to replace `clickpipes_user` and `some_secure_password` with your desired username and password. + +::: + +## SSL/TLS configuration (recommended) {#ssl-tls-configuration} + +SSL certificates ensure secure connections to your MariaDB database. Configuration depends on your certificate type: + +**Trusted Certificate Authority (DigiCert, Let's Encrypt, etc.)** - no additional configuration needed. + +**Internal Certificate Authority** - Obtain the root CA certificate file from your IT team. In the ClickPipes UI, upload it when creating a new MariaDB ClickPipe. + +**Self-hosted MariaDB** - Copy the CA certificate from your MariaDB server (look up the path via the `ssl_ca` setting in your `my.cnf`). In the ClickPipes UI, upload it when creating a new MariaDB ClickPipe. Use the IP address of the server as the host. + +**Self-hosted MariaDB starting with 11.4** - If your server has `ssl_ca` set up, follow the option above. Otherwise, consult with your IT team to provision a proper certificate. As a last resort, use the "Skip Certificate Verification" toggle in ClickPipes UI (not recommended for security reasons). + +For more information on SSL/TLS options, check out our [FAQ](https://clickhouse.com/docs/integrations/clickpipes/mysql/faq#tls-certificate-validation-error). + +## What's next? {#whats-next} + +You can now [create your ClickPipe](../index.md) and start ingesting data from your MariaDB instance into ClickHouse Cloud. +Make sure to note down the connection details you used while setting up your MariaDB instance as you will need them during the ClickPipe creation process. \ No newline at end of file diff --git a/scripts/aspell-ignore/en/aspell-dict.txt b/scripts/aspell-ignore/en/aspell-dict.txt index bf82ebcb026..1fd2c2c4d4d 100644 --- a/scripts/aspell-ignore/en/aspell-dict.txt +++ b/scripts/aspell-ignore/en/aspell-dict.txt @@ -296,6 +296,7 @@ DeprecatedBadge DestroyAggregatesThreads DestroyAggregatesThreadsActive DictCacheRequests +DigiCert DiskAvailable DiskObjectStorage DiskObjectStorageAsyncThreads diff --git a/sidebars.js b/sidebars.js index cf1b9ab776e..49e13fd39c9 100644 --- a/sidebars.js +++ b/sidebars.js @@ -747,8 +747,10 @@ const sidebars = { items: [ "integrations/data-ingestion/clickpipes/mysql/source/rds", "integrations/data-ingestion/clickpipes/mysql/source/aurora", + "integrations/data-ingestion/clickpipes/mysql/source/gcp", + "integrations/data-ingestion/clickpipes/mysql/source/generic", "integrations/data-ingestion/clickpipes/mysql/source/rds_maria", - "integrations/data-ingestion/clickpipes/mysql/source/gcp" + "integrations/data-ingestion/clickpipes/mysql/source/generic_maria", ], }, "integrations/data-ingestion/clickpipes/mysql/datatypes"