About

Horgh replicator is a tool to merge data from any databases in one storage.

How it works?

• Listening MySQL master binary log
• Binlog events parsing
• Repeat master events in slave

Database support

Quick start

• Enable MySQL master binary logs in row format. Add to MySQL master configuration:
  server-id = 1
log-slave-updates
binlog-format = row
log-bin = /var/log/mysql/mysql-bin.log
binlog_do_db=<replicated_db>

• Restart MySQL master to apply changes: service mysql restart
• Clone Horgh replicator from repo: git clone https://github.com/larsnovikov/horgh-replicator
• Copy src/.env.dist to src/.env.
  Set MySQL master credentials to src/.env
• Create slave database and tables which will be replicated
• Set slave credentials to src/.env
  Don't forget to set SLAVE_TYPE. Available values: mysql, clickhouse, vertica, postgresql
• Start container in dev mode make start-dev
• In docker container create table configs as go run main.go create-model <table_name>
• Set <table_name> to TABLE in src/.env (One instance - one table)
• In docker container init current MySQL master state: go run main.go build-slave
• In docker container start replication: go run main.go listen
• Start replication in prod mode:
  - Build docker image: make build-prod
  - Start docker container: make start-prod

Custom handlers

If you need to handle some values of fields before save, you can use special handlers.

System handlers

• system/set_value - Set field value as first value from param section

How create handlers?

• Create plugins/user/<plugin_name>/handler.go like plugins/system/set_value/handler.go
• Execute go build -buildmode=plugin -o plugins/user/<plugin_name>/handler.so plugins/user/<plugin_name>/handler.go
• Add to field description in src/system/configs/<model>.json
"beforeSave": {
  "handler": "user/<plugin_name>",
  "params": [
    "***"
  ]
}


Tools

• set-position <table_name> <binlog_name> <binlog_position> set start position of log for table listener
• load start loader for replication testing (for default tables user and post)
• create-model <table_name> create model json-file by master table structure
• build-slave create master table dump, restore this dump in slave, set start position of log for table listener
• destroy-slave truncate table, set empty position of log for table listener

Contacts

Lars Novikov
Email: larsnovikov@yandex.ru
Telegram: larsnovikov