Dolt Unable to Resolve Default Branch Head Error

[lihh1992]

Problem Description:
When running the dolt sql command, the following error occurs: "cannot resolve default branch head for database 'insight_metabase': 'main'"

Environment Information:

• Dolt Version: 1.76.2
• Database Name: insight_metabase
• Default Branch: main

root@dolt-0:/var/lib/dolt# dolt sql
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit. "\help" for help.
> exit
Bye

root@dolt-0:/var/lib/dolt# cd/db/dolt
root@dolt-0:/db/dolt# dolt sq 
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'
# "exit" or "quit" (or Ctrl-D) to exit. "\help" for help.
Error 1105 (HY000): cannot resolve default branch head for database 'insight_metabase': 'main

root@dolt-0:/db/dolt# dolt version
dolt version 1.76.2
Warning: unable to query latest released Dolt version
root@dolt-0:/db/dolt#

[timsehn]

We may need a little bit more information about how you got in this state.

For instance, both of these commands seem ill-formed:

root@dolt-0:/var/lib/dolt# cd/db/dolt
root@dolt-0:/db/dolt# dolt sq 

Maybe a copy/paste error?

You have a database that works fine at /var/lib/dolt and when you cd into /db/dolt from that directory dolt sql can't find the default branch head? Is there also a .dolt directory there?

Is there a dolt sql-server running for the /db/dolt database?

Do other commands other than dolt sql work in /db/dolt? For instance, dolt ls?

[lihh1992]

We may need a little bit more information about how you got in this state.

For instance, both of these commands seem ill-formed:

root@dolt-0:/var/lib/dolt# cd/db/dolt
root@dolt-0:/db/dolt# dolt sq 

Maybe a copy/paste error?

You have a database that works fine at /var/lib/dolt and when you cd into /db/dolt from that directory dolt sql can't find the default branch head? Is there also a .dolt directory there?

Is there a dolt sql-server running for the /db/dolt database?

Do other commands other than dolt sql work in /db/dolt? For instance, dolt ls?

I'm sorry, this is text that was OCRed from an image, and there are some missing parts.

However, this issue recurred today in another environment, and this time I was able to retrieve a complete log.

I am using the docker image dolthub/dolt-sql-server:1.76.7.

root@dolt-0:/var/lib/dolt# cd /db/dolt/

root@dolt-0:/db/dolt# dolt sql
Warning: Unknown global config option 'sqlserver.cluster.dolt_cluster_role_epoch'. Use `dolt config --global --unset sqlserver.cluster.dolt_cluster_role_epoch` to remove.
Warning: Unknown global config option 'sqlserver.cluster.dolt_cluster_role'. Use `dolt config --global --unset sqlserver.cluster.dolt_cluster_role` to remove.
# Welcome to the DoltSQL shell.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit. "\help" for help.
Error 1105 (HY000): cannot resolve default branch head for database 'insight_metabase': 'main'

root@dolt-0:/db/dolt# dolt ls
Warning: Unknown global config option 'sqlserver.cluster.dolt_cluster_role'. Use `dolt config --global --unset sqlserver.cluster.dolt_cluster_role` to remove.
Warning: Unknown global config option 'sqlserver.cluster.dolt_cluster_role_epoch'. Use `dolt config --global --unset sqlserver.cluster.dolt_cluster_role_epoch` to remove.
The current directory is not a valid dolt repository.
run: dolt init before trying to run this command

root@dolt-0:/db/dolt# cd insight_metabase/

root@dolt-0:/db/dolt/insight_metabase# dolt ls
Warning: Unknown global config option 'sqlserver.cluster.dolt_cluster_role'. Use `dolt config --global --unset sqlserver.cluster.dolt_cluster_role` to remove.
Warning: Unknown global config option 'sqlserver.cluster.dolt_cluster_role_epoch'. Use `dolt config --global --unset sqlserver.cluster.dolt_cluster_role_epoch` to remove.
Error 1105 (HY000): cannot resolve default branch head for database 'insight_metabase': 'main'

root@dolt-0:/db/dolt/insight_metabase# dolt version
Warning: Unknown global config option 'sqlserver.cluster.dolt_cluster_role'. Use `dolt config --global --unset sqlserver.cluster.dolt_cluster_role` to remove.
Warning: Unknown global config option 'sqlserver.cluster.dolt_cluster_role_epoch'. Use `dolt config --global --unset sqlserver.cluster.dolt_cluster_role_epoch` to remove.
WARN[0000] failed to load database with error: error bootstrapping chunk journal: invalid root hash record size at 0
dolt version 1.76.7

root@dolt-0:/var/lib/dolt# ls -alh
total 8.0K
drwxr-xr-x 2 root root 4.0K Nov 13 08:13 .
drwxr-xr-x 1 root root 4.0K Nov 13 08:13 ..
root@dolt-0:/var/lib/dolt# cd /db/dolt

root@dolt-0:/db/dolt# ls -alh
total 32K
drwxrwxrwx 7 root root 4.0K Oct 22 10:42 .
drwxr-xr-x 3 root root 4.0K Nov 18 02:50 ..
drwxr-xr-x 4 root root 4.0K Nov 18 02:50 .dolt
drwxr-xr-x 3 root root 4.0K Oct 22 10:41 .dolt_dropped_databases
-rw-r--r-- 1 root root 3.1K Nov 17 13:54 .sqlhistory
drwxr-xr-x 3 root root 4.0K Sep 29 16:48 insight_jobadmin
drwxr-xr-x 3 root root 4.0K Oct 22 10:53 insight_metabase
drwxr-xr-x 3 root root 4.0K Sep 29 16:48 insight_system

root@dolt-0:/db/dolt# cd .dolt

root@dolt-0:/db/dolt/.dolt# ls -alh
total 88K
drwxr-xr-x 4 root root 4.0K Nov 18 02:50 .
drwxrwxrwx 7 root root 4.0K Oct 22 10:42 ..
drwxr-xr-x 3 root root 4.0K Oct 27 11:46 .dolt
-rw------- 1 root root   43 Nov 18 02:50 sql-server.info
drwxr-xr-x 2 root root  68K Nov 18 02:50 tmp

root@dolt-0:/db/dolt/.dolt# cat sql-server.info
1:6033:0f47b9d5-fd73-4010-b204-6b189c9faeed

root@dolt-0:/db/dolt/.dolt#

config

root@dolt-1:/db/dolt# cat /etc/dolt/config/config.yaml
data_dir: /db/dolt
cfg_dir: /etc/dolt/config
log_level: info
log_format: text
behavior:
  read_only: false
  autocommit: true
  disable_client_multi_statements: false
  dolt_transaction_commit: false
  event_scheduler: "ON"
  auto_gc_behavior:
    enable: true
    archive_level: 1
listener:
  host: 0.0.0.0
  port: 6033
  max_connections: 500
  back_log: 50
  max_connections_timeout_millis: 60000
  read_timeout_millis: 28800000
  write_timeout_millis: 28800000
  tls_key: null
  tls_cert: null
  require_secure_transport: null
  allow_cleartext_passwords: null

max_logged_query_len: 0

# Advanced Configuration
metrics:
  labels: {}
  host: null
  port: -1

system_variables: {}

user_session_vars: []

jwks: []
cluster:
  standby_remotes:
    - name: standby
      remote_url_template: http://dolt-0.dolt-headless.git.svc.cluster.local:50051/{database}
  bootstrap_role: standby
  bootstrap_epoch: 1
  remotesapi:
    port: 50051

Potentially helpful troubleshooting information is that the issue occurred on dolt-0, which is the primary node, but it did not occur on dolt-1.

Is it possible that the container was killed by the operating system, resulting in incomplete or corrupted data? If so, what are the methods for recovery?

k get pod
dolt-0                               1/1     Running            0          7h42m
dolt-1                               1/1     Running            0          3d15h

[timsehn]

@macneale4 and @reltuk will dig in here.

[macneale4]

The piece that looks suspicious to me is that the .dolt_dropped_databases and insight_metabase directories have similar dates. Has this worked at all since 10/22? Also, is insight_metabase a newly created database, or did you call dolt_undrop() to restore it? Knowing the last operations on the DB would help a lot to determine how this happened.

As far as making this database work, we'll need to have a copy of the database to dissect it. The error stating that there is an invalid root hash record size at 0 is pretty bad. Can you share the database with us? I understand if that's not possible for data sensitivity reasons. If we can look at the bytes at the beginning of the journal file that would help too. The following command will show us the hex for the first 128 bytes:

od -x -N 128 /db/dolt/insight_metabase/.dolt/noms/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv

That will at least tell us if we are looking at a file full of null bytes or a file full of something else.

Finally, if you're willing, direct messages on discord can help speed the diagnosis of the problem. Here is the link to our server: https://discord.com/invite/RnNVBHrsJg, my handle is macneale

[reltuk]

@lihh1992 Based on the log messages above, it does appear that the database entered an unexpected state due to the state of the files on disk.

If dolt-1 has a current replicated copy of the database, the first step is to take a backup. On dolt-1, you can run:

use insight_metabase;
call dolt_backup('sync-url', 'file:///absolute_path_to_some_non_existant_directory_on_the_pd_the_databases_are_stored_on');

The target should be a non-existent directory which is not a subdirectory of the database data directory.

If the only appropriate mounted PD is mounted basically at the server root, it is also fine to backup to a hidden directory in that path:

use insight_metabase;
call dolt_backup('sync-url', '/db/dolt/.insight_metabase_backup_20251118');

After the database is backed up, one way to recover the database is to use cluster replication by promoting dolt-1 to the primary and recovering dolt-0 so that it runs and accepts writes.

On dolt-1, run:

CALL dolt_assume_cluster_role('primary', 2);

Note that the 2 should be higher than the current dolt_cluster_role_epoch on either dolt-0 or dolt-1.

This will promote dolt-1 to primary.

To make dolt-0 run again, move the existing broken insight_metabase database out of the way:

cd /db/dolt
mv insight_metabase .insight_metabase_fail_to_load_20251118

And then start the server as normal. dolt-0 will come up and will receive replication traffic from dolt-1. It will transition to standby and eventually insight_metabase will be fully replicated. After that is done, if you want to return to the previous state of primary and standby, you can transition it back to primary by running:

CALL dolt_assume_cluster_role('primary', 3);

on dolt-0.

[lihh1992]

The piece that looks suspicious to me is that the .dolt_dropped_databases and insight_metabase directories have similar dates. Has this worked at all since 10/22? Also, is insight_metabase a newly created database, or did you call dolt_undrop() to restore it? Knowing the last operations on the DB would help a lot to determine how this happened.

As far as making this database work, we'll need to have a copy of the database to dissect it. The error stating that there is an invalid root hash record size at 0 is pretty bad. Can you share the database with us? I understand if that's not possible for data sensitivity reasons. If we can look at the bytes at the beginning of the journal file that would help too. The following command will show us the hex for the first 128 bytes:

od -x -N 128 /db/dolt/insight_metabase/.dolt/noms/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
That will at least tell us if we are looking at a file full of null bytes or a file full of something else.

Finally, if you're willing, direct messages on discord can help speed the diagnosis of the problem. Here is the link to our server: https://discord.com/invite/RnNVBHrsJg, my handle is macneale

@macneale4

1. Database Creation Method

We use the following script to create the database. You will see that insight_metabase is restored from a database backup that already contains built-in data.

The intention of this script is: when the POD starts and the required database for the application does not exist, create the database. The insight_metabase database needs to have some built-in data.

    #!/bin/sh
    if echo "$HOSTNAME" | grep -q 'dolt-0$'; then
      echo "$HOSTNAME end with dolt-0，start to restore database..."

      # Step 1: Change to the /db/dolt directory
      cd /db/dolt || {
          echo "error：cd /db/dolt failed"
          exit 1
      }

      echo "current directory：$(pwd)"

      dolt sql-server --config /etc/dolt/config/config.yaml &
      DOLT_PID=$!
      # Step 2: Check if the insight_metabase directory exists
      echo "check insight_metabase directory ..."
      if [ -d "insight_metabase" ]; then
          echo "insight_metabase existed"
      else
          echo "insight_metabase not exist, start to restore database..."
          # Execute dolt backup restore command
          dolt backup restore file:///root/metabase_init insight_metabase
          # Check the command execution result
          if [ $? -eq 0 ]; then
              echo "insight_metabase restore success"
          else
              echo "insight_metabase restore failed"
          fi
      fi

      # Step 3: Check if the insight_jobadmin directory exists
      echo "check insight_jobadmin directory ..."
      if [ -d "insight_jobadmin" ]; then
          echo "insight_jobadmin existed"
      else
          echo "insight_jobadmin not exist, start to create database..."
          # Execute dolt create database command
          dolt sql -q "create database insight_jobadmin;"
          # Check the command execution result
          if [ $? -eq 0 ]; then
              echo "insight_jobadmin created successes"
          else
              echo "insight_jobadmin created failed"
          fi
      fi

      # Step 4: Check if the insight_system directory exists
      echo "check insight_system directory ..."
      if [ -d "insight_system" ]; then
          echo "insight_system existed"
      else
          echo "insight_system not exist, start to create database..."
          # Execute dolt create database command
          dolt sql -q "create database insight_system;"
          # Check the command execution result
          if [ $? -eq 0 ]; then
              echo "insight_system created successes"
          else
              echo "insight_system created failed"
          fi
      fi
    kill $DOLT_PID
    else
      echo "$HOSTNAME not end with dolt-0，skip to restore database..."
    fi
    echo "script done"

2. Database Copy

After the incident occurred, I backed up this erroneous database as a compressed file. However, due to data security reasons, I cannot send this database to you directly. Below is the file listing of this archive. I am not sure if it will be helpful.

[root@instance-04cghrqg-1 bad]# tar -zxvf bad.tar.gz 
insight_metabase_bak/
insight_metabase_bak/.sqlhistory
insight_metabase_bak/.dolt/
insight_metabase_bak/.dolt/config.json
insight_metabase_bak/.dolt/repo_state.json
insight_metabase_bak/.dolt/noms/
insight_metabase_bak/.dolt/noms/398ph5tti8437ejmrb4vndm2m8nbbhf5.darc
insight_metabase_bak/.dolt/noms/oldgen/
insight_metabase_bak/.dolt/noms/oldgen/64deanm44ehrij4vmufb4vulv7ofgglk.darc
insight_metabase_bak/.dolt/noms/oldgen/manifest
insight_metabase_bak/.dolt/noms/oldgen/LOCK
insight_metabase_bak/.dolt/noms/journal.idx
insight_metabase_bak/.dolt/noms/jn6e21l2n53apqb48qi5qhkoiomrntc6.darc
insight_metabase_bak/.dolt/noms/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
insight_metabase_bak/.dolt/noms/nbs_manifest_3876200906
insight_metabase_bak/.dolt/noms/nbs_manifest_1471441709
insight_metabase_bak/.dolt/noms/manifest
insight_metabase_bak/.dolt/noms/LOCK
insight_metabase_bak/.dolt/noms/7ig9au86vvjh0g5vtc7rufuffejpep20.darc
insight_metabase_bak/.dolt/noms/nbs_manifest_1593782753
insight_metabase_bak/.dolt/temptf/
insight_metabase_bak/.dolt/temptf/buffered_file_byte_sink_254997447
insight_metabase_bak/.dolt/temptf/buffered_file_byte_sink_103821519
insight_metabase_bak/.dolt/tmp/

The result of the command od -x -N 128 insight_metabase_bak/.dolt/noms/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv is as follows:

[root@instance-04cghrqg-1 bad]# od -x -N 128 insight_metabase_bak/.dolt/noms/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
0000000 0000 ea00 0201 9002 fb90 0911 158c 9cba
0000020 0f45 c4df b528 8224 ff31 03d7 02a4 1b6c
0000040 0100 2020 0000 5400 5055 184d 2400 2000
0000060 1c00 0700 1800 1400 0600 0900 0801 0008
0000100 0518 0809 0101 0502 0908 0801 0010 0500
0000120 2018 0040 0000 0048 0000 0503 1017 0014
0000140 0028 0500 2404 7b66 9003 019a 0000 1d4c
0000160 1905 001c 0008 0002 895c 4204 0014 3809
0000200

I am very sorry, but when I tried to open the link https://discord.com/invite/RnNVBHrsJg, it had already expired. I am not very familiar with Discord and might need some time to learn how to use it.

[macneale4]

Doing a dolt backup restore while a server is running is probably the root cause. I'm going to do some testing, but that command is not intended to be used while a server is running. It should error out when a server is running. I'm going to test and address that.

The journal file you provided the first few bytes for is corrupt. That first four bytes should not contain a number that large.

As far as the discord invite is concerned, sorry about that. Here is a better link: https://discord.gg/gqr7K4VNKe

[macneale4]

Until I have a better answer for you about dolt backup restore command, you should change that command to run

dolt sql -q "CALL DOLT_BACKUP('restore', 'file:///root/metabase_init', 'insight_metabase')

instead.

[macneale4]

The other thing worth noting is that the files noms/nbs_manifest_* and temptf/buffered_file_byte_sink_* indicate that there have been several failures to move temporary files into place after processing them. The fact that there are several of them in this directory is an indication of repeated failure to perform basic file system operations.

This generally only happens if your processes are being hard killed (kill -9) or if your disk is persistently full. What are the characteristics of this deployment? Are you capturing logs and seeing any errors in them if so?

[lihh1992]

@macneale4

dolt backup restore

I have been using the dolt backup restore command for deployment and have conducted multiple tests without encountering any significant errors. The only issue is: when the insight_metabase database is initially created from a backup, the application fails to recognize it. I need to restart my dolt service for the application to detect the database.

I will test using dolt sql -q "CALL DOLT_BACKUP('restore', 'file:///root/metabase_init', 'insight_metabase')". Does running this command also need to be done after dolt sql-server --config /etc/dolt/config/config.yaml &?

My Deployment Method

As mentioned in the issue, I created a dolt service using StatefulSet and used PVC for data storage. The POD did restart before the insight_metabase database encountered an exception. Unfortunately, the logs are no longer available due to multiple subsequent deployments.

[macneale4]

Running the call dolt_backup('restore'... sql operation should allow you to keep the server running.

There is another option to restore the database before your start the server.

We have an engineer working on making the dolt backup cli work correctly with a running server. That should be out early next week. If you can wait for that you won't need to change anything.