pg_probackup 2.4.4 : Cannot seek to position % in page header map

[sms1222]

Windows 2012 R2, PostgreSQL 12.4

I have a problem about overflow offset to write headers.
Value of Offset in header is negative.
Then my backup is failed.
~
2021-03-24 14:50:06 ´eCN¹I±¹ C￥AؽA [13580]: VERBOSE: Writing headers for file "base/16386/712428.1" offset: -2067342695, len: 1824706, crc: 3872237612
2021-03-24 14:50:06 ´eCN¹I±¹ C￥AؽA [13580]: VERBOSE: File "E:/PostgreSQL/12/data/base/16386/712428.1". Copied 136698888 bytes
2021-03-24 14:50:10 ´eCN¹I±¹ C￥AؽA [13580]: VERBOSE: finished segment at 2EB9/90000000 (timeline 1)
2021-03-24 14:50:13 ´eCN¹I±¹ C￥AؽA [13580]: VERBOSE: Writing headers for file "base/16386/712428" offset: -2065517989, len: 1823091, crc: 2920043281
2021-03-24 14:50:13 ´eCN¹I±¹ C￥AؽA [13580]: VERBOSE: File "E:/PostgreSQL/12/data/base/16386/712428". Copied 140730399 bytes
2021-03-24 14:50:15 ´eCN¹I±¹ C￥AؽA [13580]: VERBOSE: Writing headers for file "base/16386/712428.2" offset: -2063694898, len: 1824508, crc: 2134761632
2021-03-24 14:50:15 ´eCN¹I±¹ C￥AؽA [13580]: VERBOSE: File "E:/PostgreSQL/12/data/base/16386/712428.2". Copied 143610176 bytes
~
2021-03-24 19:04:16 ´eCN¹I±¹ C￥AؽA [13580]: VERBOSE: Validate relation blocks for file "E:/PostgreSQL/12/backup/backups/pts/QQFGC1/database/base/16386/712428"
2021-03-24 19:04:16 ´eCN¹I±¹ C￥AؽA [13580]: WARNING: Cannot seek to position 2229449307 in page header map "E:/PostgreSQL/12/backup/backups/pts/QQFGC1/page_header_map": Invalid argument
2021-03-24 19:04:16 ´eCN¹I±¹ C￥AؽA [13580]: WARNING: Cannot get page headers for file "E:/PostgreSQL/12/backup/backups/pts/QQFGC1/database/base/16386/712428"
2021-03-24 19:04:16 ´eCN¹I±¹ C￥AؽA [13580]: VERBOSE: Validate relation blocks for file "E:/PostgreSQL/12/backup/backups/pts/QQFGC1/database/base/16386/712428.1"
2021-03-24 19:04:16 ´eCN¹I±¹ C￥AؽA [13580]: WARNING: Cannot seek to position 2227624601 in page header map "E:/PostgreSQL/12/backup/backups/pts/QQFGC1/page_header_map": Invalid argument
2021-03-24 19:04:16 ´eCN¹I±¹ C￥AؽA [13580]: WARNING: Cannot get page headers for file "E:/PostgreSQL/12/backup/backups/pts/QQFGC1/database/base/16386/712428.1"
2021-03-24 19:04:16 ´eCN¹I±¹ C￥AؽA [13580]: VERBOSE: Validate relation blocks for file "E:/PostgreSQL/12/backup/backups/pts/QQFGC1/database/base/16386/712428.2"
2021-03-24 19:04:16 ´eCN¹I±¹ C￥AؽA [13580]: WARNING: Cannot seek to position 2231272398 in page header map "E:/PostgreSQL/12/backup/backups/pts/QQFGC1/page_header_map": Invalid argument
2021-03-24 19:04:16 ´eCN¹I±¹ C￥AؽA [13580]: WARNING: Cannot get page headers for file "E:/PostgreSQL/12/backup/backups/pts/QQFGC1/database/base/16386/712428.2"
2021-03-24 19:33:01 ´eCN¹I±¹ C￥AؽA [13580]: WARNING: Backup QQFGC1 data files are corrupted
2021-03-24 19:33:01 ´eCN¹I±¹ C￥AؽA [13580]: INFO: Backup QQFGC1 resident size: 774GB
2021-03-24 19:33:01 ´eCN¹I±¹ C￥AؽA [13580]: ERROR: Backup QQFGC1 failed

[gsmolk]

Hello!
Looks serious.
Can you please provide the backup_content.control file from backup QQFGC1 ?

[sms1222]

backup_content.zip

This is new one.
The negative offsets start at Line 2819 in backup_content.control file which attached.

Thanks.

Hello!
Looks serious.
Can you please provide the backup_content.control file from backup QQFGC1 ?

[gsmolk]

Is it okay if I provide you with custom installer to check that the issue is resolved ?

[gsmolk]

Installer with patched binary:
https://oc.postgrespro.ru/index.php/s/4TKSVCNGeAzlSgo

[sms1222]

Thanks, gsmol.

I will try backup with 2.4.11 that you provide tomorrow.
It takes more than 15 hours.

[gsmolk]

Any news?

[gsmolk]

This issue is the only stopper for 2.4.12 release, so if you would be so kind to provide some feedback, it will speed up the release.

[sms1222]

2.4.11
Backups are failed.
backups.zip
I attach backup log and backup_content_control files.
The different errors appeared.

I need a 2.4.12 release quickly.
Thanks.

CASE1 :
An error occured during metadata decompression for file "base/13423/2754": data error

CASE 2:
2021-03-27 13:44:39 ´eCN¹I±¹ C￥AؽA [10956]: ERROR: Interrupted during validate
2021-03-27 13:44:39 ´eCN¹I±¹ C￥AؽA [10956]: ERROR: Cannot open file "E:/PostgreSQL/12/backup/backups/pts/QQL0C1/database/base/13423/2755": Too many open files
2021-03-27 13:44:39 ´eCN¹I±¹ C￥AؽA [10956]: WARNING: Cannot get page headers for file "E:/PostgreSQL/12/backup/backups/pts/QQL0C1/database/base/13424/2679"
2021-03-27 13:44:39 ´eCN¹I±¹ C￥AؽA [10956]: ERROR: Interrupted during validate
2021-03-27 13:44:39 ´eCN¹I±¹ C￥AؽA [10956]: ERROR: Cannot open file "E:/PostgreSQL/12/backup/backups/pts/QQL0C1/database/base/16386/2579": Too many open files
2021-03-27 13:44:39 ´eCN¹I±¹ C￥AؽA [10956]: ERROR: Data files validation failed

[gsmolk]

Thank you!
I will provide new binary shortly

[gsmolk]

Installer with another patched binary:
https://oc.postgrespro.ru/index.php/s/wIiZJm99CuFDaDv

[gsmolk]

Did it helped?

[sms1222]

I don't try yet. Our backup is scheduled at every 12:00 AM (GMT+09).
I will report it, tomorrow.

Thanks.