Catchup command implementation #392
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -143,6 +143,14 @@ doc/src/sgml/pgprobackup.sgml | |
| <arg choice="plain"><option>--wal-file-name</option> <replaceable>wal_file_name</replaceable></arg> | ||
| <arg rep="repeat"><replaceable>option</replaceable></arg> | ||
| </cmdsynopsis> | ||
| <cmdsynopsis condition="12+"> | ||
| <command>pg_probackup</command> | ||
| <arg choice="plain"><option>catchup</option></arg> | ||
| <arg choice="plain"><option>-b</option> <replaceable>catchup_mode</replaceable></arg> | ||
| <arg choice="plain"><option>--source-pgdata</option>=<replaceable>path_to_pgdata_on_remote_server</replaceable></arg> | ||
| <arg choice="plain"><option>--destination-pgdata</option>=<replaceable>path_to_local_dir</replaceable></arg> | ||
| <arg rep="repeat"><replaceable>option</replaceable></arg> | ||
| </cmdsynopsis> | ||
|
|
||
| </refsynopsisdiv> | ||
|
|
||
|
|
@@ -283,6 +291,11 @@ doc/src/sgml/pgprobackup.sgml | |
| Partial restore: restoring only the specified databases. | ||
| </para> | ||
| </listitem> | ||
| <listitem condition="12+"> | ||
| <para> | ||
| Catchup: cloning a <productname>PostgreSQL</productname> instance for a fallen-behind standby server to <quote>catch up</quote> with master. | ||
| </para> | ||
| </listitem> | ||
| </itemizedlist> | ||
| <para> | ||
| To manage backup data, <application>pg_probackup</application> creates a | ||
|
|
@@ -1076,7 +1089,8 @@ GRANT SELECT ON TABLE pg_catalog.pg_database TO backup; | |
| mode: <xref linkend="pbk-add-instance"/>, | ||
| <xref linkend="pbk-backup"/>, | ||
| <xref linkend="pbk-restore"/>, | ||
| <phrase condition="12+"><xref linkend="pbk-catchup"/>,</phrase> | ||
| <xref linkend="pbk-archive-push"/>, and | ||
| <xref linkend="pbk-archive-get"/>. | ||
| </para> | ||
| </listitem> | ||
|
|
@@ -1459,6 +1473,7 @@ pg_probackup backup -B <replaceable>backup_dir</replaceable> --instance <replace | |
| </para> | ||
| </refsect3> | ||
| </refsect2> | ||
|
|
||
| <refsect2 id="pbk-verifying-a-cluster"> | ||
| <title>Performing Cluster Verification</title> | ||
| <para> | ||
|
|
@@ -1534,6 +1549,7 @@ pg_probackup checkdb --amcheck --skip-block-validation [<replaceable>connection_ | |
| higher cost of CPU, memory, and I/O consumption. | ||
| </para> | ||
| </refsect2> | ||
|
|
||
| <refsect2 id="pbk-validating-backups"> | ||
| <title>Validating a Backup</title> | ||
| <para> | ||
|
|
@@ -2101,6 +2117,7 @@ pg_probackup restore -B <replaceable>backup_dir</replaceable> --instance <replac | |
| <xref linkend="pbk-restore"/>, | ||
| <xref linkend="pbk-merge"/>, | ||
| <xref linkend="pbk-delete"/>, | ||
| <phrase condition="12+"><xref linkend="pbk-catchup"/>,</phrase> | ||
| <xref linkend="pbk-checkdb"/> and | ||
| <xref linkend="pbk-validate"/> processes can be | ||
| executed on several parallel threads. This can significantly | ||
|
|
@@ -3418,6 +3435,129 @@ pg_probackup delete -B <replaceable>backup_dir</replaceable> --instance <replace | |
| </para> | ||
|
|
||
| </refsect2> | ||
|
|
||
| <refsect2 id="pbk-creating-backup-to-catchup" condition="12+"> | ||
| <title>Cloning <productname>PostgreSQL</productname> Instance</title> | ||
| <para> | ||
| <application>pg_probackup</application> can create a copy of a <productname>PostgreSQL</productname> | ||
| instance directly, without using the backup catalog. This allows you | ||
| to add a new standby server in a parallel mode or to have a standby | ||
| server that has fallen behind <quote>catch up</quote> with master. | ||
| </para> | ||
|
|
||
| <para> | ||
| Cloning a <productname>PostgreSQL</productname> instance is different from other <application>pg_probackup</application> | ||
| processes: | ||
| <itemizedlist> | ||
| <listitem> | ||
| <para> | ||
| The backup catalog is not required. | ||
| </para> | ||
| </listitem> | ||
| <listitem> | ||
| <para> | ||
| STREAM WAL delivery mode is only supported. | ||
| </para> | ||
| </listitem> | ||
| <listitem> | ||
| <para> | ||
| Copying external directories | ||
| is not supported. | ||
| </para> | ||
| </listitem> | ||
| </itemizedlist> | ||
| </para> | ||
|
|
||
| <para> | ||
| Before cloning a <productname>PostgreSQL</productname> instance, set up the source database server as follows: | ||
| <itemizedlist> | ||
| <listitem> | ||
| <para> | ||
| <link linkend="pbk-configuring-the-database-cluster">Configure | ||
| the database cluster</link> for the instance to copy. | ||
| </para> | ||
| </listitem> | ||
| <listitem> | ||
| <para> | ||
| To copy from a remote server, <link linkend="pbk-configuring-the-remote-mode">configure the remote mode</link>. | ||
| </para> | ||
| </listitem> | ||
| <listitem> | ||
| <para> | ||
| To use the PTRACK backup mode, <link linkend="pbk-setting-up-ptrack-backups">set up PTRACK backups</link>. | ||
| </para> | ||
| </listitem> | ||
| </itemizedlist> | ||
| </para> | ||
|
|
||
| <para> | ||
| To clone a <productname>PostgreSQL</productname> instance, ensure that the source | ||
| database server is running and accepting connections and | ||
| on the server with the destination database, run the following command: | ||
| </para> | ||
| <programlisting> | ||
| pg_probackup catchup -b <replaceable>catchup-mode</replaceable> --source-pgdata=<replaceable>path_to_pgdata_on_remote_server</replaceable> --destination-pgdata=<replaceable>path_to_local_dir</replaceable> --stream [<replaceable>connection_options</replaceable>] [<replaceable>remote_options</replaceable>] | ||
| </programlisting> | ||
| <para> | ||
| Where <replaceable>catchup_mode</replaceable> can take one of the | ||
| following values: <literal>FULL</literal>, <literal>DELTA</literal>, or <literal>PTRACK</literal>. | ||
| </para> | ||
| <itemizedlist spacing="compact"> | ||
| <listitem> | ||
| <para> | ||
| FULL — creates a full copy of the <productname>PostgreSQL</productname> instance. | ||
| The destination directory must be empty for this mode. | ||
| </para> | ||
| </listitem> | ||
| <listitem> | ||
| <para> | ||
| DELTA — reads all data files in the data directory and | ||
| creates an incremental copy for pages that have changed | ||
| since the destination database was shut down cleanly. | ||
| For this mode, the destination directory must contain a previous | ||
| copy of the database that was shut down cleanly. | ||
|
There was a problem hiding this comment.
А зачем именно There was a problem hiding this comment. Из изначальной постановки задачи в PGPRO-4447 и пункт 4 раздела Assumptions в issue #277 There was a problem hiding this comment. Я считаю, что такое требование сильно уменьшает юзабельность и не дает никакого профита. There was a problem hiding this comment. Т.е. мы таким образом защищаемся от того, чтобы нам не подсунили destination, который, например, только наполовину восстановлен из бэкапа, но по-моему игра не стоит свеч, проще защититься от этого документацией. There was a problem hiding this comment. Это надо сделать в этой версии или оставим это ослабление требований на следующую версию? There was a problem hiding this comment.
Это хорошо, sanity чеков много не бывает. Но на 100% всё равно не защищает: вдруг pg_control успели донести, backup_label - нет. There was a problem hiding this comment. Так а какие проверки в таком случае будут нормальны/достаточны? There was a problem hiding this comment.
Ну по логике вещей, все остальные имеющиеся минус эта =)
Я бы не сказал There was a problem hiding this comment. И к тому же, сейчас комбинация двух проверок (на backup_label и на состояние pg_control) как раз защищает больше. There was a problem hiding this comment. Мой сценарий (а так же много каких други) мы и так не поймаем, поэтому мы не сильно много проигрываем в плане надежности, но сильно выигрываем в юзабельности. |
||
| </para> | ||
| </listitem> | ||
| <listitem> | ||
| <para> | ||
| PTRACK — creates an incremental backup tracking page | ||
| changes on the fly. | ||
| For this mode, the destination directory must contain a previous | ||
| copy of the database that was shut down cleanly. | ||
| </para> | ||
| </listitem> | ||
| </itemizedlist> | ||
| <para> | ||
| You can use <link linkend="pbk-connection-opts">connection_options</link> to specify | ||
| the connection to the source database cluster. If it is located on a different server, | ||
| also specify <link linkend="pbk-remote-server-opts">remote_options</link>. | ||
| If the source database contains tablespaces that must be located in | ||
| a different directory, additionally specify the <option>--tablespace-mapping</option> | ||
| option: | ||
| <programlisting> | ||
| pg_probackup catchup -b <replaceable>catchup-mode</replaceable> --source-pgdata=<replaceable>path_to_pgdata_on_remote_server</replaceable> --destination-pgdata=<replaceable>path_to_local_dir</replaceable> --stream --tablespace-mapping=<replaceable>OLDDIR</replaceable>=<replaceable>NEWDIR</replaceable> | ||
| </programlisting> | ||
| To run the <command>catchup</command> command on parallel threads, specify the number | ||
| of threads with the <option>--threads</option> option: | ||
| <programlisting> | ||
| pg_probackup catchup -b <replaceable>catchup-mode</replaceable> --source-pgdata=<replaceable>path_to_pgdata_on_remote_server</replaceable> --destination-pgdata=<replaceable>path_to_local_dir</replaceable> --stream --threads=<replaceable>num_threads</replaceable> | ||
| </programlisting> | ||
| </para> | ||
| <para> | ||
| For example, assume that a remote standby server with the <productname>PostgreSQL</productname> instance in <filename>/replica-pgdata</filename> data directory has fallen behind. To sync this instance with the one in <filename>/master-pgdata</filename> data directory, you can run | ||
| the <command>catchup</command> command in the <literal>PTRACK</literal> mode on four parallel threads as follows: | ||
| <programlisting> | ||
| pg_probackup catchup --source-pgdata=/master-pgdata --destination-pgdata=/replica-pgdata -p 5432 -d postgres -U remote-postgres-user --stream --backup-mode=PTRACK --remote-host=remote-hostname --remote-user=remote-unix-username -j 4 | ||
| </programlisting> | ||
| </para> | ||
| <para> | ||
| Another example shows how you can add a new remote standby server with the <productname>PostgreSQL</productname> data directory <filename>/replica-pgdata</filename> by running the <command>catchup</command> command in the <literal>FULL</literal> mode | ||
| on four parallel threads: | ||
| <programlisting> | ||
| pg_probackup catchup --source-pgdata=/master-pgdata --destination-pgdata=/replica-pgdata -p 5432 -d postgres -U remote-postgres-user --stream --backup-mode=FULL --remote-host=remote-hostname --remote-user=remote-unix-username -j 4 | ||
| </programlisting> | ||
| </para> | ||
| </refsect2> | ||
| </refsect1> | ||
|
|
||
| <refsect1 id="pbk-reference"> | ||
|
|
@@ -4290,6 +4430,120 @@ pg_probackup archive-get -B <replaceable>backup_dir</replaceable> --instance <re | |
| For details, see section <link linkend="pbk-archiving-options">Archiving Options</link>. | ||
| </para> | ||
| </refsect3> | ||
|
|
||
| <refsect3 id="pbk-catchup" xreflabel="catchup" condition="12+"> | ||
| <title>catchup</title> | ||
| <programlisting> | ||
| pg_probackup catchup -b <replaceable>catchup_mode</replaceable> | ||
| --source-pgdata=<replaceable>path_to_pgdata_on_remote_server</replaceable> | ||
| --destination-pgdata=<replaceable>path_to_local_dir</replaceable> | ||
| [--help] [--stream] [-j <replaceable>num_threads</replaceable>] | ||
| [-T <replaceable>OLDDIR</replaceable>=<replaceable>NEWDIR</replaceable>] | ||
| [<replaceable>connection_options</replaceable>] [<replaceable>remote_options</replaceable>] | ||
| </programlisting> | ||
| <para> | ||
| Creates a copy of a <productname>PostgreSQL</productname> | ||
| instance without using the backup catalog. | ||
|
|
||
| <variablelist> | ||
| <varlistentry> | ||
| <term><option>-b <replaceable>catchup_mode</replaceable></option></term> | ||
| <term><option>--backup-mode=<replaceable>catchup_mode</replaceable></option></term> | ||
| <listitem> | ||
| <para> | ||
| Specifies the catchup mode to use. Possible values are: | ||
|
|
||
| <itemizedlist spacing="compact"> | ||
| <listitem> | ||
| <para> | ||
| <literal>FULL</literal> — creates a full copy of the <productname>PostgreSQL</productname> instance. | ||
| </para> | ||
| </listitem> | ||
| <listitem> | ||
| <para> | ||
| <literal>DELTA</literal> — reads all data files in the data directory and | ||
| creates an incremental copy for pages that have changed | ||
| since the destination database was shut down cleanly. | ||
| </para> | ||
| </listitem> | ||
| <listitem> | ||
| <para> | ||
| <literal>PTRACK</literal> — creates an incremental PTRACK backup tracking | ||
|
There was a problem hiding this comment. Звучит как будто тут создается бэкап. There was a problem hiding this comment. Такой текст пойдет? There was a problem hiding this comment. Звучит норм!
Мне кажется, очень важно добавить, что и читаются тоже только изменившие страницы |
||
| page changes on the fly. | ||
| </para> | ||
| </listitem> | ||
| </itemizedlist> | ||
| </para> | ||
| </listitem> | ||
| </varlistentry> | ||
|
|
||
| <varlistentry> | ||
| <term><option>--source-pgdata=<replaceable>path_to_pgdata_on_remote_server</replaceable></option></term> | ||
| <listitem> | ||
| <para> | ||
| Specifies the path to the data directory of the instance to be copied. The path can be local or remote. | ||
| </para> | ||
| </listitem> | ||
| </varlistentry> | ||
|
|
||
| <varlistentry> | ||
| <term><option>--destination-pgdata=<replaceable>path_to_local_dir</replaceable></option></term> | ||
| <listitem> | ||
| <para> | ||
| Specifies the path to the local data directory to copy to. | ||
| </para> | ||
| </listitem> | ||
| </varlistentry> | ||
|
|
||
| <varlistentry> | ||
| <term><option>--stream</option></term> | ||
| <listitem> | ||
| <para> | ||
| Makes a <link linkend="pbk-stream-mode">STREAM</link> backup, which | ||
| includes all the necessary WAL files by streaming them from | ||
| the database server via replication protocol. | ||
| </para> | ||
| </listitem> | ||
| </varlistentry> | ||
|
|
||
| <varlistentry> | ||
| <term><option>-j <replaceable>num_threads</replaceable></option></term> | ||
| <term><option>--threads=<replaceable>num_threads</replaceable></option></term> | ||
| <listitem> | ||
| <para> | ||
| Sets the number of parallel threads for | ||
| <command>catchup</command> process. | ||
| </para> | ||
| </listitem> | ||
| </varlistentry> | ||
|
|
||
| <varlistentry> | ||
| <term><option>-T <replaceable>OLDDIR</replaceable>=<replaceable>NEWDIR</replaceable></option></term> | ||
| <term><option>--tablespace-mapping=<replaceable>OLDDIR</replaceable>=<replaceable>NEWDIR</replaceable></option></term> | ||
| <listitem> | ||
| <para> | ||
| Relocates the tablespace from the <replaceable>OLDDIR</replaceable> to the <replaceable>NEWDIR</replaceable> | ||
| directory at the time of recovery. Both <replaceable>OLDDIR</replaceable> and <replaceable>NEWDIR</replaceable> must | ||
| be absolute paths. If the path contains the equals sign (<literal>=</literal>), | ||
| escape it with a backslash. This option can be specified | ||
| multiple times for multiple tablespaces. | ||
| </para> | ||
| </listitem> | ||
| </varlistentry> | ||
|
|
||
| </variablelist> | ||
| </para> | ||
|
|
||
| <para> | ||
| Additionally, <link linkend="pbk-connection-opts">connection | ||
| options</link>, <link linkend="pbk-remote-server-opts">remote | ||
| mode options</link> can be used. | ||
| </para> | ||
| <para> | ||
| For details on usage, see the section | ||
| <link linkend="pbk-creating-backup-to-catchup">Cloning <productname>PostgreSQL</productname> Instance</link>. | ||
| </para> | ||
| </refsect3> | ||
| </refsect2> | ||
| <refsect2 id="pbk-options"> | ||
| <title>Options</title> | ||
|
|
@@ -4679,7 +4933,7 @@ pg_probackup archive-get -B <replaceable>backup_dir</replaceable> --instance <re | |
| <term><option>--no-color</option></term> | ||
| <listitem> | ||
| <para> | ||
| Disable coloring for console log messages of <literal>warning</literal> and <literal>error</literal> levels. | ||
| </para> | ||
| </listitem> | ||
| </varlistentry> | ||
|
|
@@ -4832,7 +5086,8 @@ pg_probackup archive-get -B <replaceable>backup_dir</replaceable> --instance <re | |
| <title>Connection Options</title> | ||
| <para> | ||
| You can use these options together with | ||
| <xref linkend="pbk-backup"/> | ||
| <phrase condition="12+">, <xref linkend="pbk-catchup"/>,</phrase> and | ||
| <xref linkend="pbk-checkdb"/> commands. | ||
| </para> | ||
| <para> | ||
|
|
@@ -5123,6 +5378,7 @@ pg_probackup archive-get -B <replaceable>backup_dir</replaceable> --instance <re | |
| used with <xref linkend="pbk-add-instance"/>, | ||
| <xref linkend="pbk-set-config"/>, | ||
| <xref linkend="pbk-backup"/>, | ||
| <phrase condition="12+"><xref linkend="pbk-catchup"/>,</phrase> | ||
| <xref linkend="pbk-restore"/>, | ||
| <xref linkend="pbk-archive-push"/>, and | ||
| <xref linkend="pbk-archive-get"/> commands. | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@indrups
Мой рунглиш, конечно, могуч, но может такая формуливка будет звучать лучше
... or to have a fallen-behind standby server to catch with master?