Skip to content

Commit 56632f5

Browse files
avamingliavamingli
authored
Dynamic Table. (#725)
Dynamic Table is a an auto-refreshing materialized view which could be constructed by base tables, external tables, materialized views and dynamic tables. And it could be used to answer query by AQUMV. As normal tables in CBDB, dynamic tables could also have distribution keys. The purpose of Dynamic Tables is to solve the problem often raised by customers who are big fans of a lakehouse architecture: how can we run queries on external tables as fast as internal tables? CREATE DYNAMIC TABLE: CREATE DYNAMIC TABLE dt0 SCHEDULE '5 * * * *' AS SELECT a, b, sum(c) FROM t1 GROUP BY a, b WITH NO DATA DISTRIBUTED BY(b); CREATE DYNAMIC TABLE \d List of relations Schema | Name | Type | Owner | Storage --------+------+---------------+---------+--------- public | dt0 | dynamic table | gpadmin | heap public | t1 | table | gpadmin | heap (2 rows) CREATE DYNAMIC TABLE xxx AS Query The Query allows any valid SELECT SQL of Materialized Views: from single or multiple relations, base tables, materialized views, and dynamic tables as well, joins, subquery, aggregation, group by and etc. SCHEDULE: A string used to schedule background job which auto-refreshes the dynamic table. We follow the valid string of pg_cron extension which supports linux crontab, refer https://crontab.guru ┌───────────── min (0 - 59) │ ┌────────────── hour (0 - 23) │ │ ┌─────────────── day of month (1 - 31) or last day of the month ($) │ │ │ ┌──────────────── month (1 - 12) │ │ │ │ ┌───────────────── day of week (0 - 6) (0 to 6 are Sunday to │ │ │ │ │ Saturday, or use names; 7 is also Sunday) │ │ │ │ │ │ │ │ │ │ * * * * * You can also use '[1-59] seconds' to schedule a job based on an interval. The example creates a cron job refreshing the dynamic table at minute 5 of each hour. For convenience, SCHEDULE is optional. If user didn't specific it, a default schedule is provided: at every 5th minute. WITH NO DATA: Same as Materialized View, will create an empty Dynamic Table if specified. DISTRIBUTED BY: Same as normal tables in CBDB, Dynamic Tables could support distribution keys as materialized views. Refresh Dynamic Table As seen in pg_task, we put a command to auto-refresh dynamic tables. However, if users want to do a REFRESH manually, exec command REFRESH DYNAMIC TABLE is also supported. REFRESH DYNAMIC TABLE dt0; REFRESH DYNAMIC TABLE Refresh WITH NO DATA Same as Materialized Views, Refresh with no data will truncate the Dynamic Table and make it unpopulated status. REFRESH DYNAMIC TABLE dt0 WITH NO DATA; REFRESH DYNAMIC TABLE Drop Dynamic Table Drop a Dynamic Table will drop its scheduler job automatically. DROP DYNAMIC TABLE dt0; DROP DYNAMIC TABLE Like Materialized Views, Dynamic Tables could be used to answer query too. This is limited by AQUMV. Authored-by: Zhang Mingli avamingli@gmail.com
1 parent 470a8f2 commit 56632f5

42 files changed

Lines changed: 1480 additions & 32 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎contrib/pg_stat_statements/pg_stat_statements.c‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1153,6 +1153,7 @@ pgss_ProcessUtility(PlannedStmt *pstmt, const char *queryString,
11531153
rows = (qc && (qc->commandTag == CMDTAG_COPY ||
11541154
qc->commandTag == CMDTAG_FETCH ||
11551155
qc->commandTag == CMDTAG_SELECT ||
1156+
qc->commandTag == CMDTAG_REFRESH_DYNAMIC_TABLE ||
11561157
qc->commandTag == CMDTAG_REFRESH_MATERIALIZED_VIEW)) ?
11571158
qc->nprocessed : 0;
11581159

‎doc/src/sgml/ref/allfiles.sgml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ Complete list of usable sgml source files in this directory.
6666
<!ENTITY createConversion SYSTEM "create_conversion.sgml">
6767
<!ENTITY createDatabase SYSTEM "create_database.sgml">
6868
<!ENTITY createDomain SYSTEM "create_domain.sgml">
69+
<!ENTITY createDynamicTable SYSTEM "create_dynamic_table.sgml">
6970
<!ENTITY createEventTrigger SYSTEM "create_event_trigger.sgml">
7071
<!ENTITY createExtension SYSTEM "create_extension.sgml">
7172
<!ENTITY createForeignDataWrapper SYSTEM "create_foreign_data_wrapper.sgml">
@@ -114,6 +115,7 @@ Complete list of usable sgml source files in this directory.
114115
<!ENTITY dropConversion SYSTEM "drop_conversion.sgml">
115116
<!ENTITY dropDatabase SYSTEM "drop_database.sgml">
116117
<!ENTITY dropDomain SYSTEM "drop_domain.sgml">
118+
<!ENTITY dropDynamicTable SYSTEM "drop_dynamic_table.sgml">
117119
<!ENTITY dropEventTrigger SYSTEM "drop_event_trigger.sgml">
118120
<!ENTITY dropExtension SYSTEM "drop_extension.sgml">
119121
<!ENTITY dropForeignDataWrapper SYSTEM "drop_foreign_data_wrapper.sgml">
@@ -166,6 +168,7 @@ Complete list of usable sgml source files in this directory.
166168
<!ENTITY prepare SYSTEM "prepare.sgml">
167169
<!ENTITY prepareTransaction SYSTEM "prepare_transaction.sgml">
168170
<!ENTITY reassignOwned SYSTEM "reassign_owned.sgml">
171+
<!ENTITY refreshDynamicTable SYSTEM "refresh_dynamic_table.sgml">
169172
<!ENTITY refreshMaterializedView SYSTEM "refresh_materialized_view.sgml">
170173
<!ENTITY reindex SYSTEM "reindex.sgml">
171174
<!ENTITY releaseSavepoint SYSTEM "release_savepoint.sgml">

‎doc/src/sgml/ref/create_dynamic_table.sgml‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
<!--
2+
doc/src/sgml/ref/create_dynamic_table.sgml
3+
PostgreSQL documentation
4+
-->
5+
6+
<refentry id="sql-createdynamictable">
7+
<indexterm zone="sql-createdynamictable">
8+
<primary>CREATE DYNAMIC TABLE</primary>
9+
</indexterm>
10+
11+
<refmeta>
12+
<refentrytitle>CREATE DYNAMIC TABLE</refentrytitle>
13+
<manvolnum>7</manvolnum>
14+
<refmiscinfo>SQL - Language Statements</refmiscinfo>
15+
</refmeta>
16+
17+
<refnamediv>
18+
<refname>CREATE DYNAMIC TABLE</refname>
19+
<refpurpose>define a new dynamic table</refpurpose>
20+
</refnamediv>
21+
22+
<refsynopsisdiv>
23+
<synopsis>
24+
CREATE DYNAMIC TABLE [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
25+
[ (<replaceable>column_name</replaceable> [, ...] ) ]
26+
[ USING <replaceable class="parameter">method</replaceable> ]
27+
[ WITH ( <replaceable class="parameter">storage_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
28+
[ TABLESPACE <replaceable class="parameter">tablespace_name</replaceable> ]
29+
AS <replaceable>query</replaceable>
30+
[ WITH [ NO ] DATA ]
31+
</synopsis>
32+
</refsynopsisdiv>
33+
34+
<refsect1>
35+
<title>Description</title>
36+
37+
<para>
38+
<command>CREATE DYNAMIC TABLE</command> defines a dynamic table of
39+
a query. The query is executed and used to populate the view at the time
40+
the command is issued (unless <command>WITH NO DATA</command> is used) and may be
41+
refreshed later using <command>REFRESH DYNAMIC TABLE</command>.
42+
</para>
43+
44+
<para>
45+
<command>CREATE DYNAMIC TABLE</command> is similar to
46+
<command>CREATE TABLE AS</command>, except that it also remembers the query used
47+
to initialize the view, so that it can be refreshed later upon demand.
48+
A dynamic table has many of the same properties as a table, but there
49+
is no support for temporary dynamic tables.
50+
</para>
51+
52+
<para>
53+
<command>CREATE DYNAMIC TABLE</command> requires
54+
<literal>CREATE</literal> privilege on the schema used for the dynamic
55+
table.
56+
</para>
57+
</refsect1>
58+
59+
<refsect1>
60+
<title>Parameters</title>
61+
62+
<variablelist>
63+
<varlistentry>
64+
<term><literal>IF NOT EXISTS</literal></term>
65+
<listitem>
66+
<para>
67+
Do not throw an error if a dynamic table with the same name already
68+
exists. A notice is issued in this case. Note that there is no guarantee
69+
that the existing dynamic table is anything like the one that would
70+
have been created.
71+
</para>
72+
</listitem>
73+
</varlistentry>
74+
75+
<varlistentry>
76+
<term><replaceable>table_name</replaceable></term>
77+
<listitem>
78+
<para>
79+
The name (optionally schema-qualified) of the dynamic table to be
80+
created.
81+
</para>
82+
</listitem>
83+
</varlistentry>
84+
85+
<varlistentry>
86+
<term><replaceable>column_name</replaceable></term>
87+
<listitem>
88+
<para>
89+
The name of a column in the new dynamic table. If column names are
90+
not provided, they are taken from the output column names of the query.
91+
</para>
92+
</listitem>
93+
</varlistentry>
94+
95+
<varlistentry>
96+
<term><literal>USING <replaceable class="parameter">method</replaceable></literal></term>
97+
<listitem>
98+
<para>
99+
This optional clause specifies the table access method to use to store
100+
the contents for the new dynamic table; the method needs be an
101+
access method of type <literal>TABLE</literal>. See <xref
102+
linkend="tableam"/> for more information. If this option is not
103+
specified, the default table access method is chosen for the new
104+
dynamic table. See <xref linkend="guc-default-table-access-method"/>
105+
for more information.
106+
</para>
107+
</listitem>
108+
</varlistentry>
109+
110+
<varlistentry>
111+
<term><literal>WITH ( <replaceable class="parameter">storage_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )</literal></term>
112+
<listitem>
113+
<para>
114+
This clause specifies optional storage parameters for the new
115+
dynamic table; see
116+
<xref linkend="sql-createtable-storage-parameters"/> in the
117+
<xref linkend="sql-createtable"/> documentation for more
118+
information. All parameters supported for <literal>CREATE
119+
TABLE</literal> are also supported for <literal>CREATE DYNAMIC
120+
TABLE</literal>.
121+
See <xref linkend="sql-createtable"/> for more information.
122+
</para>
123+
</listitem>
124+
</varlistentry>
125+
126+
<varlistentry>
127+
<term><literal>TABLESPACE <replaceable class="parameter">tablespace_name</replaceable></literal></term>
128+
<listitem>
129+
<para>
130+
The <replaceable class="parameter">tablespace_name</replaceable> is the name
131+
of the tablespace in which the new dynamic table is to be created.
132+
If not specified, <xref linkend="guc-default-tablespace"/> is consulted.
133+
</para>
134+
</listitem>
135+
</varlistentry>
136+
137+
<varlistentry>
138+
<term><replaceable>query</replaceable></term>
139+
<listitem>
140+
<para>
141+
A <link linkend="sql-select"><command>SELECT</command></link>, <link linkend="sql-table"><command>TABLE</command></link>,
142+
or <link linkend="sql-values"><command>VALUES</command></link> command. This query will run within a
143+
security-restricted operation; in particular, calls to functions that
144+
themselves create temporary tables will fail.
145+
</para>
146+
</listitem>
147+
</varlistentry>
148+
149+
<varlistentry>
150+
<term><literal>WITH [ NO ] DATA</literal></term>
151+
<listitem>
152+
<para>
153+
This clause specifies whether or not the dynamic table should be
154+
populated at creation time. If not, the dynamic table will be
155+
flagged as unscannable and cannot be queried until <command>REFRESH
156+
DYNAMIC TABLE</command> is used. Also, if the view is IMMV,
157+
triggers for maintaining the view are not created.
158+
</para>
159+
</listitem>
160+
</varlistentry>
161+
162+
</variablelist>
163+
</refsect1>
164+
165+
<refsect1>
166+
<title>Compatibility</title>
167+
168+
<para>
169+
<command>CREATE DYNAMIC TABLE</command> is a
170+
<productname>Cloudberry</productname> extension.
171+
</para>
172+
</refsect1>
173+
174+
<refsect1>
175+
<title>See Also</title>
176+
177+
<simplelist type="inline">
178+
<member><xref linkend="sql-createtableas"/></member>
179+
<member><xref linkend="sql-createview"/></member>
180+
<member><xref linkend="sql-dropmaterializedview"/></member>
181+
<member><xref linkend="sql-refreshmaterializedview"/></member>
182+
</simplelist>
183+
</refsect1>
184+
185+
</refentry>

‎doc/src/sgml/ref/drop_dynamic_table.sgml‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
<!--
2+
doc/src/sgml/ref/drop_dynamic_table.sgml
3+
PostgreSQL documentation
4+
-->
5+
6+
<refentry id="sql-dropdynamictable">
7+
<indexterm zone="sql-dropdynamictable">
8+
<primary>DROP DYNAMIC TABLE</primary>
9+
</indexterm>
10+
11+
<refmeta>
12+
<refentrytitle>DROP DYNAMIC TABLE</refentrytitle>
13+
<manvolnum>7</manvolnum>
14+
<refmiscinfo>SQL - Language Statements</refmiscinfo>
15+
</refmeta>
16+
17+
<refnamediv>
18+
<refname>DROP DYNAMIC TABLE</refname>
19+
<refpurpose>remove a dynamic table</refpurpose>
20+
</refnamediv>
21+
22+
<refsynopsisdiv>
23+
<synopsis>
24+
DROP DYNAMIC TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable> [, ...] [ CASCADE | RESTRICT ]
25+
</synopsis>
26+
</refsynopsisdiv>
27+
28+
<refsect1>
29+
<title>Description</title>
30+
31+
<para>
32+
<command>DROP DYNAMIC TABLE</command> drops an existing dynamic
33+
table. To execute this command you must be the owner of the dynamic
34+
table. Since every dynamic table has a auto refresh process of pg_task
35+
job, drop a dynamic table will drop that job too.
36+
</para>
37+
</refsect1>
38+
39+
<refsect1>
40+
<title>Parameters</title>
41+
42+
<variablelist>
43+
<varlistentry>
44+
<term><literal>IF EXISTS</literal></term>
45+
<listitem>
46+
<para>
47+
Do not throw an error if the dynamic table does not exist. A notice
48+
is issued in this case.
49+
</para>
50+
</listitem>
51+
</varlistentry>
52+
53+
<varlistentry>
54+
<term><replaceable class="parameter">name</replaceable></term>
55+
<listitem>
56+
<para>
57+
The name (optionally schema-qualified) of the dynamic table to
58+
remove.
59+
</para>
60+
</listitem>
61+
</varlistentry>
62+
63+
<varlistentry>
64+
<term><literal>CASCADE</literal></term>
65+
<listitem>
66+
<para>
67+
Automatically drop objects that depend on the dynamic table (such as
68+
other materialized views, or regular views or pg_task jobs),
69+
and in turn all objects that depend on those objects
70+
(see <xref linkend="ddl-depend"/>).
71+
</para>
72+
</listitem>
73+
</varlistentry>
74+
75+
<varlistentry>
76+
<term><literal>RESTRICT</literal></term>
77+
<listitem>
78+
<para>
79+
Refuse to drop the dynamic table if any objects depend on it. This
80+
is the default.
81+
</para>
82+
</listitem>
83+
</varlistentry>
84+
</variablelist>
85+
</refsect1>
86+
87+
<refsect1>
88+
<title>Examples</title>
89+
90+
<para>
91+
This command will remove the dynamic table called
92+
<literal>order_summary</literal>:
93+
<programlisting>
94+
DROP DYNAMIC TABLE order_summary;
95+
</programlisting></para>
96+
</refsect1>
97+
98+
<refsect1>
99+
<title>Compatibility</title>
100+
101+
<para>
102+
<command>DROP DYNAMIC TABLE</command> is a
103+
<productname>Cloudberry</productname> extension.
104+
</para>
105+
</refsect1>
106+
107+
<refsect1>
108+
<title>See Also</title>
109+
110+
<simplelist type="inline">
111+
<member><xref linkend="sql-createdynamictable"/></member>
112+
<member><xref linkend="sql-refreshdynamictable"/></member>
113+
</simplelist>
114+
</refsect1>
115+
116+
</refentry>

0 commit comments

Comments
 (0)