Skip to content

Commit 9249605

Browse files
kvchkvch
authored
Add new guide to documentation: Use pgroll with ORMs (#816)
This PR adds a new guide to pgroll's documentation. This is a rehashed version of the release blog post for `convert` command. It is adjusted a little, so it stands on its on as a guide. The examples are in JSON because the output format of the `convert` command is JSON.
1 parent fcf713e commit 9249605

File tree

2 files changed

+309
-1
lines changed

2 files changed

+309
-1
lines changed

‎docs/config.json

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,10 +32,15 @@
3232
"noLink": true,
3333
"items": [
3434
{
35-
"title": "Integrate `pgroll` into your project",
35+
"title": "Integrate pgroll into your project",
3636
"href": "/guides/clientapps",
3737
"file": "docs/guides/clientapps.mdx"
3838
},
39+
{
40+
"title": "Use pgroll with ORMs",
41+
"href": "/guides/orms",
42+
"file": "docs/guides/orms.mdx"
43+
},
3944
{
4045
"title": "Writing up and down migrations",
4146
"href": "/guides/updown",

‎docs/guides/orms.mdx

Lines changed: 303 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,303 @@
1+
# Use pgroll with ORMs
2+
3+
ORMs are popular tools to manage data schema from your application's code. They can translate objects from your code into database objects. Most ORMs can generate SQL migrations scripts. `pgroll` takes these generated statments and translates them into `pgroll` migrations. The generated migration can be saved to a file and you can run the `pgroll` migration using the `start` command as usual.
4+
5+
## How to convert SQL migration scripts
6+
7+
`pgroll` can translate SQL migration scripts generated by ORMs into pgroll migrations using the `convert` subcommand. It can read SQL statements from `stdin` or from a specified file. It has one flag `--name` to configure the name of the migration. If the flag is unset, the name is set to the current timestamp.
8+
9+
```sh
10+
$ pgroll convert --help
11+
Convert SQL statements to a pgroll migration. The command can read SQL statements from stdin or a file
12+
13+
Usage:
14+
pgroll convert <path to file with migrations> [flags]
15+
Flags:
16+
-h, --help help for convert
17+
-n, --name string Name of the migration (default "{current_timestamp}")
18+
```
19+
20+
## Examples
21+
22+
You can use `pgroll` with any ORM that can generate raw SQL statements for its migrations. Let's look at a few examples.
23+
24+
### Alembic
25+
26+
[Alembic](https://alembic.sqlalchemy.org/en/latest/) is a database migration tool used in SQLAlchemy projects. You can generate SQL statements from its migrations in [its offline mode](https://alembic.sqlalchemy.org/en/latest/offline.html). If you add
27+
`a --sql` flag to the `upgrade` command of `alembic` it prints the SQL statements to stdout. You can pipe this output into `pgroll convert`:
28+
29+
```sh
30+
$ alembic update {revision} --sql | pgroll convert --name {revision}
31+
{
32+
"name": "{revision}",
33+
"operations": [
34+
{
35+
"create_table": {
36+
"name": "employees",
37+
"colunms:" [
38+
{
39+
"name": "name",
40+
"type": "varchar(100)"
41+
},
42+
{
43+
"name": "joined",
44+
"type": "timestamp with time zone"
45+
},
46+
{
47+
"name": "email",
48+
"type": "varchar(254)"
49+
}
50+
]
51+
}
52+
}
53+
]
54+
}
55+
```
56+
57+
### Django
58+
59+
Django is the go-to tool for Python web development. It can connect to several databases, including PostgreSQL. If your web application uses PostgreSQL storage, you can leverage the `convert` subcommand.
60+
61+
After you've defined your migration in Python, you can use `manage.py` to extract the SQL script from Django. The [subcommand `sqlmigrate`](https://docs.djangoproject.com/en/5.1/ref/django-admin/#django-admin-sqlmigrate) prints
62+
the SQL statements to stdout.
63+
64+
Create a new model for an `Employee` with a following simplified model:
65+
66+
```python
67+
class Employee():
68+
name = models.CharField(max_length=100)
69+
joined = models.DateTimeField()
70+
email = models.EmailField()
71+
```
72+
73+
Then run `sqlmigrate` to generate the SQL statements and pipe the output into `pgroll convert`:
74+
75+
```sh
76+
manage.py sqlmigrate my_app 0000 | pgroll convert -name 0000_init
77+
{
78+
"name": "0000_init",
79+
"operations": [
80+
{
81+
"create_table": {
82+
"name": "employees",
83+
"colunms:" [
84+
{
85+
"name": "name",
86+
"type": "varchar(100)"
87+
},
88+
{
89+
"name": "joined",
90+
"type": "timestamp with time zone"
91+
},
92+
{
93+
"name": "email",
94+
"type": "varchar(254)"
95+
}
96+
]
97+
}
98+
}
99+
]
100+
}
101+
```
102+
103+
### Drizzle
104+
105+
Drizzle is a popular ORM for Typescript projects. You can extract SQL statements using its [`generate` command](https://orm.drizzle.team/docs/drizzle-kit-generate).
106+
107+
The following example schema in Drizzle can be translated using `pgroll`:
108+
109+
```ts
110+
import { pgTable, timestamp, varchar } from 'drizzle-orm/pg-core';
111+
112+
export const employees = pgTable('employees', {
113+
name: varchar({ length: 100 }),
114+
joined: timestamp({ withTimezone: true }),
115+
email: varchar({ length: 254 })
116+
});
117+
```
118+
119+
Run `pgroll convert` to get the appropriate pgroll migrations.
120+
121+
```sh
122+
drizzle-kit generate --dialect postgresql --schema=./src/schema.ts --name=init
123+
pgroll convert 0000_init.sql --name 0000_init
124+
{
125+
"name": "0000_init",
126+
"operations": [
127+
"create_table": {
128+
"name": "employees",
129+
"colunms:" [
130+
{
131+
"name": "name",
132+
"type": "varchar(100)"
133+
},
134+
{
135+
"name": "joined",
136+
"type": "timestamp with time zone"
137+
},
138+
{
139+
"name": "email",
140+
"type": "varchar(254)"
141+
}
142+
]
143+
}
144+
]
145+
}
146+
}
147+
```
148+
149+
## Limitations
150+
151+
The functionality still has some limitations. The generated pgroll migrations must be edited to provide `up` and `down` data migrations manually. You can find more details about writing these migrations in [this guide](/updown).
152+
153+
Furthermore, the SQL statements are not aggregated into single pgroll operations. Some ORMs add unique constraints in a different statement when they are creating a table with a unique column. This leads to more pgroll operations than necessary. You can resolve this manually by removing the unique constraint operation from the pgroll migration, and add it to the list of `constraints` of `create_table`.
154+
155+
```json
156+
{
157+
"create_table": {
158+
"name": "employees",
159+
"columns": [
160+
{
161+
"name": "email",
162+
"varchar(254)"
163+
}
164+
]
165+
}
166+
},
167+
{
168+
"create_constraint": {
169+
"name": "my_unique_email",
170+
"type": "unique",
171+
"columns": ["email"]
172+
"up": {
173+
"email": "TODO"
174+
}
175+
}
176+
}
177+
178+
```
179+
180+
can be simplified to
181+
182+
```json
183+
{
184+
"create_table": {
185+
"name": "employees",
186+
"columns": [
187+
{
188+
"name": "email",
189+
"varchar(254)"
190+
}
191+
],
192+
"constraints": [
193+
{
194+
"name": "my_unique_email",
195+
"type": "unique",
196+
"columns": ["email"]
197+
}
198+
]
199+
}
200+
}
201+
```
202+
203+
Also, the same applies to some cases when the ORM does the backfilling for new columns. For example, when you add a new column with a default value, an ORM produces these SQL statements:
204+
205+
```sql
206+
ALTER TABLE "adults" ADD COLUMN "age" smallint DEFAULT 18 NULL CHECK ("age" >= 18);
207+
ALTER TABLE "adults" ALTER COLUMN "level" DROP DEFAULT;
208+
209+
UPDATE adults SET age = 18;
210+
211+
ALTER TABLE adults ALTER COLUMN age SET NOT NULL;
212+
213+
UPDATE "adults" SET "age" = 18 WHERE "age" IS NULL;
214+
SET CONSTRAINTS ALL IMMEDIATE;
215+
216+
ALTER TABLE "adults" ALTER COLUMN "age" SET NOT NULL;
217+
```
218+
219+
Resulting in the following pgroll migration using the `convert` subcommand:
220+
221+
```json
222+
{
223+
"add_column": {
224+
"column": {
225+
"check": {
226+
"constraint": "age >= 18",
227+
"name": "age_check"
228+
},
229+
"default": "18",
230+
"name": "age",
231+
"nullable": true,
232+
"type": "smallint"
233+
},
234+
"table": "adults",
235+
"up": "TODO: Implement SQL data migration"
236+
}
237+
},
238+
{
239+
"alter_column": {
240+
"column": "age",
241+
"default": null,
242+
"table": "adults",
243+
"down": "TODO: Implement SQL data migration",
244+
"up": "TODO: Implement SQL data migration"
245+
}
246+
},
247+
{
248+
"sql": {
249+
"up": "update adults set age = 18"
250+
}
251+
},
252+
{
253+
"alter_column": {
254+
"column": "age",
255+
"nullable": false,
256+
"table": "adults",
257+
"up": "TODO: Implement SQL data migration"
258+
"down": "TODO: Implement SQL data migration",
259+
}
260+
},
261+
{
262+
"sql": {
263+
"up": "UPDATE \"adults\" SET \"age\" = 18 WHERE \"age\" IS NULL"
264+
}
265+
},
266+
{
267+
"sql": {
268+
"up": "SET CONSTRAINTS ALL IMMEDIATE"
269+
}
270+
},
271+
{
272+
"alter_column": {
273+
"column": "age",
274+
"nullable": false,
275+
"table": "adults",
276+
"up": "TODO: Implement SQL data migration"
277+
"down": "TODO: Implement SQL data migration",
278+
}
279+
}
280+
```
281+
282+
This can be written as a single pgroll migration:
283+
284+
```json
285+
{
286+
"add_column": {
287+
"column": {
288+
"check": {
289+
"constraint": "age \u003e= 0",
290+
"name": "age_check"
291+
},
292+
"default": "18",
293+
"name": "age",
294+
"nullable": true,
295+
"type": "smallint"
296+
},
297+
"table": "adults",
298+
"up": "18"
299+
}
300+
},
301+
```
302+
303+
Also, SQL migration scripts usually start with `BEGIN` and `COMMIT` because the ORM runs the DDLs in a single transaction. These statements show up in the list of operations in the generated migrations. These operations are safe to be deleted from the list.

0 commit comments

Comments
 (0)