-
Notifications
You must be signed in to change notification settings - Fork 108
feat: add YDB type annotations in struct field tags for runtime type validation #1936
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
Copilot
wants to merge
14
commits into
master
Choose a base branch
from
copilot/add-ydb-type-annotation-support
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 6 commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
6943e48
Initial plan
Copilot 7fc715e
feat: Add YDB type annotation support in struct fields
Copilot df58ee0
docs: Add type annotations example and documentation
Copilot a20ea51
chore: Add gitignore for example binaries
Copilot d62c880
docs: Add comprehensive documentation for type annotations
Copilot 3e85208
Final verification - all tests passing
Copilot 50f0a38
Apply suggestions from code review
asmyasnikov a181143
test: Add backward compatibility tests for struct scanning without ty…
Copilot 8d71d8a
Update internal/query/scanner/struct.go
asmyasnikov c86453b
Update internal/query/scanner/example_test.go
asmyasnikov 1af2d17
fix: Address code review feedback
Copilot 50a1634
docs: Add CHANGELOG entry for type annotations feature
Copilot 0fb1687
fix: Address linter issues
Copilot bf67b2a
Merge branch 'master' into copilot/add-ydb-type-annotation-support
asmyasnikov File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| /.golangci.yml | ||
| /.golangci.yml*.exe | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| type_annotations |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| # Type Annotations Example | ||
|
|
||
| This example demonstrates how to use YDB type annotations in struct field tags to validate column types and improve code documentation. | ||
|
|
||
| ## Features Demonstrated | ||
|
|
||
| 1. **Basic Type Annotations**: Annotate struct fields with YDB types like `Uint64`, `Text`, `Double` | ||
| 2. **Complex Type Annotations**: Use `List<T>`, `Optional<T>`, and `Dict<K,V>` annotations | ||
| 3. **Type Validation**: Automatic validation that database column types match your struct annotations | ||
| 4. **Error Detection**: Clear error messages when types don't match | ||
|
|
||
| ## Struct Tag Syntax | ||
|
|
||
| The type annotation is added to the struct tag using the `type:` prefix: | ||
|
|
||
| ```go | ||
| type Product struct { | ||
| // Column name: product_id, YDB type: Uint64 | ||
| ProductID uint64 `sql:"product_id,type:Uint64"` | ||
|
|
||
| // Column name: tags, YDB type: List<Text> | ||
| Tags []string `sql:"tags,type:List<Text>"` | ||
|
|
||
| // Column name: rating, YDB type: Optional<Double> | ||
| Rating *float64 `sql:"rating,type:Optional<Double>"` | ||
| } | ||
| ``` | ||
|
|
||
| ## Supported YDB Types | ||
|
|
||
| ### Primitive Types | ||
| - `Bool`, `Int8`, `Int16`, `Int32`, `Int64` | ||
| - `Uint8`, `Uint16`, `Uint32`, `Uint64` | ||
| - `Float`, `Double` | ||
| - `Date`, `Datetime`, `Timestamp` | ||
| - `Text` (UTF-8), `Bytes` (binary) | ||
| - `JSON`, `YSON`, `UUID` | ||
|
|
||
| ### Complex Types | ||
| - `List<T>` - List of items of type T | ||
| - `Optional<T>` - Optional (nullable) value of type T | ||
| - `Dict<K,V>` - Dictionary with key type K and value type V | ||
|
|
||
| ### Nested Types | ||
| You can nest complex types: | ||
| - `List<Optional<Text>>` - List of optional text values | ||
| - `Optional<List<Uint64>>` - Optional list of unsigned integers | ||
| - `Dict<Text,List<Uint64>>` - Dictionary mapping text to lists of integers | ||
|
|
||
| ## Benefits | ||
|
|
||
| 1. **Documentation**: Type annotations serve as inline documentation of expected database schema | ||
| 2. **Validation**: Runtime validation ensures your Go types match the database schema | ||
| 3. **Error Prevention**: Catch type mismatches early before they cause runtime errors | ||
| 4. **Code Clarity**: Makes it explicit what YDB types you expect from the database | ||
|
|
||
| ## Running the Example | ||
|
|
||
| ```bash | ||
| # Set your YDB connection string | ||
| export YDB_CONNECTION_STRING="grpc://localhost:2136/local" | ||
|
|
||
| # Run the example | ||
| go run main.go | ||
| ``` | ||
|
|
||
| ## When to Use Type Annotations | ||
|
|
||
| Type annotations are **optional** but recommended when: | ||
|
|
||
| - Working with complex types (Lists, Dicts, Optional values) | ||
| - Building a strict API where type safety is critical | ||
| - Documenting expected database schema in code | ||
| - Working in a team where schema changes need to be validated | ||
|
|
||
| ## Backward Compatibility | ||
|
|
||
| Type annotations are completely optional. Existing code without type annotations continues to work exactly as before. You can add annotations gradually to your codebase. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,206 @@ | ||
| package main | ||
|
|
||
| import ( | ||
| "context" | ||
| "flag" | ||
| "fmt" | ||
| "log" | ||
| "os" | ||
| "path" | ||
| "time" | ||
|
|
||
| environ "github.com/ydb-platform/ydb-go-sdk-auth-environ" | ||
| ydb "github.com/ydb-platform/ydb-go-sdk/v3" | ||
| "github.com/ydb-platform/ydb-go-sdk/v3/query" | ||
| "github.com/ydb-platform/ydb-go-sdk/v3/sugar" | ||
| ) | ||
|
|
||
| var connectionString = flag.String("ydb", os.Getenv("YDB_CONNECTION_STRING"), "YDB connection string") | ||
|
|
||
| func main() { | ||
| ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) | ||
| defer cancel() | ||
|
|
||
| flag.Parse() | ||
|
|
||
| if *connectionString == "" { | ||
| log.Fatal("YDB_CONNECTION_STRING environment variable or -ydb flag must be set") | ||
| } | ||
|
|
||
| db, err := ydb.Open(ctx, *connectionString, | ||
| environ.WithEnvironCredentials(), | ||
| ) | ||
| if err != nil { | ||
| log.Fatalf("failed to connect to YDB: %v", err) | ||
| } | ||
| defer func() { _ = db.Close(ctx) }() | ||
|
|
||
| prefix := path.Join(db.Name(), "type_annotations_example") | ||
|
|
||
| // Clean up any existing data | ||
| _ = sugar.RemoveRecursive(ctx, db, prefix) | ||
|
|
||
| // Create table | ||
| if err := createTable(ctx, db.Query(), prefix); err != nil { | ||
| log.Fatalf("failed to create table: %v", err) | ||
| } | ||
|
|
||
| // Insert data | ||
| if err := insertData(ctx, db.Query(), prefix); err != nil { | ||
| log.Fatalf("failed to insert data: %v", err) | ||
| } | ||
|
|
||
| // Read data with type annotations | ||
| if err := readDataWithTypeAnnotations(ctx, db.Query(), prefix); err != nil { | ||
| log.Fatalf("failed to read data: %v", err) | ||
| } | ||
|
|
||
| // Demonstrate type mismatch detection | ||
| if err := demonstrateTypeMismatch(ctx, db.Query(), prefix); err != nil { | ||
| log.Printf("Expected error (type mismatch): %v", err) | ||
| } | ||
|
|
||
| // Clean up | ||
| _ = sugar.RemoveRecursive(ctx, db, prefix) | ||
|
|
||
| fmt.Println("\nExample completed successfully!") | ||
| } | ||
|
|
||
| func createTable(ctx context.Context, c query.Client, prefix string) error { | ||
| return c.Exec(ctx, fmt.Sprintf(` | ||
| CREATE TABLE %s ( | ||
| product_id Uint64, | ||
| name Text, | ||
| description Text, | ||
| price Uint64, | ||
| tags List<Text>, | ||
| rating Optional<Double>, | ||
| metadata Dict<Text, Text>, | ||
| PRIMARY KEY (product_id) | ||
| )`, "`"+path.Join(prefix, "products")+"`"), | ||
| query.WithTxControl(query.NoTx()), | ||
| ) | ||
| } | ||
|
|
||
| func insertData(ctx context.Context, c query.Client, prefix string) error { | ||
| return c.Exec(ctx, fmt.Sprintf(` | ||
| INSERT INTO %s (product_id, name, description, price, tags, rating) | ||
| VALUES ( | ||
| 1, | ||
| "Laptop", | ||
| "High-performance laptop", | ||
| 999, | ||
| ["electronics", "computer", "portable"], | ||
| 4.5 | ||
| ); | ||
| `, "`"+path.Join(prefix, "products")+"`")) | ||
| } | ||
|
|
||
| // Product demonstrates struct with YDB type annotations | ||
| type Product struct { | ||
| // Basic types with annotations | ||
| ProductID uint64 `sql:"product_id,type:Uint64"` | ||
| Name string `sql:"name,type:Text"` | ||
| Description string `sql:"description,type:Text"` | ||
| Price uint64 `sql:"price,type:Uint64"` | ||
|
|
||
| // List type annotation | ||
| Tags []string `sql:"tags,type:List<Text>"` | ||
|
|
||
| // Optional type annotation (can be NULL) | ||
| Rating *float64 `sql:"rating,type:Optional<Double>"` | ||
| } | ||
|
|
||
| func readDataWithTypeAnnotations(ctx context.Context, c query.Client, prefix string) error { | ||
| fmt.Println("\n=== Reading data with type annotations ===") | ||
|
|
||
| return c.Do(ctx, func(ctx context.Context, s query.Session) error { | ||
| result, err := s.Query(ctx, fmt.Sprintf(` | ||
| SELECT product_id, name, description, price, tags, rating | ||
| FROM %s | ||
| `, "`"+path.Join(prefix, "products")+"`"), | ||
| query.WithTxControl(query.TxControl(query.BeginTx(query.WithSnapshotReadOnly()))), | ||
| ) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| defer result.Close(ctx) | ||
|
|
||
| for rs, err := range result.ResultSets(ctx) { | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| for row, err := range rs.Rows(ctx) { | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| var product Product | ||
|
|
||
| // ScanStruct will validate that database column types match the annotations | ||
| if err := row.ScanStruct(&product); err != nil { | ||
| return fmt.Errorf("scan error: %w", err) | ||
| } | ||
|
|
||
| fmt.Printf("\nProduct ID: %d\n", product.ProductID) | ||
| fmt.Printf("Name: %s\n", product.Name) | ||
| fmt.Printf("Description: %s\n", product.Description) | ||
| fmt.Printf("Price: $%d\n", product.Price) | ||
| fmt.Printf("Tags: %v\n", product.Tags) | ||
| if product.Rating != nil { | ||
| fmt.Printf("Rating: %.1f/5.0\n", *product.Rating) | ||
| } else { | ||
| fmt.Println("Rating: Not rated") | ||
| } | ||
| } | ||
| } | ||
|
|
||
| return nil | ||
| }) | ||
| } | ||
|
|
||
| // ProductWrongType demonstrates what happens when type annotations don't match | ||
| type ProductWrongType struct { | ||
| ProductID uint64 `sql:"product_id,type:Text"` // Wrong! Should be Uint64 | ||
| Name string `sql:"name,type:Text"` | ||
| } | ||
|
|
||
| func demonstrateTypeMismatch(ctx context.Context, c query.Client, prefix string) error { | ||
| fmt.Println("\n=== Demonstrating type mismatch detection ===") | ||
|
|
||
| return c.Do(ctx, func(ctx context.Context, s query.Session) error { | ||
| result, err := s.Query(ctx, fmt.Sprintf(` | ||
| SELECT product_id, name | ||
| FROM %s | ||
| `, "`"+path.Join(prefix, "products")+"`"), | ||
| query.WithTxControl(query.TxControl(query.BeginTx(query.WithSnapshotReadOnly()))), | ||
| ) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| defer result.Close(ctx) | ||
|
|
||
| for rs, err := range result.ResultSets(ctx) { | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| for row, err := range rs.Rows(ctx) { | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| var product ProductWrongType | ||
|
|
||
| // This will fail because product_id is Uint64 in the database | ||
| // but the annotation says it should be Text | ||
| if err := row.ScanStruct(&product); err != nil { | ||
| return fmt.Errorf("type mismatch detected: %w", err) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| return nil | ||
| }) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.