feat(ci): implement a CI/CD Pipeline

[balazs-szucs]

🚀 Pull Request

📝 Description

This PR introduces a complete CI/CD pipeline using GitHub Actions (GHA). It covers build automation, extensive testing, security scanning, and deployment processes.

🛠️ Changes Made

• Automate build and test validation for every Pull Request (PR).
• Test database migrations thoroughly to ensure data integrity.
• Validate API contracts for Kobo, KOReader, and OPDS integrations.
• Run performance and load tests for large libraries (10,000 to 50,000 books).
• Scan for security issues, including secrets, CVEs, SAST, and container vulnerabilities.
• Build Docker images for multiple architectures (AMD64, ARM64).
• Automate PR labeling and workflow management.
• Schedule weekly tests for long-running validations.

What's Included

Category	Workflows	Purpose
Core Build & Testing	3 workflows	Basic CI/CD, type checking, and compilation
Database Migration	3 workflows	Migration integrity and regression testing
API & Protocol Testing	3 workflows	Validation for Kobo, KOReader, and OPDS
Advanced Testing	6 workflows	Performance, ARM64, malformed files, and proxies
Security	3 workflows	Secrets, CVEs, SAST, and CodeQL analysis
Docker & Release	4 workflows	Multi-arch builds, releases, and cleanup
Utility & Automation	7 workflows	Auto-labeling, coverage, and orchestration

Key Workflows###Core CI/CD

• ci-comprehensive.yml: A multi-stage pipeline that runs different steps based on which files changed.
• build.yml: Validates builds for the Frontend (Angular) and Backend (Java/Gradle).
• type-checking.yml: Checks TypeScript types and runs a production build.

Database Quality

• db-migration-check.yml: Checks migration integrity, checksums, and SQL syntax.
• db-migration-regression.yml: Tests all migration paths, including fresh installs and upgrades.
• db-migration-full.yml: runs a full matrix of tests from a baseline.

Advanced Testing

• load-performance-tests.yml: Tests performance with 10k-50k books; requires P95 response times under 5 seconds.
• corpus-of-doom-tests.yml: Checks how the system handles malformed EPUBs and corrupted files.
• arm64-qemu-tests.yml: Tests compatibility with ARM64 devices like Raspberry Pi.
• reverse-proxy-tests.yml: Validates deployment behind Nginx, Traefik, and Caddy.
• docker-persistence-test.yml: Verifies that data remains after a container restarts.

API & Integration

• api-tests.yml: Validates Swagger/OpenAPI endpoints.
• integration-contract-tests.yml: Tests integrations with Kobo API, OPDS, and Calibre.
• wiremock-protocol-tests.yml: Tests specific protocol edge cases.

Security

• security-enhanced.yml: Checks dependencies (OWASP), leaks (Gitleaks), and code issues (Semgrep, Trivy).
• codeql-analysis.yml: Runs advanced GitHub CodeQL analysis.
• security.yml: Basic secret scanning and SAST.

Automation

• auto-labeler.yml: Labels PRs automatically based on file size and type.
• manual-gatekeeper.yml: Allows manual execution of all workflows.
• test-coverage.yml: Reports code coverage using JaCoCo (backend) and Karma (frontend).
• eink-visual-regression.yml: Tests for E-Ink visual regressions (currently disabled).

Testing Infrastructure

• testing/docker-integration-test.sh: A script for Docker integration testing (386 lines).
• testing/swagger_endpoint_test.py: A Python suite for API testing (1052 lines).
• testing/playwright/: A framework for E-Ink visual regression tests.
• scripts/check-migration-integrity.sh: A shell script to validate migrations.

Configuration

• .github/labeler.yml: Configures the auto-labeler.
• .github/dependabot.yml: Automates dependency updates.
• .github/owasp-suppressions.xml: Manages false positives in security scans.
• .github/actions/java-gradle-setup/: A reusable action for Java/Gradle setup.
• .dockerignore: Optimized the Docker build context.

Code Quality

• Frontend Tests: Added specs for loading, localStorage, toast, and version services.
• Backend Tests: Added FlywayMigrationIntegrationTest.java.
• Build Improvements: Updated build.gradle to include Flyway and JaCoCo.

Weekly Schedule

• Sunday: ARM64 tests and link checker.
• Monday: Migration tests, security scans, and CodeQL.
• Tuesday: Malformed file tests.
• Wednesday: Docker persistence tests.
• Saturday: Load and performance tests.

Security Hardening

• All workflows use step-security/harden-runner.
• Action versions are pinned with SHA hashes.
• Permissions are set to the minimum required (least privilege).
• Egress policy auditing is enabled.
• Includes scanning for secrets (Gitleaks), dependency CVEs (OWASP), container vulnerabilities (Trivy), and SAST analysis.

Caching and Concurrency

• Caching is enabled for Gradle, npm, and Docker layers.
• Concurrency control cancels runs that are in progress if a new one starts.
• Path-based execution ensures only necessary workflows run.
• Jobs run in parallel where possible.
• Quick checks run first to fail fast if there is an issue.

Testing Coverage & Quality Metrics

• Unit Tests: Frontend (Angular/Jest) and Backend (Java/JUnit).
• Integration Tests: API contracts, database migrations, and protocols.
• Performance Tests: Tested with 10k-50k books and 50-100 concurrent users.
• Security Tests: Secrets, CVEs, SAST, and container scanning.
• Compatibility Tests: ARM64, reverse proxies, and malformed files.

Quality Standards

• Database: Checksums, SQL syntax, sequential numbering, and upgrade/rollback safety are verified.
• Performance: P95 response time must be under 5 seconds. Memory leaks and large library support are checked.
• Security: No high/critical CVEs, no exposed secrets, and no SAST violations allowed.

Success Criteria

PR Workflows Pass When:

• All unit tests and type checks pass.
• There are no compilation errors.
• Security scans are clean.
• Migration integrity is validated.
• The Docker image builds successfully.

Advanced Tests Pass When:

• The system handles 10k+ books with P95 < 5s.
• It works on ARM64 architecture.
• It handles malformed files gracefully.
• It works correctly behind reverse proxies.
• Data persists across container restarts.

Deleted Files

• .github/workflows/docker-build-publish.yml (Replaced by new workflows)

Benefits

• Quality: Automated testing catches errors early.
• Security: Continuous scanning detects vulnerabilities and secrets.
• Performance: Tests prevent regressions in large libraries.
• Compatibility: Ensures the software works on ARM64 and behind proxies.
• Developer Experience: Provides automated labeling and quick feedback.

Mirror PR: balazs-szucs#63

🧪 Testing

• Tested manually in local environment
• Added/updated automated unit/integration tests
• All existing tests pass

📸 Visual Changes (if applicable)

✅ Pre-submission Checklist

• Code follows project style guidelines and conventions
• Tests added/updated to cover changes
• All tests pass locally (./gradlew test for backend)
• Flyway migration versioning is correct (if database changes made)
• Branch is synced with latest develop branch
• Documentation PR created at booklore-docs (for major features)

💬 Additional Notes (optional)

[acx10]

This all sounds great and the direction makes a lot of sense 👍
That said, I’m having a hard time doing a proper review because the PR is quite large and touches many areas at once.

Do you think it would be possible to break this down into smaller, incremental PRs, for example, introducing one or two workflows at a time? That would make review, feedback, and iteration much easier, and reduce the risk of subtle issues slipping through.

Happy to re-review once it’s split up, and thanks for putting in the effort on this, it’s clearly a substantial contribution.

[balazs-szucs]

Do you think it would be possible to break this down into smaller, incremental PRs, for example, introducing one or two workflows at a time? That would make review, feedback, and iteration much easier, and reduce the risk of subtle issues slipping through.

Sorry for the late response (I totally forgot! 😅).

I am currently in the process of decoupling the workflows from each other. The current plan is to introduce the workflows in the following order:

• (Utility & Automation) or at the end we'll see.
• Core Build & Testing
• Database Migration
• API & Protocol Testing
• Advanced Testing
• Security

For more detailed info see the table from the PR message.

Obviously, this PR aims to be similar what I envision as the overall structure. I am also in favor of making the changes gradually. I just got a bit overzealous with this (which admittedly happens) and didn't realize I should stop to get feedback until, well, we got here.

Happy to re-review once it’s split up, and thanks for putting in the effort on this, it’s clearly a substantial contribution.

Thanks 😄

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.