In a modern test data management approach, treating data as code is essential for reproducibility, regulatory compliance, and speed in continuous delivery processes. However, many pipelines still rely on manual tasks to provision data, which introduces inconsistencies, bottlenecks, and a loss of efficiency.

This article shows how to integrate Gigantics to fully automate test data management. You will see concrete examples of how to provision, transform, and deliver secure datasets from tools like GitLab CI, Jenkins, or GitHub Actions, ensuring consistent and automatically prepared environments in every execution.

Technical Integration Flow: Three Steps

1. Define the Dataset as Code

The team defines what data is needed in a versioned configuration file, such as data_config.yml. This file includes:

• Dataset structure


• Record volume


• Transformation rules


• Referential integrity between tables

dataset:
  name: "project-x-dev"
  schema:
    tables:
      - name: "users"
        rows: 50
        columns:
          - name: "id"
            type: "uuid"
          - name: "full_name"
            type: "string"
            rule: "gigantics.masking.fullname"
      - name: "orders"
        rows: 100
        columns:
          - name: "order_id"
            type: "uuid"
          - name: "user_id"
            type: "uuid"
            foreign_key: "users.id"

This file is stored alongside the application's source code, allowing test data to be versioned just like any other system artifact.

In addition, Gigantics connects to the data source using a secure tap, which provides access to real database schemas to build datasets that match the production structure and transform them using user-defined rules.

2. Orchestrate Provisioning from the Pipeline

The CI/CD pipeline must include a step before functional tests to provision the data automatically.

Example of a gitlab-ci.yml:

stages:
  - build
  - test
  - deploy

test_job:
  stage: test
  script:
    - ./scripts/provision-data.sh
    - pytest tests/functional_tests.py

The provision-data.sh script is responsible for invoking the Gigantics API and preparing the data.

3. Call the Gigantics API (cURL)

The following script makes an HTTP POST request to the Gigantics API, sending the data_config.yml file and using the authentication key.

#!/bin/bash

# Environment variables
GIGANTICS_API_KEY=$GIGANTICS_API_KEY
GIGANTICS_DATASET_ID="project-x-dev"
DB_CONNECTION_STRING=$DB_CONNECTION_STRING

# API call
curl --request POST \
  --url https://api.gigantics.io/v1/provision \
  --header "Authorization: Bearer ${GIGANTICS_API_KEY}" \
  --header "Content-Type: application/json" \
  --data-binary "@data_config.yml" \
  --output provisioning_log.json

# Verifies if provisioning was successful
if [[ $(jq -r .status provisioning_log.json) == "SUCCESS" ]]; then
  echo "Data provisioned successfully. Running tests..."
else
  echo "Data provisioning failed."
  exit 1
fi

This call automatically initiates the complete provisioning flow: it analyzes the configuration file, applies the defined transformations, and delivers the dataset directly to the destination database.

Technical Benefits of this Approach

• Consistent and Reproducible Test Environments: Each execution starts with a freshly provisioned dataset, eliminating data contamination between tests and enabling more effective debugging.


• Integrated Security (by design): Masking and pseudonymization rules are automatically applied from the data_config.yml file, without manual intervention.


• Speed and Efficiency: Manual environment preparation tasks are eliminated, which reduces cycle times and accelerates feedback.


• Technological Flexibility: Gigantics integrates via HTTP POST, so it's agnostic to the CI/CD tool used.


• Data as Code: The configuration is versioned and auditable, which facilitates regulatory compliance and collaboration between QA and DevOps.


• Secure Access to Real Data: Integration with TAP allows for the extraction of real production schemas to automatically transform them according to technical and regulatory criteria.

Conclusion

Automating test data within the pipeline is key to unlocking the full potential of CI/CD. Gigantics allows you to orchestrate the provisioning, transformation, and delivery of data via a REST API, directly from your DevOps workflow. This enables secure, consistent, and fast test environments, aligning quality, security, and speed without compromise.