Data Observability platform Help

Deploying DQM

Prerequisites

The minimum requirements for the server or cloud instance hosting the application are as follows:

  • CPU: 2 core

  • RAM: 4GB

  • HDD: A minimum of 10GB of free space

Deployment

Download the latest version of JAR from https://selectzero.io/downloads/

To run JAR application, use command:

java -XX:TieredStopAtLevel=1 -Dspring.jmx.enabled=false -Xmx4g -Dserver.port=8090 -jar LiTech_DQM_2024.8.jar

-Xmx4g – Specifies the amount of memory that can be used by Java – in this case 4GB

-Dserver.port=8090 – Defining DQM’s machine port (8090 is also default port if not provided)

It is recommended to run the Java file as a service.

Link to Java Service on Ubuntu

Link to Java Service on Windows

For hosting your application on your cloud instance, make sure you have a connection to docker hub.

You can find latest versions here

Deploy with docker

  1. Pull docker image

    docker pull litech/tester:latest

  2. Run container

    docker run -d -v {local_path}/logs:/usr/app/logs -v {local_path}/database:/usr/app/database -v {local_path}/drivers:/usr/app/drivers -v {local_path}/reports:/usr/app/reports -v {local_path}/keys:/usr/app/keys -p 8090:8090 -e JAVA_OPTIONS='-Xmx4g' litech/tester:2024.8

    -v {local_path}/logs:/usr/app/logs

    Logs storage

    -v {local_path}/database:/usr/app/database

    DQM local database

    -v {local_path}/drivers:/usr/app/drivers

    JAR connectors storage

    -v {local_path}/reports:/usr/app/reports

    Reports and results

    -v {local_path}/keys:/usr/app/keys

    Key files storage such as BigQuery JSON keys and application certificate keystore

    -p 8090:8090

    Port mapping for exposing DQM application at port 8090

    litech/tester:2024.8

    Image from dockerhub, latest tag is also available

  3. Validate that DQM container is running

    docker ps

Docker compose is the easiest way to deploy/manage DQM with docker containers

Deploy with docker compose

  1. Create docker-compose.yml

    version: "3.7" services: dqm: container_name: litech-tester image: litech/tester:2024.8 ports: - 8090:8090 volumes: - ./logs:/usr/app/logs - ./database:/usr/app/database - ./drivers:/usr/app/drivers - ./reports:/usr/app/reports - ./keys:/usr/app/keys environment: JAVA_OPTIONS: -Xmx4g

  2. Run docker compose

    docker compose up -d

Deploying on kubernetes

  1. Create resource file dqm.yaml

    apiVersion: v1 kind: PersistentVolume metadata: name: dqm-storage labels: type: local spec: storageClassName: hostpath capacity: storage: 20Gi accessModes: - ReadWriteOnce hostPath: path: /tmp --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: dqm-storage-claim spec: storageClassName: hostpath accessModes: - ReadWriteOnce resources: requests: storage: 20Gi --- apiVersion: apps/v1 kind: Deployment metadata: name: dqm-deployment spec: selector: matchLabels: app: dqm replicas: 1 # DQM is limited to one replica due to sessions strategy: type: RollingUpdate template: metadata: labels: app: dqm spec: volumes: - name: dqm-storage persistentVolumeClaim: claimName: dqm-storage-claim containers: - name: dqm image: litech/tester:latest resources: limits: memory: 4Gi requests: memory: 4Gi env: - name: JAVA_OPTIONS value: -XX:MaxRAMPercentage=80.0 ports: - containerPort: 8090 volumeMounts: - mountPath: "/usr/app/database" name: dqm-storage subPath: database - mountPath: "/usr/app/logs" name: dqm-storage subPath: logs - mountPath: "/usr/app/drivers" name: dqm-storage subPath: drivers - mountPath: "/usr/app/reports" name: dqm-storage subPath: reports - mountPath: "/usr/app/keys" name: dqm-storage subPath: keys - mountPath: "/usr/app/tempFiles" name: dqm-storage subPath: tempFiles --- apiVersion: v1 kind: Service metadata: name: dqm labels: app: dqm spec: type: NodePort ports: - port: 80 targetPort: 8090 selector: app: dqm

  2. Apply using kubectl

    kubectl apply -f dqm.yaml

Fixing DQM folder rights for non-root user

From version 2023.10 DQM container is running on non-root user "dqm". This is due to security reasons. "dqm" user uses default groups UID:1000 GID:1000. In case DQM container startup should give folder access rights, we have prepared a few fix scripts for assigning correct user rights to folders. This is a one time action and will work as expected in the future. It won't affect new installs, unless using direct docker bind mount to host system.

Fix folder rights

  1. Stop existing container

    docker stop {container_name}

  2. Run busybox container to fix folder permissions

    docker run --rm -v {local_path}/logs:/usr/app/logs -v {local_path}/database:/usr/app/database -v {local_path}/drivers:/usr/app/drivers -v {local_path}/reports:/usr/app/reports -v {local_path}/keys:/usr/app/keys busybox:latest sh -c "chown -R 1000:1000 /usr/app"

  3. Run the original container with fixed folder permissions

    docker run -d -v {local_path}/logs:/usr/app/logs -v {local_path}/database:/usr/app/database -v {local_path}/drivers:/usr/app/drivers -v {local_path}/reports:/usr/app/reports -v {local_path}/keys:/usr/app/keys -p 8090:8090 -e JAVA_OPTIONS='-Xmx4g' {image_id}

Fix folder rights

  1. Modify existing docker-compose.yml to include startup service

    version: "3.7" services: busybox: container_name: busybox-folders image: busybox volumes: - ./logs:/usr/app/logs - ./database:/usr/app/database - ./drivers:/usr/app/drivers - ./reports:/usr/app/reports - ./keys:/usr/app/keys command: ["chown", "-R", "1000:1000", "/usr/app/"] dqm: container_name: dqm-non-root image: "litech/tester:2024.8" ports: - "8090:8090" volumes: - ./logs:/usr/app/logs - ./database:/usr/app/database - ./drivers:/usr/app/drivers - ./reports:/usr/app/reports - ./keys:/usr/app/keys environment: JAVA_OPTIONS: "-Xmx2g -Djava.security.egd=:/dev/urandom" depends_on: busybox: condition: service_completed_successfully

  2. Re-apply docker compose

    docker compose up -d

Fix folder rights

  1. Modify deployment resource to include init container

    ... spec: volumes: - name: dqm-storage persistentVolumeClaim: claimName: dqm-storage-claim initContainers: - name: folder-rights-fix image: busybox:latest securityContext: runAsUser: 0 imagePullPolicy: Always command: ["chown", "-R", "1000:1000", "/usr/app/"] volumeMounts: - mountPath: "/usr/app/database" name: {storage-name} subPath: database - mountPath: "/usr/app/logs" name: {storage-name} subPath: logs - mountPath: "/usr/app/drivers" name: {storage-name} subPath: drivers - mountPath: "/usr/app/reports" name: {storage-name} subPath: reports - mountPath: "/usr/app/keys" name: {storage-name} subPath: keys containers: - name: dqm ...

  2. Add security context to main deployment spec

    ... apiVersion: apps/v1 kind: Deployment metadata: name: dqm-deployment spec: securityContext: runAsUser: 1000 fsGroup: 1000 ...

  3. Re-apply kubernetes resource

    kubectl apply -f dqm.yaml
Last modified: 06 September 2024
Initial loginReverse proxy