Pay attention to the difference related to the user (instead of the jenkins user, we will use the root user - below you will find an explanation why) when adding a node from Docker to Jenkins.
There is a known issue with permissions. Basically, the user you are configuring, when you are connecting to the Docker node in Jenkins node configuration, should be set as root, not as jenkins. The problem is related to GID for user inside the Docker container. If the user on a host (Docker node) has a different GID than a user inside the Docker container, you cannot copy files back and forth between the Docker container and host due to two different GIDs, leading to a permission denied error inside the Jenkins job’s log. You will find more here:
persistent volume permissions issue
1. GitLab Configuration
Generating SSL certificate
- Generate a self-signed SSL certificate and add a path for key and crt files into gitlab.rb.
2. Jenkins Configuration
Adding Docker as a Node in Jenkins
After generating an ed25519 key pair on the Jenkins server for the jenkins user, send the public key to the Docker node for the root user.
Configure the SSH connection between Jenkins and Docker node.
3. Docker Configuration
After generating SSL certificate on a GitLab server, add the certificate for Docker node.
4. Taiko and Gauge Test Automation
Docker Container Configuration for Taiko
Create a Dockerfile that installs all required dependencies.
Configure the Node.js environment and Chromium browser.
Creating Jenkinsfile
Define the pipeline in the Jenkinsfile.
Set up authentication settings such as SSH keys and API tokens.
Below are the detailed steps - note -> TLDR :) Without this, you won’t achieve the goal.
GitLab Server
This is a key step because the configuration file contains alt_names, which are essential for correctly identifying the GitLab server name.
You can perform this command only if PermitRootLogin is set to yes and Password authentication is set to yes in sshd_config on a docker node. After that revert back changes that you made in /etc/ssh/sshd_config
Configuring Jenkins:
Login to Jenkins web panel:
Then click Manage Jenkins → Nodes
Click + New Node button on the right side.
Provide a node name
Select permanent agent
Set the description the same as node name
Set number of executors to 1 (this can be increased later)
Set remote root directory to /root
Set the label docker
Usage: use this node as much as possible
Launch method: Launch agents via SSH
Host: provide the IP address of the Docker node
Credentials → add → select Jenkins
Kind - choose from the dropdown list SSH username with private key
Provide a username: root
Select enter directly
Paste the private key copied from the jenkins_docker_ed25519 on a Linux server with Jenkins
In the Description field provide a friendly name like ed25519 key for a Docker node or anything like that that will easily identify the credentials.
Provide a passphrase to this ed25519 private key you generated previously on a Linux server with Jenkins.
Click add
Select newly created credentials from a dropdown list
Host Key Verification Strategy: choose: Known hosts file Verification Strategy
Availability: Keep this agent online as much as possible
In node properties select/check Environment variables and Tools Locations
In Environment variables section add:
Name: JAVA_HOME
Value: /usr/bin/java
In Tools Locations section add:
Name: Git (default)
Value: /usr/bin/git
Click save
2. Configuring Jenkins Connection to GitLab (API Token)
The second step is to configure the Jenkins connection to GitLab using an API token.
Steps:
The technical user jenkins-ci should be created as a Regular user in GitLab, not as an Administrator. This approach adheres to the principle of least privilege, which helps in minimizing security risks by granting only the necessary permissions required for Jenkins to perform its tasks.
Steps to Create a Technical User (jenkins-ci) as a Regular User in GitLab:
Log in to GitLab as an Administrator:
Use your administrator credentials to log in to GitLab.
Navigate to the Admin Area:
Click on your profile picture or initials in the top-right corner.
Select Admin Area.
Create a New User:
In the left sidebar, click on Users.
Click the New User button.
Fill in User Details:
Username: jenkins-ci
Name: Jenkins CI
Email: Use a dedicated email address for this user.
Choose a strong password.
Ensure that Regular user is selected, not Administrator.
Set the Appropriate Permissions:
After creating the user, navigate to the project or group where Jenkins will need access.
Add jenkins-ci as a member to the specific project or group.
Assign the Developer or Maintainer role, depending on the required permissions.
Assigning Developer or Maintainer Role:
Developer: This role allows the user to write to repositories, create branches, and perform other development-related tasks.
Maintainer: This role includes all permissions of the Developer role and adds the ability to manage project settings and perform administrative tasks within the project.
For most Jenkins operations, the Developer role is sufficient. If Jenkins needs to perform additional tasks like managing project settings, use the Maintainer role.
Generating the GitLab API Token for the technical account:
Log in as jenkins-ci:
Log out of the administrator account and log in with the jenkins-ci credentials.
Generate an API Token:
Go to User Settings > Access Tokens.
Create a new token with the required scopes (api, read_repository).
Copy the generated token.
Adding the GitLab API Token in Jenkins:
Go to Jenkins Web Interface:
Navigate to Manage Jenkins > Configure System.
Add the API Token:
In the GitLab section, click Add > Jenkins.
Select GitLab API token.
Paste the token and give it a description (e.g., GitLab API Token).
Adding Project-Level Token in Jenkins:
Generate a Project-Level API Token in GitLab:
Go to the specific GitLab project.
Navigate to Settings > Access Tokens.
Create a new token with the required scopes (read_repository).
Copy the generated token.
Add the Project-Level Token in Jenkins:
Go to Manage Jenkins > Manage Credentials.
Select the appropriate domain (e.g., global).
Click Add Credentials.
Choose Username with password as the kind.
Set the username to the technical user that exists as a member in the GitLab project (e.g., project_bot).
Paste the generated token as the password.
Give it a description (e.g., Taiko Project Token).
Save the changes.
Usage in Jenkinsfile:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
pipeline{agentanystages{stage('Checkout'){steps{git(url:'https://gitlab.sysadmin.homes/developers/your-repo.git',branch:'main',credentialsId:'Taiko-token')}}// Other stages
}}
By creating jenkins-ci as a regular user and assigning the appropriate roles, you ensure that Jenkins has the necessary permissions to interact with GitLab securely and efficiently without granting excessive privileges.
The reason for using two different methods to handle API tokens in Jenkins stems from the different contexts and scopes in which they are used:
GitLab API Token for Jenkins Connection:
This token is created at the user level (e.g., the technical user jenkins-ci).
It allows Jenkins to interact with GitLab’s API for broader functionalities, such as managing projects, users, and groups, fetching repository information, triggering pipelines, etc.
This is added in Jenkins as a GitLab API token under Manage Jenkins > Configure System > GitLab.
Project-Level Token for Repository Access:
This token is specific to the project and is often used for cloning repositories during builds.
Since Jenkins needs to authenticate with GitLab to clone repositories, it uses this token as a password, and the associated technical user (e.g., project_bot) as the username.
This token is added as a Username with password credential in Jenkins under Manage Jenkins > Manage Credentials.
By using these two different methods, you ensure that Jenkins can interact with GitLab securely and efficiently, utilizing the appropriate scope and permissions for each type of interaction.
Creating a Jenkins Job Using Jenkins Pipeline and Dockerfile - Step by Step
Create a new pipeline in Jenkins.
Select GitLab for Docker as the connection to GitLab.
Scroll down to the pipeline section.
In the dropdown list, select “pipeline script from SCM”.
In the SCM dropdown list, select “Git”.
Enter the GitLab repository URL.
Select the API token for the project from the dropdown list.
Select the branch, e.g., “main”.
Leave Jenkinsfile name as it is.
In the Jenkinsfile, define the steps and specify the path to the Dockerfile located in the GitLab repository where Taiko tests and Gauge reports are stored.
pipeline{options{buildDiscarder(logRotator(numToKeepStr:'5'))disableConcurrentBuilds()}parameters{choice(name:'choose_server',choices:['ArgoCD','AWX','AdGuardHome','Confluence','GitLab','Grafana','HomeAssistant','Jenkins','NginxProxyManager','Proxmox','Synology','Wazuh','PortainerProxy','PortainerAdGuardHome'],description:'Select server')choice(name:'username',choices:['admin'],description:'Choose username')choice(name:'SPEC_FILE',choices:['test-awx.spec'],description:'Choose spec file to test a module')}environment{REPO_URL='git@gitlab.sysadmin.homes:developers/awx-taiko.git'BRANCH='main'REPORT_PATH='/workspace'REPORT_NAME='TAIKO_AUTOMATED_TESTS'username='awx-user-id'password='awx-password-id'}agent{dockerfile{filename'./Dockerfile-Taiko'label'docker'}}stages{stage('Resolve IP'){steps{script{defserverAddressMapping=['ArgoCD':'argocd.sysadmin.homes','AWX':'awx.sysadmin.homes','AdGuardHome':'10.10.0.108','Confluence':'confluence.sysadmin.homes','GitLab':'gitlab.sysadmin.homes','Grafana':'grafana.sysadmin.homes','HomeAssistant':'ha.sysadmin.info.pl','Jenkins':'jenkins.sysadmin.homes','NginxProxyManager':'sysadmin.homes','Proxmox':'proxmox.sysadmin.homes','Synology':'synology.sysadmin.homes','Wazuh':'wazuh.sysadmin.homes','PortainerProxy':'npm-portainer.sysadmin.homes','PortainerAdGuardHome':'adguard-portainer.sysadmin.homes']env.server_address=serverAddressMapping[params.choose_server]}}}stage('Init container'){steps{script{sh'mkdir -p ~/.ssh && ssh-keyscan gitlab.sysadmin.homes >> ~/.ssh/known_hosts'}checkout([$class:'GitSCM',branches:[[name:'main']],doGenerateSubmoduleConfigurations:false,extensions:[[$class:'CheckoutOption',timeout:360],[$class:'CloneOption',timeout:360],],submoduleCfg:[],userRemoteConfigs:[[credentialsId:'Taiko-token',url:"https://gitlab.sysadmin.homes/developers/awx-taiko.git"]]])}}stage('Taiko and Gauge reports'){steps{withCredentials([string(credentialsId:username,variable:'username'),string(credentialsId:password,variable:'password')]){script{sh'''
export server_address=$server_address
export username=$username
export password=$password
'''sh'''
ln -s /usr/local/lib/node_modules/ $WORKSPACE/node_modules
ln -s /usr/local/lib/node_modules/ /lib/node_modules
rm -f *.tar downloaded/*
rm -rf reports .gauge logs
'''catchError(buildResult:'UNSTABLE',stageResult:'FAILURE'){sh"""
gauge run ${WORKSPACE}/specs/${params.SPEC_FILE}
"""}}}}}stage('Archive artifacts'){steps{script{if(sh(script:"[ -d \"${WORKSPACE}/reports/\" ]",returnStatus:true)==0){defformattedDate=newDate().format("dd_MM_yyyy_HH_mm")deffilename="PASS_${REPORT_NAME}_${formattedDate}.tar"sh"""
tar -cf ${filename} ${WORKSPACE}/reports/ ${WORKSPACE}/logs/
"""archiveArtifactsartifacts:"${filename}"}}}}}post{always{cleanWs()}}}
Adding awx-user-id and awx-password-id as Secret Text in Jenkins
To securely store and manage sensitive information, add the awx-user-id and awx-password-id as secret text credentials in Jenkins. Follow these steps:
Navigate to Jenkins Credentials:
Go to the Jenkins web interface.
Click on Manage Jenkins > Manage Credentials.
Add Secret Text Credentials:
Select the appropriate domain (e.g., global).
Click on Add Credentials.
For Kind, select Secret text.
In the Secret field, enter the awx-user-id value.
Add a meaningful ID (e.g., awx-user-id).
Click OK.
Repeat for awx-password-id:
Repeat the steps to add another secret text credential.
Enter the awx-password-id value and ID.
Using the Secret Text in Jenkins Pipeline:
When configuring your Jenkinsfile, use the credentials binding to access these secrets:
1
2
3
withCredentials([string(credentialsId:'awx-user-id',variable:'username'),string(credentialsId:'awx-password-id',variable:'password')]){// Your pipeline code
}
Example Dockerfile-Taiko that is also added in the GitLab project:
RUN npm config set registry "https://nexus.sysadmin.homes/repository/npmjs.org"
is responsible for npm packages. It fetches packages to Nexus, which acts as a cache proxy for nodejs and npm.
Establishing SSH Connection to Docker Node
Before running the pipeline job in Jenkins, establish an SSH connection from the root user on the Docker node to GitLab:
1
ssh -T git@10.10.0.119
Testing and Running
Creating and Running Tests
Create Taiko and Gauge test scripts.
Run the tests from Jenkins, monitoring results and generating reports.
Troubleshooting
SSH settings and UID/GID for users in Docker.
Troubleshooting headless browser launch issues.
Summary
With the above step-by-step instructions, you should be able to automate your tests and create a concise and clear textual tutorial. If you have further questions or need additional assistance, let me know!
Explanation
API Token in Jenkins:
Jenkins does not use RSA or ED25519 keys to communicate with GitLab; it uses API tokens. This token should be created at the technical user level in GitLab, not at the project level. This means you need to create a technical user in GitLab, generate an API token for this user, and use it in the Jenkins configuration.
Project Token in Jenkinsfile:
In the Jenkinsfile, credentialsId: 'Taiko-token' is declared, which refers to a project-level token. This token is not added in Jenkins using the GitLab API token but rather as a regular username and password in Jenkins credentials. This means you need to add the token as the password, and as the ID in Jenkins, use the username copied from the members section of the GitLab project.
This solution works because Jenkins requires authentication to access the GitLab repository, and instead of full login through the API, it uses user credentials with the token as the password.
/* globals gauge*/"use strict";constpath=require('path');const{openBrowser,write,closeBrowser,goto,button,press,screenshot,above,click,checkBox,listItem,toLeftOf,link,text,into,textBox,evaluate}=require('taiko');constassert=require("assert");constheadless=process.env.headless_chrome.toLowerCase()==='true';beforeSuite(async()=>{awaitopenBrowser({headless:headless})});afterSuite(async()=>{awaitcloseBrowser();});// Return a screenshot file name
gauge.customScreenshotWriter=asyncfunction(){constscreenshotFilePath=path.join(process.env['gauge_screenshots_dir'],`screenshot-${process.hrtime.bigint()}.png`);awaitscreenshot({path:screenshotFilePath});returnpath.basename(screenshotFilePath);};step("Navigate to the AWX login page",asyncfunction(){awaitgoto("awx.sysadmin.homes");});step("Assert the login page is loaded",async()=>{assert(awaittext("Welcome to AWX!").exists());});step('Use credentials <username>:<password>',async(username,password)=>{awaitwrite(process.env.username,into(textBox("Username"),{force:true}));awaitwrite(process.env.password,into(textBox("Password"),{force:true}));});step("Click the login button",async()=>{awaitclick(button("Log In"));});step("Verify successful login",async()=>{assert(awaittext("Dashboard").exists());});step("Clear all tasks",asyncfunction(){awaitevaluate(()=>localStorage.clear());});
test-awx.spec
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# AWX login test
to execute this specification, use
npm test
This is a context step that runs before every scenario
* Navigate to the AWX login page
## Login to AWX
* Assert the login page is loaded
* Use credentials "admin":"password"
* Click the login button
* Verify successful login
___
* Clear all tasks