13 min readStep by Step Configuration of Sonarqube with Jenkins for Python

Sonarqube is an automated static code analysis tool configured with Jenkins, it will report bugs, vulnerabilities, and tech debt of the project. Sonarqube with Jenkins can be extensively used to raise the bugs that would be missed by some manual efforts. Secondly, Linting and code coverage are also handled by Sonarqube, so we don’t have to have different tools for that. Thirdly, Sonarqube is open-source software with good support and in this doc, we will be going to cover the setup, sonar scanner for one sonarqube-flask project, integration with Jenkins, and setting up the quality gate.

Consider a situation where you are the engineer who is reviewing a pull request with more than 1000 lines of code change. However, you would have a checklist where you will check the

Manual PR Checklist

• Is the PR atomic, for instance?
• In addition, Does the PR follow the single concern principle?
• Are the commit messages well-written?
• Will the code work well with the existing code and not increase duplication?
• Is the code well organised in terms of the placement of components?
• Is the PR contains the test cases for the modified code?
• New code keeping up with the idioms and code patterns of the language?
• Does the code make use of the language features and standard libraries?
• Does it comply with PEP-8?
• Are all language and project conventions followed?
• Are identifiers given meaningful and style guide-compliant names?
• Is the code free of implementation bugs that could be exploited?
• Have all the new dependencies been audited for vulnerabilities?

There’s a good probability that some points will be missed out when you are doing a manual check from the above checklist.

So here comes the Sonarqube, as it will take care of almost everything from the checklist, you just need to chill and watch Netflix.

So let’s dive into the agenda of the blog, what all things I will be covering in this blog.

Agenda

We are going to cover the following points in the blog

1. Overview of Sonarqube and where to use it.
2. Setup/Install Sonarqube.
3. Configure python project in Sonarqube.
4. Configure Jenkins with Sonarqube for automated testing.

Overview of Sonarqube

Sonarqube is a static code analysis tool, that will generate issues, bugs, vulnerabilities and tech debt of the project. But manually generating this all the time is a bit time-consuming. So Sonarqube works great with Jenkins pipelines.

Installation of Sonarqube with Jenkins

Prerequisites

Install following libraries and tools before proceeding the installation process

1. docker
2. docker-compose

Step 1

Sonarqube with Docker

Firstly, Create a directory structure just like below by creating a separate folder named sonarqube.

- sonarqube/
 - data/                  ## All the data of sonarqube is in this folder
 - docker-compose.yml     ## Sonarqube entrypoint
 - extensions/            ## All the extensions are here
 - logs/                  ## Logs are stored here
 - pg_data/               ## Sonarqube needs a db, we are using postgres for it and the data is here
 - pg_db/                 ## Postgres databases

mkdir sonarqube
cd sonarqube
mkdir data extensions logs pg_data pg_db
touch docker-compose.yml

Thirdly, Run the docker-compose file using docker swarm and use the below command for that.

docker stack deploy -c docker-compose.yml sonar

Most importantly, just crosscheck that the docker services are running fine or not by the below command

docker service ls

Step 4

Sonarqube Configuration

Open the browser and checkout localhost:9000 URL to open Sonarqube.

Step 5

Now, login the Sonarqube using below credentials

username: admin

password: admin

• Firstly, Click on Administrator’s My Account

• Secondly, Click on security
• Thirdly, Change password to something secure one

Step 6

Disable anonymous access

However, by default Sonarqube allows anonymous access to the analysis. We can also disable it by going to the administrator section. Then open the security tab and enable Force user authentication

Administrator>Security>Force User Authentication

Step 7

Enable Email authentication and Setup Email Client

To receive notifications whenever some build fails, or some issue is assigned to you or some other issues in a specific project need input from you. So we have to configure notification as per profile basis.

1. Open My Account
2. Open the Notifications tab and tick on the following rules

Setup Email Client

Open Administrator tab, then Configurations and then general

>Administrator>Configurations>General

Enter below values

1. email prefix: [SONARQUBE]
2. from address: <your email address>
3. from name: SonarQube
4. SMTP Configuration
   1. host: smtp.gmail.com
   2. password: <your password>
   3. port: 587
5. username: <your username>
6. server base URL: https://localhost:9000(your sonarqube domain)

Step 8

Configuration of Project

• Add a project manually

• Give a name to the project

• Generate a new token for analysis purpose, so that you will not use user credentials for analysing

• Store the token somewhere safe as it will not be visible once the project is set up.
• Choose the project main language, as in this case I have chosen “Other” cause the sonarqube-flask project’s main language is python. Then the OS will give you the sonar scanner command that we have to run to populate static code analysis.

• Finally, the project is setup, we can see the project listing on the projects tab without any analysis

Step 9

Create Project

As the Sonarqube needs a project to work upon, I am going to use the flask project created in Flask 2.0 blog of progress story. Below is the code of it.

Sonar Scanner – Project Setup

In this part, we will be going to set up a sonar scanner on the hello-flask project. This scanner will scan the project and generate the static code analysis of it and push that to the sonarqube server.

For sonar-scanner, we will be using docker.

1. Create a sonar-project.properties file on the root directory. This file will have all the flags or properties of the sonar scanner of that project

sonar.projectKey=sonarqube-flask
sonar.projectBaseDir=/sonarqube-flask
sonar.sources=.
sonar.host.url=http://localhost:9000
sonar.login=e21546a01f24a02b82428c8fe2d3622a5bd0fd8a
sonar.python.coverage.reportPaths=/sonarqube-flask/coverage.xml
sonar.python.xunit.reportPath=/sonarqube-flask/result.xml
sonar.coverage.dtdVerification=false
sonar.inclusions=app.py
sonar.coverage.exclusions=**/__init__.py

asgiref==3.3.4
attrs==21.2.0
beautifulsoup4==4.9.3
blinker==1.4
click==8.0.1
coverage==5.5
Flask==2.0.1
Flask-WebTest==0.0.9
iniconfig==1.1.1
itsdangerous==2.0.1
Jinja2==3.0.1
MarkupSafe==2.0.1
packaging==20.9
pluggy==0.13.1
py==1.10.0
pyparsing==2.4.7
pytest==6.2.4
pytest-cov==2.12.1
six==1.16.0
soupsieve==2.2.1
testapp==0.1.0
toml==0.10.2
waitress==2.0.0
WebOb==1.8.7
WebTest==2.0.35
Werkzeug==2.0.1

from flask_webtest import TestApp
import pytest
from app import app as current_app


@pytest.fixture
def testapp(app):
    return TestApp(app)


_app = None


@pytest.yield_fixture(scope='session')
def app():
    global _app
    if not _app:
        _app = current_app
    with _app.app_context():
        yield _app


class TestFlask(object):

    @pytest.fixture(autouse=True)
    def setup(self, testapp):
        self.testapp = testapp

    def test_comic(self):
        res = self.comic()
        assert res.status_code == 200

    def comic(self):
        return self.testapp.get(
            "/comic",
            expect_errors=True,
        )

4. Run the test case to generate the coverage report and result report using below command

py.test --cov-report xml:coverage.xml --cov=. --junitxml=result.xml  test.py

5. This will generate coverage.xml file and result.xml file in the root directory of repository

Step 10

Run Sonar Scanner on Project

For my case I will run the below command that will run the analysis on sonarqube-flask project

docker run --rm --net=host -v (PWD):/sonarqube-flask sonarsource/sonar-scanner-cli sonar-scanner -D sonar.projectBaseDir=/sonarqube-flask

Just replace sonarqube-flask with your project name

docker run --rm --net=host -v (PWD):/<your-project> sonarsource/sonar-scanner-cli sonar-scanner -D sonar.projectBaseDir=/<your-project>

Once the analysis is done, checkout the list view page of sonarqube

Open the analysis and checkout the detailed analysis

Step 11

Setup Jenkins and configure sonarqube-flask project

Install Jenkins using docker command. First create a folder for jenkins and its files

mkdir jenkins
cd jenkins
mkdir data
touch docker-compose.yml

Now open docker-compose.yml and paste the below content in it

Run the docker stack using below command

docker stack deploy -c docker-compose.yml jenkins

On the browser open localhost:7000. It will ask for initial password

pass the initial password will be located in data/secrets/initialAdminPassword

Install suggested plugins

Create a new Job

Give a name to the job sonarqube-flask then choose freestyle project

Check Git checkbox and pass the github URL and credentials in the Git SCM section.

Check Github hook trigger for GITScm polling

In the command run the shell script

Add run_sonarqube_script.sh at the root directory of the repository with the content

#!/bin/bash
docker run --rm --net=host -v ${PWD}:/sonarqube-flask sonarsource/sonar-scanner-cli sonar-scanner -D sonar.projectBaseDir=/sonarqube-flask

After commit automatically the build will be triggered and will run the script run_sonarqube_script.sh that will push the analysis to Sonarqube server.

Conclusion

If you haven’t tried Sonarqube yet and if you are working in a big team. Then its the right time to introduce Sonarqube in your team as it will take care of most of the manual stuff like linting, static analysis, test case coverage and a lot more.

I hope, it has helped you or it will help you. If you want to discuss anything or anything related to tech, you can contact me here or on the Contact Page of Progress Story. If you are interested in becoming a part of the Progress story please reach out to me or check out the Create Blog page.

See you next time! Peace Out ✌️